# A Deep Learning-Based Course Recommender System for Sustainable Development in Education
## 4 Experiments  
### Dataset and Evaluation Metric  
In this section, we specifically describe the datasets used in this study. To evaluate the performance of the proposed DECOR model, we collected a dataset consisting of user behaviors and course attributes from a Korean online recruitment website and an online certificate website, respectively. The dataset collection process is shown in Figure 2. The dataset was collected from 891 users on 11,099 courses with a sparsity of 99.73%. The descriptive statistics of the collection datasets are summarized in Table 1. The dataset also contained 26,115 interactions between five criteria: jobs, certificates, language skills, and subjects, in addition to user ID and course ID information. Specifically, there were 25 job information, 15 language skill information, 235 certificate information, and 45 subject information types. Each interaction denoted whether the user had taken the course.  
To evaluate the performance of the DECOR model, we adopted the leave-one-out evaluation technique, which has been widely used in several studies [57; 58]. We held out the users' latest interactions as the test dataset and utilized the remaining data for the model training dataset. This study followed a strategy that is common in current studies because it used too much time to rank all features for all users during the model evaluation [59; 60]. This strategy randomly samples 50 courses that do not interact with the user and ranks the test items among the 50 total items. To evaluate the performance of the recommendation list, we will rank the list and return the top \(K\) ones as recommendations. We adopted the Hit Ratio (HR)at a rank \(K\) and Normalized Discounted Cumulative Gain  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Attribute Name** & **Attribute Type** & **Description** & **Example** & **Number of Attribute** \\ \hline User ID & String & Unique identifier ID for the user & “A23123” & 891 \\ Job ID & Integer & Desired job ID of the user & 20 & 25 \\ Certificate & String & Title of Certificate of the user & “OCP” & 15 \\ Language Skill & String & Type of Language skill & “HFS 6” & 235 \\ Course ID & Integer & Unique identifier ID for the course & 3432 & 11,099 \\ Subject & String & Subject name that user taken a course & “Department of Management” & 45 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Statistics of the datasets and example.  
Figure 2: Dataset collection process.  
(_NDCG_) at a rank \(K\) metrics to evaluate the performance of a ranked list [61; 62]. When making the recommendation list, we generated the top \(K\) items for each user. If the test item was present in the top \(K\) list, it was called a hit. When \(N\) is the number of test sets, the HR is computed as follows:  
\[HR@K=\frac{\#Number\ of\ Hits@K}{N}, \tag{17}\]  
The \(HR\) is a recall-based metric, so it does not reflect the accuracy of critical top ranks. To overcome this, we also adopted the \(NDCG\), which explains the hits' position by assigning higher scores to hits with higher ranks. The \(NDCG\) is computed as follows:  
\[NDCG@K=\sum_{i=1}^{K}\frac{2^{r}i-1}{\log_{2}(i+1)}, \tag{18}\]  
where \(r_{i}\) is the graded binary relevance of the item at position \(i\). If the test item was present in the top \(K\) list, \(r_{i}\) became 1; it became 0 otherwise. For both metrics, a higher value indicates better recommendation performance. We calculated both metrics for each user and reported the average score.