# Recommendation of Learning Objects Applying Collaborative Filtering and Competencies

Sílvio César Cazella, Eliseo Berni Reategui, Patrícia Behar

► To cite this version:

Sílvio César Cazella, Eliseo Berni Reategui, Patrícia Behar. Recommendation of Learning Objects Applying Collaborative Filtering and Competencies. IFIP TC 3 International Conference on Key Competencies in the Knowledge Society (KCKS) / Held as Part of World Computer Congress (WCC), Sep 2010, Brisbane, Australia. pp.35-43, 10.1007/978-3-642-15378-5_4. hal-01054676

HAL Id: hal-01054676

https://inria.hal.science/hal-01054676

Submitted on 8 Aug 2014

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Recommendation of Learning Objects Applying Collaborative Filtering and Competencies

Sílvio César Cazella¹, Eliseo Berni Reategui², Patrícia Behar²

¹Universidade do Vale do Rio dos Sinos (UNISINOS)
Av. Unisinos, 950, CEP 93022-001, São Leopoldo, RS - Brasil
cazella@unisinos.br

²Universidade Federal do Rio Grande do Sul
Av. Paulo Gama 110, prédio 12105, Porto Alegre, RS, Brazil
eliseoreategui@gmail.com, pbehar@terra.com.br

**Abstract.** This paper presents a recommender system for learning objects which uses a collaborative filtering mechanism based on competencies. The model enables students to receive recommendations of learning objects automatically, according to students' interests but also according to competencies that have to be developed. The prototype implemented was able to recommend relevant contents to students, aiming at helping them in the development of competencies. The paper also presents a couple of experiments showing that the recommender system has a good level of accuracy for the suggestions made.

**Keywords:** Recommender Systems, Collaborative Filtering, Competencies

## 1 Introduction

One of the biggest challenges educators face nowadays is the organization of content and activities to develop certain competencies. This challenge is even bigger when one tries to identify and recommend different material to different students, based on individual needs, interests and skills.

This paper presents a recommender system for learning objects which can suggest learning materials according to students' interests as well as to skills that have to be developed. Learning objects are understood here as digital learning material constructed in a modular way so that they can be used together or separately [1]. In this sense, a scientific paper, a web page, a simulator, a question and answer software, all may be considered learning objects.

The recommendation of learning objects focusing on competencies requires a change in traditional educational paradigms where pedagogical models follow rigid curricula and favor transmissive methodologies. The focus on competencies requires that social characteristics, interests, needs and limitations of each individual also be considered. The concept of competence can be understood as “practical intelligence for situations that rely on knowledge and mobilizes, transforms them according to the complexity of the situations” [2]. In this sense, knowledge and skills (know-how) are
part of the concept of competence [3]. Although the recommendation of personalized
learning objects is a desirable feature for any area of knowledge, our focal point has
been the curricula of undergraduate programs in Information Technology (Computer
Science, Computer Engineering and Information Systems). These programs have
courses distributed throughout eight or nine semesters according to a set of
prerequisites. The documents that describe these courses usually list a series of
competencies which students should develop. For example, the databases course may
require the student to develop competencies in "multidimensional modeling". The
artificial intelligence course may require the development of competencies in
"multiagent systems design", and so on.

The research presented here is based on this organization and notion of
competencies. Our project aims to facilitate the access to learning objects that seem
to be more appropriate at certain times, according to students' features as well as to
competencies that need to be developed and distributed across the planning of a
course.

Among the computational techniques to assist in the search for relevant
information, Recommender Systems [4] are able to automatically identify contents
that are appropriate for each individual based on their characteristics or "tastes." This
paper describes a model for recommender systems that is able to suggest learning
objects relevant to undergraduate students, focusing on competencies to be developed.
This paper is structured as follows: Section 2 extends the discussion on the concept of
competence, emphasizing its implications in the development of the recommender
system. Section 3 presents the technique of Collaborative Filtering, while section 4
describes in detail the proposed model, as well as its prototype and a set of
experiments carried out to validate the model. Section 5 presents conclusions and
proposals for future work.

## 2 The Concept of Competence

It is possible to find in the literature several definitions for the concept of competency.
The Cambridge English dictionary defines it as: "an important skill that is needed to
do a job". This definition, however, does not explicit relationships between important
concepts such as: skills, issues, knowledge. Other authors enhance the definition:
competence can be defined as the set of knowledge, skills and attitudes necessary for
a person to develop their roles and responsibilities [5]. Perrenoud [6] defines
competence as the ability to mobilize a set of cognitive resources (knowledge, skills,
information, etc) to address the appropriateness and effectiveness of a variety of
situations. In this way the skills are linked to cultural, professional and social
conditions.

In all definitions, we can easily see the relationship between the concept of
competence and skills (know-how), knowledge and attitudes. Therefore, the question
arises as to when and how we can make a recommendation of a learning object which
may enable students to build knowledge related to specific issues, to develop
particular skills related to given contents, to develop a critical awareness about the
importance of competence to understand how to use it.
Thus, our goal in this research has been to use a recommender system to filter relevant information to students, and to select learning objects that are most appropriate in accordance with the competencies to be developed. The next section introduces the concept of recommender systems.

# 3 Recommender Systems

Several content retrieval applications try to assist users in identifying items of interest. However, it is common that these applications return irrelevant contents as a result [4]. Trying to minimize this problem, recommender systems have emerged, focusing on the search for relevant information according to users' features, as well as to certain requirements of the items sought. Different techniques may be used in recommender systems to find the most appropriate contents for users. In Sarwar et al. [7], for example, different algorithms are compared according to accuracy and performance. Here, our focus is the technique of Collaborative Filtering (CF) [8], a technique that is based on information collected about the entire community of users and has already proved suitable for several applications [9].

## 3.1. Collaborative Filtering

Collaborative Filtering is based on one of the most popular techniques for recommendation and is used in many systems on the Internet [10]. The technique is based on the analysis of preferences in a group of people who have common interests and have a "taste" for similar items. According to this approach, contents that may be recommended are filtered based on the evaluation of users about the items. For each user, a set of "neighbors" with similar behavior is sought [4]. The following subsection presents computational details about how to compute the coefficient of similarity between two users, one of the first steps in the collaborative filtering process. Then, the method to select subsets of users with a high similarity degree is described. In this last step, we present how to compute predictions in order to indicate how appropriate an item is for a particular user.

### 3.1.1 Computation of the Similarity and Prediction Coefficients

To compute the similarity between students, the model proposed in this paper applies the Pearson's coefficient, which is an approach widely used in Recommender Systems based on Collaborative Filtering [8]. This coefficient measures the degree of correlation between two variables, resulting in values between -1 and +1, where the value -1 represents complete lack of correlation between variables, and the value +1 represents a strong correlation between them.

Once you get the correlation between the opinions of students on certain learning objects, it is possible to compute the prediction of how much students would appreciate receiving a specific recommendation (prediction refers to the evaluation the student would give the object if he/she had access to it). This is done
independently of the coefficient used for computing the similarity degree, since the predictions are computed according to a weighted average of the ratings given by students identified as closest neighbors (individuals who obtained a similarity coefficient less than a predetermined threshold). Through equation 1 one may determine how suitable object i would be for student a (p<sub>ai</sub>), where r<sub>a</sub> is the average of grades given by student a; n is the number of students; r<sub>bi</sub> is the grade each student b gave to object i, r<sub>b</sub> is the average of grades given by student b; and corr<sub>ab</sub> is the correlation of the target student a with a particular student b.

$$p_{ai} = \bar{r}_a + \frac{\sum_{b=1}^{n} (r_{bi} - \bar{r}_b) * corr_{ab}}{\sum_{b=1}^{n} |corr_{ab}|} \quad (1)$$

### 3.1.2 The Recommendation Process applying Competencies

After the prediction value of an object has been computed, indicating how a particular student would rate that item, the rules of competencies have to be applied. These are designed to filter out learning objects according to competencies that need to be developed. Through this procedure, the system is able to identify learning objects with a high predictive factor that are likely contribuite to the development of competencies noted as important at certain times.

After applying the rules of competencies, the system checks the database for recommendations whenever a student logs in.

## 4 Proposed Model

The recommender model proposed here has been designed as a service to be made available in an application server. The model proposes the following sequence of steps for its operation:

1) Teachers plan their classes based on the competencies detailed in the course description. In their plans, learning objects for the development of those competencies are suggested. For example, in the fifth meeting of a database course, students should develop the competence of "developing projections through the use of relational algebra". Learning objects that can be used in the development of this competence should also be listed by the teacher;

2) At the first time the system is run, the similarity degree (Pearson's coefficient) between all users registered in the database has to be computed. Then, the system calculates the prediction value for all registered contents in the database. Afterwards, only new users, new contents or new objects that were rated are processed;

3) After the computation of the similarity and prediction coefficients (subsection 3.1.1), the system employs the rules of competencies to filter out the content for a
particular user. For example, the learning object for "developing projections through the use of relational algebra" may be recommended to a student as a consequence of his/her similarities with other students that rated the object positively, and because the student is starting to study topics that demand the development of this competence.

## 4.1. Prototype and Experiments

A prototype of the model was developed in order to evaluate its efficiency in making appropriate predictions. Initially, some students were invited to participate in an experiment for the evaluation of learning objects (in this case scientific papers) that were recommended by the system. The scale used to evaluate learning objects was a Likert scale of 5 points. A tool for the evaluation of learning objects was developed as a web page that could be accessed by any device with a browser and an Internet connection. Figure 1 shows the interfaces of the prototype running in a PDA (Personal Digital Assistant).

<figure><img src="image_2.png" /><figcaption>**Fig 1.** (a) Prototype interface for mobile devices papers</figcaption></figure>

<caption>(b) Interface for the evaluation of</caption>

The prototype was developed in Java, and the persistence layer was developed using the JDBC API. The database was implemented using MySQL5.
### 4.1.1 Research Method

The prototype was evaluated in two experiments with convenience samples of 10 students that were at the end of the undergraduate course of Computer Engineering. A number of papers in the area of databases were selected by a teacher, and were classified according to the competencies that had to be developed. The experiments had the following goals:

1) To evaluate whether the prediction rate calculated by the prototype matched students' rates, using the evaluation metric *MAE* (Mean Absolute Error) [11];

2) To evaluate the accuracy of the recommendations made by the system through the metrics *Recall* (coverage) and *Precision*.

In the context of recommender systems, it is said that accuracy is the relationship between the rating given by the user to an item and the rating computed by the prediction system. To evaluate the prototype, the metrics described below have been applied.

### 4.1.2 Evaluation Metrics

The *MAE* metric was used to compute the mean absolute deviation between the predictions of the recommender system and the actual rate given by students. The difference obtained was taken as the prediction error [9]. With this metric in hand, it was possible to evaluate how accurate were the recommendations made. Equation (2) presents the MAE formula, where $p_i$ represents the values predicted by the system, $r_i$ are users' ratings to items recommended, and $n$ represents the number of items considered.

$$|E| = \frac{\sum_{i=1}^{n} |p_i - r_i|}{n} \qquad (2)$$

The metric *Precision* is widely used in information retrieval, and represents the ratio between the number of relevant contents and the total contents returned by a search function. In the context of recommender systems, *Precision* is the ratio between the number of items that the user considers relevant and the number of items recommended. In turn, the metric *Recall* is based solely on the number of recommended items, while *Precision* estimates how much the contents are relevant to the user [11]. *Recall* (Equation 3) and *Precision* (Equation 4) are defined as:

$$Recall = \frac{|RA|}{R} \qquad (3) \qquad Precision = \frac{|RA|}{A} \qquad (4)$$

$RA$ is the number of relevant items recommended by the system, $R$ is the total number of items that should be recommended, $A$ is the number of items actually recommended by the system, taking into account positive results and false positives.
## 4.2 Experiments and Results

The first experiment was set in order to select an initial base of learning objects for the databases course, considering the competencies that had to be developed. Thirty papers were selected and catalogued by the teacher, relating them to the competencies that had to be developed. Thus a database with a 1:N cardinality was built, where each competence had N papers cataloged to be used during the course. The competencies were distributed according to the class plan, and the learning objects were organized according to the competencies they represented. These papers were then reviewed and rated by students, which is a way to minimize a collaborative filtering limitation known as *cold start* [9].

Students were then requested to evaluate papers that had been allocated randomly. These papers were distributed to students, thereby generating an array of initial assessments "Paper x User", totalling 103 ratings. On the average, 10.3 items were assigned to be evaluated by each student, and these initial assessments were meant to identify the preferences/profile of each student. Based on the assessments provided, the system could compute the coefficient of similarity between the students (Pearson coefficient, ranging from -1 for a weak correlation, to 1 for a strong correlation). For example, the correlation value between students U4 and U6 was 1, showing a strong correlation according to Pearson's. For students U2 and U4, a correlation value of -1 was computed, meaning a total lack of similarity between them. As a result, it was found that 27.59% of the correlations computed between students were considered strong (these students had "tastes" that were similar to the objects evaluated), 20.69% were considered weak (these students had "tastes" different from the objects evaluated). For 51.72% of the correlations computed, nothing could be said.

In a second experiment, students evaluated a number of scientific papers recommended to them based on the rates provided in the first experiment. The system recommended only items with prediction rates higher than 3, for a Likert scale of 5 points. After computing the correlation values and the similarity prediction values, the rules of competencies were applied to select the most appropriate papers. As reported earlier, the papers were cataloged by competencies and were distributed in the class plan for each meeting. For instance, the lesson about "developing projections through the use of relational algebra" was assigned for the period starting in the eighth meeting and ending in the tenth meeting. Students during this period should receive recommendations of learning objects suitable for the development of this competence.

Table 2 shows the *MAE* and *Precision* values computed for the second experiment. The *MAE* metric showed an average difference between the predictions made by the prototype and the assessments made by students (1). The accuracy achieved by the prototype's predictions was 76%, which was considered satisfactory. Table 1 shows that the integration of collaborative filtering with the filter of competencies produced 16 recommendations, the same as the number of meetings planned for the classes. All of these recommendations were tailored to the development of competencies. It should be noted that the student U4 does not appear in Table 1 as he/she did not evaluate the papers recommended by the system.
<table><caption>**Table 1.** Results of the experiment concerning Precision</caption><thead><tr><th>User</th><th>Recommended Article</th><th>Prediction</th><th>User evaluation</th><th>MAE</th><th>Precision</th></tr></thead><tbody><tr><td>U1</td><td>19</td><td>4,54</td><td>5</td><td>0,46</td><td>90,80%</td></tr><tr><td>U1</td><td>22</td><td>3,35</td><td>4</td><td>0,65</td><td>83,75%</td></tr><tr><td>U1</td><td>23</td><td>3,62</td><td>4</td><td>0,38</td><td>90,50%</td></tr><tr><td>U2</td><td>20</td><td>4,04</td><td>5</td><td>0,96</td><td>80,80%</td></tr><tr><td>U3</td><td>20</td><td>3,29</td><td>4</td><td>0,71</td><td>82,25%</td></tr><tr><td>U3</td><td>22</td><td>3,92</td><td>4</td><td>0,08</td><td>98,00%</td></tr><tr><td>U3</td><td>23</td><td>4,37</td><td>5</td><td>0,63</td><td>87,40%</td></tr><tr><td>U5</td><td>19</td><td>4,53</td><td>3</td><td>1,53</td><td>66,23%</td></tr><tr><td>U5</td><td>22</td><td>3,67</td><td>3</td><td>0,67</td><td>81,74%</td></tr><tr><td>U6</td><td>19</td><td>5</td><td>2</td><td>3</td><td>40,00%</td></tr><tr><td>U7</td><td>19</td><td>3</td><td>1</td><td>2</td><td>33,33%</td></tr><tr><td>U7</td><td>22</td><td>3</td><td>2</td><td>1</td><td>66,67%</td></tr><tr><td>U8</td><td>20</td><td>5</td><td>4</td><td>1</td><td>80,00%</td></tr><tr><td>U9</td><td>19</td><td>5</td><td>3</td><td>2</td><td>60,00%</td></tr><tr><td>U10</td><td>20</td><td>3,5</td><td>3</td><td>0,5</td><td>85,71%</td></tr><tr><td>U10</td><td>23</td><td>4,5</td><td>4</td><td>0,5</td><td>88,89%</td></tr><tr><td>Média:</td><td></td><td></td><td></td><td>1,00</td><td>76,00%</td></tr></tbody></table>

Table 2 presents the experimental results concerning the relevance of the recommendations for the students.

<table><caption>**Table 2.** Results of the experiments concerning the relevance of the recommendations</caption><thead><tr><th>User</th><th>Number of articles recommended by the CF algorithm</th><th>Number of articles recommended by CF and competency</th><th>Number of relevant art. recommended by CF and competency</th><th>Recall</th><th>Precision</th></tr></thead><tbody><tr><td>U1</td><td>7</td><td>3</td><td>3</td><td>0,75</td><td>1,00</td></tr><tr><td>U2</td><td>7</td><td>1</td><td>1</td><td>0,25</td><td>1,00</td></tr><tr><td>U3</td><td>7</td><td>3</td><td>3</td><td>0,75</td><td>1,00</td></tr><tr><td>U5</td><td>4</td><td>2</td><td>2</td><td>0,50</td><td>1,00</td></tr><tr><td>U6</td><td>5</td><td>1</td><td>0</td><td>-</td><td>-</td></tr><tr><td>U7</td><td>3</td><td>2</td><td>0</td><td>-</td><td>-</td></tr><tr><td>U8</td><td>8</td><td>1</td><td>1</td><td>0,25</td><td>1,00</td></tr><tr><td>U9</td><td>7</td><td>1</td><td>1</td><td>0,25</td><td>1,00</td></tr><tr><td>U10</td><td>5</td><td>2</td><td>2</td><td>0,50</td><td>1,00</td></tr><tr><td>Total:</td><td>53</td><td>16</td><td>13</td><td></td><td></td></tr></tbody></table>

For 7 out of the 10 students, we can say that the system succeeded to recommend appropriate items that were relevant for the users to develop competencies in the period of time established, as the *Precision* factor reached 1, and the *Recall* factor was below this value. Students U6 and U7 received, respectively, one and two recommendations each, recommendations based on collaborative filtering and competencies. However, the ratings that these users gave to the recommendations
were not satisfactory for the computation of the prediction values, which had to be at
least 3. Therefore, it was not possible to say anything about these students in
particular. Again, additional assessment would be necessary to reach any conclusion
regarding these students.

**5. Conclusion**

The main goal of this research has been to integrate collaborative filtering with a mechanism based on competencies in order to (a) help students find educational materials related to the development of specific competencies, (b) give some flexibility to the suggestion of materials to be consulted or used by students, to the extent that the interests of each are considered in the process of recommendation. By supporting students in their learning process, the system proposed here also gives support to the work of teachers, by organizing materials and learning situations, and by providing better links between knowledge and competencies.

In this sense, the proposed model tries to be aligned with these principles, recommending learning objects that best suit the interests of the student, but in accordance with competencies that need to be developed at any given time. Through empirical experimentation with a group of undergraduate students in Computer Engineering, it was found that the degree of precision achieved by the recommender system was satisfactory. The accuracy level of 76% showed that the system was able to recommend learning objects that satisfied the students for their studies, without neglecting the competencies required in the summary of the courses they were taking. As for the evaluation metrics *Precision* and *Recall*, it can be said that the prototype succeeded to get the students to have access to those materials that were relevant to the competencies to be developed in that moment, considering the set of learning resources available.

As future work, we intend to test the system with other types of learning objects to verify if its performance remains satisfactory. Using information from learning objects' metadata is another future goal. Such feature should allow us to select objects according to specific requirements also related to competence development (e.g. level of difficulty, level of interaction, etc.). We are also working on the forming of virtual communities with users with a similarity coefficient within a certain range.

**References**

1. Wiley, D. A. Learning object design and sequencing theory. Doctoral thesis, Brigham Young University. (2000)

2. Zarifian, P. La politique de la compétence et l'appel aux connaissances à partir de la stratégie d'entreprise post-fordiste. Contribution to the Coloque of Nantes, December 13, 2002. Available at: http://www.scoplepave.org/ledico/auteurs/zarifian%20competence%201.htm. (2002)

3. Fleury, A. C. C. and Fleury, M. T. L. Estratégias empresariais e formação de competências. São Paulo: Atlas. (2000).
4. Adomavicius, G. and Tuzhilin, A. Toward the Next Generation of Recommender Systems: A Survey of the State-of-the-Art and Possible Extensions. In *IEEE Transactions On Knowledge and Data Engineering*, vol. 17, no. 6, pages 734-749. (2005)

5. Dutra, J. S. Gestão por Competências. São Paulo: Editora Gente (2001).

6. Perrenoud, P. Construir as competências desde a escola. Porto Alegre: Porto Alegre. (1999).

7. Sarwar, B. G., Karypis, G., Konstan, J., Riedl, J. Analysis of recommender algorithms for ecommerce. In *ACM E-Commerce Conference*, Minneapolis, MN, pages 158-167, ACM Press. (2000).

8. Shardanand, U. and Maes, P. Social information filtering: Algorithms for automating "word of mouth", In *Human Factors in Computing Systems*, Denver, Colorado, USA. (1995).

9. Herlocker, J., Konstan, J., Terveen, L., Riedl, J. Evaluating Collaborative Filtering Recommender Systems. In *ACM Transactions on Information Systems*, Vol. 22, No. 1, pages 5-53. (2004).

10. Schafer, J. B., Konstan, J., Riedl, J. E-Commerce Recommendation Applications. In *Data Mining and Knowledge Discovery*, Hingham, MA, USA, Vol 5. Issue 5, Kluwer Academic Publishers. (2001).

11. Cazella, S.C. and Corrêa, I. and Reategui, E. Um modelo para recomendação de conteúdos baseado em filtragem Colaborativa para dispositivos móveis. *Revista Novas Tecnologias na Educação*, v. 7, pages 12-22. (2008).