# Who wants to chat on a MOOC? Lessons from a peer recommender system

François Bouchet, Hugues Labarthe, Rémi Bachelet, Kalina Yacef

► To cite this version:

François Bouchet, Hugues Labarthe, Rémi Bachelet, Kalina Yacef. Who wants to chat on a MOOC? Lessons from a peer recommender system. EMOOCs 2017, May 2017, Madrid, Spain. pp.150-159, 10.1007/978-3-319-59044-8_17. hal-01522736

HAL Id: hal-01522736

https://hal.science/hal-01522736

Submitted on 15 May 2017

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution - ShareAlike 4.0 International License
# Who wants to chat on a MOOC?
## Lessons from a peer recommender system

François BOUCHET¹, Hugues LABARTHE¹, Rémi BACHELET²
and Kalina YACEF³

¹ Sorbonne Universités, UPMC Univ Paris 06, CNRS, LIP6 UMR 7606

² Centrale Lille, University of Lille, France

³ School of Information Technologies, The University of Sydney, Australia
francois.bouchet@lip6.fr, hugues.labarthe@ac-creteil.fr,
remi.bachelet@ec-lille.fr, kalina.yacef@sydney.edu.au

**Abstract.** Peer recommender systems (PRS) in MOOCs have been shown to help reducing attrition and increase performance of those who use them. But who are the students using them and what are their motivations? And why are some students reluctant to use them? To answer these questions, we present a study where we implemented a chat-based PRS that has been used during a MOOC session involving 6,170 students. Our analyses indicate that PRS-users are students unsatisfied by other means of interactions already available (forums, social networks...), and that they seem to use it more to share emotions than to learn together, or to assess their progression against their peers.

**Keywords:** synchronous chat, peer recommender system, synchronous collaboration in MOOC, social learning

## 1 Introduction

Recommender systems (RS) have become the key to industrial success in several sectors (e.g. online stores, music or movie services...) and many of us rely on them daily. They have also been used in the field of Technology Enhanced Learning, mostly to recommend resources or other classes relevant to follow [1, 2]. A specific kind of RS is those used to recommend other persons (persons to follow on social networks, potential mates on dating services...), which in a learning context can take the form of peer recommender systems (PRS) [3]. It is therefore logical to wonder what a PRS, a system allowing students to interact with each other while learning, could bring to an MOOC. Previous studies suggest that such a tool can not only help reducing students' attrition [4], as lack or loss of relationship is a good predictor of attrition [5], but also increase success and participation [6]. Beyond the issues relative to the identification of features to be used to make a good recommendation of a learner to another, we also need to better understand the users' motivations for using such a tool [7]: in our context, this means knowing whether MOOC learners would have an interest in a PRS and actually use it, and what distinguishes these learners from those who choose not to use it.
This profiling would help us not only to improve the recommendation algorithms, but also to better understand what aspects currently missing in the MOOC experience these learners are seeking through the chat. It is even more interesting to study as reviews of the use of RS in education mention little to no use of RS as PRS [1, 2].

The question raised by a peer recommender is related to a more general issue, identified by researchers in Computer Supported Collaborative Learning (CSCL), since 1995. Social learning has been demonstrated to be, in many contexts, the best way to learn, as it allows students to identify and correct their misconceptions, as well as to improve their overall understanding by sharing it with others [8]. When dealing with thousands of students, it is obvious that some are not as involved as others for various reasons that may involve a combination of trust, confidence, shyness, willingness to share, willingness to help others... Social Learning researchers are interested in analyzing the different ways for somebody to be engaged through an analysis of the traces they leave when socializing.

In this paper, we aim at having a better understanding of this issue. To foster learners’ discussions in MOOCs, we have designed, implemented and tested a PRS. Our recommender provides each student with a list of potentially relevant persons and a way to contact them directly through an instant chat system. This list is based on what the recommender knows about the student’s profile and activities in the MOOC. We hypothesize that the students who have the keenest interest in social learning would make the most active use of the recommender. The remainder of this paper is organized as follows: in section 2, we present the context and design of our chat-based PRS and give an overview of its functioning. In section 3, we present the actual experiment, including the data collected, its preprocessing and the way it has been analyzed and the results provided by the statistical analyses performed to identify correlations between the expressions of interest for a peer recommender and its effective use. We then conclude the paper with a discussion on the lessons learned, their limits and the perspectives they open for future work.

# 2 Recommender in the GdP8 MOOC

## 2.1 Context of the experiment

In a first controlled-study conducted during a Project Management MOOC, we observed a significant improvement of students’ engagement, not only for those who interacted with the recommendations, but more largely, for all of those who accepted using the PRS [6, 9]. We also collected critical feedback from users and stakeholders concerning the design flaws of the widget (visual intrusiveness, difficulty to understand the recommendation mechanism, chat and recommendation system being two distinct elements of the interface...), the lack of interest for recommendation, and the potential features overlap with forums and social networks. Therefore, it appeared interesting to investigate the actual role of this peer recommender: who used it, who did not, and why.
This paper exploits data recently collected with a new version of our PRS imple-
mented within the 8th GdP MOOC on project management, which took place between
September and November 2016. The MOOC relies on a proprietary modified version
Canvas platform made by Unow. In this MOOC, two certification levels are available:
basic and advanced. Both require students to follow and pass 2 specialization modules
among 13 available. These modules are available on a separate specialization MOOC,
which opens on the third week of the “common core course” (cf. Figure 1 for a time-
line). The peer recommendation widget was only available on the specialization plat-
form, from its opening on week 3 to the MOOC end on week 10. On this platform,
students learn and take quizzes individually, and no social interaction is encouraged
through the pedagogy (e.g. no collaborative task or need to network with others). Not
giving access to the widget on the core MOOC allowed us to ask participants if they
were lacking interactions from the regular MOOC experience.

The GdP MOOC has a special place in the French MOOC ecosystem, as its first session (GdP1) was the first xMOOC (i.e. not run on connectivist principles) opened in France (Jan-Mar 2013). It was developed from an existing Open Course Ware (OCW) website [10] as an experience of running a distance learning course. It benefits from high enrolment, with over 130,000 distinct students having participated in the 8 sessions ran so far – some of them even returning several times [11] as the content evolves with each edition. Only the latest edition (GdP8 – Sep-Nov. 2016) is studied in this paper.

## 2.2 The peer recommender system & chat widget

The PRS and chat appear as a single widget visible on every page of the specialization module platform. It appears as minimized by default in the bottom right hand corner of the interface (cf. Fig. 2a), and can be expanded with a click. When minimized, a red cross allows users to opt out, which hides the button for the rest of the learning session. Once expanded, it consists of 3 main panels (Fig. 2c) accessible through tabs at the bottom of the interface. The first tab gives access to a list of discussion threads the student has created or has been added to. Each discussion is identifiable by the avatar of its creator and an excerpt of the first message. A click on a thread opens it (Fig. 2e), allowing the user to read the messages posted in it and to contribute by sending a message. Each student was added by default into an initial thread providing information about how the system worked. The second tab is used to initiate a discussion with a group of 20 persons chosen according to one of three sets of criteria: (1) randomly, (2) based on the student's demographics (age, location, socio-professional category...) or (3) based on the student's progression in the specialization modules (which modules they are following and how far they progressed, based on quizzes taken so far). Once a type of recommendation is chosen, students have to type their first message in the thread (Fig. 2d). The third tab gives access to a list of favorite contacts - initially empty, it can be filled by adding persons met through group chats (Fig. 2f). These persons can then be contacted individually. To ensure the widget was seen by everyone, a warning bubble (Fig. 2b) popped out after the first 3 pages visited. It disappeared forever after the student either opened the chat or ignored it for 5 pages. Each panel also benefited from a distinct help page, accessible through a question mark icon on the top right hand of
the interface. A dedicated forum thread was also opened to answer to specific questions regarding the PRS.

<figure><img src="image_2.png" /><figcaption>Fig. 1. An overview of the timeline for the GdP MOOC (8th edition)</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>Fig. 2. The peer recommender and chat widget: implementation and design</figcaption></figure>
# 3 Experiment

## 3.1 Methodology

**Participants.** 24,603 persons registered to the MOOC, among which N=6,170 logged at least once on the specialization modules platform (i.e. were still active in the MOOC by the time this platform opened in week 3) and were therefore exposed to the chat system and its associated PRS in the interface.

**Data collected.** Before the start of the MOOC and in week 3, before the start of the specialization modules, students were invited to fill in two questionnaires which included questions relative to their will to interact with others and socialize, as well as the issues encountered so far with the GdP MOOC. The questions relevant to this paper are summarized in Table 1. These questionnaires provided a few bonus points (less than 1% of the final grade) to encourage students to answer to them, but they were not mandatory, which explains a reduction of the sample size when considering students who filled them. To increase the reliability of the questionnaire, the same concept was sometimes tested through different questions: we therefore merged these questions by calculating the average answer value (between 1 and 7) for all questions corresponding to the same concept (cf. column "variable extracted"). Then the answers were dichotomized by excluding intermediate answers (e.g. if the value was between 3.5 and 4.5 for an aggregation of several 7 point Likert scale answers), which led to different sample sizes when considering each question.

In addition to this self-report data, we also considered log data collected in a database tracking the use of the PRS during the specialization modules session. We tracked whenever the MOOC participants were opening the widget, opening a discussion thread within that interface and sending a message to an existing thread or to a new one. This allows us to separate our sample of 6170 participants according to 3 criteria:

(1) whether they opened (N<sub>open</sub>) or not (N<sub>not-open</sub>) the widget at least once,

(2) whether they opened a discussion thread (N<sub>thread</sub>) or not (N<sub>no-thread</sub>) at least once,

(3) whether they sent at least 1 message to a discussion thread (N<sub>msg</sub>) or not (N<sub>no msg</sub>).

**Experimental procedure.** To compare PRS users from non-PRS users, we performed multiple Pearson's chi-squared tests for statistical independence between the 3 degrees of use of the chat (opening, reading a thread, sending a message) and the aforementioned 7 variables, i.e. we made the null hypothesis that the use of the chat was independent from each variable.

<table><caption>**Table 1.** List of variables extracted from research questionnaires</caption><thead><tr><td>Question (translated)</td><td>Answers</td><td>Variable extracted</td></tr></thead><tbody><tr><td>- It is important for me to succeed more than other students</td><td></td><td>GoalBetterThanOthers</td></tr></tbody></table>
<table><tr><td>- It is important for me to succeed well compared to others in this MOOC<br/>- My goal in this MOOC is to have a better grade than most of the other students</td><td>7 point<br/>Likert<br/>scale</td><td></td></tr><tr><td>- I'm not concerned by what others might think of me<br/>- I'm not concerned by the judgment of others<br/>- I don't worry about what others think of me</td><td>7 point<br/>Likert<br/>scale</td><td>FearOfBeingJudged*</td></tr><tr><td>- When I think of the MOOC activities, I feel an emotion I want to share</td><td>7 point<br/>Likert<br/>scale</td><td>WillToShareFeelings*</td></tr><tr><td>- I went to the forums to share my issues with other students<br/>- When I am stuck in the course, I seek advice from other students<br/>- I talk with other students about parts of the course that appear unclear<br/>- I interact with other students to know if we have understood the same thing<br/>- I go to social networks to share my issues with other students<br/>- I interact with other students to know how to handle the online courses</td><td>7 point<br/>Likert<br/>scale</td><td>WillToInteractToLearn*</td></tr><tr><td>- For you, how useful are the forums in this MOOC?<br/>- How useful are social networks (Facebook, Google Plus, Twitter, blogs...) to help you learn?</td><td>5 point<br/>Likert<br/>scale</td><td>PerceivedSocializationUsefulness</td></tr><tr><td>- I have contributed to the discussion forums<br/>- I have shared on social networks to ask or answer to questions, share my experience...</td><td>5 point<br/>Likert<br/>scale</td><td>PerformedSocialization</td></tr><tr><td>- One of the problems I face with this MOOC is that I lack being in contact with other learners</td><td>Yes/No</td><td>LackContactWithOthers</td></tr></table>

## 3.2 Results

**Use of the chat with the recommender system.** When separating the different participant samples based on the type of use of the widget, we found the following results:

- N<sub>open</sub>=3025 (49.03%) participants opened the widget at least once

- N<sub>thread</sub>=570 (9.24%) participants opened a thread at least once

- N<sub>msg</sub>=206 (3.34%) participants sent at least one message in a discussion thread (84 of which created at least one thread).

Although they represent a minority of the participants in the specialization modules, these figures are similar to the percentage of learners who tend to use social features in MOOCs – usually from 3 to 15% according to a literature review from [12], higher in platforms explicitly encouraging conversational learning (e.g. 24 to 45% of Future-Learn's learners post at least one comment [13]). But more importantly, the sample sizes are large enough to allow us to perform a more detailed study in order to identify the features that distinguish these learners from the ones who did not use the PRS.

<footnote>* for these variables, one or several attention filters were used, i.e. a fake question asking participants not to answer anything was used to filter out those who answered anything to it</footnote>
**Comparing PRS users and non-users.** The results (cf. Table 2) reveal that PRS users and non-users differ according to numerous characteristics. In particular, participants interested in doing better than others were more likely to send a message (although they opened the widget and threads in equal proportion to others). Conversely, being afraid of others' judgment was negatively correlated with the likelihood of opening a thread (moderately, p=.043) and sending a message (strongly, p=.002). Participants who expressed interest in sharing how they feel about the MOOC were more likely to welcome the PRS, opening it and using it very significantly more. The will to interact with others to learn was less correlated with the chat use, as only the action consisting in sending a message was significantly more performed by those who liked learning with others. Finally, learners who consider socialization to be useful, those who already socialized either through the forum and/or through social networks as well as those who declared not having enough contact with others were all very significantly more likely to engage with the PRS (by opening it, reading the threads and sending messages).

<table><caption>**Table 2.** 3x7 Chi-squared test results: contingency table and p-values</caption><thead><tr><th></th><th></th><th colspan="2">Opened the chat</th><th colspan="2">Opened a thread</th><th colspan="2">Sent a message</th><th rowspan="2">n</th></tr><tr><th></th><th></th><th>0</th><th>1</th><th>0</th><th>1</th><th>0</th><th>1</th></tr></thead><tbody><tr><td>GoalBetter-ThanOthers</td><td>0</td><td>1222</td><td>1375</td><td>2323</td><td>274</td><td>2514</td><td>83</td><td rowspan="2">3748</td></tr><tr><td></td><td>1</td><td>511</td><td>640</td><td>1008</td><td>143</td><td>1090</td><td>61</td></tr><tr><td></td><td></td><td colspan="2">p &gt; 0.05</td><td colspan="2">p &gt; 0.05</td><td colspan="2">p &lt; 0.01**</td></tr><tr><td>FearOfBeing-Judged</td><td>0</td><td>230</td><td>315</td><td>471</td><td>74</td><td>513</td><td>32</td><td rowspan="2">3098</td></tr><tr><td></td><td>1</td><td>1177</td><td>1376</td><td>2283</td><td>270</td><td>2474</td><td>79</td></tr><tr><td></td><td></td><td colspan="2">p = 0.097</td><td colspan="2">p = 0.043*</td><td colspan="2">p = 0.002**</td></tr><tr><td>WillToShare-Feelings</td><td>0</td><td>596</td><td>589</td><td>1114</td><td>71</td><td>1176</td><td>9</td><td rowspan="2">2701</td></tr><tr><td></td><td>1</td><td>596</td><td>920</td><td>1271</td><td>245</td><td>1425</td><td>91</td></tr><tr><td></td><td></td><td colspan="2">p = 0.000***</td><td colspan="2">p = 0.000***</td><td colspan="2">p = 0.000***</td></tr><tr><td>WillToInteract-ToLearn</td><td>0</td><td>1069</td><td>1300</td><td>2105</td><td>264</td><td>2295</td><td>74</td><td rowspan="2">2844</td></tr><tr><td></td><td>1</td><td>201</td><td>274</td><td>415</td><td>60</td><td>450</td><td>25</td></tr><tr><td></td><td></td><td colspan="2">p = 0.261</td><td colspan="2">p = 0.352</td><td colspan="2">p = 0.020*</td></tr><tr><td>Perceived-Socialization-Usefulness</td><td>0</td><td>691</td><td>695</td><td>1266</td><td>120</td><td>1359</td><td>27</td><td rowspan="2">3176</td></tr><tr><td></td><td>1</td><td>674</td><td>1116</td><td>1521</td><td>269</td><td>1667</td><td>123</td></tr><tr><td></td><td></td><td colspan="2">p = 0.000***</td><td colspan="2">p = 0.000***</td><td colspan="2">p = 0.000***</td></tr><tr><td>Performed-Socialization</td><td>0</td><td>1176</td><td>1236</td><td>2208</td><td>204</td><td>2368</td><td>44</td><td rowspan="2">3799</td></tr><tr><td></td><td>1</td><td>457</td><td>930</td><td>1126</td><td>261</td><td>1264</td><td>123</td></tr><tr><td></td><td></td><td colspan="2">p = 0.000***</td><td colspan="2">p = 0.000***</td><td colspan="2">p = 0.000***</td></tr><tr><td>LackContact-WithOthers</td><td>0</td><td>1921</td><td>2250</td><td>3733</td><td>438</td><td>4023</td><td>148</td><td rowspan="2">4506</td></tr><tr><td></td><td>1</td><td>105</td><td>230</td><td>267</td><td>68</td><td>300</td><td>35</td></tr><tr><td></td><td></td><td colspan="2">p = 0.000***</td><td colspan="2">p = 0.000***</td><td colspan="2">p = 0.000***</td></tr></tbody></table>

**Why don't some people want to use a PRS in a MOOC?** Until now, we have focused on students who used the PRS, but in order to get a complementary perspective, we also asked the 1,631 participants who did not use the recommender system but who

<footnote>* p < 0.05; ** p < 0.01; *** p < 0.001</footnote>
noticed it to explain their decision not to give it a try (surprisingly, despite the warning bubble, 718 students declared not having noticed it, and 406 did not remember whether they had seen it or not). Participants were allowed to choose one or more options (or none, if none fitted their situation). Results are summarized in Figure 3. Overall, it appears that the main reason was the lack of a relevant question to ask, followed by a preference for the forums (although it is unclear whether it is to reach more persons or simply because the interface was more familiar to them as it was already accessible in the main section of the MOOC). Around 8% of them mentioned issues, either technical (e.g. the recommender might have not worked well on some exotic devices or browsers) or practical (e.g. difficulties to understand how to use the chat, despite the provided help system). Finally, a few participants mentioned they did not think others could help them with the issues they were having. No participants felt intimidated by the widget, which was the case in a previous version [9] where chat users had to choose individually the persons they wanted to interact with, based on their avatar and short bio (if available). This result is confirmed by the fact that nobody reported having issues about being in contact with unknown fellow learners.

<figure><img src="image_4.png" /><figcaption>Fig. 3. The identified causes for not using the chat interface</figcaption></figure>

## 3.3 Discussion

Chat users differed significantly from learners who chose not to use it. In particular, many were already users of other interaction media (forums, social networks...) and apparently were not fully satisfied by the interactions offered – as opposed to the non-users who stated that forums were enough for them. Moreover, combining the fact that a quarter of the non-users had no relevant question to ask and the fact that chat users needed to share feelings more than learning with others (which is consistent with [14]), it appears that chat users are mostly seeking a way to track other students' progress and emotions to reassure themselves. This seems even more crucial for those who like performing better than others, as MOOCs rarely provide mechanisms to benchmark oneself against others the way classrooms do. It suggests the chat was used by participants who feared being late compared with others: it would explain why the fear of being judged is highly correlated with the chances of sending a message.

It is interesting to confront the benchmark hypothesis with another observation: participants fearing judgment of others opened discussion threads in proportions similar to participants who did not, although they sent less messages. As we know from the questionnaire filled by people who did not use the chat, the fear to talk to unknown persons is not a valid explanation of this behavior. It could therefore be consistent with the aforementioned hypothesis, indicating a form of lurking behavior: they wanted to check
how well others were doing, without necessarily sharing their own progress – either
because they were doing better and it was enough for them to know it, or because they
were lagging behind and feeling too ashamed to let others know about it.

4 Conclusion

In this paper, we have reported the results of the use of a peer recommender chat
system in a MOOC, showing that participants interested in interacting with recom-
mended peers differed significantly from those who did not. The chat seemed to fill a
gap experienced by some of the MOOC participants who complain about the lack of
interaction with others despite the tools already available (forums, social networks...).
A detailed analysis of the answers also suggests the tool was mostly used to share emo-
tions more than to actually learn together, and that it seemed interesting even to the
eyes of some participants who did not want to post a message, as a way of reassurance
on how well or not the others are doing.

Although implementing the PRS only in the second half of the MOOC allowed us
to measure what participants felt was lacking, the fact it was not a feature available on
day 1 may have reduced the overall use of the tool. In particular, as we know it may be
used as a way to keep social links and not to drop off, we have probably lost participants
who might have stayed around longer with this tool, but who had already dropped off
by the time it became accessible in week 3. Another limitation of this work, relative to
the analysis, is that we did not consider the final grade obtain by students, which could
be an additional indication for explaining some of the observed behaviors. An analysis
of the sequences of actions could also be relevant to confirm some of the hypotheses
we are making here. Finally, the actual content of the discussion between participants
would also need to be analyzed and maybe compared to the content of the forums.

In future work, we intend to use a similar version of the PRS in a new session of this
MOOC that would be accessible from day 1 and to possibly encourage its use for learn-
ing activities (since it did not appear to be the most common use this time). We also
wish to make our PRS system an open source plugin so that it can easily be embedded
in any MOOC; we do not foresee any issue as it relies on information that is not specific
to the GdP MOOC. The results of this experiment also raise questions about the interest
of having an embedded tool to better keep track of other participants’ progress.

Acknowledgments

The authors would like to thank Unow, and in particular Régis Millet for their tech-
nical assistance.

References

1. Manouselis, N., Drachsler, H., Vuorikari, R., Hummel, H., Koper, R.: Recommender Systems in Technology Enhanced Learning. In: Ricci, F., Rokach, L.,
Shapira, B., and Kantor, P.B. (eds.) Recommender Systems Handbook. pp. 387–415. Springer US (2011).

2. Klašnja-Milićević, A., Ivanović, M., Nanopoulos, A.: Recommender systems in e-learning environments: a survey of the state-of-the-art and possible extensions. Artificial Intell. Rev. 44, 571–604 (2015).

3. Tang, T.Y., Daniel, B.K., Romero, C.: Recommender systems in social and online learning environments. Expert Syst. 32, 261–263 (2015).

4. Ferschke, O., Yang, D., Tomar, G., Rosé, C.P.: Positive Impact of Collaborative Chat Participation in an edX MOOC. In: Conati, C., Heffernan, N., Mitrovic, A., and Verdejo, F.M. (eds.) Artificial Intelligence in Education: 17th International Conference, AIED 2015, Madrid, Spain, June 22-26, 2015. Proceedings. pp. 115–124. Springer International Publishing, Cham (2015).

5. Yang, D., Wen, M., Rosé, C.P.: Peer Influence on Attrition in Massive Open Online Courses. In: Proceedings of the 7th International Conference on Educational Data Mining. pp. 405–406. , London, UK (2014).

6. Labarthe, H., Bouchet, F., Bachelet, R., Yacef, K.: Does a Peer Recommender Fos-ter Students’ Engagement in MOOCs? In: Barnes, T., Chi, M., and Feng, M. (eds.) Proc. of the 9th International Conf. on Educational Data Mining. pp. 418–423. In-ternational Educational Data Mining Society (IEDMS), Raleigh, NC, USA (2016).

7. Zheng, S., Rosson, M.B., Shih, P.C., Carroll, J.M.: Understanding Student Motiva-tion, Behaviors and Perceptions in MOOCs. In: Proceedings of the 18th ACM Con-ference on Computer Supported Cooperative Work & Social Computing. pp. 1882–1895. ACM, New York, NY, USA (2015).

8. Kirschner, F., Paas, F., Kirschner, P.A.: A Cognitive Load Approach to Collabora-tive Learning: United Brains for Complex Tasks. Educ. Psychol. Rev. 21, 31–42 (2009).

9. Labarthe, H., Bachelet, R., Bouchet, F., Yacef, K.: Increasing MOOC completion rates through social interactions: a recommendation system. In: Khalil, M., Ebner, M., Kopp, M., Lorenz, A., and Kalz, M. (eds.) Proc. of the European Stakeholder Summit on experiences and best practices in and around MOOCs. pp. 471–480. Graz, Austria (2016).

10. Bachelet, R.: Cours et MOOC de gestion de projet : formations en vidéo, ppt, pdf et modèles de documents, http://gestiondeprojet.pm.

11. Bouchet, F., Bachelet, R.: Do MOOC students come back for more? Recurring Stu-dents in the GdP MOOC. In: Proc. of the European MOOCs Stakeholders Summit 2015. pp. 174–182. , Mons, Belgium (2015).

12. Alario-Hoyos, C., Perez-Sanagustin, M., Delgado-Kloos, C., G, H.A.P., Munoz-Or-ganero, M.: Delving into Participants’ Profiles and Use of Social Tools in MOOCs. IEEE Trans. Learn. Technol. 3, 260–266 (2014).

13. Measuring our first eight courses, https://about.futurelearn.com/blog/measuring-our-first-eight-courses/.

14. Coetzee, D., Fox, A., Hearst, M.A., Hartmann, B.: Chatrooms in MOOCs: All Talk and No Action. In: Proceedings of the First ACM Conference on Learning @ Scale Conference. pp. 127–136. ACM, New York, NY, USA (2014).