Mining for Confusion : Classifying Affect in MOOC Learners ’ Discussion Forum Posts

Interest in educational data mining (EDM) has grown exponentially in recent years [5]. The Journal of Educational Data Mining was established to accommodate the surge in EDM research activity; in the following years, Learning at Scale and the International Conference on Learning Analytics cropped up as well. Broadly, EDM ”is concerned with methods to explore unique types of data in educational settings and . to better understand students and the settings in which they learn,” [1]. Its applications cut across many domains within the educational setting, ranging from visualization of course activity to automating course construction. The advent of Massive Open Online Course (MOOC) platforms edX and Coursera in 2012 unleashed vast troves of data for researchers to mine. Much effort has gone into developing ”student models” – predicting students’ nextquestion-accuracy and modeling their acquired skills. Some argue that such research is reaching the point of diminishing returns [1]. In contrast, relatively little research has been devoted to digging into MOOC discussion forums. Learners often resort to discussion forums in order to better understand the subject matter at hand. MOOC enrollments, however, are large: The typical Coursera class, for example, consists of more than 40,000 students [3]. These learners’ posts get drowned out in a sea of other, non-academic posts, especially since some learners use forums as conversational chatrooms. As an unfortunate result, instructors might overlook questions posed by struggling learners, discouraging both forum participation and learner motivation [4]. Ideally, the MOOC platform would automatically feed instructors with posts relevant to them. Instructors could then simply answer posts out of this intelligent queue, relieving them of the burden of sifting through thousands of discussion posts by hand. Such a platform would empower learners, too, and would more generally boost the usefulness of discussion forums. In this paper, we take an initial step towards an intelligent queue by presenting a pipeline that automatically surfaces posts that exhibit confusion. We say that a post exhibits confusion if it appears that its author either explicitly requests for clarification about a course topic, or if his language implicitly reveals a gap in comprehension. In contrast, we say that a post exhibits knowledge if it delivers factual information relevant to the topics studied in the course. If the post conveys neither confusion nor knowledge, we say that it is neutral. While EDM work on analyzing sentiment in MOOC forums exists [6], to the best of our knowledge, no previous work has explicitly attempted to identify confusion in posts. Other EDM efforts have attempted to predict instructor intervention patterns [2]; such an approach will only be successful in generating an intelligent queue if the instructor is already effective in answering relevant forum posts, which might not be the case. We instead take the approach of discovering content that we a priori believe will be relevant to instructors, regardless of their discussion forum history. We frame the problem of retrieving confused posts as a multilabel classification problem, of the following form: Given the body of a discussion forum post P with a true unknown label L in { knowledgeable, neutral, confused }, apply some hypothesis h that correctly divines L. That the problem is a multilabel one is of significance: Solving the easier binary classification problem in which posts are labeled as confused or not-confused would force us to forfeit potentially useful information about where a post author lies on the knowledge-confusion spectrum. The remainder of this paper is organized as follows. We describe our dataset in section two, detail our preprocessing, feature extraction and feature selection pipelines in section three, list our models in section four, present our results in section five, interpret them and discuss insights gleaned in section six, and propose future work in section seven.