Automatically Grading the Use of Language in Learner Summaries

Abstract: A summary is a short clear description that gives the main facts or ideas about something. Summaries are widely used in traditional teaching as an educational diagnostic strategy to infer comprehension, or how much information from text is retained in memory. From its early beginning Computer Supported Learning Systems research has faced open-ended learners’ response evaluation. Global summary grading includes partial discourse assessment on relevant issues such as coherence, cohesion, comprehension, adequacy and use of language. This paper describes the procedure followed searching for the best available approach to model the use of language grading of learner summaries written in Basque language. Keywords: Automatic summary grading, use of language Introduction In educational contexts, a summary contains the main ideas which learners recall from text. Thus, summaries are widely used in traditional teaching as an educational diagnostic strategy to infer comprehension, or how much information from reading text is retained in memory [1-3]. Broadly speaking, there are two ways to approach learning diagnosis: close-ended and open-ended. The close-ended evaluation (multiple choice test, ordering exercises, etc.) has the advantage of being countable, assures the possibility of objective assessment and makes possible accurate computation. Therefore, it has been most widely used in Computer Supported Learning Systems (CSLSs) for learning evaluation. But, the close-ended mode also involves a greater probability of scoring by chance and restricts the freedom and richness of learner responses. Open-ended evaluation assesses free text. This assessment mode, although less accurate, has also been present in Artificial Intelligence and education since the very early works in Socratic dialog systems [4-8]. By using free text learners have the freedom to write anything a nd can construct their own answer. Therefore, it is more likely to obtain a better approximation to real learners’ knowledge in open-ended modes. However, automatic evaluation in open-ended modes is complex and has to face uncertainty. Still, new developments in Natural Language Processing (NLP) allowed a rebirth with a variety of open-ended approaches in various applications: dialogue systems [9-12], feedback in narratives [13], etc. The work presented in this paper has been carried out in the context of an automatic summary grading environment, LEA [14]. Relevant variables identified when producing a summarisation environment are: text related (text type, text present/absent, theme and text length), discourse related (adequacy, coherence, cohesion, use of language and comprehension), learner related (learner level and learner’s prior knowledge) and aid tools (dictionaries, spell and grammar check, etc.). The above variables have been identified after a deep analysis of both summary grading and human summary grading performance to model their criteria [15]. In the context of this work, global summary grading decisions are gained by means of a Bayesian Network model based on discourse measures on cohesion, coherence, comprehension, adequacy and