[AIED 2023] Who and How: Using Sentence-level NLP to Evaluate Idea Completeness
Channel: Martin Ruskov
Date: 2023-06-23

A poster presentation at the AIED 2023 conference in Tokyo.

Presented here is a study of grammatical analysis using natural language processing to evaluate the completeness of ideas proposed within an online learning environment in order to provide real-time feedback. We examine how well an open source Natural Language Processing framework is able to measure completeness of free-form user contributions. More precisely, we aim to classify two features in ideas generated by learners during security training: 1) whether the idea indicates a responsible stakeholder (who) and 2) what is the relevant necessary action (how).

To perform these classifications, we apply Part-of-Speech Tagging and Dependency Parsing on contextualised short texts, written by learners and collected during the use of a publicly accessible website, called the CCO Toolkit. We propose simple grammatical heuristics for the two classification tasks and test them on a dataset of 1174 contributions in the domains of information security and community safety. We compare the results of the classification against a ground-truth, annotated by two security experts.

Our results suggest that for the purposes of providing feedback in free input problem-solving exercises, generic transformer pipelines without fine-tuning can achieve very good performance on the identification of missing stakeholder and moderately good performance on missing relevant action. For the purposes of real-time online feedback both of these are satisfactory if it is provided in a non-intrusive manner. A further usability study would investigate acceptance and learning effects of such a new functionality. 

Further information at https://cco.works/opendata/