## 3 Methodology: Information Extraction and Teaching Intervention
### Annotation Scheme Development  
The first stage was to develop an annotation scheme to categorize students' free-text responses, not only to provide a better understanding of the areas that concerned them but to enhance the performance of the later text-mining methods to identify issues automatically. Although there is no existing scheme for text mining (as this study is considered to be the first attempt automatically to analyze NSS open-text responses), [PERSON] et al. 's [20] findings on NSS free-text data provide a good starting point. After manually analyzing about a thousand responses (both positive and negative) to the NSS open-ended questions, [PERSON]'s study devised 29 categories to cover the whole range of data, with the top-10 most common categories covering 80% of the data. In our study, we too developed an annotation scheme with 10 categories, as shown Table 1.  
Our annotation scheme (Table 1) provided full coverage of the NSS core questionnaire categories (The NSS questionnaire was revised in 2017, after it was first introduced in 2005. The National Student Survey website ([[https://www.thestudentsurvey.com](https://www.thestudentsurvey.com)]([https://www.thestudentsurvey.com](https://www.thestudentsurvey.com)), accessed on 20 August 2021) provides only the latest version. The pre-2016 version can be found available online: [[http://www.bristol.ac.uk/academic-quality/ug/nss/nssqs05-16.html/](http://www.bristol.ac.uk/academic-quality/ug/nss/nssqs05-16.html/)]([http://www.bristol.ac.uk/academic-quality/ug/nss/nssqs05-16.html/](http://www.bristol.ac.uk/academic-quality/ug/nss/nssqs05-16.html/)) (accessed on 20 August 2021)). This is because, upon manually examining 10% of the free-text responses in our NSS dataset (used as case study from a UK university, consisting of 4400 responses 2014-2017), we found that because the NSS questionnaire uses a Likert scale that provides no additional fields for further comments, rather than to discuss new topics the students often used any opportunity for a free-text response to elaborate on their thoughts. We added three additional categories (Categories 6-8) for more specific topics, based on our examination of our NSS dataset, as well as [PERSON] et al. 's findings [20]. Although these open-ended questions invited negative comment (positive comments were already covered), we found that a large proportion of responses (29%) were either positive or general, for instance, 'Overall, it's been a good course' or 'No, negatives at this time'. We considered that this was due to students tending to want to start with a positive statement \begin{table}
\begin{tabular}{c c c} \hline \hline
**Category** & **Proposed Annotation Scheme** & **Covers NSS Core Questionnaire Categories** \\ \hline
1 & Overall quality of teaching &
\begin{tabular}{c} The teaching on my course \\ Personal development * \\ \end{tabular} \\ \hline
2 & Overall level of support &
\begin{tabular}{c} Academic support \\ Learning opportunities ** \\ \end{tabular} \\ \hline
3 & Assessment and feedback & Assessment and feedback \\ \hline
4 & \
\begin{tabular}{c} Organization, management, \\ and responses \\ \end{tabular} &
\begin{tabular}{c} Organization and management \\ Student voice ** \\ \end{tabular} \\ \hline
5 & Learning resources & Learning resources \\ \hline
6 & Teaching materials and curricula & N/A \\ \hline
7 & Placement and employability & N/A \\ \hline
8 & Student life and social support & Learning community ** \\ \hline
9 & Overall dissatisfaction & N/A \\ \hline
10 & Positive or general statements & Overall satisfaction \\ \hline \hline \end{tabular}  
* Category removed after 2016; ** Category added since 2017.  
\end{table} 
Table 1: Annotation scheme of problem statement categories used in this study and how it covers the NSS categories. or perhaps to think that comment was mandatory. Therefore, since we were targeting negative feedback, we created Category 10 for those statements that were nonessential to our study.