This is the final peer-reviewed accepted manuscript of:

Zanellati, A., Zingaro, S.P., Gabbrielli, M. (2022). Student Low Achievement Prediction. In: Rodrigo, M.M., Matsuda, N., Cristea, A.I., Dimitrova, V. (eds) Artificial Intelligence in Education. AIED 2022. Lecture Notes in Computer Science, vol 13355. Springer, Cham, pp. 737–742

The final published version is available online at
[https://dx.doi.org/10.1007/978-3-031-11644-5_76](https://dx.doi.org/10.1007/978-3-031-11644-5_76)

Rights / License:

The terms and conditions for the reuse of this version of the manuscript are specified in the publishing policy. For all terms of use and more information see the publisher's website.
# Student Low Achievement Prediction

Andrea Zanellati<fnref n="1" />[0000-0001-6171-0397], Stefano Pio Zingaro<fnref n="1" />[0000-0002-8462-5651], and Maurizio Gabbrielli³[0000-0003-0609-8662]

University of Bologna, Bologna, Italy
{andrea.zanellati2,stefanopio.zingaro,maurizio.gabbrielli}@unibo.it

**Abstract.** In this paper, we propose a method for assessing the risk of low achievement in primary and secondary school. We train three machine learning models with data collected by the Italian Ministry of Education through the INVALSI large-scale assessment tests. We compare the results of the trained models and evaluate the effectiveness of the solutions in terms of performance and interpretability. We test our methods on data collected in end-of-primary school mathematics tests to predict the risk of low achievement at the end of compulsory schooling (5 years later). The promising results of our approach suggest that it is possible to generalise the methodology for other school systems and for different teaching subjects.

**Keywords:** low achievement · performance prediction · assessment tests

## 1 Introduction

Low achievement at school is a widespread phenomenon which has long-term consequences, both for the individual and for society as a whole. In 2016, above 28% of students across OECD <fnref n="1" /> countries underscored the minimum level of proficiency in at least one of the three core subjects according to the Programme for International Student Assessment (PISA), which are English reading and comprehension, mathematics, and science[14]. Low achievement is strongly related to school dropout, i.e., the discontinuation of education [7], and impact on the cultural and professional growth of the individual and citizen [3,12]. Indeed, school performance in first grade is already a significant indicator of future high dropout risk. In 2019, a study conducted by the National Institute for Assessment of the Education System (INVALSI) found that 20% of Italian students had a lower-than-expected achievement and, eventually, dropped out of school [17].

To counteract dropout as soon as possible and to detect low achievement, we address the following research questions:

**RQ1** Is it possible to quantitatively represent a student's knowledge level and build a model of his or her skill attainment?

**RQ2** Is it possible to develop a suitable AI-tool to predict, at an early stage, the risk of low achievement at secondary school for primary school students?

<footnote n="1">¹ OECD stands for Organization for Economic Co-operation and Development.</footnote>
In the following, we present a case study, focusing on the Italian context and using data collected from the INVALSI national large-assessment tests in mathematics. In particular, from these tests, we aim to extract the relevant features related to students' learning in terms of their skill and competence level performance.

In RQ2, we refer to “early stage” meaning to detect risk as soon as possible, i.e., several years in advance, so that appropriate countermeasures can be taken, and to design an intervention aimed at reducing risk when it is detected. Concretely, we develop three models able to predict the risk of low achievement at K-10, using student data at K-5.

In selecting the predictive model, we strive for a balance between interpretability and performance. Hence, we consider state-of-the-art machine learning techniques that proved to be effective in preliminary experiments [19]: random forests and neural networks [5,9]. On the one hand, we exploit random forests to extract rules that facilitate the process of interpreting the outcomes of the research and, on the other hand, we test neural networks for flexibility, e.g., exploring non-linear correlations, and performance gain.

## 2 Related Work

The topic of students' low achievement is a widely studied phenomenon in the social sciences and education [6,8]. The problem has also been addressed in terms of predictive models for low achievement or dropout risk for both high school and college students. These models exploit different machine learning techniques, including supervised learning, e.g., random forests, support vector machine and Bayesian network, unsupervised learning, e.g., k-means and hierarchical clustering, and recommender systems, e.g., collaborative filtering [16,2]. Moreover, several kinds of data have been used to tackle the problem. In [13] the dataset for building the predictive model uses demographic data of the students and their grades. Other studies are based on students performance, i.e., grades, collected during first semester courses [1,11]. Some datasets include behavioural data supplemented with other features related to learning results [18], in a mix of cognitive and non-cognitive characteristics. In some studies data collected through large-scale assessment tests were used to design predictive models of student performance through several machine learning techniques. In [15], for example, the authors refer to data collected through the PISA international large-scale assessment tests.

We aim to contribute to the research field of Artificial Intelligence-based education solutions by presenting a case study for predicting the risk of low achievement of high school students using their performance data collected during primary school. In addition, we extract features directly related to students' learning in terms of knowledge and skills, privileged indicators for the study of learning [4], thus proposing a knowledge-based method for encoding students' learning. We believe that this element can improve the interpretability of the results and make this tool useful for students, teachers and instructional coordinators.
## 3 Methodology

The INVALSI dataset is the result of a large-scale assessment administered in Italy since the school year 2002/03 at the levels K-2, K-5, K-8, K-10, and K-13. In our case study, we considered data on maths test from two cohorts of students: K-5 of the 2012/13 school year and K-5 of the 2013/14 school year. For the same students, we collected data from five years later at grade K-10, to be used for the definition of the low achievement target, i.e., the students grade in the test is less than or equal to 2 on a scale from 1 to 5. After merging K-5 datasets with their correspondent K-10 targets, the K-5 2012/13 cohort is made up of 351746 students, while the K-5 2013/14 cohort of 354987 students.

There are several features in the dataset and we applied a feature selection process to determine a subset of relevant features. The datasets also contain a boolean feature for each test item, where the students' answers correctness are recorded. To enable the use of our predictive models on different cohorts of students it is necessary to release the dataset from the individual items that constitute a certain test. Therefore, we used a knowledge-based approach considering the items classification in terms of areas, processes and macro-processes according to the INVALSI framework for the design of math tests. In Table 1, we give for reference an overview of the areas, processes, and macro-processes that have been used in the encoding of the questions.

<table><caption>**Table 1.** Maths INVALSI framework for question encoding.</caption><thead><tr><th></th><th></th></tr></thead><tbody><tr><td><strong>Areas</strong></td><td></td></tr><tr><td>(NU) Numbers</td><td></td></tr><tr><td>(SF) Space and figures</td><td></td></tr><tr><td>(DF) Data and forecasts</td><td></td></tr><tr><td>(RF) Relations and functions</td><td></td></tr><tr><td><strong>Process</strong></td><td></td></tr><tr><td>(P1) Know and master the specific contents of mathematics</td><td></td></tr><tr><td>(P2) Know and use algorithms and procedures</td><td></td></tr><tr><td>(P3) Know different forms of representation and move from one to the other</td><td></td></tr><tr><td>(P4) Solve problems using strategies in different fields</td><td></td></tr><tr><td>(P5) Recognize the measurable nature of objects and phenomena in different contexts and measure quantities</td><td></td></tr><tr><td>(P6) Progressively acquire typical forms of mathematical thought</td><td></td></tr><tr><td>(P7) Use tools, models and representations in quantitative treatment information in the scientific, technological, economic and social fields</td><td></td></tr><tr><td>(P8) Recognize shapes in space and use them for problem solving</td><td></td></tr><tr><td><strong>Macro-process</strong></td><td></td></tr><tr><td>(MP1) Formulating</td><td></td></tr><tr><td>(MP2) Interpreting</td><td></td></tr><tr><td>(MP3) Employing</td><td></td></tr></tbody></table>
We define one new variable for each area, process, and macro-process. Each of these new features takes the value corresponding to the percentage of correct answers provided by the student for that specific group of items, namely, correctness rate. Last, we concatenate the computed values to obtain a new flattened representation of learning, where each item is a possible indicator and not its unique representative. Following our strategy, we represent each student's learning in the space of fifteen (15) dimensions, as shown in Table 2.

We use two techniques to develop our AI-tool. The first one is Random forest (RF) [5], which is widely used in Educational Data Mining for the high degree of explainability and effortless interpretation of the results. We trained our models through bootstrap aggregating (bagging) to reduce the overfitting of dataset and increase precision. To tune the model, we performed a grid search.

The second technique is based on neural networks, which has recently become widespread also in the field of Educational Data Mining and has also been applied in predictive models for student performance [10]. We firstly include a preprocessing step, aimed at encoding the values of categorical variables into numerical values with a “one-hot” encoding algorithm. After preprocessing, we implemented two neural networks based on different data transformation approaches. Categorical Embeddings (CE), is a neural network that treats the input depending on its type: categorical inputs are passed through an embedding layer, numerical ones are fed to a dense layer. Feature Tokenizer Transformer (FTT) [9] is able to identify the input or the group of inputs that most influence the output, thanks to attention maps.

<table><caption>**Table 2.** Example of the student's learning final encoding.</caption><thead><tr><th>Id</th><th>NU</th><th>SF</th><th>DF</th><th>RF</th><th>P1</th><th>P2</th><th>P3</th><th>P4</th><th>P5</th><th>P6</th><th>P7</th><th>P8</th><th>MP1</th><th>MP2</th><th>MP3</th></tr></thead><tbody><tr><td>1</td><td>0.86</td><td>0.75</td><td>0.90</td><td>0.80</td><td>0.71</td><td>0.80</td><td>1.00</td><td>0.89</td><td>1.00</td><td>0.67</td><td>0.91</td><td>0.75</td><td>0.81</td><td>0.73</td><td>0.94</td></tr><tr><td>2</td><td>0.50</td><td>0.25</td><td>0.50</td><td>0.53</td><td>0.29</td><td>0.60</td><td>0.50</td><td>0.22</td><td>1.00</td><td>0.33</td><td>0.73</td><td>0.25</td><td>0.50</td><td>0.47</td><td>0.44</td></tr></tbody></table>

# 4 Experimental Results

We carried out all the experiments using the Google Colaboratory Notebook environment, with the Python programming language and popular machine learning libraries, such as scikit-learn and pandas.

The dataset for all the experiments was preprocessed cleaning features with many missing values, highly correlation (computed by R² measure above 0.5) or specifically referred to a cohort of students, preventing the model to be transferred to new cohorts (e.g., identification code for a class). This features selection process, together with the engineering of the features related to the items in the tests, results in a set of 34 features, which refers both to socio-economic and cultural context, demographic data and learning dimension. For the definition
of the training set we used the data from 2012/13 K-5 cohort. For the models based on neural networks we split this cohort to generate both training and validation sets (split in 80% and 20% respectively). Finally, we used the K-5 2013/14 cohort to test and measure the model performance. The dataset is unbalanced between underachievement/non-underachievement classes; therefore balancing techniques were applied. In the development of the RF models, a random undersampling technique was used, implemented in the imblearn library. We trained neural networks using a weighted random sampler, that samples the data to balance classes ratio in the training batches.

In Table 3, we present the overall results on the test dataset of the above mentioned models: RF, CE, and FTT. For RF, we considered the best hyper-

<table><caption>Table 3. Performance on test set</caption><thead><tr><td>Models</td><td>Accuracy</td><td>Precision</td><td>Recall</td></tr></thead><tbody><tr><td>Random Forest</td><td>0.77</td><td>0.62</td><td>0.67</td></tr><tr><td>CE neural network</td><td>0.76</td><td>0.76</td><td>0.76</td></tr><tr><td>FTT neural network</td><td>0.78</td><td>0.77</td><td>0.78</td></tr></tbody></table>

parameters setting determined with the grid search technique: 50 estimators in the forest, trained with 30% of random samples, 60% of random features and max depth set to 11. The FTT outperforms the other predictive models with accuracy, precision and recall between 77% and 78%.

# 5 Conclusion and Future Work

Our results suggest that the challenge of predicting low achievement risk for primary and secondary school students can be effectively addressed through the use of well-curated datasets and the choice of reliable predictive models. Our abstract representation of (INVALSI) tests and the related encoding for the student achievement allowed us to transfer the trained models on different cohorts and therefore to obtain a accurate prediction. We believe that the ability to predict low school achievement with reasonable accuracy five years in advance offers a practical tool for policy makers, managers and educators.

We are interested in extending our work in several directions. First, we want to verify the transferability of the proposed methodology to other disciplines, using a representation for students' learning similar to the one proposed in this work. Second, we want to increase the quality of the information provided as input to the predictive models, e.g., by collecting more data and by integrating new data sources. We aim to improve the learning encoding —thus the feature extraction process— in a way that is not knowledge-based to limit the bias. Finally, we want to deepen the interpretability of the results of our models, by analysing the feature importance computed on RF model and comparing it with the interpretation of the weights that define the neural networks we have used.
References

1. Al-Barrak, M.A., Al-Razgan, M.: Predicting students final gpa using decision trees: a case study. International journal of information and education technology **6**(7), 528 (2016)

2. Albreiki, B., Zaki, N., Alashwal, H.: A systematic literature review of student performance prediction using machine learning techniques. Education Sciences **11**(9), 552 (2021)

3. Alexander, K.L., Entwisle, D.R., Olson, L.S.: Schools, achievement, and inequality: A seasonal perspective. Educational evaluation and policy analysis **23**(2), 171–191 (2001)

4. Baartman, L.K., De Bruijn, E.: Integrating knowledge, skills and attitudes: Conceptualising learning processes towards vocational competence. Educational Research Review **6**(2), 125–134 (2011)

5. Breiman, L.: Random forests. Machine learning **45**(1), 5–32 (2001)

6. Cassen, R., Kingdon, G.: Tackling low educational achievement. Joseph Rowntree Foundation (2007)

7. Curtis, D.D., McMillan, J.: School non-completers: Profiles and initial destinations (2008)

8. Geary, D.C.: Consequences, characteristics, and causes of mathematical learning disabilities and persistent low achievement in mathematics. Journal of developmental and behavioral pediatrics: JDBP **32**(3), 250 (2011)

9. Gorishniy, Y., Rubachev, I., Khrulkov, V., Babenko, A.: Revisiting deep learning models for tabular data (2021)

10. Hernández-Blanco, A., Herrera-Flores, B., Tomás, D., Navarro-Colorado, B.: A systematic review of deep learning approaches to educational data mining. Complexity (2019)

11. Ibrahim, Z., Rusli, D.: Predicting students’ academic performance: comparing artificial neural network, decision tree and linear regression. In: 21st Annual SAS Malaysia Forum, 5th September (2007)

12. Ingels, S.J., Curtin, T.R., Kaufman, P., Alt, M.N., Chen, X., et al.: Coming of Age in the 1990s: The Eighth-Grade Class of 1988 12 Years Later. Eric (2002)

13. Kotsiantis, S., Pierrakeas, C., Pintelas, P.: Predicting students' performance in distance learning using machine learning techniques. Applied Artificial Intelligence **18**(5), 411–426 (2004)

14. OECD: Who and Where are the Low-Performing Students? OECD Publishing (2016)

15. Pejić, A., Molcer, P.S., Gulači, K.: Math proficiency prediction in computer-based international large-scale assessments using a multi-class machine learning model. In: 2021 IEEE 19th International Symposium on Intelligent Systems and Informatics (SISY). pp. 49–54. IEEE (2021)

16. Rastrollo-Guerrero, J.L., Gomez-Pulido, J.A., Durán-Domínguez, A.: Analyzing and predicting students' performance by means of machine learning: A review. Applied sciences **10**(3), 1042 (2020)

17. Ricci, R.: La dispersione scolastica implicita (2019)

18. Sultana, S., Khan, S., Abbas, M.A.: Predicting performance of electrical engineering students using cognitive and non-cognitive features for identification of potential dropouts. International Journal of Electrical Engineering Education **54**(2), 105–118 (2017)

19. Zanellati, A., Zingaro, S.P., Gabbrielli, M.: Student low achievement prediction - report (2022)