# Studying the Diagnostic Effectiveness of English Education Based on Rule Mining and Question Type Association Analysis

Authors: Yan Wu; Baohua Gao
Journal: Wireless Communications and Mobile Computing
 (2022)

---

## Abstract

The greatest difficulty in implementing diagnostic assessment lies in the tracking, recording, real-time feedback, and correction of English learners’ learning process, and these difficulties are difficult to be solved when human beings (teachers or students) are the assessment subjects. To this end, this paper uses data mining to build a question type association model, machine learning to build an English education prediction model, and finally adds the existing knowledge point association model to the system to obtain a diagnostic evaluation model. Based on the diagnostic evaluation model, a question association rule-based algorithm is designed and implemented and validated in three aspects: grouping time, test recommendation, and performance improvement. Based on the requirements analysis, the authors designed and implemented a diagnostic assessment subsystem using the diagnostic assessment model and the related paper grouping algorithm and added it to the university English diagnostic practice system. The diagnostic assessment model for English education proposed in this paper can accurately evaluate the learning status of English learners, dynamically diagnose the learning obstacles of English learners, and effectively provide better practice guidance and test recommendations for English learners’ learning status and knowledge-based question type obstacles.


---

## Body

## 1. Introduction

The goal of English education is “to develop students’ comprehensive English application skills, especially listening and speaking skills, so that they can communicate effectively in English in their future studies, work and social interactions, and at the same time enhance their independent learning ability and improve their overall cultural quality to meet the needs of our social development and international exchanges” [1]. However, the actual results are far from satisfactory. According to Hu Junsheng, “From a pedagogical point of view, the current English teaching activities and evaluation system are contrary to the laws of teaching and learning, and are a vicious extension of secondary school examination-based education in universities.” [2] The “teaching activities and assessment system are contrary to the laws of teaching and learning” mentioned here can be seen as both a drawback of the current English education and a breakthrough point for reform, especially the assessment system, which, as a gateway to testing the effectiveness of learning, actually determines what paths are chosen for teaching and learning activities, and the extent to which teachers and students pay attention to the process [3]. As a result, a number of scholars have carried out a great deal of research, starting with the reform of the assessment system, and the most fruitful of which is the process assessment [4].However, after trying these systems, we found that the practice process of these systems is still the same as the traditional problem solving process, and they do not provide diagnostic evaluation, nor do they provide accurate study suggestions or test recommendations, which makes the learning efficiency of English learners not very high [5]. This makes it difficult for English learners to assess their true level of performance. In contrast to the traditional classroom, where students are evaluated mainly through the teacher’s subjective assessment and the students’ test scores, in e-learning, there is no teacher’s tutoring and students are completely dependent on their own learning, which requires data mining and machine learning technologies to support us in making effective diagnostic assessments and accurate test recommendations in order to effectively improve learning outcomes and rapidly increase learning performance [6].Educational assessment can be divided into two types of assessment in educational measurement: formative assessment and summative assessment [7]. Formative assessment is the process of diagnosing the progress of English learners, identifying gaps in their learning, and providing them with appropriate suggestions to improve their learning [8]. The assessment of learning outcomes is the summative assessment. From these statements, it is clear that formative assessment is an assessment of the learning process, while summative assessment is an assessment of the learning outcomes [9].The diagnostic assessment designed in this study is a way of helping English learners to automatically diagnose barriers to learning, evaluate their learning, be alerted to their learning status, and provide dynamic feedback on their learning outcomes when they are engaged in independent learning [10]. For independent English learners, diagnostic assessment can help them to adjust their learning strategies and improve their learning methods [11]. Therefore, as a way to improve the learning process, diagnostic assessment should provide dynamic feedback on the problems of English learners in the learning process so that it can better help English learners to improve their learning style and adjust their learning pace [12].


## 2. Framework of the Diagnostic Evaluation Model

Our proposed diagnostic assessment model is a dynamic assessment model that diagnoses learning barriers, evaluates learning, regulates learning styles, and provides early warning of learning status for English learners [13]. Accordingly, the framework of the diagnostic assessment model is shown in Figure 1.Figure 1
Framework of the diagnostic evaluation model.Of how the diagnostic assessment model operates is discussed in the following sections.


### 2.1. Counting English Learners’ Test and Practice Information and Carrying Out Analysis and Calculation

The information on English learners’ tests and exercises is extracted from the system’s database, processed, and then analysed and calculated using the diagnostic assessment model.


### 2.2. Analyse and Evaluate the Overall Learning Status of English Learners

English learners are analysed and evaluated in terms of their scores on knowledge points and question types, as well as their stability in learning. Through the quantitative judgement of English learners’ learning status, the system will give early warning to English learners whose status is unstable and whose learning progress is lagging behind, so as to stimulate and promote English learners’ learning motivation [14].


### 2.3. Diagnosis of High English Learner Knowledge Barriers and Question Types

The system already has a list of knowledge association rules that have been verified to be accurate and reasonable, so they can be added directly to the diagnostic assessment model. The question type association rules can be used to correlate ELLs’ question type scores to infer ELLs’ question type barriers and help ELLs find their own deficiencies [15].


### 2.4. Predicting English Learners’ Level 4 Scores

Feature extraction is carried out on English learners’ test and practice information, and two models, random forest and multiple linear regression, are used to train and fuse to build a prediction model for Grade 4 scores, to predict Grade 4 scores for English learners, so that English learners can understand their own English level and urge them to strengthen their practice.


### 2.5. Recommending Test Questions to English Learners

In order to provide guidance and recommendations on English language learners’ learning strategies and pacing, the diagnostic assessment model provides English language learners with targeted test questions. In Section4, we design and validate two algorithms for grouping papers based on the diagnostic assessment model to be use by the English language learners.As can be seen from the above process, we need to research and design the learning state assessment model, the question type correlation analysis, and the grade 4 score prediction model separately and finally integrate them to obtain the diagnostic assessment model.


### 2.6. GP Model Construction for Teaching Quality Evaluation

The GP model is constructed by first using the training data set and finding the maximum posterior likelihood estimates of the hyperparameters based on Bayesian principles [16]. Then, based on the GP model, the negative log-likelihood function of the conditional probabilities of the training samples is established. The optimal hyperparameter solution of the GP model is determined adaptively by using the conjugate gradient optimization method, and the Gaussian regression model is finally constructed based on the optimal hyperparameters [17].In the process of building the teaching quality evaluation model, the GP algorithm is used to learn the training data set in order to determine the hyperparameters of the GP model. Based on the hyperparameters, the teaching quality evaluation model can be built to achieve teaching quality evaluation estimation and estimate the variance of the evaluation results. The specific steps are as follows:(1)
Create a training data set. Let the training data setxi,ti:i=1Λm,where xi=xi1,xi2,Λ,xid is the value of the d teaching quality evaluation indicators for the ith sample as input to the GP model, ti is the corresponding teaching quality evaluation result as output to the GP model, and m is the number of samples in the training data set(2)
Normalization of modeled data. In order to eliminate the influence of the data due to dimensionality, the training data set is normalized by the normalization formula shown in (1).(1)xi∗=xi−xminxmax−xminIn the above equation,xi, xi∗ denote the values before and after normalization of the data, respectively, and xmin, xmax denote the minimum and maximum values of the sample data, respectively
(3)
GP model hyperparameter adaptive computation. The optimal hyperparameters of the GP are obtained by adapting the log-likelihood maximization of the learning samples,θ(4)
Teaching quality evaluation model building. Based on the optimal hyperparameterθ of GP, the teaching quality results Y∗ of input X∗ were estimated(5)
Reliability analysis of the results of the GP method for teaching quality evaluationThe GP calculates the expectation (y∗=ux∗) and variance σi∗2 corresponding to the input x∗ of teaching quality evaluation. Taking the confidence interval of 95% confidence as an example, zθ/2=1.96. When the theoretical value of teaching quality evaluation results (generally given by experts) falls within the confidence interval of GP estimation, it is confirmed that the reliability of GP results is high.
(2)θ=Yj∗−σyj·zθ/zYj∗+σyj∗zθ/2=θ¯.


### 2.7. Learning Status Assessment Model

The assessment of the learning status of English learners should be based on two dimensions: the level of competence and the degree of stability of the English learners.Teachers often use staged tests to assess ELLs’ recent learning. However, it can be seen that test scores do not fully indicate the true level of English language learners, and that some English language learners may get some easy questions wrong but get more difficult questions right, for example because of carelessness [18]. This situation indicates that the English learners’ learning status is not stable, and their mastery of some knowledge points or question types is not solid.The S-P table was constructed by extracting the English learners’ question type scores and knowledge point scores from the system database, recording those scores greater than or equal to the average as 1 and those less than the average as 0. The S-P table analysis method was used to analyse the English learners’ mastery and calculate the English learners’ attention coefficients, and based on their attention coefficients and scores on the questions, the English learners’ learning status was analysed and evaluated. The criteria for evaluating ELLs’ learning status are shown in Figure2.Figure 2
Learning status assessment criteria.In the original theory of S-P table analysis, English learners were classified into 3 categories according to their mastery of learning, namely, 0-50%, 50%-75%, and 75%-100%. Table1 shows that the average scores of the English learners in this project were mainly in the range of 30%-60% for the knowledge points and question types. If the theoretical categories were followed, nearly half of the ELLs would be classified in the 0-50% range, which is inaccurate for this application and would result in the ELLs not being evaluated accurately [19]. Therefore, ELLs will be classified into 5 main categories based on their knowledge of the topic and stability of learning.


### 2.8. Learning Status Evaluation Algorithm

The learning state evaluation algorithm calculates the ELL’s knowledge of the topic and the ELL’s attention factor and determines the ELL’s learning type to provide targeted guidance for the ELL. The steps are in the following sections.


#### 2.8.1. Reading ELLs’ Knowledge Point and Question Type Scores

Read the knowledge points and scores of English learners from the database and store them in the form of a matrix. Each ELL has a knowledge point and question type score ofM items, and element xij of the matrix represents the score of the ith ELL of size j. The formula of the matrix is as follows:
(3)xij=x11x12⋯x1mx21x22⋯x2m⋮⋮⋮xn1xn1⋯xnm.


#### 2.8.2. Processing Continuous Data

The learning state evaluation algorithm can only process binary data, and the knowledge point and question type scores are decimals distributed in the interval [0, 1], so these data need to be processed. The mean value of each knowledge point and question type score is set as a threshold, which is also used as a criterion for determining mastery. The formula for determining mastery is as follows:(4)Linexij=0,x<∑i=1nxijn1,x≥∑i=1nxijn.


#### 2.8.3. S-P Table Row Calculation

Letxij be the score of the ith English learner on question j. The sum of the scores of English learner i is xi, and the sum of the number of correct answers on question j is xj, with the following formula:
(5)xi=∑j=1mxij,(6)xj=∑i=1nxij.


#### 2.8.4. Calculation of ELLs’ Knowledge of Question Types and ELLs’ Attention Coefficients

Letxij be the score of the ith ELL on question j; xi be the total score of ELLs, xj be the sum of the number of correct answers on question j, and μ be the average score of ELLs, the simplified formula for calculating the ELL attention factor is as follows:
(7)CSi=1−∑j=1nxijxj−xiμ∑j=1xixj−xiμ.


#### 2.8.5. Determining the Type of English Learner

The ELL mastery and attention factor derived in step (4) were determined according to the ELL type classification criteria.


#### 2.8.6. Output Results

The output is the ELL’s knowledge score, the ELL’s attention factor, and the ELL type.In summary, these are the specific steps of the learning state evaluation algorithm, and the flow chart is shown in Figure3.Figure 3
Flow chart of the learning state evaluation algorithm.


## 2.1. Counting English Learners’ Test and Practice Information and Carrying Out Analysis and Calculation

The information on English learners’ tests and exercises is extracted from the system’s database, processed, and then analysed and calculated using the diagnostic assessment model.


## 2.2. Analyse and Evaluate the Overall Learning Status of English Learners

English learners are analysed and evaluated in terms of their scores on knowledge points and question types, as well as their stability in learning. Through the quantitative judgement of English learners’ learning status, the system will give early warning to English learners whose status is unstable and whose learning progress is lagging behind, so as to stimulate and promote English learners’ learning motivation [14].


## 2.3. Diagnosis of High English Learner Knowledge Barriers and Question Types

The system already has a list of knowledge association rules that have been verified to be accurate and reasonable, so they can be added directly to the diagnostic assessment model. The question type association rules can be used to correlate ELLs’ question type scores to infer ELLs’ question type barriers and help ELLs find their own deficiencies [15].


## 2.4. Predicting English Learners’ Level 4 Scores

Feature extraction is carried out on English learners’ test and practice information, and two models, random forest and multiple linear regression, are used to train and fuse to build a prediction model for Grade 4 scores, to predict Grade 4 scores for English learners, so that English learners can understand their own English level and urge them to strengthen their practice.


## 2.5. Recommending Test Questions to English Learners

In order to provide guidance and recommendations on English language learners’ learning strategies and pacing, the diagnostic assessment model provides English language learners with targeted test questions. In Section4, we design and validate two algorithms for grouping papers based on the diagnostic assessment model to be use by the English language learners.As can be seen from the above process, we need to research and design the learning state assessment model, the question type correlation analysis, and the grade 4 score prediction model separately and finally integrate them to obtain the diagnostic assessment model.


## 2.6. GP Model Construction for Teaching Quality Evaluation

The GP model is constructed by first using the training data set and finding the maximum posterior likelihood estimates of the hyperparameters based on Bayesian principles [16]. Then, based on the GP model, the negative log-likelihood function of the conditional probabilities of the training samples is established. The optimal hyperparameter solution of the GP model is determined adaptively by using the conjugate gradient optimization method, and the Gaussian regression model is finally constructed based on the optimal hyperparameters [17].In the process of building the teaching quality evaluation model, the GP algorithm is used to learn the training data set in order to determine the hyperparameters of the GP model. Based on the hyperparameters, the teaching quality evaluation model can be built to achieve teaching quality evaluation estimation and estimate the variance of the evaluation results. The specific steps are as follows:(1)
Create a training data set. Let the training data setxi,ti:i=1Λm,where xi=xi1,xi2,Λ,xid is the value of the d teaching quality evaluation indicators for the ith sample as input to the GP model, ti is the corresponding teaching quality evaluation result as output to the GP model, and m is the number of samples in the training data set(2)
Normalization of modeled data. In order to eliminate the influence of the data due to dimensionality, the training data set is normalized by the normalization formula shown in (1).(1)xi∗=xi−xminxmax−xminIn the above equation,xi, xi∗ denote the values before and after normalization of the data, respectively, and xmin, xmax denote the minimum and maximum values of the sample data, respectively
(3)
GP model hyperparameter adaptive computation. The optimal hyperparameters of the GP are obtained by adapting the log-likelihood maximization of the learning samples,θ(4)
Teaching quality evaluation model building. Based on the optimal hyperparameterθ of GP, the teaching quality results Y∗ of input X∗ were estimated(5)
Reliability analysis of the results of the GP method for teaching quality evaluationThe GP calculates the expectation (y∗=ux∗) and variance σi∗2 corresponding to the input x∗ of teaching quality evaluation. Taking the confidence interval of 95% confidence as an example, zθ/2=1.96. When the theoretical value of teaching quality evaluation results (generally given by experts) falls within the confidence interval of GP estimation, it is confirmed that the reliability of GP results is high.
(2)θ=Yj∗−σyj·zθ/zYj∗+σyj∗zθ/2=θ¯.


## 2.7. Learning Status Assessment Model

The assessment of the learning status of English learners should be based on two dimensions: the level of competence and the degree of stability of the English learners.Teachers often use staged tests to assess ELLs’ recent learning. However, it can be seen that test scores do not fully indicate the true level of English language learners, and that some English language learners may get some easy questions wrong but get more difficult questions right, for example because of carelessness [18]. This situation indicates that the English learners’ learning status is not stable, and their mastery of some knowledge points or question types is not solid.The S-P table was constructed by extracting the English learners’ question type scores and knowledge point scores from the system database, recording those scores greater than or equal to the average as 1 and those less than the average as 0. The S-P table analysis method was used to analyse the English learners’ mastery and calculate the English learners’ attention coefficients, and based on their attention coefficients and scores on the questions, the English learners’ learning status was analysed and evaluated. The criteria for evaluating ELLs’ learning status are shown in Figure2.Figure 2
Learning status assessment criteria.In the original theory of S-P table analysis, English learners were classified into 3 categories according to their mastery of learning, namely, 0-50%, 50%-75%, and 75%-100%. Table1 shows that the average scores of the English learners in this project were mainly in the range of 30%-60% for the knowledge points and question types. If the theoretical categories were followed, nearly half of the ELLs would be classified in the 0-50% range, which is inaccurate for this application and would result in the ELLs not being evaluated accurately [19]. Therefore, ELLs will be classified into 5 main categories based on their knowledge of the topic and stability of learning.


## 2.8. Learning Status Evaluation Algorithm

The learning state evaluation algorithm calculates the ELL’s knowledge of the topic and the ELL’s attention factor and determines the ELL’s learning type to provide targeted guidance for the ELL. The steps are in the following sections.


### 2.8.1. Reading ELLs’ Knowledge Point and Question Type Scores

Read the knowledge points and scores of English learners from the database and store them in the form of a matrix. Each ELL has a knowledge point and question type score ofM items, and element xij of the matrix represents the score of the ith ELL of size j. The formula of the matrix is as follows:
(3)xij=x11x12⋯x1mx21x22⋯x2m⋮⋮⋮xn1xn1⋯xnm.


### 2.8.2. Processing Continuous Data

The learning state evaluation algorithm can only process binary data, and the knowledge point and question type scores are decimals distributed in the interval [0, 1], so these data need to be processed. The mean value of each knowledge point and question type score is set as a threshold, which is also used as a criterion for determining mastery. The formula for determining mastery is as follows:(4)Linexij=0,x<∑i=1nxijn1,x≥∑i=1nxijn.


### 2.8.3. S-P Table Row Calculation

Letxij be the score of the ith English learner on question j. The sum of the scores of English learner i is xi, and the sum of the number of correct answers on question j is xj, with the following formula:
(5)xi=∑j=1mxij,(6)xj=∑i=1nxij.


### 2.8.4. Calculation of ELLs’ Knowledge of Question Types and ELLs’ Attention Coefficients

Letxij be the score of the ith ELL on question j; xi be the total score of ELLs, xj be the sum of the number of correct answers on question j, and μ be the average score of ELLs, the simplified formula for calculating the ELL attention factor is as follows:
(7)CSi=1−∑j=1nxijxj−xiμ∑j=1xixj−xiμ.


### 2.8.5. Determining the Type of English Learner

The ELL mastery and attention factor derived in step (4) were determined according to the ELL type classification criteria.


### 2.8.6. Output Results

The output is the ELL’s knowledge score, the ELL’s attention factor, and the ELL type.In summary, these are the specific steps of the learning state evaluation algorithm, and the flow chart is shown in Figure3.Figure 3
Flow chart of the learning state evaluation algorithm.


## 2.8.1. Reading ELLs’ Knowledge Point and Question Type Scores

Read the knowledge points and scores of English learners from the database and store them in the form of a matrix. Each ELL has a knowledge point and question type score ofM items, and element xij of the matrix represents the score of the ith ELL of size j. The formula of the matrix is as follows:
(3)xij=x11x12⋯x1mx21x22⋯x2m⋮⋮⋮xn1xn1⋯xnm.


## 2.8.2. Processing Continuous Data

The learning state evaluation algorithm can only process binary data, and the knowledge point and question type scores are decimals distributed in the interval [0, 1], so these data need to be processed. The mean value of each knowledge point and question type score is set as a threshold, which is also used as a criterion for determining mastery. The formula for determining mastery is as follows:(4)Linexij=0,x<∑i=1nxijn1,x≥∑i=1nxijn.


## 2.8.3. S-P Table Row Calculation

Letxij be the score of the ith English learner on question j. The sum of the scores of English learner i is xi, and the sum of the number of correct answers on question j is xj, with the following formula:
(5)xi=∑j=1mxij,(6)xj=∑i=1nxij.


## 2.8.4. Calculation of ELLs’ Knowledge of Question Types and ELLs’ Attention Coefficients

Letxij be the score of the ith ELL on question j; xi be the total score of ELLs, xj be the sum of the number of correct answers on question j, and μ be the average score of ELLs, the simplified formula for calculating the ELL attention factor is as follows:
(7)CSi=1−∑j=1nxijxj−xiμ∑j=1xixj−xiμ.


## 2.8.5. Determining the Type of English Learner

The ELL mastery and attention factor derived in step (4) were determined according to the ELL type classification criteria.


## 2.8.6. Output Results

The output is the ELL’s knowledge score, the ELL’s attention factor, and the ELL type.In summary, these are the specific steps of the learning state evaluation algorithm, and the flow chart is shown in Figure3.Figure 3
Flow chart of the learning state evaluation algorithm.


## 3. Analysis of the Association of Question Types

The English diagnostic practice system has been well researched in terms of knowledge points, including knowledge point association rules and the ability to group papers according to knowledge point association rules. However, there is a lack of research on the question types of Level 4 examinations, and only a simple way of grouping papers by question type is available. As an important part of the University English IV exam, the question types have been reformed over the years and are very different from those of the past, so they must be taken seriously. Therefore, this section uses a large amount of raw data collected in the use of the system, and after data processing and data stratification, two levels of correlation analysis are carried out on the question type data, and finally, a more complete and reliable table of question type correlation rules is derived [20].


### 3.1. Data Collection

In 2021, the University English Diagnostic Practice System went live at a university. Students at this university, who are all at an average level in their GCSEs and have an average level of English, are very keen to pass their University English Level 4 exams and therefore use the University English Diagnostic Practice System frequently to practice and work on questions.In the 2 years that the system has been online, the system has saved over 6,600 students’ practice data in the back-end database. In order to improve the accuracy of the correlation analysis, the richness of the question types and the accuracy of the predictions, our laboratory has expanded the existing question bank of the system by adding a total of 12 sets of question papers for 2020 and 2021 and 8 sets of mock questions for the system’s English learners to practice with.Among them, the comparison table between the new and old types of objective questions of University English Level 4 is shown in Table2, and the number of each type of questions in the system is shown in Table 3.Table 1
Comparison of the old and new types of objective questions in University English Level 4.
Old question typeNew question typeShort dialogue listeningShort newsLong dialogue listeningLong dialogue listeningPassage listening comprehensionPassage listening comprehensionRead and fill in the blanksRead and fill in the blanksReading choice questionsReading choice questionsFast readingInformation matchingFill in the blanksTable 2
Number of questions in the University English Diagnostic Practice System by type.
Question typeQuantityFast reading157Short news60Short dialogue listening1550310Long dialogue listening310Passage listening comprehension340Read and fill in the blanks153Reading choice questions720Fill in the blanks136Information matching55


### 3.2. Data Layering

During the years when the English Diagnostic Practice System has been online, the type of questions in the Level 4 exam has been reformed and has changed slightly. In the listening section, short conversation listening questions have been removed and short news questions have been added, while in the reading section, quick reading and completion questions have been removed and information matching questions have been added. As a result, there were no short news and information matching questions in the early practice questions, so some English learners were missing out on these two types of questions. After sorting the questions in descending order, the number of English learners for each question type and the proportion of the total number of learners are shown in Table1.Table 3
Sample size and proportion of each question type.
Question type nameNumber of samplesProportion in total populationRead and fill in the blanks6604100%Reading choice questions6604100%Long dialogue listening652598.8%Short dialogue listening652398.7%Passage listening comprehension652398.77%Fast reading650398.47%Cloze test650398.47%Information matching542482.13%Short news542482.13%As can be seen from Table1, the sample size of all the questions was above 98%, except for the new type of information matching and short news. Since the new question type is now used in the Level 4 examination, the new question type and the old question type should be correlated and analysed separately. As the reform of Level 4 has just begun, the number of questions is small and the quality of the practice questions varies, so there is still value in practising old questions such as short conversation listening. It is still possible to improve English language skills by practising these old questions. Therefore, new questions should be added to the old question types when they are stratified, the correlation between the old and new questions should be explored, and the system’s question bank should be fully utilised in view of the limited number of real questions, so that English learners can be exposed to more practice questions.Therefore, to address the above situation, we decided to stratify the question types, with the first layer containing all new and old question types and the second layer containing only new question types, and to carry out separate data mining work on these two layers so as to combine them to obtain a table of question type association rules. The specific stratification results are shown in Table4.Table 4
Results of stratification of question types.
LayeredQuestion type nameNumber of samplesFirst floorRead and fill in the blanks6604Reading choice questions6604Long dialogue listening6525Short dialogue listening6523Passage listening comprehension6523Fast reading6503Cloze test6503Information matching5424Short news5424The second floorRead and fill in the blanks6604Reading choice questions6604Long dialogue listening6525Passage listening comprehension6523Information matching5424Short news5424From Table4, it can be seen that the sample size for each question type is not the same, which will affect the number of frequent item sets, so the sample size should be unified to a minimum sample size of 5424 for all question types.


### 3.3. Association Rule Mining

Extensive experiments were conducted on the above data based on the Apriori algorithm, including symmetric analysis and hierarchical analysis. Figure4 shows the operational flow of the Apriori algorithm data stream, with the following steps:
(1)
Import data. As shown in Figure4, the node “question score data xsv” is the original data to be analysed in each run, and the source file options of the Apriori algorithm support a variety of formats such as CSV and EXCEL [21].(2)
Field filtering. As shown in Figure4, the node “filter” is used to clean up irrelevant data. In this paper, the English learner codes are irrelevant data and need to be removed(3)
Binary conversion. As shown in Figure4, the node “binary conversion” is used to binary discretize the data by comparing the score of each question type with its average and then converting the data into T or F(4)
Apriori algorithm. As shown in Figure4, in this step, the results of the correlation analysis are adjusted by changing the three conditions of support, confidence, and number of priors(5)
After hitting run, the results of the analysis in Figure4 are obtainedFigure 4
Operational data flow.


## 3.1. Data Collection

In 2021, the University English Diagnostic Practice System went live at a university. Students at this university, who are all at an average level in their GCSEs and have an average level of English, are very keen to pass their University English Level 4 exams and therefore use the University English Diagnostic Practice System frequently to practice and work on questions.In the 2 years that the system has been online, the system has saved over 6,600 students’ practice data in the back-end database. In order to improve the accuracy of the correlation analysis, the richness of the question types and the accuracy of the predictions, our laboratory has expanded the existing question bank of the system by adding a total of 12 sets of question papers for 2020 and 2021 and 8 sets of mock questions for the system’s English learners to practice with.Among them, the comparison table between the new and old types of objective questions of University English Level 4 is shown in Table2, and the number of each type of questions in the system is shown in Table 3.Table 1
Comparison of the old and new types of objective questions in University English Level 4.
Old question typeNew question typeShort dialogue listeningShort newsLong dialogue listeningLong dialogue listeningPassage listening comprehensionPassage listening comprehensionRead and fill in the blanksRead and fill in the blanksReading choice questionsReading choice questionsFast readingInformation matchingFill in the blanksTable 2
Number of questions in the University English Diagnostic Practice System by type.
Question typeQuantityFast reading157Short news60Short dialogue listening1550310Long dialogue listening310Passage listening comprehension340Read and fill in the blanks153Reading choice questions720Fill in the blanks136Information matching55


## 3.2. Data Layering

During the years when the English Diagnostic Practice System has been online, the type of questions in the Level 4 exam has been reformed and has changed slightly. In the listening section, short conversation listening questions have been removed and short news questions have been added, while in the reading section, quick reading and completion questions have been removed and information matching questions have been added. As a result, there were no short news and information matching questions in the early practice questions, so some English learners were missing out on these two types of questions. After sorting the questions in descending order, the number of English learners for each question type and the proportion of the total number of learners are shown in Table1.Table 3
Sample size and proportion of each question type.
Question type nameNumber of samplesProportion in total populationRead and fill in the blanks6604100%Reading choice questions6604100%Long dialogue listening652598.8%Short dialogue listening652398.7%Passage listening comprehension652398.77%Fast reading650398.47%Cloze test650398.47%Information matching542482.13%Short news542482.13%As can be seen from Table1, the sample size of all the questions was above 98%, except for the new type of information matching and short news. Since the new question type is now used in the Level 4 examination, the new question type and the old question type should be correlated and analysed separately. As the reform of Level 4 has just begun, the number of questions is small and the quality of the practice questions varies, so there is still value in practising old questions such as short conversation listening. It is still possible to improve English language skills by practising these old questions. Therefore, new questions should be added to the old question types when they are stratified, the correlation between the old and new questions should be explored, and the system’s question bank should be fully utilised in view of the limited number of real questions, so that English learners can be exposed to more practice questions.Therefore, to address the above situation, we decided to stratify the question types, with the first layer containing all new and old question types and the second layer containing only new question types, and to carry out separate data mining work on these two layers so as to combine them to obtain a table of question type association rules. The specific stratification results are shown in Table4.Table 4
Results of stratification of question types.
LayeredQuestion type nameNumber of samplesFirst floorRead and fill in the blanks6604Reading choice questions6604Long dialogue listening6525Short dialogue listening6523Passage listening comprehension6523Fast reading6503Cloze test6503Information matching5424Short news5424The second floorRead and fill in the blanks6604Reading choice questions6604Long dialogue listening6525Passage listening comprehension6523Information matching5424Short news5424From Table4, it can be seen that the sample size for each question type is not the same, which will affect the number of frequent item sets, so the sample size should be unified to a minimum sample size of 5424 for all question types.


## 3.3. Association Rule Mining

Extensive experiments were conducted on the above data based on the Apriori algorithm, including symmetric analysis and hierarchical analysis. Figure4 shows the operational flow of the Apriori algorithm data stream, with the following steps:
(1)
Import data. As shown in Figure4, the node “question score data xsv” is the original data to be analysed in each run, and the source file options of the Apriori algorithm support a variety of formats such as CSV and EXCEL [21].(2)
Field filtering. As shown in Figure4, the node “filter” is used to clean up irrelevant data. In this paper, the English learner codes are irrelevant data and need to be removed(3)
Binary conversion. As shown in Figure4, the node “binary conversion” is used to binary discretize the data by comparing the score of each question type with its average and then converting the data into T or F(4)
Apriori algorithm. As shown in Figure4, in this step, the results of the correlation analysis are adjusted by changing the three conditions of support, confidence, and number of priors(5)
After hitting run, the results of the analysis in Figure4 are obtainedFigure 4
Operational data flow.


## 4. Case Studies

### 4.1. Analysis of the First Layer

After several experiments, the minimum support was set to 40%, the maximum confidence level to 90%, and the maximum number of antecedent items to 3. The number of association rules obtained in the association analysis between the questions in the first level is shown in Table5.Table 5
Number of first level rules.
Analysis categoryQuantityT analysis39F analysis12Total51It can be seen that the number of association rules for the T analysis is more in line with expectations, while the number of rules for the F analysis is low. Therefore, in order to obtain more association rules, the value of the minimum support can be reduced to obtain them. Thus, for the F analysis, when the minimum support is reduced to 35%, the number of rules that can be obtained is 43.


### 4.2. Analysis of the Second Level

After several experiments, the minimum support was set to 20%, the maximum confidence level to 90%, and the maximum number of antecedents to 3. The number of association rules obtained in the association analysis between the questions in the second level is shown in Table6.Table 6
Number of second tier rules.
Analysis categoryQuantityT analysis26F analysis37Total63It can be seen that the number of association rules for the T analysis is more in line with expectations, while the number of rules for the F analysis is on the high side. Therefore, the number of association rules can be reduced by increasing the minimum support appropriately. Thus, for the F analysis, when the minimum support is increased from 20% to 25%, the number of association rules that can be mined is 28.


### 4.3. Analysis of the Results of the Question Type Association Rule

After the above data processing, data stratification, and association analysis, the final number of association rules was obtained as shown in Table7.Table 7
Final table of the number of association rules.
Unified settingNumber of layersMinimum confidenceMinimum supportNumber of association rulesTotalThe maximum number of top items is 3;lift>1T analysisFirst floor90403965136The second floor902026F analysisFirst floor90354371The second floor902528According to Table7, it can be seen that a total of 136 rules were obtained. The number of occurrences of each question type in the 136 association rules in the preceding and following terms is shown in Table 8.Table 8
Number of occurrences of each question type in the pre- and postitems.
Question type nameAppears in the following itemAppears in the preceding itemTotalRead and fill in the blanks102030Reading choice questions262349Long dialogue listening222749Passage listening comprehension21021Fast reading302151Cloze test13214Information matching101929Short news033Due to space constraints, it is not possible to show and explain all 136 association rules, so only the association rules with the highest confidence and support at each level were selected for a total of four levels of analysis on T and four levels of analysis on F. These rules are shown in Table9.Table 9
Example of the results of the question type correlation rule.
Number of layersConsequentThe aforesaidDegree of supportConfidenceT analysisFirst floorReading choiceInformation matching and quick reading42.68%97.23%The second floorReading choiceInformation matching47.18%96.13%F analysisFirst floorShort dialogue listeningLong dialogue listening and short passage listening comprehension46.18%99.69%The second floorInformation matchingReading choice and reading choice33.44%96.53%The support in Table9 indicates the probability that the antecedent and the consequent terms occur together, while the confidence level indicates the probability that when the consequent term occurs, the antecedent term is also present.In t-analysis, the total number of samples in association rules is 5424, that is, in the 5424 sample data, the problem types include reading and selection, while in all samples, the proportion that the scores of information matching and fast reading are greater than or equal to their average scores is 42.68%, while the proportion of all samples in which the scores of the two question types of antecedent information matching and quick reading. The probability of the latter reading choice scoring at a rate greater than or equal to the average score is 97.23%.The second level of association rule for the T analysis indicates that the total number of samples in the second level is 5424, i.e., among the 5424 samples, the proportion of all samples in which both the score of the postitem reading choice and the score of the information match of the preitem are greater than or equal to their average scores is 47.18%, while the score of the postitem reading choice is 47.18% when the score of the information match of the preitem is greater than or equal to its average score. The probability that the score for the latter question was also greater than or equal to the average score was 96.13%.The correlation rule for the first level of the F analysis indicates that the total number of samples in the first level is 5424, i.e., among the 5424 samples, the proportion of all samples in which the scores for the postitem short conversation listening and the scores for the preitem long conversation listening and short text listening comprehension are both less than their average scores is 46.81%, while the proportion of samples in which the scores for the preitem long conversation listening and short text listening comprehension are both less than their average scores is 46.81%. The probability of the latter being less than the average is 99.69%.The second level of association rules for the F analysis indicated that the total number of samples in the second level was 5424, i.e., the proportion of these sample data in which the scores for the postitem information matching and the scores for the preitem reading choice and reading word choice were both less than their mean scores was 33.44% in all samples, and in which the scores for the preitem reading choice and reading word choice were both less than their mean scores, and the probability that the scores for the postitem information matching choice were also less than their mean scores was 96.53%. The probability that the score for the latter question is also less than the average score is 96.53%.The four rules with the highest levels of confidence and support above show that listening question types, such as those belonging to the same broad category, are closely related to each other, and that when ELLs have higher error rates for both long conversation listening and short text listening comprehension, they also have higher error rates for short conversation listening. This also confirms the previous prediction that the old and new types of questions are closely related to each other, and that by practising past questions from the old types, you can also improve your mastery of the new types of questions.


### 4.4. Question Type Experimental Verification

In order to evaluate whether the test recommendations of the two diagnostic assessment-based paper grouping algorithms can effectively target the actual learning situation of users and whether the two paper grouping algorithms can actually improve users’ performance, we designed an experiment to evaluate the effectiveness of the algorithms in terms of grouping time, test recommendations, and performance improvement.Two sophomore classes supervised by the same teacher at the university where the system was used were selected for the comparison experiment, the experimental class, and the control class. The two classes had comparable initial scores and were about to take their Level 4 exams. The control class will follow the normal teaching schedule, while the experimental class will complete six sets of questions in each of the two sets of papers in addition to the normal teaching schedule. The degree of correlation between the different question types, as shown in Figure5, has resulted in high average scores and pass rates for the first year English exams in both classes.Figure 5
Question Type Correlation Chart.As shown in Figure6, the process of both grouping algorithms based on diagnostic evaluation is relatively complex, so the grouping time had to be tested. We ran the two algorithms 10 times in both the local test environment and the production server environment and recorded the running time for each run. When the client gets a response after 10 seconds, the user will feel that the speed of the web page is bad and the user will leave the web page.Figure 6
Effect of categorising questions in different scenarios.From Figure6, we can see that in the local test environment, due to the lack of bandwidth limitation, the response time of reminder categorization of different algorithms is less than 2 seconds, which fully meets user requirements; in the server production environment, due to bandwidth limitation and server configuration, the grouping time of two grouping algorithms is also slightly more than 2 seconds and does not exceed 3 seconds, which is also in line with user requirements.


## 4.1. Analysis of the First Layer

After several experiments, the minimum support was set to 40%, the maximum confidence level to 90%, and the maximum number of antecedent items to 3. The number of association rules obtained in the association analysis between the questions in the first level is shown in Table5.Table 5
Number of first level rules.
Analysis categoryQuantityT analysis39F analysis12Total51It can be seen that the number of association rules for the T analysis is more in line with expectations, while the number of rules for the F analysis is low. Therefore, in order to obtain more association rules, the value of the minimum support can be reduced to obtain them. Thus, for the F analysis, when the minimum support is reduced to 35%, the number of rules that can be obtained is 43.


## 4.2. Analysis of the Second Level

After several experiments, the minimum support was set to 20%, the maximum confidence level to 90%, and the maximum number of antecedents to 3. The number of association rules obtained in the association analysis between the questions in the second level is shown in Table6.Table 6
Number of second tier rules.
Analysis categoryQuantityT analysis26F analysis37Total63It can be seen that the number of association rules for the T analysis is more in line with expectations, while the number of rules for the F analysis is on the high side. Therefore, the number of association rules can be reduced by increasing the minimum support appropriately. Thus, for the F analysis, when the minimum support is increased from 20% to 25%, the number of association rules that can be mined is 28.


## 4.3. Analysis of the Results of the Question Type Association Rule

After the above data processing, data stratification, and association analysis, the final number of association rules was obtained as shown in Table7.Table 7
Final table of the number of association rules.
Unified settingNumber of layersMinimum confidenceMinimum supportNumber of association rulesTotalThe maximum number of top items is 3;lift>1T analysisFirst floor90403965136The second floor902026F analysisFirst floor90354371The second floor902528According to Table7, it can be seen that a total of 136 rules were obtained. The number of occurrences of each question type in the 136 association rules in the preceding and following terms is shown in Table 8.Table 8
Number of occurrences of each question type in the pre- and postitems.
Question type nameAppears in the following itemAppears in the preceding itemTotalRead and fill in the blanks102030Reading choice questions262349Long dialogue listening222749Passage listening comprehension21021Fast reading302151Cloze test13214Information matching101929Short news033Due to space constraints, it is not possible to show and explain all 136 association rules, so only the association rules with the highest confidence and support at each level were selected for a total of four levels of analysis on T and four levels of analysis on F. These rules are shown in Table9.Table 9
Example of the results of the question type correlation rule.
Number of layersConsequentThe aforesaidDegree of supportConfidenceT analysisFirst floorReading choiceInformation matching and quick reading42.68%97.23%The second floorReading choiceInformation matching47.18%96.13%F analysisFirst floorShort dialogue listeningLong dialogue listening and short passage listening comprehension46.18%99.69%The second floorInformation matchingReading choice and reading choice33.44%96.53%The support in Table9 indicates the probability that the antecedent and the consequent terms occur together, while the confidence level indicates the probability that when the consequent term occurs, the antecedent term is also present.In t-analysis, the total number of samples in association rules is 5424, that is, in the 5424 sample data, the problem types include reading and selection, while in all samples, the proportion that the scores of information matching and fast reading are greater than or equal to their average scores is 42.68%, while the proportion of all samples in which the scores of the two question types of antecedent information matching and quick reading. The probability of the latter reading choice scoring at a rate greater than or equal to the average score is 97.23%.The second level of association rule for the T analysis indicates that the total number of samples in the second level is 5424, i.e., among the 5424 samples, the proportion of all samples in which both the score of the postitem reading choice and the score of the information match of the preitem are greater than or equal to their average scores is 47.18%, while the score of the postitem reading choice is 47.18% when the score of the information match of the preitem is greater than or equal to its average score. The probability that the score for the latter question was also greater than or equal to the average score was 96.13%.The correlation rule for the first level of the F analysis indicates that the total number of samples in the first level is 5424, i.e., among the 5424 samples, the proportion of all samples in which the scores for the postitem short conversation listening and the scores for the preitem long conversation listening and short text listening comprehension are both less than their average scores is 46.81%, while the proportion of samples in which the scores for the preitem long conversation listening and short text listening comprehension are both less than their average scores is 46.81%. The probability of the latter being less than the average is 99.69%.The second level of association rules for the F analysis indicated that the total number of samples in the second level was 5424, i.e., the proportion of these sample data in which the scores for the postitem information matching and the scores for the preitem reading choice and reading word choice were both less than their mean scores was 33.44% in all samples, and in which the scores for the preitem reading choice and reading word choice were both less than their mean scores, and the probability that the scores for the postitem information matching choice were also less than their mean scores was 96.53%. The probability that the score for the latter question is also less than the average score is 96.53%.The four rules with the highest levels of confidence and support above show that listening question types, such as those belonging to the same broad category, are closely related to each other, and that when ELLs have higher error rates for both long conversation listening and short text listening comprehension, they also have higher error rates for short conversation listening. This also confirms the previous prediction that the old and new types of questions are closely related to each other, and that by practising past questions from the old types, you can also improve your mastery of the new types of questions.


## 4.4. Question Type Experimental Verification

In order to evaluate whether the test recommendations of the two diagnostic assessment-based paper grouping algorithms can effectively target the actual learning situation of users and whether the two paper grouping algorithms can actually improve users’ performance, we designed an experiment to evaluate the effectiveness of the algorithms in terms of grouping time, test recommendations, and performance improvement.Two sophomore classes supervised by the same teacher at the university where the system was used were selected for the comparison experiment, the experimental class, and the control class. The two classes had comparable initial scores and were about to take their Level 4 exams. The control class will follow the normal teaching schedule, while the experimental class will complete six sets of questions in each of the two sets of papers in addition to the normal teaching schedule. The degree of correlation between the different question types, as shown in Figure5, has resulted in high average scores and pass rates for the first year English exams in both classes.Figure 5
Question Type Correlation Chart.As shown in Figure6, the process of both grouping algorithms based on diagnostic evaluation is relatively complex, so the grouping time had to be tested. We ran the two algorithms 10 times in both the local test environment and the production server environment and recorded the running time for each run. When the client gets a response after 10 seconds, the user will feel that the speed of the web page is bad and the user will leave the web page.Figure 6
Effect of categorising questions in different scenarios.From Figure6, we can see that in the local test environment, due to the lack of bandwidth limitation, the response time of reminder categorization of different algorithms is less than 2 seconds, which fully meets user requirements; in the server production environment, due to bandwidth limitation and server configuration, the grouping time of two grouping algorithms is also slightly more than 2 seconds and does not exceed 3 seconds, which is also in line with user requirements.


## 5. Conclusions

This paper examines the diagnostic study of test questions in the English language education process, and the Level 4 performance validation experiment can go some way towards verifying the practical effectiveness of a group paper algorithm based on diagnostic assessment. A comparison of student performance shows that the experimental class, although initially at the same level as the control class, showed a small increase in performance through the use of the diagnostic assessment system and an increase in the number of Level 4 passes. At the same time, the average score of the experimental class was also higher than the average score of the whole school, and the percentage of Level 4 passes also increased compared to the whole school, so it can be proved to some extent that the actual use of diagnostic-based assessment is effective.



---

*Source: 5500107-2022-04-06.xml*