Yamauchi et al. Smart Learning Environments
https://doi.org/10.1186/s40561-023-00271-9

RESEARCH(2023) 10:51

Smart Learning Environments

Open Access

Automated labeling of PDF mathematical
exercises with word N‑grams VSM classification
Taisei Yamauchi1, Brendan Flanagan2* , Ryosuke Nakamoto1, Yiling Dai3, Kyosuke Takami4 and Hiroaki Ogata3*Correspondence:flanagan.
brendanjohn.4n@kyoto-u.ac.jp
1
Graduate School of Informatics,
Kyoto University, Kyoto, Japan
2
Center for Innovative Researchand Education in Data Science,
Institute for Liberal Arts
and Sciences, Kyoto University,
Kyoto, Japan
3
Academic Centerfor Computing and Media
Studies, Kyoto University, Kyoto,Japan
4
Education Data Science Center,
National Institute for Educational
Policy Research, Tokyo, Japan

Abstract
In recent years, smart learning environments have become central to modern education and support students and instructors through tools based on prediction
and recommendation models. These methods often use learning material metadata,such as the knowledge contained in an exercise which is usually labeled by domain
experts and is costly and difficult to scale. It recognizes that automated labeling eases
the workload on experts, as seen in previous studies using automatic classification
algorithms for research papers and Japanese mathematical exercises. However, thesestudies didn’t delve into fine-grained labeling. In addition to that, as the use of materials in the system becomes more widespread, paper materials are transformed into PDF
formats, which can lead to incomplete extraction. However, there is less emphasison labeling incomplete mathematical sentences to tackle this problem in the previous
research. This study aims to achieve precise automated classification even from incomplete text inputs. To tackle these challenges, we propose a mathematical exercise
labeling algorithm that can handle detailed labels, even for incomplete sentences,using word n-grams, compared to the state-of-the-art word embedding method.
The results of the experiment show that mono-gram features with Random Forestmodels achieved the best performance with a macro F-measure of 92.50%, 61.28%
for 24-class labeling and 297-class labeling tasks, respectively. The contribution of thisresearch is showing that the proposed method based on traditional simple n-grams
has the ability to find context-independent similarities in incomplete sentences
and outperforms state-of-the-art word embedding methods in specific tasks like classifying short and incomplete texts.
Keywords: Automatic labeling, Word n-gram, Random forest, Incomplete textclassification, Word embedding, Mathematical education, Mathematical education inJapan

Introduction
Labeling learning materials is a key problem in scaling smart learning environments(Contractor et al., 2015). The availability of knowledge metadata for learning materials is
critical as important decisions, such as what to recommend for study for the next time,
are usually made based on the metadata and the learners’ previous experience (Vovides
et al., 2007). Each exercise in a textbook for each subject usually has a set of course units
that clarify the category of each exercise and are very useful in educational situations and© The Author(s) 2023. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permitsuse, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original
author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third
party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or
exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://
creativecommons.org/licenses/by/4.0/.

Yamauchi et al. Smart Learning Environments(2023) 10:51

the framework of educational problems. Recently, there has also been a growing trend in
the adoption of nationwide curriculum or studying guidelines, such as: the Australian
digital curriculum called Australian Curriculum, Assessment and Reporting Authority(ACARA) in Australia (Ditchburn, 2012), Common Core Standards (Porter et al., 2011;
Ritter, 2009) in America, Mathematics Curriculum Standards for Compulsory Education(MOE, 2012) in China, and the Courses of Study (MEXT, 2018) in Japan. These guidelines provide regulations for education and instruction, as well as standard units for eachsubject (MEXT, 2018). Educators select learning materials based on these guidelines tomeet the requirements of the compulsory curriculum. Therefore, learning materials that
do not contain knowledge metadata are difficult to incorporate into the course of study,
and the automated assignment of labels to learning materials could help overcome thisproblem.
In this study, the task of labeling learning materials has two main objectives: yielding
high accuracy for detailed classification and labeling incomplete texts. First, as is common with other labeling tasks, the performance of the classification task is very important as the aim of labeling materials is to reduce the burden of domain experts who areusually manually tackling the knowledge classification task. Detailed labeling of learning materials is very useful in the educational field, but assigning the classification to
problems manually is a hard task that requires the cooperation of experts, and the burden could be alleviated through automation. Schubotz et al. (2020), examined the taskof automatically assigning coarse labels according to a mathematical subject classification scheme for retrieving research papers and literature on mathematics in English. It
was found that the support provided by the proposed automatic classification algorithm
resulted in a reduced manual classification burden for domain experts. Another study
proposed the WE-KE model, which combines word embedding and knowledge components, to achieve accurate unit classification of Japanese mathematical exercises (Tianet al., 2022). With the shift to ICT education, researchers label the exercises to utilize
them for learning pattern analysis (Wang et al., 2022). While more detailed classifications may be necessary depending on the intended use, such detailed labeling was notconducted in those studies.
Second, as extracting complete text is sometimes difficult due to the format of learningmaterials, another approach for labeling incomplete text is required. With the increased
digitization of learning materials and their use in smart learning environments, teachersand publishers are migrating existing non-digital materials to these systems. As theselearning materials were usually not created while considering digitization, it is often seenthat publishers will provide publication-quality PDFs directly to teachers or educational
institutes. Problems are caused when uploading and analyzing such materials in learning
environments as it is difficult to extract all of the information, such as: text, formulas,graphs, and images, from publication-quality PDFs, resulting in incomplete information
extraction (Abekawa & Aizawa, 2016). While researchers have tried labeling with sentences, images, formulas, or a combination of them (Bhartiya et al., 2016; Shen, et al.,2021; Tian et al., 2022; Wang et al., 2022), there has been less focus on classification withincomplete information of mathematical sentences. In this study, we propose a mathematical exercise labeling algorithm that can deal with detailed labels, even for incomplete sentences, by focusing on the exact match of a set of mathematical exercises andPage 2 of 30

Yamauchi et al. Smart Learning Environments(2023) 10:51

predicting a unit using an existing machine learning method or calculating the similarityof any given exercise to a set of weighted word n-grams. Therefore, we aim to answer thefollowing research question:
RQ: What are the best features and models that can assign detailed and precise labelsfrom incomplete mathematical exercise text?
We propose an algorithm to automatically provide classification results for preprocessed exercise sentences that have been extracted from publication-quality PDFs that
include incomplete text. In the experiments of this study, two different levels of labels
are assigned to each exercise for validation. We then predict the labels to evaluate theperformance of the proposed algorithm and compare it to state-of-the-art word embedding models.

Literature review
Labeling learning materials
National labeling standards for mathematical exercises

Often learning materials are labeled to notice easily what kind of knowledge is containedin an exercise. Government standards often provide some norms of mathematical exercise classification, for example, in Japan the government provides common standards ofsubjects and directions for each unit of study that aim to develop the qualities and abilities to think mathematically through mathematical activities in the Guidelines for the
Course of Study for Senior High Schools (MEXT, 2009, 2018), and teachers prepare exercises by following these directions. In the US, the Common Core State Standards (CCSS)
classification refers to the learning standards for K-12 education that was developed incollaboration with teachers, school administrators, and professionals to provide a clear
and consistent framework for preparing children for college and career success (Ritter,2009). It includes 11 units that students will study over the course of nine years, plus
appendices that cover counting and radix, operations and algebraic thinking, decimal
numbers and operations, fraction operations, measurement and data, ratios and proportion relationships, number systems, expressions and equations, functions, geometry, statistics, and probability, as well as content taught in higher grades (Shintani, 2014). In
Mathematics Curriculum Standards for Compulsory Education in China (MOE, 2012),
learning items are distributed into one of up to four main parts and assigned categories
10 keywords, including: number sense, symbolic awareness, space concept, geometry
intuitive, data analysis concept, computation ability, reasoning ability, model idea, application awareness, and innovative awareness (Guo et al., 2018). In Australian Curriculum,
Assessment and Reporting Authority, treated as an Australian digital curriculum (Ditchburn, 2012), units are called “content strands” and consist of number and algebra, measurement and geometry, and statistics and probability. Each of these strands has 6, 5, and2 units, respectively, and the structure can be described as hierarchical (ACARA). Thereis also a specialized system called Zentralblatt MATH (zbMATH) that is a mathematicsrelated bibliographic database and literature search engine. The Mathematics Subject
Classification (MSC) which zbMATH helps maintain is used to classify items in mathematical sciences literature. Every 10 years, two editorial groups solicit input from thePage 3 of 30

Yamauchi et al. Smart Learning Environments(2023) 10:51

mathematical community. The new MSC (MSC2020) includes 63 two-digit classifications, 529 three-digit classifications, and 6006 five-digit classifications (Dunne & Hulek,
2020; Kühnemund, 2016).
As the topic standards mentioned above can be important rules when classifying manymathematical materials, some researchers decided to tackled labeling math exercises
based on the standard automatically. One study has attempted to classify according to
the CCSS (Ritter, 2009), and this study used 385 different labels to classify 12 years ofmathematics materials from kindergarten through to high school (Shen et al., 2021).
One study also proposed the MathBERT model (Shen et al., 2021), which is a model
created by preparing a large mathematical corpus ranging from the pre-kindergarten to
the graduate level and training a base BERT model (Devlin et al., 2019). However, thesestudies did not tackle the problem of incomplete text classification. In this study, we use
information from MEXT to label exercise data in both a coarse and detailed method
while focusing on incomplete exercise text labeling.
Labeling for analysis of how students learn

There is a trend toward analyzing learning behavior in a new way using labels assigned
to teaching materials. Regarding the use of features in the analysis of learning effectiveness, a study reported that the proposed system automatically assigned labels withlearning materials and the study shows the assigned labels can assist in the discovery ofstudents’ learning patterns (Wang et al., 2022). While the analysis using labels is novel inthe research, the labeling conducted in this research was only for one class in the university and was not generalized using a common standard.
Giving labels to exercises for knowledge tracing is also a hot research topic. One study,
using multiple real data sets consisting of tens of thousands of users and items, showedthat regression classification models could accurately and rapidly estimate student
knowledge, even when student data is sparsely observed. In addition, the study showedthat the model can handle multiple knowledge elements and side information such as
the number of trials of items and skill levels (Vie et al., 2019). If no labels were given toeach exercise, the study could not accurately predict the student’s performance.
It is also useful to categorize any exercise for recommending a specific exercise toenhance students’ understanding. One study discusses the application of a topic-based
tree structure to personalized adaptive educational systems for its transparency for theusers (Sosnovsky & Brusilovsky, 2015). Another study focuses on the visualization of
the relationship between any combination of two topics to notify the achievements ofeach student individually, which aims to be consistent among the assessments in different courses, to do meaningful feedback to individual, and to grasp the students’ longterm progress (Khosravi & Cooper, 2018). There has also been research into extracting
labels from learning materials to form knowledge structure representations that learners
can use to increase their awareness of the study process (Flanagan et al., 2019). These
research examples show that it is easier to obtain or utilize detailed information about
the characteristics of the material if they are labeled in advance. In addition, there is
one system, called BookRoll, that any learner can post the PDF materials freely without
selecting any topics (Flanagan & Ogata, 2018), so in this context the automatic labelingsystem helps the materials to obtain some topics.

Page 4 of 30

Yamauchi et al. Smart Learning Environments(2023) 10:51

In this study, we tackle the task of text classification to automate knowledge labeling process for incomplete text by proposing a more detailed and highly accurate
method based on n-grams. The proposed method could improve the use of materials
with knowledge labeling and assist in the analysis of how students study using thesematerials.

Labeling to reduce the burden on domain experts

Automatic labeling and classification of learning materials is a prominent area of classification research in education. Schubotz et al., (2020), proposed an automatic classification method in a mathematical subject classification scheme for organizing
mathematical literature, achieving a classification agreement rate of 81% with very close
accuracy in two large peer-review services. It also enabled an 86% reduction in laborwhen compared to the manual classification task. The result shows the advantage oflabeling automatically, although the research has a different context when compared tothe present paper. Tian et al. (2022), proposed a unit classification method that combines natural language processing techniques with a method for extracting keywords
from mathematical exercises, and this resulted in a 25% labor reduction compared tomanual classification. While the paper provides a mostly accurate classification of units,
it only provides as detailed a classification as the Courses of Study even though moredetailed labeling may be necessary depending on the intended use.
Automated detailed labeling must be accurate in order to reduce the burden on
domain experts and assist in assigning labels to exercises. In this study, we developed
a more detailed automated classification that has high accuracy even when labelingexercises that contain incomplete text.

Hierarchical and automatic labeling of teaching materials

Hierarchical text classification (HTC) is a method that can classify objects into multilevel detailed classifications, and this aims to assign one or more optimal categories
to text documents from a hierarchical category space (Graovac, 2017) and literaturein this area has applied this method to many different types of domains (Silla & Freitas, 2011). Another study proposes a method of categorizing and labeling educational materials with various academic learning objectives (Bhartiya et al., 2016). Thismethod selected words in the materials as labels and achieved extensive labeling invarious grades and subjects.
When labeling the exercises, the granularity that is required depends on how thelabels will be used, so by assigning different labels to each exercise the scope of use
can be broadened. In the experiments, we assigned two labels to each exercise, suchas: 1st level unit and 2nd level unit and measured the classification accuracy of eachlabel. Previous studies related to labeling materials for use in Japanese schools don’t
consider the hierarchical label. Tian et al. (2022) uses 24 labels for the Japanese highschool curriculum, and Wang et al. (2022) uses 47 for a course at a university in Japan.
Our study uses the most detailed labeling scheme of all previous studies into Japanesemathematical exercise classification with a total of 297 items at the 2nd unit level.

Page 5 of 30

Yamauchi et al. Smart Learning Environments(2023) 10:51

Text vectorization method for classification tasks
N‑gramWe often use text mining, machine learning, and natural language processing to classify
many kinds of text data, such as: electronic documents, online news, blogs, emails, anddigital libraries, to obtain meaningful knowledge, and many classification methods havebeen proposed (Khan et al., 2010). Previously, Suen (1979) showed that n-gram classification is effective to classify incomplete sentences from OCR. Text classification must
work reliably for all input, and therefore must allow some tolerance for various typesof text error problems, such as misspellings and grammatical errors in e-mail and character recognition errors in OCR-processed documents, and Cavnar and Trenkle (1994)
argued that n-grams is an effective way to meet this requirement. Graovac (2014) proposed an n-gram method for topic-based text classification using the characters in a text
so that the method is independent of language and topic.
The task of classification using n-grams has been investigated in various studies. A
study on the results of using an n-gram-based algorithm for Bangla text classification(Mansur, 2006) and a study that attempted to statistically estimate the expressive quality of an article by using word n-grams and part-of-speech n-grams in the article (Kobayashi et al., 2012). Despite the loss of semantic information, bag-of-n-grams-based
methods have been shown to perform well in sentiment analysis (Li et al., 2016). Many
studies have also found n-grams to be an effective tool for classification tasks in a variety
of fields, such as in music analysis (Zheng et al., 2017).
However, there are still few studies that use n-gram to classify Japanese mathematical
exercise materials. Our study uses n-grams and applies it as a novel method of Japanesemathematical text classification.
Word embedding

Recently, word embedding methods have become a popular text vectorization method,and one of the most representative and popular word embedding methods is Word2Vec(Mikolov et al., 2013). This method trains a model on context-independent distributed
representations for words. Considering the context of the sentence using RNN or LSTM,
machine learning improves the understanding of sentences, such as: ELMo (Peters et al.,2018) that uses LSTM for a contextualized word embedding model. Moreover, OpenAI’s
GPT model (Radford et al., 2019) is a model that can have enhanced flexibility for finetuned tasks, which allows an AI to consider words at a distance and to compute it not
as a Markov method, but in parallel. BERT (Devlin et al., 2019) is also a popular natural
language model created by Google which has an attention mechanism instead of RNN
and applies a masked language model for learning.
Prior studies have demonstrated the efficacy of word embedding for label classification tasks. For instance, Dharma et al. (2022) utilized the Fasttext method to classify adataset of 19,977 news articles and 20 news topics with 97.2% accuracy, outperforming other word embedding techniques. However, in the case of short sentence exercises,
the sentence vectorization methods using word embedding has been found to be lesseffective. Tian et al. (2022) applied word embedding for the classification of short Japanese exercise texts, achieving an accuracy of 72.87%. The combination of this method
along with the extraction of keywords, called the WE-KE model, further enhanced thePage 6 of 30

Yamauchi et al. Smart Learning Environments(2023) 10:51

accuracy to 79.57%. These findings suggest that word embedding may not be as effective
for short exercise texts. It is worth noting that for this experiment, incomplete sentenceswere employed as inputs.
The objective of this study is to introduce an automated classification algorithm capable of effectively categorizing short Japanese sentences found in mathematical exercises. To accomplish this, we concentrate on achieving the best agreement between setsof mathematical exercises through the calculation of similarity using weighted word
n-gram variance representations. The algorithm is then assessed by comparing it to similar experiments conducted using prediction models, and its accuracy is calculated.
Morphological analysis and relation to reading comprehension

As a study of mathematical morphological analysis, it is popular to investigate the relationship between learners’ reading comprehension and their mathematical skills. It issuggested that general vocabulary may serve as a proxy for mathematics-specific vocabulary in studies that do not include measures of mathematics-specific vocabulary (Chow& Ekholm, 2019). Much of the research investigating the relationship between language
proficiency and math outcomes focuses specifically on vocabulary for reasons such as
memorizing large numbers as words (Spelke & Tsivkin, 2001) and the need to understand oral instruction (Chow & Ekholm, 2019).
While the present study does not specifically address the learners’ reading comprehension skills, but we use morphology to analyze the Japanese sentence and to create a vector representation.
Classification of incomplete exercise texts

According to previous research, exercise texts for classification task, which is called“TREC” in the paper, contains the least number of sentences and even the least numberof vocabularies of all 7 dataset types, including movie review, sentiment classification
dataset and subjectivity dataset (Liu & Guo, 2019). This fact indicates that an exercise
text consists of relatively less characters. Previous studies have also shown that it is difficult to achieve adequate performance on the classification of short text by word embedding which was also discussed in Sect. 2.2.2, and therefore another approach is required
for this task.
Unlike other natural-language-presented subjects such as languages, history, andsocial science, mathematical learning materials involve the presentation of notations,formulas, and figures. Using the common PDF format, the processing of non-language
information in the mathematical learning materials is costly and complex. Although
prior studies have shown that formula processing is detectable if the layout and format
are defined (Date & Isozaki, 2015; Fateman et al., 1996), it is difficult to detect when
they are not. Such issues arise during the uploading and analyzing of these materials in
educational settings due to the challenge of fully extracting content like text, formulas,graphs, and images from published PDFs, leading to incomplete information retrieval(Abekawa & Aizawa, 2016). Hence, other methods should be investigated for the labeling from incomplete text.
In this study, we aim to automatically label the mathematical learning materials by
analyzing textual information which is readily extractable from PDF files.

Page 7 of 30

Yamauchi et al. Smart Learning Environments(2023) 10:51

Mathematical education in Japan

Japanese students’ performance in mathematics is the highest level among countries in
the world, which is said to be due to the influence of students’ confidence in mathematics, student Socio-Economic Status (SES), and school emphasis on academic success(Wang et al., 2023).
Japan’s Courses of Study are curriculum standards established by the Ministry of
Education, Culture, Sports, Science and Technology (MEXT) to ensure that standards
are maintained in all schools throughout Japan. They are revised approximately every
10 years. In recent years, the decline in Japan’s performance in the PISA 2003 international achievement test has triggered a shift in educational policy toward improving academic achievement (Onishi, 2011). MEXT revision of that standard in 2009
strengthened English foreign language learning and introduced task-based learning(MEXT, 2009). The latest revision, issued in 2018, set three items as learning objectives:“knowledge and skills,” “ability to think, judge, express” and “ability to learn and humanity” (MEXT, 2018). Students’ textbooks, exercises, and in-class learning are based on
the Courses of Study. In mathematics, the curriculum guidelines divide mathematicalknowledge and skills into categories, each of which has its own meaning. Table 1 shows
the organization of mathematics units and their objectives as defined by the Courses ofStudy revised in 2009 and exercises in the materials in this study are prepared based on
this. Because these standards are used all around Japan, the categorization of exercises
can affect mathematical education throughout the country.
Technology is helping researchers better understand how students learn mathematics inorder to improve studies on mathematical education (Fishback & Schlicker, 1996; Hussein,2023). In the context of mathematics in Japan, units on mathematics related to statistics
have been introduced at every grade level, as indicated by the enhancement of statisticaleducation, and learning activities using computers and other tools. A recent study has proposed the use of programming environments to support the learning of statistics according
to learner’s grade (Kayama et al., 2022). To support learners use of such environments, it isimportant for learners to be able to figure out which exercises are in which grade level ofTable 1 Explanation of each organization part of the units (MEXT, 2009)Part Description
I

To provide students with an understanding of numbers and expressions, figures and measurements,
quadratic functions, and data analysis, and to cultivate the acquisition of basic knowledge and proficiency
in these skills, as well as to cultivate the ability to consider events mathematically, to recognize the meritsof mathematics, and to develop an attitude of utilizing these skills

II

To develop understanding of various expressions, figures and equations, exponential and logarithmic
functions, trigonometric functions, and differential and integral calculus, to acquire basic knowledge andskills, to develop the ability to consider and express phenomena mathematically, and to foster an attitude
to make use of such knowledge

III

To deepen students’ understanding of curves on a plane, the complex plane, limits, differential and integral
calculus, to develop their knowledge and skills, to develop their ability to consider and express phenomena mathematically, and to foster an attitude to actively utilize these skills

A

To make students understand the number of cases and probability, properties of integers, and propertiesof figures, to acquire basic knowledge and skills, to cultivate the ability to consider events mathematically,
to recognize the goodness of mathematics, and to cultivate an attitude to utilize these skills

B

To develop an understanding of probability distributions and statistical inferences, number sequencesand vectors, to acquire basic knowledge and skills, to develop the ability to consider and express events
mathematically, and to cultivate an attitude of using these skills

Page 8 of 30

Yamauchi et al. Smart Learning Environments(2023) 10:51

Page 9 of 30

similar statistical units without requiring teacher intervention. Another study has proposed
the method to explain the unit structure of textbooks in order to relate knowledge in learning (Taniguchi & Itoh, 2023). However, without knowledge labeling of textbooks and exercises, it is difficult to make use of such unit structures in educational settings.
In this study, we focus on the labeling of mathematics units and verify the assignment ofunits to textbooks and exercises, which have been the subject of much research. In addition, we focus on the Japanese context of mathematical education and use the most common standards in Japan.

MethodOur goal is to find an algorithm that can assign appropriate labels to educational materials using characters extracted from math teaching material PDFs. In particular, we use thecharacters extracted from the math teaching material PDFs as input, vectorize them using
natural language processing, train the vectors as features, and output the labels lpred.
We defined the method of predicting labels with the following two functions:tfvec fpredvl∈L→→ pred(1)where t is a set of characters from a mathematical PDF material, v is a vector from t
by vectorization. In the following section, we defined the functions fvec1 , fvec2 andfpred1 , fpred2 respectively as the methods of vectorization from characters and the methods of prediction from the vector. In other words, we defined fvec1 or fvec2 as the featureselecting method and did fpred1 or fpred2 as the model-selecting method. Note that there
are a set of labels L that lpred can be selected from L. Figure 1 shows the experimental
overview from inputting exercise PDF to outputting a prediction.
Data preparation

The input data in this experiment is a Japanese math exercise contained in a PDF file. To
use the characters’ information of exercises, we first extract text and create a text set fromthe exercise PDF files. We defined datasets Q = {q1 , q2 , ..., qi , ..., qn } for each qi ∈ Q as an
exercises’ text data set. Each q has its label li = {li1 , li2 } in advance, where li1 , li2 represent
the 1st level label, and the 2nd level label, respectively. A relation between a unit label and a
subunit label can be formulated as follows:∀li1 2 ∈li1 1 ,li2 2 ∈li2 1 i1 1 �= i2 1 ⇒ i1 2 � = i2 2Fig. 1 Overview of experiment(2)

Yamauchi et al. Smart Learning Environments(2023) 10:51

We divide the obtained characters into meaningful chucks before converting theminto vectors. This preprocessing provides us with word sets Ti = {ti1 , ti2 , ..., tij } of each Qi .
n(Ti ) equals j where n(X) represents the number of elements in the set X .
The exercise texts used are electronic pdf versions of each of the following exercise
books:••••••“Supplementary and Revised Edition Charting Mathematics from the Basics I + A”
“Supplementary and Revised Edition Charting Mathematics from the Basics II + B”
“Supplementary and Revised Edition Charting Mathematics from the Basics III”
“Succeeding Mathematics I + A for Textbook Sidelines”“Succeeding Mathematics II + B for Textbook Sidelines”“Succeeding Mathematics III for Textbook Sidelines”

These exercise books are designed for high school students and align with the textbooks approved by the Japanese government (MEXT, 2021). They are produced by thesame company responsible for the widely used textbooks in Japan.
We prepared text files by reading text data using the Python library Pdf2text (Palmer,2021). Note that PDF files are more difficult to obtain in their complete text form thanHTML-formatted files (Ramakrishnan et al., 2012; Smith, 2007).
Japanese high school mathematics teachers created one 1st level unit label and one
2nd level unit label for each exercise by referring to sections in their textbooks and mapping them to each other. There was a total of 2775 exercises, consisting of 24 1st level
units and 297 2nd level units. The same 2nd level unit is never assigned across multiple
1st level units. Each 1st level unit consists of between 25 and 200 exercises, with a minimum of five exercises assigned per 2nd level unit. Table 2 shows the content of each 1stlevel unit, the organization part the unit belongs to, the number of 2nd level units it contains (n Ll 2 ), the number of exercises it contains (n(Ql )), and the mean and standard−
deviation for the morphemes contained in each exercise (n(Tl ), sTl ). All of these 1st level
units are math common standard in Japan. They are categorized into 5 big meaningful
sets. The column “part” of Table 2 represents one of the five organization parts (refer toTable 1) that is assigned to the unit. Figure 2 shows an example of the hierarchical structure of 1st level unit and 2nd level unit. Figure 3 shows an example of an exercise and the
1st level unit and 2nd level unit that has been assigned to it.
We used pdf2txt to extract the characters from the mathematical exercise PDF. Figure 4 shows an example of what was extracted from the mathematical exercise PDF. In
the figure, (a) represents the raw PDF data of the exercise, (b) represents the extracted
Japanese texts from (a), and (c) is an English translation of (b). As shown in (a) of the
figure, while the information about the diagram in the PDF cannot be extracted, also theletters highlighted in blue in the PDF do not appear in the extracted text. These words
consist of mathematical formulas “GH = 2OG”, figures such as “3” of “3点” (3 points),
and symbols such as “ABC”. It is difficult to extract significant sentences from extracted
texts because of the few of the text and symbols related to math equations could be
extracted. We can see from (b) or (c) that we could not get the full sentence from PDF
text, and it was also somewhat meaningless and difficult to comprehend.

Page 10 of 30

Yamauchi et al. Smart Learning Environments(2023) 10:51

Page 11 of 30

Table 2 Detailed information of each 1st level unit
Unit NamePart

English translation of the Unit Name

数と式

I

Number and formula

集合と命題

I

二次関数

 
n Ll2

n(Ql )

18

Set theory

9

I

Quadratic function

図形と計量

I

データの分析

−n(Tl )

s Tl

125

51.8

43.3

60

124.9

79.4

15

200

123.3

42.6

Measuring graphics

10

85

78.7

49.7

I

Data analysis

3

25

107.0

60.2

式と証明

II

Formulas and proofs

10

95

50.3

40.9

複素数と方程式

II

Complex numbers and equations

12

115

66.5

45.1

図形と方程式II

Geometric equations

16

135

88.1

62.2

三角関数
指数関数と対数関数

II

Trigonometric function

11

90

87.7

42.6

II

Exponential or logarithmic function

8

70

84.1

44.1

微分法

II

Differentiation

6

100

125.0

51.9

積分法

II

Integration

6

70

92.3

41.6

複素数平面

III

Complex plane

8

100

111.7

59.5

極限

III

Limit

12

120

83.2

43.0

式と曲線

III

Equations and curves

18

175

145.2

64.9

微分法の応用III

High-leveled differentiation

17

175

110.1

46.2

積分法の応用III

High-leveled integration

18

200

91.6

54.6

場合の数

A

Number of cases

13

120

121.9

104.3

確率

A

Probability

11

90

115.3

74.9

図形の性質

A

Geometric properties

16

110

87.4

54.6

整数の性質

A

Properties of integers

16

160

81.8

61.7

平面上のベクトルB

Two-dimensional vector

16

125

89.0

52.5

空間のベクトル

B

Three-dimensional vector

10

95

104.4

47.8

数列

B

Sequence

18

135

114.3

57.9

297

2775

98.5

61.4

All exercises

Fig. 2 Example of the hierarchical structure of 1st and 2nd level unit

As Japanese text does not contain word boundaries, preprocessing to extract morphemes is required and we used a package called Nagisa (Ikeda, 2021) for the morphological analysis of text data. Nagisa is a package for the morphological analysis ofJapanese sentences. One feature of Nagisa is that it can assign a part of speech to eachYamauchi et al. Smart Learning Environments(2023) 10:51

Fig. 3 Example of exercises in the dataset. The 1st level unit “two-dimensional vector” and 2nd level unit “useof inner product” are assigned to an exercise in the figure

Fig. 4 Examples of PDF and extracted texts

segmented morpheme and can exclude words with a specific part of speech. Someparts of the text cannot precisely be divided into morphemes and therefore someparts of deviation are incorrect.
Vectorization methods
VSM created from N‑gram

We assumed that the words or a sequence of words in a sentence which has the samelabel will be similar to one and another, so we developed the n-gram word extracting
method and compared the performance to methods using state-of-the-art word embedding. As we will compare both word embeddings and n-grams in the same context, wehave to convert the n-grams into a vector which represents the n-gram features.

Page 12 of 30

Yamauchi et al. Smart Learning Environments(2023) 10:51

Page 13 of 30

Fig. 5 Example of the way to extract n-grams

We first define vector VGi,k that is created from the specific exercise tokens Ti with all
exercise text token T and the number of consecutive tokens of k -gram k , such as:

fvec1 (T , Ti , k) → VGi,k(3)

Figure 5 shows the overview of method to convert n-gram of sentence into vector.
The method of creating n-grams is as follows:
We created a word k -grams gi,k,l (1 ≤ l ≤ n(Ti ) − k + 1) from the tokenized exercise
sentences of t ∈ Ti . This means that k consecutive tokens from til to til+k−1 were taken
and stored in a single tuple:
gi,k,l = til , til+1 , . . . , til+k−1(4)Then we made Gi,k aggregating all l of gi,k,l.

Gi,k = gi,k,1 , gi,k,2 , . . . , gi,k,n(Ti )−k+1(5)

For vectorization using word n-grams, we prepared a list Gk that includes all gi,k,l in
all Gi,k . Then, we made a list called k -gram-list that indicates if each component of the n-grams included the query n-grams. We defined the mth elements of k -gram-list:
Gk = gx,k,z |∃x,z (x = i) ∧ (z = l)(6)k − gram − list[m] = gm gm ∈ Gk , ∀x,y , x � = y ⇒ gx � = gy , 0 ≤ m < n(Gk )(7)

For each i , the qi should have one vector whose length is the same as n(Gk ). The i thvalue of v at k -gram, VGi,k ,m ∈ VGi,k , is determined by the following formula: 
1 gm ∈ Gi,k
VGi,k ,m =(8)0(otherwise)
When Nagisa morphologically analyzes numbers, it recognizes each number as a
one-digit noun. In mathematical texts, different numerals are treated as different morphemes, so we created an algorithm that treats digits as a single number, as shown in

Fig. 6 Example formula processing added to Nagisa

Yamauchi et al. Smart Learning Environments(2023) 10:51

Page 14 of 30

Table 3 The number of n-gram elements in the vectors
Numberof n-gramelements

n

1

2451

2

20,424

3

56,324

4

95,536

5

130,034

6

157,546

Fig. 7 Processing flow of creating n-grams with Nagisa

Fig. 6, and treated all numbers as the same thing. This process makes easier to find thesame exercise except for numbers or formula.
We collected the n-gram data of the exercise texts. In n-grams, it is necessary to determine the value of n for good classification accuracy. Although there are some studies that
explore appropriate values of n for each task, as research has shown that large n-grams
have advantages in generating features that can be interpreted in malware analysis (Raffet al., 2018), in almost all previous studies n values are very small, and n > 6 is extremely
rare. Larger values of n are not tested due to the computational burden and the risk ofoverfitting. So in this study, we conducted n-g extraction for 1 ≤ n ≤ 6. Table 3 shows
the results of the number of n-grams with 1 ≤ n ≤ 6. Figure 7 shows the overall flow ofcreating n-grams with Nagisa. In the figure, (a) represents the extracted full text data.
The item (b) represents a list of morphemes with part of the speech of each morpheme:n, p, v, s stands for noun, particle, verb, suffix, respectively. The item (c) represents an
obtained list of morphemes processed numbers by the method illustrated in Fig. 6. The
item (d) represents the completely obtained bi-gram from (a).
Word embedding vectorization We defined vector VEi that is created from the specificexercise tokens Ti (1 ≤ i ≤ n(Q)) and the model for word embedding model, i.e.

fvec2 (Ti , model) → VEi(9)For vectorization with word embedding, we used a model called fastText (Joulin et al.,2017). There is a website, https://​fastt​ext.​cc/​docs/​en/​crawl-​vecto​rs.​html, which has
pre-trained models for 157 languages. In this experiment, we used the Japanese model,

Yamauchi et al. Smart Learning Environments(2023) 10:51

Page 15 of 30

which combines three methods to represent input sentence data in 300 dimensions:character 5-gram, weighting by position, and Word2Vec (Church, 2017).
Label prediction by vectorized sentencePrediction by calculating cosine similarity

For any exercises text T , we use score s(Ta , Tb ) to measure the similarity of texts between
Ta and Tb. The higher s(Ta , Tb ) is, the more similar Ta and Tb are. The answer of predicting labels with finding similarity of exercises can be formulated as: Given a set of query
exercise text vector vquery , a labeled-exercise text vlabeled that has the label llabeled , weightparameters function fw , our goal is to integrate these heterogeneous materials to measure the similarity scores of exercise pairs and predict the 1st level unit label or the 2ndlevel unit label for any vquery by selecting the candidate label lpred with a predicted label,i.e.
fpred1 vquery , Vlabeled , Llabeled , fw → lpred ∈ L(10)where Vlabeled , Llabeled is the set of vectors of labeled data and labels of them respectively, fw is the weight parameters function, and L is the domain of labels in the data. The
selected label for query lpred is the prediction label of the exercises.
In this algorithm, as shown in Fig. 1, the data set is divided into label data andquery, and the similarity between the set of word n-grams in the label data and the setof word n-grams in the query is calculated. Here, the similarity of the vectors is thevalues vlX , vquery obtained using the cosine similarity method, where vlX , vquery represent the vector of word n-grams of the labeled data with the label lX (1 ≤ X ≤ n(L)) andthe vector of word n-grams of the query, respectively.


v ·vquerys vlX , vquery = �v lX��vquery �

(11)

lX



We then compute slX ,query by aggregating s vlX , vquery of all vector vlX with label lX , andsubstitute them all into the determined weight function fw . Previous studies improveaccuracy by weighting for realistic non-homogeneous data sets. One study successfully
achieved high accuracy using cosine similarity with added weighting to effectively train
CNNs in realistic learning situations such as class imbalance, small size, and label noise(Kobayashi, 2021). Weighting explanatory variables with generated n-grams is said to bean effective means of improving text classification accuracy (Graovac et al., 2015). The
calculation formula of slX ,query is as follows:
slX ,query = fw ssetlX ,query(12)
ssetlX ,query =




 

s vlX1 , vquery , s vlX2 , vquery , . . . , s vlX n VlabeledX, vquery(13)

where VlabeledX represents the labeled vector assigned label X . Finally, we find slX ,query for
all lX and determine lpred,query as follows:Yamauchi et al. Smart Learning Environments(2023) 10:51

Page 16 of 30

Fig. 8 How to find the weight at which label lX assigns to a query

lpred,query =argmaxslX (X = 1, 2, . . . , n(L)) lX ,query(14)

What this formula means is that the predicted label is the same label as the problem
with higher similarity. Various changes in the function fw are used to determine amore suitable weighting for classification. In this experiment, we defined the functions as follows:fmean ssetlX ,query =n ssetl ,query
X


k=1

sl ,query Xk
n ssetl ,query

(15)

X



fmax ssetlX ,query = HX,1

(16)



m

HX,k
ftopm ssetlX ,query =
m

(17)

k=1



m

(m−k+1)H X,kmfrank m ssetlX ,query =
k
k=1

k=1

(18)

where HX,k represents the k th highest value in ssetlX ,query . The prediction vector for
query, vquery , is defined as the array of values [sl1 ,query , sl2 ,query , . . . , sln(L) ,query ] obtainedby the function fw.
We created these functions based on the sentence similarity which has the samelabel: the more similar the sentences are, the more likely to have the same label. There
are two assumptions as follows:• Assumption 1: Any pair of two exercises that have the same label are similar to eachother. Therefore, we created to find the most appropriate label considering all labeled
exercises’ similarities, fmean.
• Assumption 2: Specific exercises with the same label have high similarity with eachother. Therefore, we created to find the most appropriate label considering m labeled
exercises’ similarities, ftopm , frankm.
Figure 8 shows the overview of how to find the weight of specific label assigns to aquery.

Yamauchi et al. Smart Learning Environments(2023) 10:51

Page 17 of 30

Prediction by machine learning

The problem of finding similar exercises can be formulated as follows: Given a set oftest exercise text vector vtest , a set of training text vectors vtrain that have the true label
set, our goal is to integrate these heterogeneous materials to predict the 1st level unitor the 2nd level unit for any vector from query exercise text vquery by selecting thecandidate label lpred with a predicted label, i.e.

(19)

fpred2 (vtest , vtrain , model) → lpred ∈ L

where model is a package that can classify these vectors into the specific number ofcategories and L is the domain of labels in the data. The selected label for test data vtest is
the prediction label of the exercises, described as lpred.
• XGBoost (Chen et al., 2015; Chen & Guestrin, 2016): This model, which merges
boosting with decision trees, has demonstrated promising outcomes in diverse natural language processing assignments, making it an appropriate choice for employment in this paper’s context.
• Random Forest (Breiman, 2001): This is a model that employs numerous decision
trees trained using randomly selected training data. It performs effectively even witha considerable number of explanatory variables, enabling it to handle a 300-dimensional vector.
• Logistic Regression (Cox, 1958), Perceptron (Rosenblatt, 1958): Both models are used
for statistical regression with variables that follow a Bernoulli distribution. However,the former employs coordinate descent or quasi-Newtonian methods for parameter
determination in optimization problems, whereas the latter utilizes the stochastic
gradient descent method.

Evaluation

We conducted experiments using fivefold cross validation for training and prediction.
The use of fivefold reduces over-training on training and label data. In addition, accuracy
AL, macro F-measure FL and weighted F-measure FwL were used to evaluate this experimental algorithm. Let TPl , FPl , TNl and FNl denote that the true prediction for a label lis correct or wrong, and that the false prediction for a label l is correct or wrong, thenaccuracy Al and precision Pl , recall Rl and the f score Fl can be expressed as follows.
TPl
TPl +TNl
2Pl Rll
Pl = TPTP+FP , Rl = TP +FN , Al = TP +TN +FP +FN , Fl = P +R
l

PL =



l∈L Plln(L) , RL =ll∈L Rlln(L) , AL =ll∈L Allln(L) , FL =

l

l

l



2PL RLl∈L Fln(L) , FwL = PL +RL

(20)
(21)

We used AL, FL, and FwL to evaluate the performance of the prediction.

ResultClassification results with selecting features and methods

We take n-grams of 1 ≤ n ≤ 6 and vector with w2vec into consideration. We also prepare a cosine similarity model with the weighted function formula (15), (16), (17), (18)Yamauchi et al. Smart Learning Environments(2023) 10:51

Page 18 of 30

Table 4 Classification in 1st level unit between a feature and accuracy AL in n-grams and machinelearning methods
n=1

n=2

n=3

n=4

n=5

n=6

w2vec

Similarity (fmean)

0.71210.73980.70590.70490.68250.68580.3834

Similarity (fmax)

0.80250.81840.81300.79530.77910.75390.6944

Similarity (ftop2)

0.83280.83860.83140.81050.78380.76470.7063

Similarity (ftop3)

0.84110.84290.83570.80720.7791

0.76250.7081

Similarity (ftop4)

0.84070.84940.83280.81050.77300.75390.7049Similarity (ftop5)

0.84320.85050.83240.80220.76790.7506

0.6955

Similarity (ftop6)

0.84430.84940.83210.79600.76070.74090.6818Similarity (ftop7)

0.84040.85230.83030.78880.75960.73730.6728Similarity (ftop8)

0.83890.85080.8256

0.78590.75530.73480.6595

Similarity (ftop9)

0.83780.84830.8202

0.78310.75240.72430.6541

Similarity (ftop10)

0.83500.84650.81620.77980.74810.71960.6436

Similarity (frank2)

0.82310.83820.82810.80900.78410.76470.7095

Similarity (frank3)

0.83530.84110.83320.81230.78520.76650.7114

Similarity (frank4)

0.84000.84720.83640.81300.78520.76720.7106

Similarity (frank5)

0.84320.85230.83820.81190.78130.76540.7085

Similarity (frank6)

0.84610.85330.83710.81410.77980.75890.7067

Similarity (frank7)

0.84580.85370.83860.81230.77800.75930.6998Similarity (frank8)

0.84830.85510.83750.81010.77440.75570.6951
0.6923

Similarity (frank9)

0.84790.85550.83530.80720.77590.7517

Similarity (frank10)

0.84790.85370.83350.80290.77190.74630.6836xgb

0.86050.8274

0.6681

0.5831

0.48250.42630.6468rf

0.92500.90130.83100.74270.6526

0.59680.7139mlp

0.79100.87320.86310.86740.83820.82090.8065lr

0.91930.89300.86410.83710.80250.74920.8537(2 ≤ m ≤ 10), and machine learning method Xgboost, Random Forest, Perceptron andLogistic Regression. Tables 4, 5, 6, 7, 8 and 9 show the three kinds of prediction result,accuracy AL, macro F-measure FL and weighted F-measure FwL, when we used the combination of each feature and the model. In the tables, the best performance rate in eachfeature is bolded, and the best overall performance is underlined. We also draw a graphthat represents all recalls of each feature and model selection in Figs. 9 and 10. The tables
show that at the both 1st level unit and 2nd level unit prediction, the algorithm yielded
the best AL, FL of all when using mono-gram features with the Random Forest model,
and best FwL when bi-gram features were used with the Random Forest model, whencompared to the use of word embedding, n-grams of the other n features, and the othermodels such as cosine similarity or machine learning methods.
Unlike word embedding, n-grams can be analyzed literally without considering thecontext. It is a suitable feature for this experiment in that we are using text data poorly
extracted from PDF files. In addition, since the experiment using cosine similarity considers textual similarity, the text is likely to be classified into the units that contain manytexts with high similarity. Therefore, the higher the textual similarity of the texts, thehigher the similarity at a larger n is likely to be. However, if n is too large, there will be
fewer matching n-gram words and less textual similarity. Considering these conditions,Yamauchi et al. Smart Learning Environments(2023) 10:51

Page 19 of 30

Table 5 Classification in 1st level unit between a feature and macro F-measure FL in n-grams andmachine learning methods
n=1

n=2

n=3

n=4

n=5

n=6

w2vec

Similarity (fmean)

0.72110.74790.71220.70580.68270.68350.3894

Similarity (fmax)

0.80150.81240.80510.78640.76840.74370.7031

Similarity (ftop2)

0.83090.83450.82360.80040.77460.75500.7133

Similarity (ftop3)

0.83690.83750.82820.79690.76950.75200.7133

Similarity (ftop4)

0.83700.84410.82490.80070.76070.74270.7094

Similarity (ftop5)

0.83920.84570.8252

0.79280.75420.73690.7014

Similarity (ftop6)

0.83940.84440.82430.78360.74440.72570.6868Similarity (ftop7)

0.83380.84640.82120.77280.74380.71990.6767

Similarity (ftop8)

0.83280.84570.81600.76990.73890.7154

0.6619Similarity (ftop9)

0.83120.84210.80910.76690.73300.70040.6563

Similarity (ftop10)

0.82860.83870.80130.75980.7226

0.69130.6440Similarity (frank2)

0.82090.83430.8196

0.79990.77400.7546

0.7163

Similarity (frank3)

0.83300.83790.82540.80180.77560.75720.7181

Similarity (frank4)

0.83790.84330.82880.80270.77520.75700.7162

Similarity (frank5)

0.8406

0.84840.83040.80200.77080.75520.7139Similarity (frank6)

0.84330.84850.82970.8046

0.76660.74720.7119Similarity (frank7)

0.84240.84920.83100.80280.76490.74730.7050Similarity (frank8)

0.84390.85020.82980.79950.76050.74210.70020.6965

Similarity (frank9)

0.84180.85040.82750.79670.75860.7375

Similarity (frank10)

0.84140.84900.82510.79040.75510.73030.6865xgb

0.85720.82120.61710.51950.39380.33120.6303rf

0.92500.89960.82050.71330.6176

0.55680.7098mlp

0.79140.87190.86230.86650.83930.81780.8079lr

0.92180.89030.85700.82470.78800.73580.8566mono-grams turn out to be the most suitable n-size since it is the size of the n-gram thatis most likely to be used in the experiment.
Figures 11 and 12 compare the graph of AL, FL, FwL between selected weighted similarity models and MLP models. The reason of selecting the model in the figure is clarified
as follows: ftop3 is the best prediction model of all ftopm models, frank3 is the best prediction model of all frankm models in 2nd level unit prediction, and frank9 is the best prediction model of all frankm models in 1st level unit prediction.
As shown in Figs. 11 and 12, in both experiments, we could see the results using
weighted similarity models are similar to that using MLP models from the point of the
shape of the figure, while the result between MLP and the other machine learning methods’ results are not so similar; the former doesn’t have a peak when n = 1, and the latter does when n = 1. This suggests that weighted similarity models are taking the same
method as MLP, like aggregating the number in the way of calculating the prediction.
This also shows that as for the optimal value of n for n-grams, n = 2 was optimal for
prediction by searching for similar sentences using cosine similarity. This means that thesmaller the value of n, the greater the number of matching components, while the larger
the value of n, the higher the degree of similarity of sentences with the higher agreement, suggesting that n = 2 is a moderate value that covers both aspects.

Yamauchi et al. Smart Learning Environments(2023) 10:51

Page 20 of 30

Table 6 Classification in 1st level unit between a feature and weighted F-measure FwL in n-grams
and machine learning methods
n=1

n=2

n=3

n=4

n=5

n=6

w2vec

Similarity (fmean)

0.71270.73920.70550.70630.68310.6864

0.3962

Similarity (fmax)

0.80240.81910.81400.79600.77970.7541

0.6939Similarity (ftop2)

0.83310.83930.83220.81090.78370.76380.7064

Similarity (ftop3)

0.84140.84400.83650.80730.77850.76090.7091

Similarity (ftop4)

0.84080.85050.83370.81050.77190.75140.7059Similarity (ftop5)

0.84340.85140.83320.80190.76590.7476

0.6961

Similarity (ftop6)

0.84460.85020.83310.79540.75840.73780.6825

Similarity (ftop7)

0.84070.85310.83130.78860.75690.7344

0.6735

Similarity (ftop8)

0.83940.85150.82650.78520.75280.73210.6607

Similarity (ftop9)

0.83850.84920.8212

0.78230.75040.72230.6556

Similarity (ftop10)

0.83580.84750.81750.77950.7466

0.71840.6457

Similarity (frank2)

0.82320.83890.82910.80930.78440.76430.7097

Similarity (frank3)

0.83570.84190.83400.8126

0.78500.7656

0.7118Similarity (frank4)

0.84030.84810.83720.81340.7846

0.76590.7116

Similarity (frank5)

0.84340.85320.83910.81210.78070.76380.7094

Similarity (frank6)

0.84620.85440.83770.81390.77920.75680.7076

Similarity (frank7)

0.84600.85470.83920.81210.77670.75660.7006

Similarity (frank8)

0.84840.85620.83830.80990.77260.75270.69590.6929Similarity (frank9)

0.84830.85630.83640.80690.77390.7486

Similarity (frank10)

0.84840.85450.83440.80250.76980.74310.6846xgb

0.86010.82790.67770.60370.52740.48000.6530rf

0.92550.90200.83150.74300.65140.60160.7191mlp

0.79240.87450.86360.86730.83670.81920.8066lr

0.91930.89350.86550.83870.80290.74530.8537

Random forest mono‑gram feature analysis

To examine the predictions in detail, we performed feature analysis on the random forestmodel that was trained using monograms as it had the highest accuracy of all of the models
that were evaluated. Table 10 (a) contains the most influential monograms and their degreeof influence. The words ‘I’, ‘III’, ‘II’, ‘A’, and ‘B’ appear to be highly influential. This is because,
as shown in the figure, the classifications of the units fall into one of these five patterns.
Therefore, when these classifications are listed in the PDFs, it was found that these words
can be used to classify the unit more.
Also, not all PDFs contain a classification indicating these five categories. The word “解説” (solution) is not a word that describes the math exercises or the solutions themselves.
Therefore, by omitting these as stop-words, shaded in gray in Table 10 (a), the prediction
can be performed to obtain a more general classification prediction result. This prediction
resulted in AL of 82.88%, FL of 82.82%, and FwL of 83.08%. Table 10 (b) shows the most influential words and their degree of influence in this prediction. The top five words were words
representing “ベクトル” (vector), “数” (number), “関数” (function), “確率” (probability) and “複素” (complex) respectively. All of these words are used as part of more than one name ofa specific unit. Therefore, it is likely that these words were helpful in classifying the text into
broad categories. Note that the assertion of the organization part in a specific place in theYamauchi et al. Smart Learning Environments(2023) 10:51

Page 21 of 30

Table 7 Classification in 2nd level unit between a feature and accuracy AL in n-grams and machinelearning methods
n=1

n=2

n=3

n=4

n=5

n=6

w2vec

Similarity (fmean)

0.43170.46990.44470.44180.42490.41510.2584

Similarity (fmax)

0.55890.60760.5946

0.57800.55640.53050.5121

Similarity (ftop2)

0.58450.62520.60500.59140.56400.53370.4941

Similarity (ftop3)

0.57950.61690.60000.58230.5514

0.52830.4688Similarity (ftop4)

0.55530.59680.58450.56070.53590.51640.4328Similarity (ftop5)

0.48400.5586

0.55100.5431

0.52320.50740.3640Similarity (ftop6)

0.46630.5276

0.5276

0.52220.50850.49840.3495

Similarity (ftop7)

0.45440.5016

0.50990.50450.49050.48900.3240Similarity (ftop8)

0.4411

0.48290.49080.49150.4861

0.4836

0.2998Similarity (ftop9)

0.37950.44900.46880.47820.47640.4782

0.2378Similarity (ftop10)

0.34670.42310.44680.4631

0.46810.47100.2288Similarity (frank2)

0.58950.63350.60830.59240.5676

0.54340.5114

Similarity (frank3)

0.5917

0.63640.61190.59930.56790.54090.4915

Similarity (frank4)

0.58130.62990.60540.59210.56040.54020.4742

Similarity (frank5)

0.55140.61190.59890.58090.55570.53590.3968Similarity (frank6)

0.51240.58590.58410.57050.54670.53190.3838Similarity (frank7)

0.49230.57120.57230.56400.5416

0.52500.3726

Similarity (frank8)

0.4782

0.5532

0.55820.55530.53370.51820.35820.2847

Similarity (frank9)

0.45840.53770.54670.54090.52790.5114

Similarity (frank10)

0.43210.51780.53150.53300.52140.50950.2656xgb

0.14410.08680.06590.05410.04900.03750.1643

rf

0.68500.68290.63140.59570.54520.48610.4987mlp

0.44180.62810.65440.6566

0.64140.61690.4840lr

0.6404

0.63390.62700.62200.59240.56360.6245

PDF would be helpful in classifying exercises, however less generalizable as it would rely on
a consistent format that might not be realistic.

Discussion
Feature selection of extracted incomplete text from PDFs

Labeling incomplete text has been tackled in previous research by using n-grams,
which was shown to be an effective way to meet this problem (Cavnar & Trenkle,
1994; Graovac, 2014; Suen, 1979). In the present research, we investigated using
n-grams on the extracted texts from a PDF of mathematical exercises for which complete texts were difficult to obtain and categorized them into different leveled units.
First, the extracted text could not pick up any information such as mathematical equations, symbols, or numbers. When predicting the topic of incomplete texts, we foundthat vector classification, which involves only information on whether the text is composed of similar elements and does not involve contextual analysis such as n-grams,
was more effective than models that involve contextual analysis. However, we foundthat mono-grams which are similar to more traditional methods, such as n-grams or
bag of words, provided the best classification performance, contradicting results fromprevious research for this specific task. Therefore, we assume that the use of n-gramsin the classification of incomplete texts may depend on the target of the task, whichYamauchi et al. Smart Learning Environments(2023) 10:51

Page 22 of 30

Table 8 Classification in 2nd level unit between a feature and macro F-measure FL in n-grams andmachine learning methods
n=1

n=2

n=3

n=4

n=5

n=6

w2vec

Similarity (fmean)

0.4046

0.44610.42880.43190.41900.40940.2459Similarity (fmax)

0.51030.55730.54110.51530.49580.4723

0.4817

Similarity (ftop2)

0.51670.5616

0.53800.52160.49790.47250.4388Similarity (ftop3)

0.49030.53800.51820.49740.47260.45890.3904

Similarity (ftop4)

0.44190.49330.4832

0.46660.45110.43950.3307

Similarity (ftop5)

0.26940.40820.42090.43230.42970.42630.1895

Similarity (ftop6)

0.24310.3376

0.37310.39680.40380.41500.1797

Similarity (ftop7)

0.23390.29050.34130.36750.37990.39940.1592

Similarity (ftop8)

0.22150.26490.31100.34850.36870.39270.1388Similarity (ftop9)

0.16040.23100.28830.32830.35480.38470.0740Similarity (ftop10)

0.12470.20610.26190.30610.34340.37890.0704

Similarity (frank2)

0.52750.57690.5462

0.52740.50180.4826

0.4661

Similarity (frank3)

0.52020.57030.5422

0.52730.50060.47720.4291

Similarity (frank4)

0.49140.55100.52530.5121

0.48280.47250.3919Similarity (frank5)

0.41770.51790.50630.49170.47700.46200.2148Similarity (frank6)

0.32830.45820.47700.47480.46340.45650.2050Similarity (frank7)

0.28140.42500.45510.46400.45220.4476

0.1958Similarity (frank8)

0.2552

0.39240.42670.44880.44140.43910.18290.1047

Similarity (frank9)

0.23670.36090.41070.42590.43490.4317

Similarity (frank10)

0.21130.32840.38780.41500.42470.42800.0862xgb

0.03330.01500.00870.00360.00260.00140.0480rf

0.61280.59670.54780.5151

0.46230.41280.4285mlp

0.39190.57250.59990.60840.59500.57070.4377lr

0.58580.58280.57520.57450.54990.53850.5922in this case was Japanese mathematical exercises. As the previous research that successfully utilized n-grams to classify incomplete text (Cavnar & Trenkle, 1994; Graovac, 2014; Suen, 1979) neither targeted Japanese nor mathematical exercises, this may
have implications for future research into the classification of incomplete Japanese ormathematical texts.

Model selection for more precise prediction

We aimed at labeling Japanese math text more precisely. A previous study treating Japanese mathematical exercises’ text classification yields 79.57% accuracy with WE-KE
model (Tian et al., 2022). In this experiment, proposed algorithms predicted different
leveled units by two methods: search by similarity sentences using cosine similarity andclassification using machine learning. The results concluded that the best prediction
accuracy was achieved using Random Forest, which resulted in 92.50%. This shows that
our method using mono-grams and Random Forest performed well when it comes toJapanese mathematical text classification.
The result indicates that mono-grams yielded the best classification when we used themethod of Random Forest classification. The reason that mono-gram performs well is,
as Fig. 3 shows, there are incomplete parts of the text when extracting from PDF files,
so there is some meaningless parts that consist of multiple words within a chunk. In

Yamauchi et al. Smart Learning Environments(2023) 10:51

Page 23 of 30

Table 9 Classification in 2nd level unit between a feature and weighted F-measure FwL in n-grams
and machine learning methods
n=1

n=2

n=3

n=4

n=5

n=6

w2vec

Similarity (fmean)

0.44680.47990.44950.43880.41590.40510.2879Similarity (fmax)

0.56410.61400.60210.58600.56170.53490.5183

Similarity (ftop2)

0.60250.64190.62290.60710.57680.54390.5136

Similarity (ftop3)

0.60840.64220.62490.6036

0.56750.54060.5016

Similarity (ftop4)

0.5966

0.63300.6186

0.58830.55630.53300.4755

Similarity (ftop5)

0.56050.61150.59490.57680.54720.52590.4319Similarity (ftop6)

0.54780.59610.58120.56100.53670.51810.4178Similarity (ftop7)

0.53680.57880.57070.5496

0.52180.51180.3932

Similarity (ftop8)

0.52650.56530.55650.54010.52090.50810.3711

Similarity (ftop9)

0.47800.53680.53740.52960.51290.5036

0.3232

Similarity (ftop10)

0.45510.51490.51840.51740.50650.49630.3119Similarity (frank2)

0.60450.64700.62380.60620.58000.5544

0.5261

Similarity (frank3)

0.61310.6561

0.63190.6171

0.58210.55190.5152

Similarity (frank4)

0.61160.65560.63030.61340.5772

0.55300.5075

Similarity (frank5)

0.59990.64540.62930.60620.57300.55190.4640Similarity (frank6)

0.5782

0.63110.62110.59850.5674

0.54880.4511

Similarity (frank7)

0.56720.6232

0.61310.59380.56450.54320.4410Similarity (frank8)

0.55810.61020.60430.5886

0.55840.5376

0.42810.3735

Similarity (frank9)

0.54120.60240.59500.57810.55280.5310Similarity (frank10)

0.52030.5886

0.58390.5716

0.54850.52940.3570xgb

0.21620.13780.11050.09710.08950.07000.2308rf

0.70640.70990.65920.62130.56600.49690.5260mlp

0.44790.64610.66920.67020.65330.62800.4867lr

0.65330.63970.63490.63050.59390.55250.6311

Fig. 9 Relationship between n value in n-gram and evaluation values with some machine learning models in
1st level unit prediction

Fig. 10 Relationship between n value in n-gram and evaluation values with some machine learning models
in 2nd level unit prediction

Yamauchi et al. Smart Learning Environments(2023) 10:51

Page 24 of 30

Fig. 11 Relationship between n value in n-gram and evaluation values with MLP and aggregate function
methods in 1st level unit prediction

Fig. 12 Relationship between n value in n-gram and evaluation values with MLP and aggregate function
methods in 2nd level unit prediction

Table 10 Feature analysis in mono-gram and Random Forest(a) without omitting any words(b) omitting meaningless words

Mono-gramMono-gram (English)

Importance

Mono-gramMono-gram (English)

ImportanceI

I

0.027101ベクトル

Vector

0.012692

IIIIII

0.026868

数

Number

0.011921

IIII

0.025151

関数

Formula

0.011126

AA

0.018989

確率

Probability

0.009463

BB

0.016693

複素Complex0.008825

数

Number

0.012049

点

Point

0.008570

関数

Function

0.011044

式

Formula0.008534

解説

Solution

0.010463

よっTherefore

0.008256

ベクトル

Vector

0.009900

極限

Limit

0.008139

点

Point

0.009046

定積

Constant volume

0.007628

確率

Probability

0.008046

列

Column (or Sequence)

0.006980

複素Complex

0.007726

値

Value0.006849

式

Formula0.007594

次

Next

0.006669

列Column (or Sequence)

0.006998

求め

Find [the value]

0.006547

積分

Integral

0.006740

するDo

0.006483

三角

Triangle

0.006716

積分

Integral

0.006374

定積

Constant volume

0.006563

三角

Triangle

0.006342

極限

Limit

0.006439

がBe

0.006325

addition, a previous study documented good results when using the Bag-of-Words
method and Random Forest (Montoliu et al., 2015). This is also possibly a reason why
this model yielded the best performance.
Since Random Forest uses decision trees, it is easy to create accurate decision techniques for binary vectors. Therefore, we believe that classification using Random Forest
was able to accurately classify binary vectors with numerous dimensions. In addition,

Yamauchi et al. Smart Learning Environments(2023) 10:51

Fig. 13 The top of a sentence in a PDF. There is a character representing the part of organization in the firstline (in this example, ‘B’ surrounded by a red circle)

the fact that characters such as ‘I’, ‘II’, ‘III’, ‘A’, and ‘B’ existed as typical classification indices in Japanese mathematics and had a significant influence on classification, suggeststhat Random Forest with mono-grams produced the best prediction accuracy. This also
indicates that the organization characters can be useful when classifying the exercises: if
the PDF sentence contains such characters, as shown in Fig. 13, it can be easier to automatically classify it.
Selection of evaluation method in the educational context

Three indices, AL, FL, and FwL, were used in the experiment as evaluation indices. AL isdesired to be evaluated with a more reliable index, since in the present data set, there
are much more data that are true-negative than true-positive data and tend to rate themodel that is false for all data highly (Manning et al., 2008). In this case, the indicator
F-measure is often used for two-class classification, but there are two ways to obtain it
for multi-class classification. In the case of multiclass classification, there are two ways
to obtain the F-measure: FL and FwL.
FL returns the average of the Fl obtained for each class l , which is equal to the averageof the Fl obtained for all classes, even if the number of data in each class is not uniform.
Therefore, it is possible to treat all units equally even if the number of data in each unit
is not uniform. In other words, it is an effective indicator for labeling biased data sets.
For example, FL is useful when a teacher in a school setting selects three 1st level units
to create a test (ignoring the rest) and automatically assigns 2nd level units to exercises
within those units. FwL is the F-measure calculated from the sum of Precision Pl and
Recall Rl for each class l . This is a weighted index that takes into account the number of
each data set. Therefore, it can be said that the index accurately reflects the distributionof this data set. The index FwL is useful for labeling a uniform data set, i.e., the mathematical material studied in three years of Japanese high school at a time.
From the experimental results, we can say that the combination of mono-gram andRandom Forest, which has the largest FL, is effective when limiting the unit, and the
combination of bigram and Random Forest is effective for a uniform data set of three
years of high school. However, the accuracy rate is not much different when using thefeature n = 1 or n = 2 in Random Forest prediction.
Practical educational implications in this research

Automatic labeling can help reduce teacher workload (Tian et al., 2022) and develop
mathematics workmanship (Fishback & Schlicker, 1996) by introducing systems that
require labels (Vie et al., 2019; Wang et al., 2022). For further development of mathematics in Japan, a programming environment on supporting units on mathematics using
data (Kayama et al., 2022) and clarification of unit structure for knowledge association inlearning (Taniguchi & Itoh, 2023) have been proposed.

Page 25 of 30

Yamauchi et al. Smart Learning Environments(2023) 10:51

The labeling method in this study allows labels to be assigned to unlabeled mathematical instructional materials by learning the text of the labeled materials, even though they
are not formatted suitably for extracting text, such as handmade materials by mathematics teachers. These can facilitate unit learning based on the national standard curriculum
guidelines. For example, when students study on their own, the system can suggest different exercises than the ones they have solved, with the explanation that they are part ofthe same unit, which can promote student understanding. Therefore, it can be said that asystem using units is more easily utilized in the school contexts. In other words, the contribution of this study is that the automatic assignment of unit information to systems in
the educational field will expand the range of support without burdening teachers.
In addition, although we have chosen to use mathematical exercises as the subject
matter, we believe that such a method could be applied to other subjects as well, given
the uniform treatment of equations, terminology, and other information as textual information. To do so, we need clearly shared criteria and examples of exercises to which they
are pre-assigned (i.e., we can use the method proposed in Sect. 3 if the data set is in a
usable format).

Limitations and future research
In this study, we proposed an algorithm to solve the problem of classifying incomplete
texts of mathematical exercises into different leveled units. However, if more detailed
text were available, a context-aware classification algorithm is expected to produce better accuracy.
In this experiment, we limited ourselves to one topic of the same level to be assigned toeach mathematical exercise, but there also are mathematical exercises that span multiple
topics. In order to properly assign topics to such mathematical exercises, a system that
can assign multiple units using the algorithm verified in this study is needed. Multi-label
classification is also widely used in machine learning (Sorower, 2010; Tsoumakas & Katakis, 2007). Once such a system is completed, it would be possible to recommend similar
exercises using mathematical topics and analyze student learning based on topics.
This experiment showed that even if it is not possible to read mathematical expressions, numbers, or symbols, it is possible to classify with high accuracy using only the
textual information obtained. Once such a system is developed, it would be possible torecommend similar exercises based on mathematical topics and analyze student learning based on topics. Since the system would be able to assign common labels to different
teaching materials, it would be possible to develop a textbook recommendation systemthat assigns textbook subsections to exercises so that students can review them in the
textbook when they make a mistake on an exercise. The information collected using
these systems would then create a learning support environment that takes into accountthe degree of difficulty and understanding of the 1st level unit and 2nd level unit itself.
In addition, since the experiment was conducted independently of student learning, there are no results on the contribution to learner and teacher activities. It will
be necessary to verify the educational effects in future experiments by using the automatic labeling of the unit to recommend teaching materials or to analyze the behaviorof the learners, especially with predicting entire difficulty of the exercises and learners’

Page 26 of 30

Yamauchi et al. Smart Learning Environments(2023) 10:51

complete rate of them, or considering students reading comprehension. In addition tothis, as one study developed a recommendation system which uses student’s action as aparameter of the system (Takami et al., 2022), there is also room for combining following two approaches: topic-based model driven approach (i.e., the labeling the exercises)
and student’s behavior data driven approach (i.e., using the student’s achievement of theexercise into the system).

Conclusion
This paper proposes an algorithm that uses several techniques to correctly assign topics to the incomplete mathematical text obtained from PDF text. The extracted text
showed that all information on numbers, mathematical expressions, and symbols wasomitted when converted from PDF to text. Furthermore, we compared the prediction
accuracy of the two methods at the stage of predicting topics from the obtained vectors.
Two methods were used to compare their prediction accuracy: one using cosine similarity and the other using machine learning. We attempted to predict with all features andmodels and found that the best prediction accuracy was achieved by using mono-grams
as features and applying Random Forest (92.5% and 68.5% for 1st level unit and 2nd level
unit, respectively). We conclude that the reason for the higher accuracy was the ability
to find context-independent similarities even in incomplete sentences by using n-grams
to find matches in which the remaining words are used, and the existence of organization parts (‘I’, ‘II’, ‘III’, ‘A’, ‘B’) representing common national classifications for Japanesemathematical exercises. Given that PDFs are not necessarily assigned such national symbols, we conducted a similar experiment omitting them as stop words and found that theaccuracy dropped a little, but important mathematical knowledge elements appeared in
the key features, which are important for the classification of mathematical exercises.
The contribution in the research is the discovery that mono-grams, a simpler approachsimilar to traditional methods like n-grams or bag of words, outperformed state-of-theart methods in classifying incomplete texts, particularly in the context of Japanese mathematics exercises. These findings challenge previous research results and suggests thatthe choice of text analysis techniques may depend on the specific task or target domain.
Abbreviations
ACARA​	Australian curriculum, assessment and reporting authority
BERT	Bidirectional encoder representations from transformers
CCSS	Common core state standard
CNN	Convolutional neural network
ELMo	Embeddings from language models
GPT	Generative pretrained transformer
HTC	Hierarchical text classification
HTML	Hyper text markup language
ICT	Information and communication technology
K-12	Kindergarten through 12th gradeLR	Logistic regression
LSTM	Long short term memory
MEXT	Ministry of Education, Culture, Sports, and Technology
MLP	Multi layered perceptron
MOE	Ministry of Education
MSC	Mathematics subject classification
OCR	Optical character recognition
PDF	Portable document format
PISA	Programme for international student assessment
RF	Random forest

Page 27 of 30

Yamauchi et al. Smart Learning Environments(2023) 10:51

RNN	Recurrent neural network
SES	Socio-economic status
US	United States
VSM	Vector space model
WE-KE	Word embedding and knowledge extracting
XGB	EXtreme gradient boostingzbMATH	Zentralblatt MATH
Author contributions
RN, BF, and HO contributed to the research conceptualization and methodology. TY wrote the manuscript. RN, YD, KT, BF,
and HO provided comments to improve the manuscript. All authors read and approved the final manuscript.
Funding
This work was partly supported by JSPS Grant-in-Aid for Scientific Research (B) JP23H01001, JP22H03902, JP20H01722,
JSPS Grant-in-Aid for Scientific Research (Exploratory) JP21K19824, and NEDO JPNP20006.
Availability of data and materials
The data of this study is not open to the public due to participant privacy.

Declarations
Competing interests
The author declares no competing interests.
Received: 4 July 2023 Accepted: 9 October 2023

References
Abekawa, T., & Aizawa, A. (2016). SideNoter: Scholarly paper browsing system based on PDF restructuring and text
annotation. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: System
Demonstrations, 136–140.
Australian Curriculum, Assessment and Reporting Authority (ACARA). F-10 curriculum mathematics structure. Retrieved 01
September, 2023 from https://​www.​austr​alian​curri​culum.​edu.​au/f-​10-​curri​culum/​mathe​matics/​struc​ture/.
Bhartiya, D., Contractor, D., Biswas, S., Senjupta, B., & Mohania, M. (2016). Document segmentation for labeling with academic learning objectives. In Paper presented at the International Conference on Educational Data Mining, 282–287.
Breiman, L. (2001). Random forests. Machine Learning, 45, 5–32.
Cavnar, W. B., & Trenkle, J. M. (1994). N-gram-based text categorization. In Proceedings of SDAIR-94, 3rd annual symposium
on document analysis and information retrieval, 161175.
Chen, T., & Guestrin, C. (2016). XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 785–794.
Chen, T., He, T., Benesty, M., Khotilovich, V., Tang, Y., Cho, H., & Chen, K. (2015). Xgboost: Extreme gradient boosting. R Package Version, 1(4), 1–4.
Chow, J. C., & Ekholm, E. (2019). Language domains differentially predict mathematics performance in young children.
Early Childhood Research Quarterly, 46, 179–186.
Church, K. W. (2017). Word2Vec. Natural Language Engineering, 23(1), 155–162.
Contractor, D., Popat, K., Ikbal, S., Negi, S., Sengupta, B., & Mohania, M. K. (2015). Labeling educational content with academic learning standards. In Proceedings of the 2015 SIAM International Conference on Data Mining, pp. 136–144.
Cox, D. R. (1958). The regression analysis of binary sequences. Journal of the Royal Statistical Society: Series B (methodological), 20(2), 215–232.
Date, I., & Isozaki, H. (2015). Detection of mathematical formula regions in images of scientific papers by using deeplearning and OCR. IEICE Technical Report, 2015(4), 1–6. in Japanese.
Devlin, J., Chang, M., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language
understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1, 4171–4186.
Dharma, E. M., Gaol, F. L., Warnars, H. L. H. S., & Soewito, B. (2022). The accuracy comparison among word2vec, glove, andfasttext towards convolution neural network (CNN) text classification. Journal of Theoretical and Applied Information
Technology, 100(2), 349–359.
Ditchburn, G. (2012). A national Australian curriculum: In whose interests? Asia Pacific Journal of Education, 32(3), 259–269.
Dunne, E., & Hulek, K. (2020). Mathematics subject classification 2020. EMS Newsletter, 115, 5–6.
Fateman, R. J., Tokuyasu, T., Berman, B. P., & Mitchell, N. (1996). Optical character recognition and parsing of typeset mathematics. Journal of Visual Communication and Image Representation, 7(1), 2–15.
Fishback, P., & Schlicker, S. (1996). The impact of technology on mathematics education. Grand Valley Review, 14(1), 27.
Flanagan, B., Majumdar, R., Akçapınar, G., Wang, J., & Ogata, H. (2019). Knowledge map creation for modeling learning
behaviors in digital learning environments. In Companion Proceedings of the 9th International Conference on Learning
Analytics and Knowledge, 428–436.
Flanagan, B., & Ogata, H. (2018). Learning analytics platform in higher education in Japan. Knowledge Management &E-Learning: An International Journal, 10(4), 469–484.

Page 28 of 30

Yamauchi et al. Smart Learning Environments(2023) 10:51

Graovac, J. (2014). Text categorization using n-gram based language independent technique. Intelligent Data Analysis,
18(4), 677–695.
Graovac, J., Kovačević, J., & Pavlović-Lažetić, G. (2015). Language independent n-gram-based text categorization withweighting factors: A case study. Journal of Information and Data Management, 6(1), 4–17.
Graovac, J., Kovačević, J., & Pavlović-Lažetić, G. (2017). Hierarchical vs. flat n-gram-based text categorization: Can we dobetter? Computer Science and Information Systems, 14(1), 103–121.
Guo, Y., Silver, E. A., & Yang, Z. (2018). The latest characteristics of mathematics education reform of compulsory education
stage in China. American Journal of Educational Research, 6(9), 1312–1317.
Hussein, H. B. (2023). Global trends in mathematics education research. International Journal of Research in Educational
Sciences., 6(2), 309–319.
Ikeda, T. (2021). nagisa (0.2.7). https://​github.​com/​taishi-​i/​nagisa.
Joulin, A., Grave, E., Bojanowski, P., & Mikolov, T. (2017). Bag of tricks for efficient text classification. In Proceedings of the
15th Conference of the European Chapter of the Association for Computational Linguistics, 2, 427–431.
Kayama, M., Nagai, T., & Asuke, T. (2022). A proposal for a visual programming environment for “use of data” related units
in primary and secondary education. Journal of Japanese Society for Information and Systems in Education, 39(2),224–234. in Japanese.
Khan, A., Baharudin, B., Lee, L. H., & Khan, K. (2010). A review of machine learning algorithms for text-documents classification. Journal of Advances in Information Technology, 1(1), 4–20.
Khosravi, H., & Cooper, K. (2018). Topic dependency models: Graph-based visual analytics for communicating assessment
data. Journal of Learning Analytics, 5(3), 136–153.
Kobayashi, T. (2021). T-vMF similarity for regularizing intra-class feature distribution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 6616–6625.
Kobayashi, Y., Tanaka, S., & Tomiura, Y. (2012). Pattern recognition of english scientific papers using n-grams. Information
Fundamentals and Access Technologies, 12(1), 1–6.
Kühnemund, A. (2016). The role of applications within the reviewing service zbMATH. PAMM, 16(1), 961–962.
Li, B., Liu, T., Du, X., Zhang, D., & Zhao, Z. (2016). Learning document embeddings by predicting n-grams for sentimentclassification of long movie reviews. In The Eleventh International Conference on Learning Representations.
Liu, G., & Guo, J. (2019). Bidirectional LSTM with attention mechanism and convolutional layer for text classification.
Neurocomputing, 337, 325–338.
Manning, C. D., Raghavan, P., & Schütze, H. (2008). Introduction to information retrieval. Cambridge University Press.
Mansur, M. (2006). Analysis of n-gram based text categorization for Bangla in a newspaper corpus (Doctoral dissertation,
BRAC University).
Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., & Dean, J. (2013). Distributed representations of words and phrases andtheir compositionality. In Proceedings of Advances in Neural Information Processing Systems 26.
Ministry of Arts and Sciences (MEXT). (2009). 高等学校学習指導要領(平成21年3月告示). [High school curriculum guidelines (announced in March 2009)]. https://​erid.​nier.​go.​jp/​files/​COFS/​h20h/​index.​htm. in Japanese.
Ministry of Arts and Sciences (MEXT). (2018). 数学編・理数編 高等学校学習指導要領(平成30年告示). [In mathematics
and science, high school curriculum guidelines (announced in 2018)]. https://​www.​mext.​go.​jp/​conte​nt/​20230​217-​mxt_​
kyoik​u02-​10000​2620_​05.​pdf. in Japanese.
Ministry of Arts and Sciences (MEXT). (2021). 高等学校用教科書目録(令和4年度使用) [For higher education textbook
catalog (for fiscal year 2021)]. https://​www.​mext.​go.​jp/​conte​nt/​20210​604-​mxt_​kyoka​syo02-​00001​4470_4.​pdf. inJapanese.
Ministry of Education of the People’s Republic of China (MOE). (2012). Mathematics curriculum standards for compulsory
education (2011th ed.). Beijing Normal University Press.
Montoliu, R., Martín-Félez, R., Torres-Sospedra, J., & Martínez-Usó, A. (2015). Team activity recognition in association football using a bag-of-words-based method. Human Movement Science, 41, 165–178.
Ohnishi, T. (2011). Task-based learning in high school mathematics. Japan Society for Science Education Research Report,
26(8), 45–48. in Japanese.
Palmer, J. A. (2021). pdftotext (2.2.2). https://​github.​com/​jalan/​pdfto​text.
Peters, M., Neumann, M., Zettlemoyer, L., & Yih, W. (2018). Dissecting contextual word embeddings: Architecture and representation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 1499–1509.
Porter, A., McMaken, J., Hwang, J., & Yang, R. (2011). Common core standards: The new US intended curriculum. Educational Researcher, 40(3), 103–116.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are unsupervised multitask
learners. OpenAI Blog, 1(8), 9.
Raff, E., Richard, Z., Cox, R., Sylvester, J., Yacci, P., Ward, R., Tracy, A., Mclean, M., & Nicholas, C. (2018). An investigation of byte
n-gram features for malware classification. Journal of Computer Virology and Hacking Techniques, 14, 1–20.
Ramakrishnan, C., Patnia, A., Hovy, E., & Burns, G. A. (2012). Layout-aware text extraction from full-text PDF of scientificarticles. Source Code for Biology and Medicine, 7(1), 1–10.
Ritter, B. J. (2009). Update on the common core state standards initiative. National Governors Association.
Rosenblatt, F. (1958). The perceptron: A probabilistic model for information storage and organization in the brain. Psychological Review, 65(6), 386.
Schubotz, M., Scharpf, P., Teschke, O., Kühnemund, A., Breitinger, C., & Gipp, B. (2020). Automsc: Automatic assignment ofmathematics subject classification labels. In International Conference on Intelligent Computer Mathematics, 237–250.
Shen, J. T., Yamashita, M., Prihar, E., Heffernan, N., Wu, X., McGrew, S., & Lee, D. (2021). Classifying math knowledge components via task-adaptive pre-trained BERT. In International Conference on Artificial Intelligence in Education, 408–419.
Shintani, R. (2014). The development process and contents of the common core state standards: Based on a comparative study with the Japanese Course of Study for lower secondary school. Journal of Japan Association of American
Educational Studies, 25, 15–27. in Japanese.
Silla, C. N., Jr., & Freitas, A. A. (2011). A survey of hierarchical classification across different application domains. Data Mining and Knowledge Discovery, 22(1–2), 31–72.

Page 29 of 30

Yamauchi et al. Smart Learning Environments(2023) 10:51

Smith, R. (2007). An overview of the tesseract OCR engine. In Ninth International Conference on Document Analysis andRecognition, 2, 629–633
Sorower, M. S. (2010). A literature survey on algorithms for multi-label learning. Oregon State University, Corvallis, 18(1), 25.
Sosnovsky, S., & Brusilovsky, P. (2015). Evaluation of topic-based adaptation and student modeling in quizguide. User
Modeling and User-Adapted Interaction, 25, 371–424.
Spelke, E. S., & Tsivkin, S. (2001). Language and number: A bilingual training study. Cognition, 78(1), 45–88.
Suen, C. Y. (1979). N-gram statistics for natural language understanding and text processing. IEEE Transactions on PatternAnalysis and Machine Intelligence, 2, 164–172.
Takami, K., Dai, Y., Flanagan, B., & Hiroaki Ogata. (2022). Educational explainable recommender usage and its effectiveness in high school summer vacation assignment. In 12th International Learning Analytics and Knowledge Conference,
458–464.
Taniguchi, Y., & Itoh, T. (2023). Unit association method with symbolization in high school mathematics textbook. Journalof Information Processing, 64(1), 256–269. in Japanese.
Tian, Z., Flanagan, B., Dai, Y., & Ogata, H. (2022). Automated matching of exercises with knowledge components. In 30th
International Conference on Computers in Education Conference Proceedings, 24–32.
Tsoumakas, G., & Katakis, I. (2007). Multi-label classification: An overview. International Journal of Data Warehousing andMining, 3(3), 1–13.
Vie, J. J., & Kashima, H. (2019). Knowledge tracing machines: Factorization machines for knowledge tracing. Proceedings of
the AAAI Conference on Artificial Intelligence, 33(1), 750–757.
Vovides, Y., Sanchez-Alonso, S., Mitropoulou, V., & Nickmans, G. (2007). The use of e-learning course management systems
to support learning strategies and to improve self-regulated learning. Educational Research Review, 2(1), 64–74.
Wang, F., King, R. B., & Leung, S. O. (2023). Why do east Asian students do so well in mathematics? A machine learning
study. International Journal of Science and Mathematics Education, 21(3), 691–711.
Wang, J., Minematsu, T., Okubo, F., & Shimada, A. (2022). Topic-wise representation of learning activities for new learning
pattern analysis. In 30th International Conference on Computers in Education Conference Proceedings, 1, 268–278.
zbMATH OPEN, The first resource for mathematics. Mathematics subject classification—MSC2020. Retrieved 03 September,
2023 from https://​zbmath.​org/​class​ifica​tion/.
Zheng, E., Moh, M., & Moh, T. S. (2017). Music genre classification: A n-gram based musicological approach. In 2017 IEEE
7th International Advance Computing Conference, 671–677.

Publisher’s NoteSpringer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Page 30 of 30