# Pre-Training Multi-Concept Question Embeddings for Knowledge Tracing
## 1 Introduction  
In intelligent educational systems, knowledge tracing is a crucial task aimed at modeling students' proficiency levels based on their historical answering records and assessing their current knowledge state [1, 2]. The task is typically evaluated by the ability to predict whether a student will answer the next question correctly. To address this problem, numerous methods have been developed. A key element in the definition of the knowledge tracing task is the embedding of the questions themselves during student interactions. Most existing methods do not focus on the embedding of questions but instead jointly train the representation of questions within the framework of a predictive model using supervised methods. However, due to the potential insufficiency of interaction data for some questions or concepts, the issue of data sparsity arises, limiting the expressiveness of embeddings obtained through supervised methods. As illustrated in Figure 1, knowledge tracing involves analyzing students' mastery of different concepts at each time step based on recorded interactions with a sequence of questions, which include information about thequestions, the concepts covered, and the correctness of the responses. Typically, answering correctly indicates an improvement in the student's mastery of the corresponding concepts, as shown with \(q_{4}\), where the student's proficiency in concept \(c_{2}\) increases. Conversely, an incorrect response, such as with \(q_{3}\), suggests a decline in proficiency for concepts \(c_{1}\) and \(c_{3}\). The objective of this study is to construct question embeddings from a multi-concept perspective, allowing for a more accurate reflection of the diverse concepts covered by a question.  
Current pre-training methods for question embeddings do not consider the textual information of concepts or questions, nor do they account for the possibility of a question encompassing multiple concepts.  
Based on this analysis and consideration, we propose a pre-training model that integrates textual embeddings and structural information to uniformly model the relationships between questions and concepts, providing high-quality embeddings for knowledge tracing and its downstream tasks. Specifically, we initially employ large language models to obtain textual embeddings of concepts. Then, we construct a question-concept graph from the interaction history and use graph convolutional methods to update the embeddings of questions and concepts, subsequently reconstructing the graph and optimizing embeddings through graph loss calculation. The experimental results on two real-world datasets demonstrate that compared to traditional supervised question embedding methods in knowledge tracing, our pre-trained embeddings can enhance model performance and improve interpretability.  
Our contributions are as follows:  
* We developed a method to obtain both question embeddings and question-concept weight matrices, facilitating downstream tasks such as knowledge tracing and question recommendation. This approach transitions from traditional single-concept to multi-concept modeling.
* By leveraging the relatively fixed nature of concept texts, we utilized methods from other text embedding models to represent concepts, enriching the structured information of the question-knowledge bipartite graph with additional textual semantic information.
* We designed a difficulty analyzer that assesses question difficulty based on its embeddings and auxiliary information, which can be utilized for question evaluation and incorporated into downstream tasks [3, 4, 5].