Prob2Vec: Mathematical Semantic Embedding for Problem Retrieval in Adaptive Tutoring

1 INTRODUCTION . The traditional teaching methods that are widely used at universities for science , technology , engineering , and mathematic ( STEM ) courses do not take different abilities of learners into account . Instead , they provide learners with a fixed set of textbooks and homework problems . This ignorance of learners ’ prior background knowledge , pace of learning , various preferences , and learning goals in current education system can cause tremendous pain and discouragement for those who do not keep pace with this inefficient system Chen ( 2008 ) ; Brusilovsky & Vassileva ( 2003 ) ; Chen et al . ( 2005 ) ; Hübscher ( 2000 ) ; Weber & Specht ( 1997 ) . Hence , e-learning methods are given considerable attention in an effort to personalize the learning process by providing learners with optimal and adaptive curriculum sequences . Over the years , many web-based tools have emerged to adaptively recommend problems to learners based on courseware difficulty . These tools tune the difficulty level of the recommended problems for learners and push them to learn by gradually increasing the difficulty level of recommended problems on a specific concept . The downside of such methods is that they do not take the concept continuity and mixture of concepts into account , but focus on the difficulty level of single concepts . Note that a learner who knows every individual concept does not necessarily have the ability to bring all of them together for solving a realistic problem on a mixture of concepts . As a result , the recommender system needs to know similarity/dissimilarity of problems with mixture of concepts to respond to learners ’ performance more effectively as described in the next paragraph , which is something that is missing in the literature and needs more attention . Since it is difficult for humans to determine a similarity score consistent across a large enough training set , it is not feasible to simply apply supervised methods to learn a similarity score for problems . In order to take difficulty , continuity , and mixture of concepts into account for similarity score used in a personalized problem recommender system in an adaptive practice , we propose to use a proper numerical representation of problems on mixture of concepts equipped with a similarity measure . By virtue of vector representations for a set of problems on both single and mixture of concepts ( problem embedding ) that capture similarity of problems , learners ’ performance on a problem can be projected onto other problems . As we see in this paper , creating a proper problem representation that captures mathematical similarity of problems is a challenging task , where baseline text representation methods and their refined versions fail to work . Although the state-of-the-art methods for phrase/sentence/paragraph representation are doing a great job for general purposes , their shortcoming in our application is that they take lexical and semantic similarity of words into account , which is totally invalid when dealing with text related to math or any other special topic . The words or even subject-related keywords of problems are not completely informative and can not contribute to embedding of math problems on their own ; as a result , the similarity of two problems is not highly correlated with the wording of the problems . Hence , baseline methods perform poorly on the problem similarity detection test in problem recommender application . We find that instead of words or even subject-related keywords , conceptual ideas behind the problems determine their identity . The conceptual particles ( concepts ) of problems are mostly not directly mentioned in problem wording , but there can be footprints of them in problems . Since problem wording does not capture the similarity of problems , we propose an alternative hierarchical approach called Prob2Vec consisting of an abstraction and an embedding step . The abstraction step projects a problem to a set of concepts . The idea is that there exists a concept space with a reasonable dimension N , with N ranging from tens to a hundred , that can represent a much larger variety of problems of order O ( 2N ) . Each variety can be sparsely inhabited , with some concept combination having only one problem . This is because making problems is itself a creative process : The more innovative a problem is , the less likely it has exactly the same concept combination as another problem . The explicit representation of problems using concepts also enables state-dependent similarity computation , which we will explore in future work . The embedding step constructs a vector representation of the problems based on concept cooccurrence . Like sentence embedding , not only does it capture the common concepts between problems , but also the continuity among concepts . The proposed Prob2Vec algorithm achieves 96.88 % accuracy on a problem similarity test , where human experts are asked to label the relative similarity among each triplet of problems . In contrast , the best of the existing methods , which directly applies sentence embedding , achieves 75 % accuracy . It is surprising that Prob2Vec is able to distinguish very fine-grained differences among problems , as the problems in some triplets are highly similar to each other , and only humans with extensive training in the subject are able to identify their relative order of similarity . The problem embedding obtained from Prob2Vec is being used in the recommender system of an e-learning tool for an undergraduate probability course for four semesters with successful results on hundreds of students , specially benefiting minorities who tend to be more isolated in the current education system . In addition , the sub-problem of concept labeling in the abstraction step is interesting in its own right . It is a multi-label problem suffering from dimensionality explosion , as there can be as many as 2N problem types . This results in two challenges : First , there are very few problems for some types , hence a direct classification on 2N classes suffers from a severe lack of data . Second , per-concept classification suffers from imbalance of training samples and needs a very small per-concept false positive in order to achieve a reasonable per-problem false positive . We propose pre-training of the neural network with negative samples ( negative pre-training ) that beats a similar idea to oneshot learning Fei-Fei et al . ( 2006 ) , where the neural network is pre-trained on classification of other concepts to have a warm start on classification of the concept of interest ( transfer learning ) . 1.1 RELATED WORK . Embedding applications : the success of simple and low-cost word embedding technique using the well-known neural network ( NN ) based word embedding method , Word2Vec by Mikolov et al . ( 2013b ; a ) , compared to expensive natural language processing methods , has motivated researchers to use embedding methods for many other areas . As examples , Doc2Vec Lau & Baldwin ( 2016 ) , Paper2Vec Tian & Zhuo ( 2017 ) ; Ganguly & Pudi ( 2017 ) , Gene2Vec Cox , Graph2Vec Narayanan et al . ( 2017 ) , Like2Vec , Follower2Vec and many more share the same techniques originally proposed for word embedding with modifications based on their domains , e.g . see Ganguly & Pudi ( 2017 ) ; Grover & Leskovec ( 2016 ) ; Barkan & Koenigstein ( 2016 ) ; Narayanan et al . ( 2016 ) ; Dhingra et al . ( 2016 ) ; Ma et al . ( 2016 ) ; Ring et al . ( 2017 ) ; Dong et al . ( 2017 ) ; Ribeiro et al . ( 2017 ) ; Chen et al . ( 2017 ) ; Chung et al . ( 2016 ) ; Niu et al . ( 2015 ) ; Vasile et al . ( 2016 ) ; Lin et al . ( 2017 ) ; Ristoski & Paulheim ( 2016 ) ; Melamud et al . ( 2016 ) ; Shi et al . ( 2018 ) ; Liu et al . ( 2018 ) . In this work , we propose a new application for problem embedding for personalized problem recommendation . Word/Phrase/Sentence/Paragraph embedding : the prior works on word embedding include learning a distributed representation for words by Bengio et al . ( 2003 ) , multi-task training of a convolutional neural network using weight-sharing by Collobert & Weston ( 2008 ) , the continuous Skip-gram model by Mikolov et al . ( 2013b ) , and the low rank representation of word-word co-occurrence matrix by Deerwester et al . ( 1990 ) ; Pennington et al . ( 2014 ) . Previous works on phrase/sentence/paragraph embedding include vector composition that is operationalized in terms of additive and multiplicative functions by Mitchell & Lapata ( 2008 ; 2010 ) ; Blacoe & Lapata ( 2012 ) , uniform averaging for short phrases by Mikolov et al . ( 2013b ) , supervised and unsupervised recursive autoencoder defined on syntactic trees by Socher et al . ( 2011 ; 2014 ) , training an encoderdecoder model to reconstruct surrounding sentences of an encoded passage by Kiros et al . ( 2015 ) , modeling sentences by long short-term memory ( LSTM ) neural network by Tai et al . ( 2015 ) and convolutional neural networks by Blunsom et al . ( 2014 ) , and the weighted average of the words in addition to a modification using PCA/SVD by Arora et al . ( 2016 ) . The simple-tough-to-beat method proposed by Arora et al . ( 2016 ) beats all the previous methods and is the baseline in text embedding . The rest of the paper is outlined as follows . The description of the data set for which we do problem embedding in addition to our proposed Prob2Vec method for problem embedding are presented in section 2 . Negative pre-training for NN-based concept extraction is proposed in section 3 . Section 4 describes the setup for similarity detection test for problem embedding , evaluates the performance of our proposed Prob2Vec method versus baselines , and presents the results on the negative pre-training method . Section 5 concludes the paper with a discussion on opportunities for future work . 2 PROB2VEC : CONCEPT-BASED PROBLEM EMBEDDING . Consider a set of M problems P = { P1 , P2 , · · · , PM } for an undergraduate probability course , where each problem can be on a single or mixture of concepts among the set of all N concepts C = { C1 , C2 , · · · , CN } . Note that these concepts are different from keywords of problem wordings and are not originally included in problems , but labeling problems with concepts is a contribution of this work that is proposed for achieving a proper problem representation . Instead , problems are made of words from the setW = { W1 , W2 , · · · , WL } ; i.e . Pi =Wi for 1 ≤ i ≤M , whereWi ⊂ W |Wi| . In the following subsection , we propose the Prob2Vec method for problem embedding that uses an automated rule-based concept extractor , which relieves reliance on human labeling and annotation for problem concepts . 2.1 PROB2VEC PROBLEM EMBEDDING . As shown in section 4 , using the set of wordsWi|Mi=1 or even a subset of keywords to represent problems , text embedding baselines fail to achieve high accuracy in similarity detection task for triplets of problems . In the keyword-based approach , all redundant words of problems are ignored and the subject-related and informative words such as binomial , random variable , etc. , are kept . However , since the conceptual ideas behind problems are not necessarily mapped into normal and mathematical words used in problems , even the keyword-based approach fails to work well in the similarity detection task that is explained in section 4 . Alternatively , we propose a hierarchical method consisting of abstraction and embedding steps that generates a precise embedding for problems that is completely automated . The block diagram of the proposed Prob2Vec method is depicted in figure 1 . ( i ) Abstraction step : similarity among mathematical problems is not captured by the wording of problems ; instead , it is determined by the abstraction of the problems . Learners who have difficulty solving mathematical problems mostly lack the ability to do abstraction and relate problems with appropriate concepts . Instead , they try to remember procedure-based rules to fit problems in them and use their memory to solve them , which does not apply to solving hard problems on mixture of concepts . We observe the same pattern in problem representation ; i.e . problem statements do not necessarily determine their identity , instead abstraction of problems by mapping them into representative concepts moves problem embedding from lexical similarity to conceptual similarity . The concepts of a problem are not mostly mentioned directly in its text , but there can be footmarks of concepts in problems . A professor and two experienced teaching assistants are asked to formulate rule-based mappings from footmarks to concepts for automation of concept extraction . As an example , the rule for labeling a problem with concept “ nchoosek ” is \\\\choose|\\\\binom|\\\\frac\ { \s * \w+\ ! \s * \ } \ { \s * \w+\ ! \s * \\\\times\s * \w+\ ! \s * \ } . By applying the rule-based concept extractor to problems , we have a new representation for problems in concept space instead of word space ; i.e . Pi ≡ Ci for 1 ≤ i ≤M , where Ci ⊂ C. ( ii ) Embedding step : a method similar to Skip-gram in Word2Vec is used for concept embedding . The high-level insight of Skip-gram is that a neural network with a single hidden layer is trained , where its output relates to how likely it is to have each concept co-occurred in a problem with the input concept . As an example , if concept “ law-of-total-probability ” of a problem is input of the neural network , we expect the neural network to state it more likely to have concept “ conditional-probability ” in the problem than for unrelated concepts like “ Poisson-process ” . However , the neural network is not used for this task , but the goal is to use weights of the hidden layer for embedding . Recall the set of all concepts as { C1 , C2 , · · · , CN } , where a problem is typically labeled with a few of them . Consider onehot coding from concepts to real-valued vectors of size N that are used for training of the neural network , where the element corresponding to a concept is one and all other elements are zero . We consider 10 neurons in hidden layer with no activation functions ( so the embedded concept vectors have 10 features ) and N neurons in the output that form a softmax regression classifier . In order to clarify on input-output pair of neural network , assume a problem that has a set of concepts { C1 , C2 , C5 } . The neural network is trained on all pairs ( C1 , C2 ) , ( C1 , C5 ) , ( C2 , C1 ) , ( C2 , C5 ) , ( C5 , C1 ) , and ( C5 , C2 ) , where the one-hot code of the first element of a pair is the input and the one-hot code of the second element of a pair is the output of the neural network in the training phase . Hence , the neural network is trained over ∑M i=1 |Ci| × ( |Ci| − 1 ) number of training data . This way , the neural network learns the statistic from the number of times that a pair is fed into it ( the neural network is probably fed with more training pairs of ( “ law-of-total-probability ” , “ conditional-probability ” ) than the pair ( “ law-of-total-probability ” , “ Poisson-process ” ) ) . Note that during training phase , input and output are one-hot vectors representing the input and output concepts , but after training when using the neural network , given a one-hot input vector , the output is a probability distribution on the set of all concepts . Finally , since input concepts are coded as one-hot codes , rows of hidden layer weight matrix , which is of size N by 10 , are concept vectors ( concept embedding ) which we are really after . Denoting embedding of concept c ∈ C by E ( c ) , problem embedding denoted by Ei for problem Pi is obtained as follows : Ei = 1 |Ci| ∑ c∈Ci 1 fc · E ( c ) , ( 1 ) where fc is frequency of concept c ∈ C in our data set of M = 635 problems . Concept embedding is scaled by its corresponding frequency to penalize concepts with high frequency that are not as informative as low frequency ones . For example , concept “ pmf ” is less informative than concept “ ML-parameter-estimation ” . Given problem embedding , similarity between two problems is defined as cosine of the angle between corresponding vector representations . Remark . We choose rule-based concept extractor for the abstraction step over any supervised/unsupervised classification methods for concept extraction because of two main reasons . First , there is a limited number of problems for most concepts as few as a couple of problems which makes any supervised classification method inapplicable due to lack of training data . Second , there are N concepts , so potentially there can be 2N −1 categories of problems which makes classification challenging for any supervised or unsupervised methods . Consider the maximum number of concepts in a problem to be k , then there are on the order of O ( Nk ) categories of problems . Even if we consider possessing acceptable number of problems for each of the O ( Nk ) categories , false positive needs to be on the order of O ( 1 Nk ) so that the number of false positives for each category be in the order of O ( 1 ) . Given that there are N = 96 concepts , utilizing a supervised or unsupervised approach to achieve such a low false positive is not feasible . Exploiting the rule-based classifier for problem labeling though , we succeed to achieve as low as average 0.98 % false positive and average 9.17 % false negative for all concepts , where problem concepts annotated by experts are considered to be ground truth . Although 100.00 % accuracy is observed in similarity detection test when utilizing problem concepts annotated by experts , those concepts are not necessarily the global optimum ground truth or the only one . Thinking of increasing accuracy in similarity detection task as an optimization problem , there is not necessarily a unique global optimum for problem concepts that lead to good performance . Hence , not having a very low false negative for rule-based does not necessarily mean that such labels are not close to a local/global optimum problem concepts . In fact , rule-based extracted concepts achieve a high 96.88 % accuracy on a similarity test as is mentioned in section 4 .

Summary: This paper proposes a method for mathematical problem embedding, which firstly decomposes problems into concepts by an abstraction step and then trains a skip-gram model to learn concept embedding. A problem can be represented as the average concept (corresponding to those in the problem) embeddings. To handle the imbalanced dataset, a negative pre-training method is proposed to decrease false and false positives. Experimental results show that the proposed method works much better than baselines in similar problem detection, on an undergraduate probability data set.