Prob2Vec: Mathematical Semantic Embedding for Problem Retrieval in Adaptive Tutoring

1 INTRODUCTION . The traditional teaching methods that are widely used at universities for science , technology , engineering , and mathematic ( STEM ) courses do not take different abilities of learners into account . Instead , they provide learners with a fixed set of textbooks and homework problems . This ignorance of learners ’ prior background knowledge , pace of learning , various preferences , and learning goals in current education system can cause tremendous pain and discouragement for those who do not keep pace with this inefficient system Chen ( 2008 ) ; Brusilovsky & Vassileva ( 2003 ) ; Chen et al . ( 2005 ) ; Hübscher ( 2000 ) ; Weber & Specht ( 1997 ) . Hence , e-learning methods are given considerable attention in an effort to personalize the learning process by providing learners with optimal and adaptive curriculum sequences . Over the years , many web-based tools have emerged to adaptively recommend problems to learners based on courseware difficulty . These tools tune the difficulty level of the recommended problems for learners and push them to learn by gradually increasing the difficulty level of recommended problems on a specific concept . The downside of such methods is that they do not take the concept continuity and mixture of concepts into account , but focus on the difficulty level of single concepts . Note that a learner who knows every individual concept does not necessarily have the ability to bring all of them together for solving a realistic problem on a mixture of concepts . As a result , the recommender system needs to know similarity/dissimilarity of problems with mixture of concepts to respond to learners ’ performance more effectively as described in the next paragraph , which is something that is missing in the literature and needs more attention . Since it is difficult for humans to determine a similarity score consistent across a large enough training set , it is not feasible to simply apply supervised methods to learn a similarity score for problems . In order to take difficulty , continuity , and mixture of concepts into account for similarity score used in a personalized problem recommender system in an adaptive practice , we propose to use a proper numerical representation of problems on mixture of concepts equipped with a similarity measure . By virtue of vector representations for a set of problems on both single and mixture of concepts ( problem embedding ) that capture similarity of problems , learners ’ performance on a problem can be projected onto other problems . As we see in this paper , creating a proper problem representation that captures mathematical similarity of problems is a challenging task , where baseline text representation methods and their refined versions fail to work . Although the state-of-the-art methods for phrase/sentence/paragraph representation are doing a great job for general purposes , their shortcoming in our application is that they take lexical and semantic similarity of words into account , which is totally invalid when dealing with text related to math or any other special topic . The words or even subject-related keywords of problems are not completely informative and can not contribute to embedding of math problems on their own ; as a result , the similarity of two problems is not highly correlated with the wording of the problems . Hence , baseline methods perform poorly on the problem similarity detection test in problem recommender application . We find that instead of words or even subject-related keywords , conceptual ideas behind the problems determine their identity . The conceptual particles ( concepts ) of problems are mostly not directly mentioned in problem wording , but there can be footprints of them in problems . Since problem wording does not capture the similarity of problems , we propose an alternative hierarchical approach called Prob2Vec consisting of an abstraction and an embedding step . The abstraction step projects a problem to a set of concepts . The idea is that there exists a concept space with a reasonable dimension N , with N ranging from tens to a hundred , that can represent a much larger variety of problems of order O ( 2N ) . Each variety can be sparsely inhabited , with some concept combination having only one problem . This is because making problems is itself a creative process : The more innovative a problem is , the less likely it has exactly the same concept combination as another problem . The explicit representation of problems using concepts also enables state-dependent similarity computation , which we will explore in future work . The embedding step constructs a vector representation of the problems based on concept cooccurrence . Like sentence embedding , not only does it capture the common concepts between problems , but also the continuity among concepts . The proposed Prob2Vec algorithm achieves 96.88 % accuracy on a problem similarity test , where human experts are asked to label the relative similarity among each triplet of problems . In contrast , the best of the existing methods , which directly applies sentence embedding , achieves 75 % accuracy . It is surprising that Prob2Vec is able to distinguish very fine-grained differences among problems , as the problems in some triplets are highly similar to each other , and only humans with extensive training in the subject are able to identify their relative order of similarity . The problem embedding obtained from Prob2Vec is being used in the recommender system of an e-learning tool for an undergraduate probability course for four semesters with successful results on hundreds of students , specially benefiting minorities who tend to be more isolated in the current education system . In addition , the sub-problem of concept labeling in the abstraction step is interesting in its own right . It is a multi-label problem suffering from dimensionality explosion , as there can be as many as 2N problem types . This results in two challenges : First , there are very few problems for some types , hence a direct classification on 2N classes suffers from a severe lack of data . Second , per-concept classification suffers from imbalance of training samples and needs a very small per-concept false positive in order to achieve a reasonable per-problem false positive . We propose pre-training of the neural network with negative samples ( negative pre-training ) that beats a similar idea to oneshot learning Fei-Fei et al . ( 2006 ) , where the neural network is pre-trained on classification of other concepts to have a warm start on classification of the concept of interest ( transfer learning ) . 1.1 RELATED WORK . Embedding applications : the success of simple and low-cost word embedding technique using the well-known neural network ( NN ) based word embedding method , Word2Vec by Mikolov et al . ( 2013b ; a ) , compared to expensive natural language processing methods , has motivated researchers to use embedding methods for many other areas . As examples , Doc2Vec Lau & Baldwin ( 2016 ) , Paper2Vec Tian & Zhuo ( 2017 ) ; Ganguly & Pudi ( 2017 ) , Gene2Vec Cox , Graph2Vec Narayanan et al . ( 2017 ) , Like2Vec , Follower2Vec and many more share the same techniques originally proposed for word embedding with modifications based on their domains , e.g . see Ganguly & Pudi ( 2017 ) ; Grover & Leskovec ( 2016 ) ; Barkan & Koenigstein ( 2016 ) ; Narayanan et al . ( 2016 ) ; Dhingra et al . ( 2016 ) ; Ma et al . ( 2016 ) ; Ring et al . ( 2017 ) ; Dong et al . ( 2017 ) ; Ribeiro et al . ( 2017 ) ; Chen et al . ( 2017 ) ; Chung et al . ( 2016 ) ; Niu et al . ( 2015 ) ; Vasile et al . ( 2016 ) ; Lin et al . ( 2017 ) ; Ristoski & Paulheim ( 2016 ) ; Melamud et al . ( 2016 ) ; Shi et al . ( 2018 ) ; Liu et al . ( 2018 ) . In this work , we propose a new application for problem embedding for personalized problem recommendation . Word/Phrase/Sentence/Paragraph embedding : the prior works on word embedding include learning a distributed representation for words by Bengio et al . ( 2003 ) , multi-task training of a convolutional neural network using weight-sharing by Collobert & Weston ( 2008 ) , the continuous Skip-gram model by Mikolov et al . ( 2013b ) , and the low rank representation of word-word co-occurrence matrix by Deerwester et al . ( 1990 ) ; Pennington et al . ( 2014 ) . Previous works on phrase/sentence/paragraph embedding include vector composition that is operationalized in terms of additive and multiplicative functions by Mitchell & Lapata ( 2008 ; 2010 ) ; Blacoe & Lapata ( 2012 ) , uniform averaging for short phrases by Mikolov et al . ( 2013b ) , supervised and unsupervised recursive autoencoder defined on syntactic trees by Socher et al . ( 2011 ; 2014 ) , training an encoderdecoder model to reconstruct surrounding sentences of an encoded passage by Kiros et al . ( 2015 ) , modeling sentences by long short-term memory ( LSTM ) neural network by Tai et al . ( 2015 ) and convolutional neural networks by Blunsom et al . ( 2014 ) , and the weighted average of the words in addition to a modification using PCA/SVD by Arora et al . ( 2016 ) . The simple-tough-to-beat method proposed by Arora et al . ( 2016 ) beats all the previous methods and is the baseline in text embedding . The rest of the paper is outlined as follows . The description of the data set for which we do problem embedding in addition to our proposed Prob2Vec method for problem embedding are presented in section 2 . Negative pre-training for NN-based concept extraction is proposed in section 3 . Section 4 describes the setup for similarity detection test for problem embedding , evaluates the performance of our proposed Prob2Vec method versus baselines , and presents the results on the negative pre-training method . Section 5 concludes the paper with a discussion on opportunities for future work . 2 PROB2VEC : CONCEPT-BASED PROBLEM EMBEDDING . Consider a set of M problems P = { P1 , P2 , · · · , PM } for an undergraduate probability course , where each problem can be on a single or mixture of concepts among the set of all N concepts C = { C1 , C2 , · · · , CN } . Note that these concepts are different from keywords of problem wordings and are not originally included in problems , but labeling problems with concepts is a contribution of this work that is proposed for achieving a proper problem representation . Instead , problems are made of words from the setW = { W1 , W2 , · · · , WL } ; i.e . Pi =Wi for 1 ≤ i ≤M , whereWi ⊂ W |Wi| . In the following subsection , we propose the Prob2Vec method for problem embedding that uses an automated rule-based concept extractor , which relieves reliance on human labeling and annotation for problem concepts . 2.1 PROB2VEC PROBLEM EMBEDDING . As shown in section 4 , using the set of wordsWi|Mi=1 or even a subset of keywords to represent problems , text embedding baselines fail to achieve high accuracy in similarity detection task for triplets of problems . In the keyword-based approach , all redundant words of problems are ignored and the subject-related and informative words such as binomial , random variable , etc. , are kept . However , since the conceptual ideas behind problems are not necessarily mapped into normal and mathematical words used in problems , even the keyword-based approach fails to work well in the similarity detection task that is explained in section 4 . Alternatively , we propose a hierarchical method consisting of abstraction and embedding steps that generates a precise embedding for problems that is completely automated . The block diagram of the proposed Prob2Vec method is depicted in figure 1 . ( i ) Abstraction step : similarity among mathematical problems is not captured by the wording of problems ; instead , it is determined by the abstraction of the problems . Learners who have difficulty solving mathematical problems mostly lack the ability to do abstraction and relate problems with appropriate concepts . Instead , they try to remember procedure-based rules to fit problems in them and use their memory to solve them , which does not apply to solving hard problems on mixture of concepts . We observe the same pattern in problem representation ; i.e . problem statements do not necessarily determine their identity , instead abstraction of problems by mapping them into representative concepts moves problem embedding from lexical similarity to conceptual similarity . The concepts of a problem are not mostly mentioned directly in its text , but there can be footmarks of concepts in problems . A professor and two experienced teaching assistants are asked to formulate rule-based mappings from footmarks to concepts for automation of concept extraction . As an example , the rule for labeling a problem with concept “ nchoosek ” is \\\\choose|\\\\binom|\\\\frac\ { \s * \w+\ ! \s * \ } \ { \s * \w+\ ! \s * \\\\times\s * \w+\ ! \s * \ } . By applying the rule-based concept extractor to problems , we have a new representation for problems in concept space instead of word space ; i.e . Pi ≡ Ci for 1 ≤ i ≤M , where Ci ⊂ C. ( ii ) Embedding step : a method similar to Skip-gram in Word2Vec is used for concept embedding . The high-level insight of Skip-gram is that a neural network with a single hidden layer is trained , where its output relates to how likely it is to have each concept co-occurred in a problem with the input concept . As an example , if concept “ law-of-total-probability ” of a problem is input of the neural network , we expect the neural network to state it more likely to have concept “ conditional-probability ” in the problem than for unrelated concepts like “ Poisson-process ” . However , the neural network is not used for this task , but the goal is to use weights of the hidden layer for embedding . Recall the set of all concepts as { C1 , C2 , · · · , CN } , where a problem is typically labeled with a few of them . Consider onehot coding from concepts to real-valued vectors of size N that are used for training of the neural network , where the element corresponding to a concept is one and all other elements are zero . We consider 10 neurons in hidden layer with no activation functions ( so the embedded concept vectors have 10 features ) and N neurons in the output that form a softmax regression classifier . In order to clarify on input-output pair of neural network , assume a problem that has a set of concepts { C1 , C2 , C5 } . The neural network is trained on all pairs ( C1 , C2 ) , ( C1 , C5 ) , ( C2 , C1 ) , ( C2 , C5 ) , ( C5 , C1 ) , and ( C5 , C2 ) , where the one-hot code of the first element of a pair is the input and the one-hot code of the second element of a pair is the output of the neural network in the training phase . Hence , the neural network is trained over ∑M i=1 |Ci| × ( |Ci| − 1 ) number of training data . This way , the neural network learns the statistic from the number of times that a pair is fed into it ( the neural network is probably fed with more training pairs of ( “ law-of-total-probability ” , “ conditional-probability ” ) than the pair ( “ law-of-total-probability ” , “ Poisson-process ” ) ) . Note that during training phase , input and output are one-hot vectors representing the input and output concepts , but after training when using the neural network , given a one-hot input vector , the output is a probability distribution on the set of all concepts . Finally , since input concepts are coded as one-hot codes , rows of hidden layer weight matrix , which is of size N by 10 , are concept vectors ( concept embedding ) which we are really after . Denoting embedding of concept c ∈ C by E ( c ) , problem embedding denoted by Ei for problem Pi is obtained as follows : Ei = 1 |Ci| ∑ c∈Ci 1 fc · E ( c ) , ( 1 ) where fc is frequency of concept c ∈ C in our data set of M = 635 problems . Concept embedding is scaled by its corresponding frequency to penalize concepts with high frequency that are not as informative as low frequency ones . For example , concept “ pmf ” is less informative than concept “ ML-parameter-estimation ” . Given problem embedding , similarity between two problems is defined as cosine of the angle between corresponding vector representations . Remark . We choose rule-based concept extractor for the abstraction step over any supervised/unsupervised classification methods for concept extraction because of two main reasons . First , there is a limited number of problems for most concepts as few as a couple of problems which makes any supervised classification method inapplicable due to lack of training data . Second , there are N concepts , so potentially there can be 2N −1 categories of problems which makes classification challenging for any supervised or unsupervised methods . Consider the maximum number of concepts in a problem to be k , then there are on the order of O ( Nk ) categories of problems . Even if we consider possessing acceptable number of problems for each of the O ( Nk ) categories , false positive needs to be on the order of O ( 1 Nk ) so that the number of false positives for each category be in the order of O ( 1 ) . Given that there are N = 96 concepts , utilizing a supervised or unsupervised approach to achieve such a low false positive is not feasible . Exploiting the rule-based classifier for problem labeling though , we succeed to achieve as low as average 0.98 % false positive and average 9.17 % false negative for all concepts , where problem concepts annotated by experts are considered to be ground truth . Although 100.00 % accuracy is observed in similarity detection test when utilizing problem concepts annotated by experts , those concepts are not necessarily the global optimum ground truth or the only one . Thinking of increasing accuracy in similarity detection task as an optimization problem , there is not necessarily a unique global optimum for problem concepts that lead to good performance . Hence , not having a very low false negative for rule-based does not necessarily mean that such labels are not close to a local/global optimum problem concepts . In fact , rule-based extracted concepts achieve a high 96.88 % accuracy on a similarity test as is mentioned in section 4 .

Summary: The paper proposed a hierarchical framework for problem embedding and intended to apply it to adaptive tutoring. The system first used a rule-based method to extract the concepts for problems and then learned the concept embeddings and used them for problem representation. In addition, the paper further proposed negative pre-training for training with imbalanced data sets to decrease false negatives and positives. The methods are compared with some other word-embedding based methods and showed 100% accuracy in a similarity detection test on a very small dataset.