Title: Prob2Vec: Mathematical Semantic Embedding for Problem Retrieval in Adaptive Tutoring
Authors: Du Su
Ali Yekkehkhany
Yi Lu
Wenmiao Lu
Venue: ICLR 2019
Keywords: personalized learning
e-learning
text embedding
Skip-gram
imbalanced data set
data level classification methods

Abstract: We propose a new application of embedding techniques to problem retrieval in adaptive tutoring. The objective is to retrieve problems similar in mathematical concepts. There are two challenges: First, like sentences, problems helpful to tutoring are never exactly the same in terms of the underlying concepts. Instead, good problems mix concepts in innovative ways, while still displaying continuity in their relationships. Second, it is difficult for humans to determine a similarity score consistent across a large enough training set. We propose a hierarchical problem embedding algorithm, called Prob2Vec, that consists of an abstraction and an embedding step. Prob2Vec achieves 96.88\% accuracy on a problem similarity test, in contrast to 75\% from directly applying state-of-the-art sentence embedding methods. It is surprising that Prob2Vec is able to distinguish very fine-grained differences among problems, an ability humans need time and effort to acquire. In addition, the sub-problem of concept labeling with imbalanced training data set is interesting in its own right. It is a multi-label problem suffering from dimensionality explosion, which we propose ways to ameliorate. We propose the novel negative pre-training algorithm that dramatically reduces false negative and positive ratios for classification, using an imbalanced training data set.

TL;DR: We propose the Prob2Vec method for problem embedding used in a personalized e-learning tool in addition to a data level classification method, called negative pre-training, for cases where the training data set is imbalanced.

Review discussion:
This paper proposes a new application of embedding techniques for mathematical problem retrieval in adaptive tutoring. The proposed method performs much better than baseline sentence embedding methods.
Another contribution is on using negative pre-training to deal with an imbalanced training dataset.
To me this paper is just not good enough - the method essentially i) use "a professor and two teaching assistants" to build a "rule-based concept extractor" for problems, then ii) map problems into this "concept space" and simply treat them as words. There are several problems with this approach.
First, doing so does not touch the core of the proposed application. For tutoring applications, the most important thing is to select a problem that can help students improve;
even if you can indeed select a problem that is the most similar to another problem, is it the best one to show a student? There are no evaluations on real students in the paper.
Moreover, the main difference between math problems and other problems is that there are math expressions; I do not think that using words/concept labels only is enough without touching on the math expressions.
Second, the proposed method does not sound scalable - the use of a professor and two teaching assistants to construct the concept extractor, and the use of an expert TA to select a small set of informative words. I am not sure how this will generalize to a larger number of problem spanning many different domains.
I also had a hard time going through the paper - there aren't many details. Section 2.1 is where the method is proposed, yet most of the descriptions there are unclear. Without these details it is impossible to judge the novelty of the "rule-based concept extractor", which is the key technical innovation.
1- We briefly mentioned the way problem embedding with similarity metric is used in the recommendation system in this work, but here is more explanation on that. The most similar problem is not necessarily recommended to a student. On a high level, if a student performs well on problems, we assume he/she performs well on similar problems as well, so we recommend a dissimilar problem and vice versa. More specifically, we project the performance of students on problems they solved onto the problems that they have not solved. This way, we have an evaluation of the performance of students on unseen problems. A problem is recommended that is within the capacity of students close to their boundary to help them learn, and at the same time recommendation is done so that all the concepts necessary for students are practiced by them.
An evaluation on real students is presented in part 2 of the comment titled “Response to questions about Prob2Vec” on this page, and we observed that similar problems are more likely to be solved correctly at the same time or wrong at the same time.
The math expressions are not ignored in our proposed Prob2Vec method. In the example given in the last paragraph on page 3 for example, math expressions are used to extract the concept n-choose-k. We both use math expressions and text to label problems with appropriate concepts.
2- Prob2Vec only uses expert knowledge for rule-based concept extractor, but does not use selected informative words. The effort put for rule-based concept extractor is negligible compared to effort needed for annotation of all problems with their corresponding concepts. We both annotated all problems manually and used rule-based concept extractor for annotation. In the former method, we observed 100% accuracy in the similarity detection test and observed 96.88% accuracy in the latter method.
However, the rule-based concept extractor needs much less manual effort than manual problem annotation and is capable to provide us with relatively high level of accuracy we need in our application. Note that our method is scalable as long as problems are in the same domain as the rule-based concept extractor is automated for a single domain, but for the case that problems span many different domains, it is the natural complexity of the data set that requires a more sophisticated rule-based concept extractor.
Furthermore, in most realistic cases for education purposes, problems span a single domain not multiple ones.
We also like to grab your attention to the negative pre-training method proposed for training on imbalanced data sets. You may want to refer to part 2 of comment titled “Response to Question on Negative Pre-Training” and part 1 of our response to reviewer2.
This paper proposes a method for mathematical problem embedding, which firstly decomposes problems into concepts by an abstraction step and then trains a skip-gram model to learn concept embedding. A problem can be represented as the average concept (corresponding to those in the problem) embeddings. To handle the imbalanced dataset, a negative pre-training method is proposed to decrease false and false positives. Experimental results show that the proposed method works much better than baselines in similar problem detection, on an undergraduate probability data set.
Strong points:
(1)	The idea of decomposing problems into concepts is interesting and also makes sense.
(2)	The training method for imbalanced datasets is impressive.
Concerns or suggestions:
1.	The main idea of using contents to represent a problem is quite simple and straightforward. The contribution of this paper seems more on the training method for imbalanced data sets.
But there are no comparisons between the proposed training method and previous related works. Actually, imbalance data sets are common in machine learning problems and there are many related works. The comparisons are also absent in experiments.
2.	The experimental data set is too small, with only 635 problems. It is difficult to judge the performance of the proposed model based on so small data set.
3.	The proposed method, which decomposes a problem into multiple concepts, looks general for many problem settings.
For example, representing a movie or news article by tags or topics. In this way, the proposed method can be tested in a broader domain and on larger datasets.
4.	For the final purpose, comparing problem similarity, I am wondering what the result will be if we train a supervised model based problem-problem similarity labels?
1- The idea of using concepts to represent a problem is simple, but using it along with neural network based embedding gives us the opportunity to gain concept continuity as discussed on the last paragraph on page 7 and table 2, which is an active field of research in education.
The focus of this work is on problem embedding and its application in a recommendation system that uses problem embedding to project students’ performance for the problems they solved onto the problems that they have not solved yet. Using the evaluation on unseen problems, a problem is recommended that is within the capacity of students close to their boundary to help them learn, and at the same time we cover all the concepts necessary for them to learn. In the meanwhile, we got the interesting idea of negative pre-training on training with imbalanced training data and tested our hypothesis and included in the paper.
Due to space limit, we did not include the literature review and comparison of other methods in terms of memory use and training complexity, but you can find them in the response of a previous comment below titled “Response to Question on Negative Pre-Training” on this page to see the comparison. We can include the literature review for training on imbalanced data sets as well as comparison of other methods with negative pre-training in terms of memory use and training complexity in the final version.
In summary, a) oversampling extremely suffers from over-fitting, b) SMOTE method that generates synthetic data sample is not feasible in word space, so the generated synthetic data (that are mathematical problems) are not of use for our training purpose, c) borderline-SMOTE both suffers from the same issue as SMOTE and its high complexity for finding the pairwise distance between all data samples, which is a burden in high dimensional data, and d) hybrid methods need m >> 1 weak learners in contrast to negative pre-training that uses a single learner. Memory use and training time is an issue for hybrid method when the weak learners are deep neural networks with too many parameters.
We are currently running a broader experiment for negative pre-training on other data sets to gain more insight on it, but for the purpose of the task proposed in this work, it outperforms one-shot learning, which cannot be said that is the state-of-the art, but is a common practice. There is no notion of state-of-the-art in training on imbalanced data sets since due to our best knowledge, there is no method that outperforms all the other ones, and the performance of different methods depends more on the nature of the data set.
2- The data set being small is the nature of the application since creating mathematical problems is a creative process, so it is hard to have a very big data set. The Prob2Vec method is performing well on this not relatively big data set, which is our goal, but if we have a bigger data set (as we have right now with more than 2400 problems), Prob2Vec may even have a better performance since with more data we can have a more precise concept and problem embedding.
3- Thanks for your suggestion.
4- It is difficult for humans to determine a similarity score consistent across a large enough training set, so it is not feasible to simply apply supervised methods to learn a similarity score for problems.
Even if problem-problem similarity annotation is feasible, a lot of effort should go into the annotation, which is not scalable.
The paper proposed a hierarchical framework for problem embedding and intended to apply it to adaptive tutoring. The system first used a rule-based method to extract the concepts for problems and then learned the concept embeddings and used them for problem representation.
In addition, the paper further proposed negative pre-training for training with imbalanced data sets to decrease false negatives and positives. The methods are compared with some other word-embedding based methods and showed 100% accuracy in a similarity detection test on a very small dataset.
In sum, the paper has a very good application but not good enough as a research paper. Some of the problems are listed as follows:
1.	Lack of technical novelty.  It seems to me just a combination of several mature techniques. I do not see much insight into the problem.
For example, if the rule-based concept extractor can already extract concepts very well, the “problem retrieval” should be solved by searching with the concepts as queries. Why should we use embedding to compare the similarity?
Also, the title of the paper is about problem retrieval but the experiments are about similarity comparison, there seems a gap.
2.	Data size is too small, and the baselines are not state-of-the-art. There are some unsupervised sentence embedding methods other than the word-embedding based models.
Some clarity issues.
For example, Page 6. “is pre-trained on a pure set of negative samples”— what is the objective function? How to train on only negative samples?
1- There are two reasons that concept and problem embedding are performed in this work. Considering concept continuity is an important matter in education. Having concept embedding, concept continuity can be reached as is discussed in the last paragraph on page 7 and some other examples are given in table 2. By just having the most sophisticated concept extractor, the concept continuity cannot be retrieved.
Furthermore, problem embedding is used by the recommender system to project the performance of students on the problems they solved onto other problems that they have not solved. This way, we have an idea of what problems should be recommended to them and which problems should not by having an evaluation of their ability to solve unseen problems and recommend problems in the boundary of their capacity, not way beyond, and to recommend problems in a way that covers all concepts necessary for students to learn. We have observed interesting patterns, e.g. similar problems are more likely to be solved correctly at the same time or wrong at the same time. Note that by just having the concepts of problems that are not in numerical form, performance projection may not be feasible and there is a need for using other methods like embedding.
2- The data size being small is just the nature of the application. Creating new problems is a creative process and is not easy, given that with the insight we have on the application, the data size seems to suffice.
Furthermore, since Prob2Vec is performing well for not a relatively big data set, it would definitely do well for big data sets since the more data we have, the more precise the concept and problem embedding are. The easy-tough-to-beat method proposed by Arora et al. is the state of the art in unsupervised sentence embedding that we compared our algorithm with. Please let us know if we missed anything.
Pre-training is a common practice in transfer learning (one-shot learning). The objective function does not differ from the objective function used for post training. Training on only negative samples with lower training epochs than the training epochs in post training just adjusts the weights of the neural network to a better starting point. If the training epochs in pre-training is relatively smaller than the training epochs in post training, due to curse of dimensionality, the warm start for post training results in better performance for NN classifier. To make it more clear what it means to train the neural network on a pure set of negative data samples, think about batch training.
It's not likely, but possible, that a batch only has negative or positive samples. In the pre-training phase of our method, we intentionally used a pure set of negative samples (with fewer training epochs) to have a warm start for post training. As table 3 shows, our proposed method outperforms one-shot learning. Please look at part 1 of our response to reviewer2 and part 2 of comment titled "Response to Question on Negative Pre-Training" below.

Meta-review:
role: AC
data: metareview: I tend to agree with reviewers. This is a bit more of an applied type of work and does not lead to new insights in learning representations. 
Lack of technical novelty
Dataset too small

Decision: Reject