Reference Distance Estimator
Authors: Yanpeng Li
ICLR 2014

Abstract: A theoretical study is presented for a simple linear classifier called reference distance estimator (RDE), which assigns the weight of each feature j as P(r|j)-P(r), where r is a reference feature relevant to the target class y. The analysis shows that if r performs better than random guess in predicting y and is conditionally independent with each feature j, the RDE will have the same classification performance as that from P(y|j)-P(y), a classifier trained with the gold standard y. Since the estimation of P(r|j)-P(r) does not require labeled data, under the assumption above, RDE trained with a large number of unlabeled examples would be close to that trained with infinite labeled examples. For the case the assumption does not hold, we theoretically analyze the factors that influence the closeness of the RDE to the perfect one under the assumption, and present an algorithm to select reference features and combine multiple RDEs from different reference features using both labeled and unlabeled data. The experimental results on 10 text classification tasks show that the semi-supervised learning method improves supervised methods using 5,000 labeled examples and 13 million unlabeled ones, and in many tasks, its performance is even close to a classifier trained with 13 million labeled examples. In addition, the bounds in the theorems provide good estimation of the classification performance and can be useful for new algorithm design.


Peer review:
[Review]:
The paper describes a heuristic algorithm for learning representation features from labeled and unlabeled data. These features are then used as inputs to a classifier trained on labeled data. The main motivation of the algorithm is semi-supervised learning when unlabeled data can be used for estimating and selecting the representation features. Properties of these new features are analyzed under different hypothesis and in particular, a bound is provided for selecting the new features among a set of candidates. Experiments are then performed on a large dataset of Medline abstract and the proposed approach is compared to supervised and semi-supervised classifiers.
The paper introduces new ideas for learning intermediate representations.  They appear to be quite effective in the experimental comparison and the proposed method performs better than alternative semi-supervised baselines. The author also develops a series of properties of the learned features. On the other hand, the paper presents different weaknesses. The development of these features is neither intuitive nor theoretically founded. The final classifier performs a linear combination of the initial data features, and it is hard to understand why it should perform better than state of the art linear methods. The proposed heuristic could be interpreted as a prior on classifier weights, but again in the context of large training sets like the ones used in the experiments, this should not be a real advantage. Some of the experimental results are surprising. It is said that logistic regression and linear SVMs cannot be trained on large amounts of data and the author does not present results with these classifiers beyond 200 k training data. This should be reconsidered.  Results of semi-supervised classifiers are also doubtful. The baselines used for comparison are not able to leverage the use of unlabeled data when usually for text classification, semi-supervised learning is quite effective.
The form of the paper should also be improved.  In particular the experimental illustration of the distance and bound statistics is not clearly explained.
Globally, there are interesting ideas in this method, but also conceptual and experimental weaknesses that should be improved upon.