Reference Distance Estimator
Authors: Yanpeng Li
ICLR 2014

Abstract: A theoretical study is presented for a simple linear classifier called reference distance estimator (RDE), which assigns the weight of each feature j as P(r|j)-P(r), where r is a reference feature relevant to the target class y. The analysis shows that if r performs better than random guess in predicting y and is conditionally independent with each feature j, the RDE will have the same classification performance as that from P(y|j)-P(y), a classifier trained with the gold standard y. Since the estimation of P(r|j)-P(r) does not require labeled data, under the assumption above, RDE trained with a large number of unlabeled examples would be close to that trained with infinite labeled examples. For the case the assumption does not hold, we theoretically analyze the factors that influence the closeness of the RDE to the perfect one under the assumption, and present an algorithm to select reference features and combine multiple RDEs from different reference features using both labeled and unlabeled data. The experimental results on 10 text classification tasks show that the semi-supervised learning method improves supervised methods using 5,000 labeled examples and 13 million unlabeled ones, and in many tasks, its performance is even close to a classifier trained with 13 million labeled examples. In addition, the bounds in the theorems provide good estimation of the classification performance and can be useful for new algorithm design.


Peer review:
[Review]:
The paper proposes a feature selection algorithm and learn a new representation based on the selected features.  The new representation is then used for constructing classifiers. 

This is a very difficult paper to read, due to the exposition style. As far as I can gather, the feature selection algorithm just selects features are (on average) as correlated to other features as possible, cf. step 1 in section 2.2. The new representation is then weighted combination of the old features. 

I have difficulty in understanding either contributions.  There is little motivation on using a reference feature --- which is not available in practice.  Moreover, how Theorem 3's condition is generally true?