ODELING FOR RARE CLASSES

This paper presents a new predictive modeling framework designed to learn classification models from imperfectly labeled samples, in the absence of expertannotated training samples, for identifying rare classes. Our results show that, under some reasonable assumptions, the classifiers trained from imperfectly labeled training data using this approach have performance comparable to the models trained using expert-annotated training data. This capability of learning from imperfect supervision is advantageous in a wide range of applications where the target class of interest is relatively rare and obtaining a precise labeling of even a small number of training samples is infeasible. We present the application of the framework for creating historical maps of forest fires from satellite data for the tropical forests. This new forest fire product identifies approximately 1 million sq. km. of burned areas in the tropical forests in South America and South-east Asia during years 2001-2014, which is more than three times of the total burned area reported by the state-of-art NASA products in these regions. We show validation of these results using burn-scars visible in satellite images to confirm the veracity of these forest fires. I. MOTIVATION The traditional classification approach uses labeled training data to select the best classification model from a family of models. Since collecting labeled samples is often tedious and sometimes even infeasible, recent research in machine learning [1, 2, 3] has focused on developing algorithms to train classification models in scarcity of labeled training samples. In contrast, the focus of this paper is to address problem settings where acquiring even a small number of expert-annotated gold standard labeled samples for supervision is infeasible. The problem of learning in the complete absence of labels is relevant in many important domains. Consider the task of mapping forest fires globally from satellite data in which the relationship between the explanatory variables and target variable varies in space and time. Corresponding author: V Mithal, University of Minnesota, Twin Cities MN mithal@cs.umn. edu. 1 University of Minnesota 2 NASA AMES This necessitates learning separate customized classification models for each homogeneous data partition because learning a single global model, or applying a model trained in one partition on another heterogeneous partition leads to suboptimal performance. Since expert labeled data is not available in most parts of the world, there is a need to train classifiers in absence of labeled training data. The problem of mapping forest fires from satellite data is also challenging due to the ultra-skew class distribution. Traditional training algorithms, eg. logistic regression, are designed to learn the model parameters to achieve highest classification accuracy when the conditional probability (P (y = 1|x)) is thresholded at 0.5. For rare class problem settings, a more appropriate objective is to maximize the recall and precision of the rare class. Varying the threshold used on P (y = 1|x) changes the precision and recall; increasing the threshold value decreases the recall while decreasing the threshold value decreases the precision. Incorrect choice of threshold can lead to very poor performance (because either recall or precision is too low) and thus it is critical to find the threshold that maximizes a desired combination of precision and recall. II. METHOD In absence of expert provided labels, we propose a new framework RAPT (RAre class Prediction in absence of True labels), which makes use of imperfect labels to train classifiers. In previous research it has been shown that if the imperfect labels are conditionally independent of the explanatory variables, the conditional probability from model trained using expert labeled training samples and the conditional probability from model trained using imperfectly labeled training samples give identical rankings for test instances. In case of expert labeled training samples, a threshold of 0.5 on the conditional probability is known to maximize accuracy. However, if the imperfect labels are perturbations of the true labels, then the threshold corresponding to maximum classification accuracy is not 0.5 and needs to be identified during training phase (Natarajan et al. [4]).