Before revision:  This paper considers the following questions: (1) Why should learning the distribution of natural language help with downstream classificationtasks ? (2) Why do features learned using language modelinghelp solve downstream tasks with linear classifiers ?
After revision:  This paper initiates a mathematical study of this phenomenon for the downstream task of text classification by considering the following questions: (1) Why should learning the distribution of natural language help with downstream classificationtasks ? (2) Why do features learned using language modelinghelp solve downstream tasks with linear classifiers ?