Before revision:  For (2) , we analyze properties of the cross-entropy objective to show that \epsilon-optimal language models in cross-entropy (log-perplexity) learn features that are \mathcal{O}(\sqrt{\epsilon}) -good on natural linear classification tasks , thus demonstrating mathematically that doing well on language modeling can be beneficial for downstream tasks.
After revision:  With a mathematical formalization of this hypothesis, we make progress towards (2) , we analyze properties of the cross-entropy objective to show that \epsilon-optimal language models in cross-entropy (log-perplexity) learn features that are \mathcal{O}(\sqrt{\epsilon}) -good on natural linear classification tasks , thus demonstrating mathematically that doing well on language modeling can be beneficial for downstream tasks.