Before revision:  However, to what extent is the information encoded in a sentence representation actually used for the task which the encoder is trained on ? In this work, we examine this probing paradigm through a case-study in Natural Language Inference, showing that models  learn to encode linguistic properties even when not needed for a task . We identify that pre-trained word embeddings play a considerable role in encoding these properties rather than the training task itself, highlighting the importance of careful controls when designing probing experiments.
After revision:  However, to what extent is the information encoded in a sentence representation actually used for the task which the encoder is trained on ? In this work, we examine this probing paradigm through a case-study in Natural Language Inference, showing that models learn to encode linguistic properties even when not needed for the task on which the model was trained. We further identify that pretrained word embeddings play a considerable role in encoding these properties rather than the training task itself, highlighting the importance of careful controls when designing probing experiments.