text: 5,62,1106930520301150210,2785337469,Sebastian Ruder,"New paper with @mattthemathman & @nlpnoah on adapting pretrained representations: We compare feature extraction & fine-tuning with ELMo and BERT and try to give several guidelines for adapting pretrained representations in practice. <LINK> <LINK> @michalwols @mattthemathman @nlpnoah For BERT, we mostly modified the provided Colaboratory notebooks, while for ELMo, we modified scripts in AllenNLP, so both of those should be fairly easily reproducible using existing resources. I might upload the analysis scripts if there's interest.",https://arxiv.org/abs/1903.05987,"While most previous work has focused on different pretraining objectives and architectures for transfer learning, we ask how to best adapt the pretrained model to a given target task. We focus on the two most common forms of adaptation, feature extraction (where the pretrained weights are frozen), and directly fine-tuning the pretrained model. Our empirical results across diverse NLP tasks with two state-of-the-art models show that the relative performance of fine-tuning vs. feature extraction depends on the similarity of the pretraining and target tasks. We explore possible explanations for this finding and provide a set of adaptation guidelines for the NLP practitioner. ","To Tune or Not to Tune? Adapting Pretrained Representations to Diverse