Domain Adaptation / Adapting Pretrained Model, Part 2 (WING Reading Group – Week Recess, Sem 2010)
Channel: Min-Yen Kan
Date: 25/09/2020

Lecturers:  Yap Jia Qing, Zhang Ruixi, Laiba Mehnaz, Brian Formento, Zhang Yifan, Khush Jammu
Scribe Notes: Taha Aksu, Shreyas Kutanoor Prakash, Abhinav Ramesh Kashyap, Bhavitvya Malik

0:00 1. Introduction to Fine-tuning on BERT by Brian:
* Devlin et al. (2019) "BERT: Pre-training of Deep Bidirectional Transformers for
Language Understanding" https://arxiv.org/pdf/1810.04805.pdf
* Sanh et al. (2020) "DistilBERT, a distilled version of BERT: smaller,
faster, cheaper and lighter" https://arxiv.org/pdf/1910.01108.pdf
* Lan et al. (2020) "ALBERT: A LITE BERT FOR SELF-SUPERVISED
LEARNING OF LANGUAGE REPRESENTATIONS" ICLR'20 https://arxiv.org/pdf/1909.11942.pdf
38:35 2. Introduction to Fine-tuning on GPT by Khush:
* Brown et al. (2020) "Language Models are Few-Shot Learners", https://arxiv.org/abs/2005.14165
* Radford et al. (2019) "Language Models are Unsupervised Multitask Learners" https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
* Radford et al. (2018) "Improving Language Understanding by Generative Pre-Training" https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf
* "How does a Generalized Autoregressive Pretraining Method Compare to an Autoencoder Language Model on NLP Tasks?"
1:17:14 3. Transfer Learning for Pretrained Models by Ruixi and Jia Qing:
* Housby et al. (2019) "Parameter-Efficient Transfer Learning" PMLR  97 http://proceedings.mlr.press/v97/houlsby19a/houlsby19a.pdf 
* Yang et al. (2017) "Transfer Learning for Sequence Tagging with Hierarchical Recurrent Network" ICLR'17 https://openreview.net/pdf?id=ByxpMd9lx

Session Notes and Paper list will be available at the website: 
https://wing-nus.github.io/cs6101