Can we use GPT-2 sentence embedding for classification tasks?
Author: shamanez
Pull request: no
Labels: name: Discussion

name: wontfix

I am experimenting on the use of transformer embeddings in sentence classification tasks **without finetuning them**. I have used BERT embeddings and those experiments gave me very good results. Now I want to use GPT-2 embeddings (without fine-tuning). So I have two questions,

1. Can I use GPT-2 embeddings like that (because I know Gpt-2 is trained on the left to right)
2. Is there any example uses of GPT-2 in classification tasks other than generation tasks?
3.  If I can use GPT-2 embeddings, how should I do it ?

Comments:
GPT-2 and BERT are both transformer networks with very similar architectures. You can use the GPT-2 embeddings the same way you used BERT embeddings.

As you said, GPT-2 only handles left context. You can read [the paper](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) where the authors showcase results on several tasks in a zero-shot setting (section 3).

I recently imported the GPT2Model and built a simple classifier. I think the model is too naive. And could use some improvements. If you notice any mistakes, please correct me. :) 
```
class SimpleGPT2SequenceClassifier(nn.Module):
    def __init__(self, hidden_size: int, num_classes:int ,max_seq_len:int, gpt_model_name:str, 
                 cache_dir:str):
        super(SimpleGPT2SequenceClassifier,self).__init__()
        self.gpt2model = GPT2Model.from_pretrained(
            gpt_model_name, cache_dir = cache_dir
        )
        self.fc1 = nn.Linear(hidden_size, num_classes)
        
    def forward(self, x_in):
        """
        Args:
                x_in: encoded inputs ids of sent.
        """
        
        gpt_out = self.gpt2model(x_in)[0] #returns tuple
        batch_size = gpt_out.shape[0]
        prediction_vector = self.fc1(gpt_out.view(batch_size,-1)) #(batch_size , max_len, num_classes)
    
        return prediction_vector
```
For preprocessing the text before encoding them with the tokenizer.

```
punkt_sentence_detector = nltk.data.load('tokenizers/punkt/english.pickle')
class GPT2Preprocessor:
    def __init__(self, transformer_tokenizer, sentence_detector):
        self.transformer_tokenizer = transformer_tokenizer
        self.sentence_detector = sentence_detector

    def add_eos_tokens(self, text):
        eos_token = " " + self.transformer_tokenizer.eos_token + " "
        sentences = self.sentence_detector.tokenize(text)
        eos_added_text = (
            eos_token.join(sentences) + " " + self.transformer_tokenizer.eos_token
        )
        return eos_added_text
```

I tried GPT-2 embeddings and compare them with Roberta embeddings for the task of sentiment classification (both networks were frozen during the training). GPT-2 couldn't outperform the results of Roberta.

@cozek from the code, it isn't obvious whether you've frozen gpt2 layers or not ?

> @cozek from the code, it isn't obvious whether you've frozen gpt2 layers or not ?

Of course, I have not frozen any layers. It is not always necessary to freeze the layers. If required you can easily freeze the layers as necessary. 

> I tried GPT-2 embeddings and compare them with Roberta embeddings for the task of sentiment classification (both networks were frozen during the training). GPT-2 couldn't outperform the results of Roberta.

Do you still have the notebooks? I would be interested to see how you implemented a classification head on top of gpt-2. 

> > I tried GPT-2 embeddings and compare them with Roberta embeddings for the task of sentiment classification (both networks were frozen during the training). GPT-2 couldn't outperform the results of Roberta.
> 
> Do you still have the notebooks? I would be interested to see how you implemented a classification head on top of gpt-2.

https://github.com/cozek/OffensEval2020-code/blob/master/notebooks/Eng%20Task%20A%20-%20Ensemble%20DistilGPT2.ipynb

Here you go. I used it for OffenEval 2020, Hate Speech Detection. I used the distilled version. Feel free to swap it out and take the full GPT-2. We got 0.90 Macro f1 with this model.  

You can add a CLS token to the vocabulary 

`tokenizer.add_special_tokens({'cls_token': '[CLS]'})
  model.resize_token_embeddings(len(tokenizer))`

Then append this CLS token at the end of your input 
Then use the representation of this CLS token for classification as done in BERT.
cc @cozek 

> > > I tried GPT-2 embeddings and compare them with Roberta embeddings for the task of sentiment classification (both networks were frozen during the training). GPT-2 couldn't outperform the results of Roberta.
> > 
> > 
> > Do you still have the notebooks? I would be interested to see how you implemented a classification head on top of gpt-2.
> 
> https://github.com/cozek/OffensEval2020-code/blob/master/notebooks/Eng%20Task%20A%20-%20Ensemble%20DistilGPT2.ipynb
> 
> Here you go. I used it for OffenEval 2020, Hate Speech Detection. I used the distilled version. Feel free to swap it out and take the full GPT-2. We got 0.90 Macro f1 with this model.

Thanks a lot. Very helpful! 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@cozek I see in your code that you concatenate all the token embeddings together to produce the sentence representation and then pass that through `fc1`:


```python
        gpt_out = self.gpt2model(x_in)[0] #returns tuple
        batch_size = gpt_out.shape[0]
        prediction_vector = self.fc1(gpt_out.view(batch_size,-1))
```

Instead of concatenating all the token embeddings, did you try:

1. pooling over all the tokens to get the sentence representation? For example, max pooling or mean pooling?
2. using the embedding of the last token?

@AsmirMumin 

> @cozek I see in your code that you concatenate all the token embeddings together to produce the sentence representation and then pass that through `fc1`:
> 
> ```python
>         gpt_out = self.gpt2model(x_in)[0] #returns tuple
>         batch_size = gpt_out.shape[0]
>         prediction_vector = self.fc1(gpt_out.view(batch_size,-1))
> ```
> 
> Instead of concatenating all the token embeddings, did you try:
> 
> 1. pooling over all the tokens to get the sentence representation? For example, max pooling or mean pooling?
> 2. using the embedding of the last token?

I did not try 1 or 2. Option 1 seems logical as it would reduce the size of the FC layer and increase training speed. 
I am not familiar with option 2.