Is there an easy way to access the multiple choice head of the RobertaForMultipleChoice?
Author: h56cho
Pull request: no

Hello,

This question is specifically for the ```RobertaForMultipleChoice``` model.
I know how to extract the hidden state vectors from each layer of the ```RobertaForMultipleChoice``` model, but is there any way that I can directly feed the hidden state vectors of layer `j` as an input to the multiple-choice head of the model?

In other words, I would like to know if there is any way that I can control the input of the multiple-choice head of the ```RobertaForMultipleChoice``` model (or if there is any easy way that I can access the multiple-choice head directly).

Thank you,

Comments:
There is no direct way to do this from the model, you will have to edit the code directly. You can just copy the code from the file `modeling_roberta.py` and then change the forward method of `RobertaForMultipleChoice`. You can then directly feed the hidden layer of your choice to the final layer.

Hello,
Would the code below do the job that I am looking for?
```python
# get the pre-trained HuggingFace RobertaForMultipleChoice and resize the token embeddings after adding the special token
best_model_roberta = RobertaForMultipleChoice.from_pretrained('roberta-base', output_hidden_states = True)

# for each layer j = 1,...,12, extract the hidden states at the layer j
hidden_states = best_model_roberta(input_ids=input_ids, token_type_ids=token_type_ids, attention_mask=attention_mask, labels=mc_labels)[2][j][:,:,:].detach()

mc_logits = best_model_roberta.classifier(hidden_states).detach()
```

Thank you,


Yes, but if you detach the history, you won't be able to do a backward pass through this model.

Hello,

Thank you for your reply. If I want to calculate the mc_loss generated after inputting the hidden state vector directly into the multiple-choice head, would the code below be appropriate?
```python
import torch
from torch.nn import CrossEntropyLoss
from matplotlib import pyplot as plt
from transformers import RobertaTokenizer, RobertaForMultipleChoice, AdamW, get_constant_schedule
from transformers import PreTrainedTokenizer
import numpy as np
import pandas as pd
import pickle
import dill
from matplotlib.pyplot import plot, savefig, xlim, figure, ylim, legend, boxplot, setp, axes, xlabel, ylabel, xticks
import gc 
import math
import time
from random import seed
from random import randint
import sys
import statistics
from numpy import nan
import scipy.stats as ss

# import the pre-trained HuggingFace RobertaTokenizer
tokenizer = RobertaTokenizer.from_pretrained('roberta-base')
        
# get the encoding for the special tokens
pub2_pad_token_id = tokenizer.convert_tokens_to_ids(tokenizer.pad_token)
        
# sanity check
len(tokenizer) 

# get the pre-trained HuggingFace RobertaForMultipleChoice and resize the token embeddings after adding the special token
best_model_roberta = RobertaForMultipleChoice.from_pretrained('roberta-base', output_hidden_states = True)

# Turn on the evaluation mode
best_model_roberta.eval()
            
#  extract the hidden states at the layer 1
hidden_states = best_model_roberta(input_ids=input_ids, attention_mask=attention_mask, labels=mc_labels, output_hidden_states=True)[2][1][:,:,:].detach()

# access the multiple-choice head            
mc_logits = best_model_roberta.classifier(hidden_states).detach()
  
# define the loss function
loss_fct = CrossEntropyLoss()

# calculate the mc_loss
mc_loss = loss_fct(mc_logits.view(1,-1), mc_labels)
```

The code above works without error, but I am particularly wondering if `mc_logits.view(1,-1)` is correct. The original HuggingFace code for RobertaForMultipleChoice uses  `mc_logits.view(-1,num_choice)` to calculate the resulting error, but I am wondering if it is correct to specify  `mc_logits.view(1,-1)` instead, if I am inputting the hidden state vectors directly into the multiple-choice head (i.e. instead of `pooled_output`...not sure what the `pooled_output` in the HuggingFace code is).

Thank you,

        

I think you should use the pooler layer too, which is there to convert all your sequence tokens in one summary state. So
```python
pooled_output = self.roberta.pooler(hidden_state)
logits = self.classifier(pooled_output)
reshaped_logits = logits.view(-1, num_choices)
loss_fct = CrossEntropyLoss()
mc_loss = loss_fct(reshaped_logits, mc_labels)
```