only fine tune the encoder part of BART
Author: lytum
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Description of your issue -->
May i ask how can I fine-tune only the BART-encoder?

from transformers import BartTokenizer, BartModel

tokenizer = BartTokenizer.from_pretrained('facebook/bart-base')
model = BartModel.from_pretrained('facebook/bart-base', return_dict=True)

inputs = tokenizer(["i am a girl", "i am from germany"], return_tensors="pt", padding=True)
outputs = model(**inputs).encoder_last_hidden_state

i load the pretrained BART, and get the encoder_last_hiddene_state.

however, during fine-tuning, how about the decoder part?

how could i only load the part of encoder and fine-tune?

Thanks a lot
<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:

Comments:
What task are you working on? Or to rephrase, what do you do once you have ` encoder_output = model(**inputs).encoder_last_hidden_state`?

Thanks for your feedback. In my case, I only want the bart encoder and combine it to my own decoder.  Thanks for further feedback

How do you want to fine-tune the encoder? You'll either need to add a task specific head or pair it with your decoder and train both encoder and decoder. I think you should be able to Initialize only the encoder using following snippet 

```python
from torch import nn

from transformers.modeling_bart import BartEncoder, PretrainedBartModel, PretrainedBartModel
from transformers import BartConfig

class Encoder(PretrainedBartModel):
  def __init__(self, config: BartConfig):
    super().__init__(config)

    padding_idx, vocab_size = config.pad_token_id, config.vocab_size
    self.shared = nn.Embedding(vocab_size, config.d_model, padding_idx)

    self.encoder = BartEncoder(config, self.shared)
  
  def forward(
      self, input_ids, attention_mask=None, output_attentions=False, output_hidden_states=False, return_dict=False
  ):

    output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
    output_hidden_states = (
        output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
    )
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict

    encoder_outputs = self.encoder(
        input_ids=input_ids,
        attention_mask=attention_mask,
        output_attentions=output_attentions,
        output_hidden_states=output_hidden_states,
        return_dict=return_dict,
    )

    return encoder_outputs

enc = Encoder.from_pretrained("facebook/bart-base")
```

@sshleifer  does this makes sense ?

The text makes sense. Does that code run!?

> How do you want to fine-tune the encoder? You'll either need to add a task specific head or pair it with your decoder and train both encoder and decoder. I think you should be able to Initialize only the encoder using following snippet
> 
> ```python
> from torch import nn
> 
> from transformers.modeling_bart import BartEncoder, PretrainedBartModel, PretrainedBartModel
> from transformers import BartConfig
> 
> class Encoder(PretrainedBartModel):
>   def __init__(self, config: BartConfig):
>     super().__init__(config)
> 
>     padding_idx, vocab_size = config.pad_token_id, config.vocab_size
>     self.shared = nn.Embedding(vocab_size, config.d_model, padding_idx)
> 
>     self.encoder = BartEncoder(config, self.shared)
>   
>   def forward(
>       self, input_ids, attention_mask=None, output_attentions=False, output_hidden_states=False, return_dict=False
>   ):
> 
>     output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
>     output_hidden_states = (
>         output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
>     )
>     return_dict = return_dict if return_dict is not None else self.config.use_return_dict
> 
>     encoder_outputs = self.encoder(
>         input_ids=input_ids,
>         attention_mask=attention_mask,
>         output_attentions=output_attentions,
>         output_hidden_states=output_hidden_states,
>         return_dict=return_dict,
>     )
> 
>     return encoder_outputs
> 
> enc = Encoder.from_pretrained("facebook/bart-base")
> ```
> 
> @sshleifer does this makes sense ?

 Thanks for your code @patil-suraj . Really appreciate!!

> The text makes sense. Does that code run!?

In my case, the code runs without problems, so I think it"s correct! Thanks a lot!