text: 
  
  TITLE
  EncoderDecoderModel: tie weights between different classes of models
  
  COMMENTS
  3
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  # 🚀 Feature request
  
  Tie weights between different classes of models, tie embedding matrices, update tutorial.
  
  ## Motivation
  
  I have been following the Longformer2Roberta tutorial https://github.com/huggingface/transformers/blob/master/model_cards/patrickvonplaten/longformer2roberta-cnn_dailymail-fp16/README.md and it seems like the crucial part of tying weights is missing.
  The EncoderDecoderModel is initialized with "allenai/longformer-base-4096" and "roberta-base", and "allenai/longformer-base-4096" in its turn was initialized from "roberta-base". It seems natural to be able to tie their attention and FFNN weights, although it might be problematic to deal with positional embedding. Anyway, I think one feature that definitely should be implemented is the tying embedding matrices. 
  
  ## Your contribution
  
  For now I solve the issue with
  ```
  model.encoder.embeddings.word_embeddings.weight = model.decoder.roberta.embeddings.word_embeddings.weight
  ```
  @patrickvonplaten 