text: 
  
  TITLE
  Porting Compressive Transformer to Huggingface
  
  COMMENTS
  0
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  # 🌟 New model addition
  
  As part of my Master-Thesis I will implement or adapt different transformer architecture for LM that are specifically designed for long-context situation. As part of this I started with porting the Compressive Transformer to the huggingface interface. And will probably do the same for other architectures in the future. Let me know if you're interested for a pull-request.
  
  ## Model description
  [Paper](https://arxiv.org/pdf/1911.05507.pdf)
  
  The Compressive Transformer is an extension of the Transformer-XL architecture with an additional compressed memory. Memories that would get discarded in the Transformer-XL get compressed and added to the compressed memory. 
  The compression function can take different forms but the best performance on word-level LM is a Conv1d compression. 
  Training of the C-Transformer happens the same way to how the Transformer-XL architecture is trained. But additional to that there is an "attention-reconstruction-loss" which compares the attention that we get from using the memory with the attention we get from its compressed counter-part. Using the MSE-loss we can perform gradient updates on the compression function
  
  
  ## Open source status
  
  * [x] the model implementation is available: 
  https://nn.labml.ai/transformers/compressive/index.html Is an open source implemention under the MIT license
  https://github.com/deepmind/pg19 The data-set used in parts of the experiment by the Authors
  https://github.com/vilmarzti/long_context_transformers/blob/main/longcontext/transformers/compressive_transformer.py My humble start of porting the architecture to the huggingface-format.
  * [x] the model weights are available: 
  None that I could find.
  Weights (for wikitext-2 and 103) might become available as my thesis progresses and I start training
  * [x] who are the authors: 
  Jack W. Rae (https://github.com/dm-jrae)
  Anna Potapenko (https://github.com/AnyaP)
  Siddhant M. Jayakumar (github-profile not found)
  Chloe Hillier (github profile not found)
  Timothy P. Lillicrap (github profile not found)
  