changing config.axial_pos_shape for 'ReformerModelWithLMHead' when fine-tuning 
Author: D-i-l-r-u-k-s-h-i
Pull request: no
Labels: name: wontfix

name: reformer

# ❓ Questions & Help

I'm trying to fine-tune the Reformer for the language generation task and I padded the sequence lengths to be a multiple of least common multiple chunk_length 64, and now I'm asked to pad the sequence to 524288(512 * 1024), which will give me an out of memory error. 

I would like to know a workaround for this, since the error message also gives an alternative to 'pad_to_max_length', which is 'changing config.axial_pos_shape' and specially since this is known to be a memory efficient transformer. Thank you.
**A link to original question on Stack Overflow**: [https://stackoverflow.com/questions/61986452/fine-tuning-reformer-gives-out-of-memory-error-when-sequence-length-is-padded-t](https://stackoverflow.com/questions/61986452/fine-tuning-reformer-gives-out-of-memory-error-when-sequence-length-is-padded-t)


Comments:
I would not recommend to set `axial_pos_shape` to (512 * 1024). In the notebook I just used that to demonstrate how far the limits can be pushed for Reformer. Half a million token is extremely long and usually unnecessary. 

Make sure you have read and understood how AxialPostionEmbeddings work: https://huggingface.co/transformers/model_doc/reformer.html#axial-positional-encodings . 

For "normal" language modeling it might make much more sense to start from the Reformer-wiken8 model and finetune it: https://huggingface.co/google/reformer-enwik8

Greetings,
Would fine tuning https://huggingface.co/google/reformer-enwik8 work normally with run_language_modeling.py script?
Thanks

Hmm, for the most part but you will have to define your own tokenzer function as can be seen here: https://huggingface.co/google/reformer-enwik8#reformer-language-model-on-character-level-and-trained-on-enwik8


So instead of sticking to the script, I would recommend slightly changing this notebook: https://github.com/patrickvonplaten/notebooks/blob/master/PyTorch_Reformer.ipynb. Instead of creating the dataset by using a tokenizer, you should use the function linked above. Does that make sense? Also linking: https://github.com/huggingface/transformers/pull/4480. If someone has an easy script for Reformer Char LM it'd be great to post it here or add a notebook. 

Ok, thanks. So the function _flatten_and_tokenize_ (in the notebook) shall be replaced by the _encode_
function (in the enwik8 model card), am I following right?

> I would not recommend to set `axial_pos_shape` to (512 * 1024). In the notebook I just used that to demonstrate how far the limits can be pushed for Reformer. Half a million token is extremely long and usually unnecessary.
> 
I've been using 'google/reformer-crime-and-punishment' model from [https://huggingface.co/transformers/model_doc/reformer.html#reformermodelwithlmhead](url)

I get this error after I padded the sequence lengths to be a multiple of least common multiple chunk_length 64.

```
...
for epoch in range(EPOCHS):
    print(f"EPOCH {epoch} started" + '=' * 30)
    for idx,article in tqdm_notebook(enumerate(article_loader)):
        
        article_tens = tokenizer.encode(article[0], return_tensors='pt').to(device)
      
        print(article_tens.shape)
        #multiple of least common multiple chunk_length 64.
        pads_to_be_filled=getNoOfPads(article_tens.size()[1])
        
        padded_tens= torch.cat((article_tens[0],Variable(torch.zeros((pads_to_be_filled),dtype=torch.long).cuda())) )
        
        print(padded_tens.unsqueeze(0).shape)

        outputs = model(padded_tens.unsqueeze(0), labels=padded_tens.unsqueeze(0))[0]
        ...
```
```

EPOCH 0 started==============================
0/? [00:00<?, ?it/s]
torch.Size([1, 131])
torch.Size([1, 192])

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-11-81c445097515> in <module>()
     29         print(padded_tens.unsqueeze(0).shape)
     30 
---> 31         outputs = model(padded_tens.unsqueeze(0), labels=padded_tens.unsqueeze(0))[0]
     32         print(outputs)
     33 

7 frames
/usr/local/lib/python3.6/dist-packages/transformers/modeling_reformer.py in forward(self, position_ids)
    127                 reduce(mul, self.axial_pos_shape) == sequence_length
    128             ), "If training, make sure that config.axial_pos_shape factors: {} multiply to sequence length. Got prod({}) != sequence_length: {}. You might want to consider padding your sequence length to {} or changing config.axial_pos_shape.".format(
--> 129                 self.axial_pos_shape, self.axial_pos_shape, sequence_length, reduce(mul, self.axial_pos_shape)
    130             )
    131             if self.dropout > 0:

AssertionError: If training, make sure that config.axial_pos_shape factors: (512, 1024) multiply to sequence length. Got prod((512, 1024)) != sequence_length: 192. You might want to consider padding your sequence length to 524288 or changing config.axial_pos_shape.

```

>If training, make sure that config.axial_pos_shape factors: (512, 1024) multiply to sequence length. Got prod((512, 1024)) != sequence_length: 384. You might want to consider padding your sequence length to 524288 or changing config.axial_pos_shape.

So I guess that is because its default set to  (512, 1024), and if so, how can I change it to a smaller value? 

ReformerConfig {
  "architectures": [
    "ReformerModelWithLMHead"
  ],
  "attention_head_size": 64,
  "attention_probs_dropout_prob": 0.1,
  "attn_layers": [
    "local",
    "lsh",
    "local",
    "lsh",
    "local",
    "lsh"
  ],
  "axial_norm_std": 1.0,
  "axial_pos_embds": true,
  "axial_pos_embds_dim": [
    64,
    192
  ],
  "axial_pos_shape": [
    512,
    1024
  ],
  "chunk_size_feed_forward": 0,
  "chunk_size_lm_head": 0,
  "eos_token_id": 2,
  "feed_forward_size": 512,
  "hash_seed": null,
  "hidden_act": "relu",
  "hidden_dropout_prob": 0.05,
  "hidden_size": 256,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "is_decoder": true,
  "layer_norm_eps": 1e-12,
  "local_attention_probs_dropout_prob": 0.05,
  "local_attn_chunk_length": 64,
  "local_num_chunks_after": 0,
  "local_num_chunks_before": 1,
  "lsh_attention_probs_dropout_prob": 0.0,
  "lsh_attn_chunk_length": 64,
  "lsh_num_chunks_after": 0,
  "lsh_num_chunks_before": 1,
  "max_position_embeddings": 524288,
  "model_type": "reformer",
  "num_attention_heads": 2,
  "num_buckets": [
    64,
    128
  ],
  "num_chunks_after": 0,
  "num_chunks_before": 1,
  "num_hashes": 1,
  "num_hidden_layers": 6,
  "output_past": true,
  "pad_token_id": 0,
  "task_specific_params": {
    "text-generation": {
      "do_sample": true,
      "max_length": 100
    }
  },
  "vocab_size": 320
}

Given above is the default configuration of the model before training/finetuning

> For "normal" language modeling it might make much more sense to start from the Reformer-wiken8 model and finetune it: https://huggingface.co/google/reformer-enwik8

Will try that too.

Thank you.



yeah the google/crime-and-punishment is not a good model for fine-tuning. It assumes you use a sequence length of > 500K tokens, which is not really reasonable.

> Ok, thanks. So the function _flatten_and_tokenize_ (in the notebook) shall be replaced by the _encode_
> function (in the enwik8 model card), am I following right?

exactly. You should be able to just enwik8 function I linked above. The enwik8 model has a maximum length of ~65K tokens, which is very long but very feasible for reformer.

> yeah the google/crime-and-punishment is not a good model for fine-tuning. It assumes you use a sequence length of > 500K tokens, which is not really reasonable.

Oh okay. Thank you very much for the clarification. Will try finetuning reformer-enwik8.

It would be awesome if you could upload your training script here - people seem very interested in it :-) 


 @patrickvonplaten, Sure, will do when everything is sorted.

> 
> 
> > Ok, thanks. So the function _flatten_and_tokenize_ (in the notebook) shall be replaced by the _encode_
> > function (in the enwik8 model card), am I following right?
> 
> exactly. You should be able to just enwik8 function I linked above. The enwik8 model has a maximum length of ~65K tokens, which is very long but very feasible for reformer.

From the notebook am struggling to adapt the DataCollator, how to define it properly in this context? 
Thanks

Someone effectivelly fine tune on enwiki8 pre trained model? using colab with P100 gpu i was not able to load model yet due to memory limitation

Unfortunately facing the same issue now.

Can you add a link to your notebook here @lucashueda ?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@lucashueda Did you manage to fine-tune enwiki8 pre trained model ? or other datasets ? Would you mind sharing your Colab? 

@epetros did you manage to perform the fine-tuning ? 

@patrickvonplaten any update on this? 

Os is there a notebook where we can pre-train this model from wiki or huge corpus ourselves and then fine-tune it to downstream tasks? 

Hi, I'm trying to fine-tune the ReformerModelWithLMHead (google/reformer-enwik8) for NER. I used the padding sequence length same as in the encode method (max_length = max([len(string) for string in list_of_strings])) along with attention_masks. And I got this error: 

**ValueError:** If training, make sure that config.axial_pos_shape factors: (128, 512) multiply to sequence length. Got prod((128, 512)) != sequence_length: 2248. You might want to consider padding your sequence length to 65536 or changing config.axial_pos_shape.

1) When I changed the sequence length to 65536, my colab session crashed by getting all the inputs of 65536 lengths.  
2) According to the second option(changing config.axial_pos_shape), I cannot change it. 

I would like to know, Is there any chance to change config.axial_pos_shape while fine-tuning the model? Or I'm missing something in encoding the input strings for reformer-enwik8? Are there any additional steps to forward the input to the model after encoding?

Thanks!