Reformer model axial.position.shape config not working
Author: as-stevens
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->
I am trying to play around with the config parameters of the Reformer model;
I change the axial position shape parameters to ;
"axial_pos_shape": [
    64,
    256
  ]
and the "max_position_embeddings": 8192 (multiplication of axial_pos_shape)

But it get the following error;
**RuntimeError: Error(s) in loading state_dict for ReformerForSequenceClassification:
	size mismatch for reformer.embeddings.position_embeddings.weights.0: copying a param with shape torch.Size([512, 1, 64]) from checkpoint, the shape in current model is torch.Size([64, 1, 64]).
	size mismatch for reformer.embeddings.position_embeddings.weights.1: copying a param with shape torch.Size([1, 1024, 192]) from checkpoint, the shape in current model is torch.Size([1, 256, 192]).**

The rest of parameters are unchanged. What's wrong with the config?
## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Comments:
Either my calculator and me are wrong or the max pos embedding is not matching the axial pos shape

![Screenshot_20200626_211633_com.android.calculator2.jpg](https://user-images.githubusercontent.com/47894090/85893082-715b5100-b7f2-11ea-9d85-fd341fcc5340.jpg)

> Either my calculator and me are wrong or the max pos embedding is not matching the axial pos shape
> 
> ![Screenshot_20200626_211633_com.android.calculator2.jpg](https://user-images.githubusercontent.com/47894090/85893082-715b5100-b7f2-11ea-9d85-fd341fcc5340.jpg)

Definitely your calculator looks like in learning phase! :)

I am wondering if I load the models;

model = ReformerForSequenceClassification.from_pretrained("./cnp", num_labels = 2, output_attentions = False,  output_hidden_states = False,)
I assume the config will not be overridden, I mean it will complain about the config not being default.

Not sure but, I am trying;
model = ReformerForSequenceClassification(ReformerConfig())
model.load_state_dict(torch.load("./cnp"))

which will load the custom model.



Could you upload the trained model ?
Then it's easier to reproduce the error and play around with the code

> Could you upload the trained model ?
> Then it's easier to reproduce the error and play around with the code

The model is trained on a very few samples (~100) and in the office network, which unfortunately does not allow to upload to outside world.

I will try to upload one model after training it on collab.


config = ReformerConfig()
config.max_position_embeddings = 8192
config.axial_pos_shape=[64, 128]

model = ReformerForSequenceClassification(config)
model.load_state_dict(torch.load("./cnp/pytorch_model.bin"))

This is how I am trying to load the config, I tried an other approach but that did not work as well.


@as-stevens are you training a Reformer Model from scratch?

Regarding your code:

```python
config = ReformerConfig()
config.max_position_embeddings = 8192
config.axial_pos_shape=[64, 128]

model = ReformerForSequenceClassification(config)
model.load_state_dict(torch.load("./cnp/pytorch_model.bin"))
```

please note that it is not recommended instantiating a config and then later changing already set attributes. It's better to do the following:
```
config = ReformerConfig(max_position_embeddings=8192, axial_pos_shape=[64,128])
```

This code for example works:
```python 
from transformers import ReformerModel, ReformerConfig
config = ReformerConfig(max_position_embeddings=8192, axial_pos_shape=[64, 128])
model = ReformerModel(config)
model.eval()

input_ids = torch.tensor([10 * [2, 4]])
model(input_ids)
```

If you load a model from your checkpoint: "./cnp/pytorch_model.bin" the axial position embedding weights have to have the same dimensions in order for this to work

Closing this for now. Feel free to re-open if you encounter other problems. Also make sure to have read the corresponding documentation of Reformer: https://huggingface.co/transformers/model_doc/reformer.html#axial-positional-encodings .

@patrickvonplaten  Thank you! Let me try your suggestion and run the classifier. Update you with the findings.