Multi class classification using Reformer Model
Author: as-stevens
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->
Hi All:

I am trying to implement a **multi-class classification** using the **ReformerModel, ReformerModelWithLMHead** But I don't see any API implementation for the same.
I have 10+ class of text data and wanted to use the pre-trained ReformerModel, ReformerModelWithLMHead to classify the text.

I see classes like RobertaForSequenceClassification have support for text classification but could not find any for Reformer.

Please, let me if it is implemented in the Reformer model or is it work in progress. I tried to find any implementation for the same but could not find any.

ps: I am referring to this paper https://web.stanford.edu/class/cs224n/reports/custom/report21.pdf
where they have implemented text classification using Reformer.
Thank you
Amit
<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Comments:
In the reformer projects there is only an card for QA task.
Nothing about reformerforsequenceclassification or other model heads.
@patrickvonplaten has assigned the tasks to himself.
Are there any more steps to do when I want to pretrain reformer on larger datasets using the notebook ?
Specifically we would like to train it on c4 dataset as described in the open tasks.

**@flozi00** thank you for the reply. So there is no way at this point that I can implement multi-class classification using Reformer? Or is there a workaround that I can use.

@as-stevens you could try to write the classification head yourself.
At the moment there is no ready to use solution given.

There are currently no pretrained weights for a bidirectional reformer, so adding a QA model extension is not a high priority at the moment. PRs with a clean implementation of ReformerForQA would be welcome :-) 

> @as-stevens you could try to write the classification head yourself.
> At the moment there is no ready to use solution given.

**@flozi00** could point some reference that I can use to write a classification head?

I have free computing capacities to train bidirectional reformer model on larger datasets, but no time to do so.
Any advice or ready to use scripts to do so ? @patrickvonplaten 

@flozi00 - this sounds nice! Let me come back to you on this. There is currently no script to do so, but I will think about it :-)

@patrickvonplaten we could talk about the details in private chat ?
The development should not pause cause missing resources

> > @as-stevens you could try to write the classification head yourself.
> > At the moment there is no ready to use solution given.
> 
> **@flozi00** could point some reference that I can use to write a classification head?

1. https://github.com/ThilinaRajapakse/simpletransformers/blob/master/simpletransformers/custom_models/models.py

2. https://github.com/ThilinaRajapakse/simpletransformers/tree/master/simpletransformers/classification/transformer_models

> > > @as-stevens you could try to write the classification head yourself.
> > > At the moment there is no ready to use solution given.
> > 
> > 
> > **@flozi00** could point some reference that I can use to write a classification head?
> 
> 1. https://github.com/ThilinaRajapakse/simpletransformers/blob/master/simpletransformers/custom_models/models.py
> 2. https://github.com/ThilinaRajapakse/simpletransformers/tree/master/simpletransformers/classification/transformer_models

** @flozi00 ** Thank you for sharing the above the links, let me go through the links and try to understand the flow/architecture. I am new to this and would appreciate any other pointers as well.

**@flozi00** Please let me know who do I  verify my changes. By creating the model on a classification data set or is there a way better way to get the code verified? I am new to this hence
some of my questions may seem basic.

Just open an pull request with your changes.
patrickvonplaten is the author of the implementation of reformer in this repository, I think he would have a look on it.

Training the classification model on an dataset would be a good proof that it is working.

@patrickvonplaten 
I have implemented the ReformerForSequenceClassification and ReformerForClassificationHead. I have taken RobertaForSequenceClassification and other classification head as a reference.
Further, I have not opened a pull request as I wanted to make sure that I have a working sample code before I raise a PR. Test before raising a PR. I am using the IMDB review dataset.

The link to the collab;
https://colab.research.google.com/drive/1KFsQxLqsMB6vBF4_bRmTFGhdGwkgx0zI?usp=sharing

The 3rd cell has all the code related to the reformer classification head.
Further, I am getting;
**AssertionError: If training, make sure that config.axial_pos_shape factors: (128, 512) multiply to sequence length. Got prod((128, 512)) != sequence_length: 2048. You might want to consider padding your sequence length to 65536 or changing config.axial_pos_shape**.

I looked at https://github.com/huggingface/transformers/issues/4565 but that looks like it is an LM model and could find any solution for the same.
Any thoughts/suggestions?

From the docs
```
In practice, the parameter config.axial_pos_embds_dim is set to list(d1,d2)(d1,d2) which sum has to be equal to config.hidden_size and config.axial_pos_shape is set to list(n1s,n2s)(ns1,ns2) and which product has to be equal to config.max_embedding_size which during training has to be equal to the sequence length of the input_ids.
```

The axial pos shape is new in reformer model.
The product of it values have to equal to the sequence length.
You could change the sequence length or set the axial pos shape to the right values. In this case it could be an list of (32,64)

> From the docs
> 
> ```
> In practice, the parameter config.axial_pos_embds_dim is set to list(d1,d2)(d1,d2) which sum has to be equal to config.hidden_size and config.axial_pos_shape is set to list(n1s,n2s)(ns1,ns2) and which product has to be equal to config.max_embedding_size which during training has to be equal to the sequence length of the input_ids.
> ```
> 
> The axial pos shape is new in reformer model.
> The product of it values have to equal to the sequence length.
> You could change the sequence length or set the axial pos shape to the right values. In this case it could be an list of (32,64)

When I try to change the axial position shape(assuming max sequence length to be 512), I get error

---> 12 model =  ReformerForSequenceClassification.from_pretrained('google/reformer-enwik8', num_labels = 2, axial_pos_shape= (16,32))

/usr/local/lib/python3.6/dist-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    751                 raise RuntimeError(
    752                     "Error(s) in loading state_dict for {}:\n\t{}".format(
--> 753                         model.__class__.__name__, "\n\t".join(error_msgs)
    754                     )
    755                 )

RuntimeError: Error(s) in loading state_dict for ReformerForSequenceClassification:
	**size mismatch for reformer.embeddings.position_embeddings.weights.0: copying a param with shape torch.Size([128, 1, 256]) from checkpoint, the shape in current model is torch.Size([16, 1, 256]).
	size mismatch for reformer.embeddings.position_embeddings.weights.1: copying a param with shape torch.Size([1, 512, 768]) from checkpoint, the shape in current model is torch.Size([1, 32, 768]).**


> > From the docs
> > ```
> > In practice, the parameter config.axial_pos_embds_dim is set to list(d1,d2)(d1,d2) which sum has to be equal to config.hidden_size and config.axial_pos_shape is set to list(n1s,n2s)(ns1,ns2) and which product has to be equal to config.max_embedding_size which during training has to be equal to the sequence length of the input_ids.
> > ```
> > 
> > 
> > The axial pos shape is new in reformer model.
> > The product of it values have to equal to the sequence length.
> > You could change the sequence length or set the axial pos shape to the right values. In this case it could be an list of (32,64)
> 
> When I try to change the axial position shape(assuming max sequence length to be 512), I get error
> 
> ---> 12 model = ReformerForSequenceClassification.from_pretrained('google/reformer-enwik8', num_labels = 2, axial_pos_shape= (16,32))
> 
> /usr/local/lib/python3.6/dist-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
> 751 raise RuntimeError(
> 752 "Error(s) in loading state_dict for {}:\n\t{}".format(
> --> 753 model.**class**.**name**, "\n\t".join(error_msgs)
> 754 )
> 755 )
> 
> RuntimeError: Error(s) in loading state_dict for ReformerForSequenceClassification:
> **size mismatch for reformer.embeddings.position_embeddings.weights.0: copying a param with shape torch.Size([128, 1, 256]) from checkpoint, the shape in current model is torch.Size([16, 1, 256]). size mismatch for reformer.embeddings.position_embeddings.weights.1: copying a param with shape torch.Size([1, 512, 768]) from checkpoint, the shape in current model is torch.Size([1, 32, 768]).**

@flozi00 any idea what may be wrong?

Sorry, I have pretty much to do at the moment.
I will have a look on it, but in worst case it take time up to Sunday evening in MEZ Timezone.
Maybe someone else can answer you earlier

> Sorry, I have pretty much to do at the moment.
> I will have a look on it, but in worst case it take time up to Sunday evening in MEZ Timezone.
> Maybe someone else can answer you earlier

Thank you so much for the quick response! I appreciate it. I am just wondering how else could I get some one to throw light on this issue.


Just link someone from huggingface team to this as done with patrickvonplaten earlier.
in my experience the team is very nice and helpful all the time.
Maybe you should open an PR with your code and write [WIP] in front of it, so it gets better seen than an issue and you could get faster help by more people

> Just link someone from huggingface team to this as done with patrickvonplaten earlier.
> in my experience the team is very nice and helpful all the time.
> Maybe you should open an PR with your code and write [WIP] in front of it, so it gets better seen than an issue and you could get faster help by more people

Makes sense, Thank you for the advice!

Hey, sorry.
I still found no time to have a look on your colab notebook.
Did you opened the pull request?

> Hey, sorry.
> I still found no time to have a look on your colab notebook.
> Did you opened the pull request?

@flozi00 I created a pull request. https://github.com/huggingface/transformers/pull/5198
Please have a look at it and provide your feedback/suggestions.

Hey, just have seen that you already got feedback.
I still didn't had time to run your code cause my calendar is very full until Friday, sorry

> Hey, just have seen that you already got feedback.
> I still didn't had time to run your code cause my calendar is very full until Friday, sorry

Hey, I got caught up with work, could not reply earlier!

I received the feedback and I have implemented the initial feedback as well. Trying to implement the test case for classification head, it is taking time as I need to understand the underlying test framework and also the architecture of the test cases.
Further, as the implementation of the classification head, did not had much review comment I decided to test the changes on the IMDB Dataset, but I have not been successful! I am getting CUDA error! link to the notebook;

https://colab.research.google.com/drive/1KFsQxLqsMB6vBF4_bRmTFGhdGwkgx0zI#scrollTo=vOyStELCX8VA&uniqifier=2

https://discuss.pytorch.org/t/runtimeerror-cuda-error-cublas-status-alloc-failed-when-calling-cublascreate-handle/78545/6

There are similar issues here.
It seems like there is something out of bounds / out of index range

> https://discuss.pytorch.org/t/runtimeerror-cuda-error-cublas-status-alloc-failed-when-calling-cublascreate-handle/78545/6
> 
> There are similar issues here.
> It seems like there is something out of bounds / out of index range

@flozi00 
Though the issue was different, I was able to solve the issue, and finally, the classification head is working, Thanks

Further, I am trying to play with the sequence length parameter but the model throws an error;
posted a separate issue https://github.com/huggingface/transformers/issues/5320 for the same. Please let me know if you have an idea about this one.

ps: I will update the notebook once I am done with the sequence length setting.

Any Idea how I can get sentence representation using Reformer Model? ( 1, 1024 ) shape using reformer-enwik8?

Thanks!

you can just use the output of `ReformerModel` no? 

@patrickvonplaten  It's returning sentences generation output instead of vector?

@patrickvonplaten 
This post is a little longer, I appreciate your time and sorry for the long post! But I have been trying to make the classifier work and hence could help myself.

I have implemented a classification model using the Plain Reformer model,
Link to collab notebook.
[https://colab.research.google.com/drive/1l61NccWTGMfNFPj1T8kvMjnik2iEQWfe?usp=sharing](url)
I have used the pre-trained crime and punishment (CP) tokenizer sequence tokenization. But, I am not able to improve the accuracy of the model form **~50%**, Which is equal to random classification as it is a binary classifier. I tried to play around with the learning rate, batch size, epochs, and sequence length but it does not help. 

I have implemented a classifier using Roberta and that seems to work fine, giving me an accuracy of ~94%. 
[https://colab.research.google.com/drive/10vv8YgwJzbKDpd0Q-pXupP86b1pOZJg8?usp=sharing](url)
So, I started comparing the difference between the two.

- For Reformer, I am not able to use any existing pre-trained model for fine-tuning unlike Roberta.
- The output of the tokenizer.tokenize(<sentence>) is also much different in both the cases. I mean in the case of Roberta the sentence get tokenized more or less in words, while in case of Reformer the sentence is mostly broken down in characters except for very common words like the, and, it ..etc

`Roberta output: '
', 'ĠOne', 'Ġof', 'Ġthe', 'Ġother', 'Ġreviewers', 'Ġhas', 
Reformer output: '▁', '<', 's', '>', '▁', 'O', 'n', 'e', '▁of', '▁the', '▁o', 'ther', '▁re', 'v', 'i', 'e', 'w', 'er',
`

So, I am wondering if the CP tokenizer still needs to be trained on larger data.
I tried to use the pre-trained XL tokenizer as that is also a sentence piece tokenization but that stated giving me memory issues. Can, I use a different classifier, I read in one of the blogs that tokenizer is a by-product of the model training. Hence, it is tried to the pre-trained model.

Are there any pre-trained weights available for Reformer to be used to fine-tune for classification. Like it is there for other models, If not; do we have anything planned for it or not?

Thanks
Amit




Hey @as-stevens,

Yeah Reformer sadly still does not have a bi-directional pretrained model, so that it will be hard to fine-tune a Reformer Model for your task. The `enwik-8` model was trained using the CLM objective as a char-lm, which means that it does not really help at all for classification. Also the model was trained on chars and not tokens....over all I would recommend just using RoBERTa for now.

I hope that the original authors will open-source pretrained weights at some point. We are also discussing internally whether it makes sense to pre-train Reformer ourselves.

@patrickvonplaten  thank you for addressing all the queries! This helps.

After playing with the different tokenizers(Xl_net, Bert and Roberta), I could sense that the Reformer tokenizer is a char level and not a word level. And after playing with the different hyper parameter and no change is model state, it looked like the model is training from scratch.

Thanks for answering all the question.
Any ETA for the pre-trained model?

@patrickvonplaten do you have any updates on this ? " We are also discussing internally whether it makes sense to pre-train Reformer ourselves."
Thank you!

Hey @mihaidobri , Reformer has a `ReformerForMaskedLM` that can be used for pretraining :-)

@as-stevens, @patrickvonplaten  
I've improved the performance by classifying the average of token representations instead of classifying the < s >  token. 
You only need to change this line  of code  

>   x = features[:, 0, :]  # take < s > token (equiv. to [CLS])

  in the ReformerClassificationHead class to this  

> x = torch.mean(features, dim=1)



@shramezani Would it be possible to share the notebook for reference. Also, what accuracy you achieved for the classification task?

This is a link to https://colab.research.google.com/drive/1kTMNT8-UprrWEb-Vpjji4Qbs8gURZiFs?usp=sharing
I changed the representation and got an accuracy score of 0.69.

@shramezani  the notebook link is not accessible, permission denied, plz check.