Model not learning when using albert-base-v2 -- ALBERT
Author: trevorpfiz
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): AlbertForSequenceClassification

Language I am using the model on (English, Chinese....): English

The problem arise when using:
When I use `albert-base-v2` instead of `albert-base-v1` for the model and tokenizer, the model does not learn during training.

The tasks I am working on is:
https://colab.research.google.com/drive/1Y4o3jh3ZH70tl6mCd76vz_IxX23biCPP

## To Reproduce

Steps to reproduce the behavior:

1. Open the colab notebook I have referenced above
2. Change the word Bert to Albert in necessary places
3. Run 4.3. Training Loop, you will see that the model does not learn

## Expected behavior

I would expect the model to learn on the given task.

## Environment

Colab

## Additional context

I can not pinpoint exactly what the problem is, but it almost seems like the data that is being fed to the model is not understood.  If I use the AlbertTokenizer for a BertForSequenceClassification model, which the BERT model would obviously not understand, the same behavior is exhibited.  


Comments:
The interesting things is that why do u use another model’s tokenizer to
process data .

Do u know each model tokenizer is a map function which map ID to token ?
So for each model , the same word for them is mapping to different iD . So
it will not learning . Do u read paper ?

On Fri, Jan 17, 2020 at 03:19 Trevor Pfizenmaier <notifications@github.com>
wrote:

> 🐛 Bug
>
> Model I am using (Bert, XLNet....): AlbertForSequenceClassification
>
> Language I am using the model on (English, Chinese....): English
>
> The problem arise when using:
> When I use albert-base-v2 instead of albert-base-v1 for the model and
> tokenizer, the model does not learn during training.
>
> The tasks I am working on is:
> https://colab.research.google.com/drive/1Y4o3jh3ZH70tl6mCd76vz_IxX23biCPP
> To Reproduce
>
> Steps to reproduce the behavior:
>
>    1. Open the colab notebook I have referenced above
>    2. Change the word Bert to Albert in necessary places
>    3. Run 4.3. Training Loop, you will see that the model does not learn
>
> Expected behavior
>
> I would expect the model to learn on the given task.
> Environment
>
> Colab
> Additional context
>
> I can not pinpoint exactly what the problem is, but it almost seems like
> the data that is being fed to the model is not understood. If I use the
> AlbertTokenizer for a BertForSequenceClassification model, which the BERT
> model would obviously not understand, the same behavior is exhibited.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/2553?email_source=notifications&email_token=AIEAE4DURWPRQ22RW6PWX5DQ6CXMXA5CNFSM4KHZJIJ2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IGXUZ2Q>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AIEAE4DTEA5RJ36IGEJQVGTQ6CXMXANCNFSM4KHZJIJQ>
> .
>


I know that you can not use a different tokenizer, as I said "which the BERT model would obviously not understand".  I did this to extend my understanding of the problem, which yields the exact same behavior (loss does not go down/accuracy does not go down/model does not learn) as using `albert-base-v2` + `AlbertTokenizer` with `albert-base-v2` + `AlbertForSequenceClassification`.

I'm getting similar bad results. seems like ALBERT v2 isn't converging on the hyperparameters published in the original paper.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Has anyone figured this out? I've attempted the same thing (in tensorflow) by swapping out bert model & tokenizer for Albert V1 and V2 and no learning is done. Bert works just fine, but Albert is a no-go.


@1337-Pete, current version of ALBERT V1 and V2 work well but are very sensitive to the training data and hyperparameters. If you use the hyperparameters from the paper you will get similar results for both models on all GLUE tasks.