noamaanMulla-03/transformers-issues huggingface dataset:
train loss is not decreasing using TFBertModel
Author: ZahraAbbasiantaeb
Pull request: no

I have used the TFBertModel and AutoModel from the transformer library for training a two-class classification task and the training loss is not decreasing. 

`bert = TFBertModel.from_pretrained('bert-base-uncased')`
`input_ids = tf.keras.layers.Input(shape=(SEQ_LEN,), name='input_ids', dtype='int32')`
`mask = tf.keras.layers.Input(shape=(SEQ_LEN,), name='attention_mask', dtype='int32')`
`embeddings = bert(input_ids, attention_mask=mask)[1]`
`X = tf.keras.layers.Dropout(0.1)(embeddings)`
`X = tf.keras.layers.Dense(128, activation='relu')(X)`
`y = tf.keras.layers.Dense(1, activation='sigmoid', name='outputs')(X)`
`bert_model = tf.keras.Model(inputs=[input_ids, mask], outputs=y)`
`bert_model.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['accuracy'])`



But when I use the TFBertForSequenceClassification model the model converges fast and the training loss reaches zero.

`bert_model = TFBertForSequenceClassification.from_pretrained('bert-base-uncased',num_labels=2)`
`loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)`
`metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')`
`optimizer = tf.keras.optimizers.Adam(learning_rate=2e-5,epsilon=1e-08)`
`bert_model.compile(loss=loss, optimizer=optimizer, metrics=[metric])`

I want to use the sequence output of BERT and hence I need to load the model with TFBertModel or something similar which returns the outputs of BERT.

@Rocketknight1 

Comments:
Hi,

For training-related questions, please refer to the [forum](https://discuss.huggingface.co/). We like to keep Github issues for bugs/feature requests.

Closing this for now.

Thank you!
[tokenizer] use use_auth_token for config
Author: stas00
Pull request: yes

This PR forwards `use_auth_token` to `AutoConfig.from_pretrained()` when `AutoTokenizer.from_pretrained(mname, use_auth_token=True)` is used. Otherwise `AutoConfig` fails to retrieve `config.json` with 404.

Model's `from_pretrained` does the right thing.

@sgugger, @LysandreJik 

Comments:
Layoutlm onnx support (Issue #13300)
Author: nishprabhu
Pull request: yes

# What does this PR do?

This PR extends ONNX support to LayoutLM as explained in https://huggingface.co/transformers/serialization.html?highlight=onnx#converting-an-onnx-model-using-the-transformers-onnx-package

Fixes Issue #13300


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [X] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@mfuntowicz 


Comments:
Push to hub when saving checkpoints
Author: sgugger
Pull request: yes

# What does this PR do?

This PR enables automatic checkpointing on the Hub by introducing a new training argument called `hub_strategy`. This argument can take several values:

- `"end"`: push the model, its configuration, the tokenizer (if passed along to the `Trainer`) and a draft of a model card at the end of training.
- `"every_save"`: push the model, its configuration, the tokenizer (if passed along to the`Trainer`) and a draft of a model card each time there is a model save. The pushes are asynchronous to not block training, and in case the save are very frequent, a new push is only attempted if the previous one is finished. A last push is made with the final model at the end of training.
- `"checkpoint"`: like `"every_save"` but the latest checkpoint is also pushed in a subfolder named last-checkpoint, allowing a user to resume training easily with `trainer.train(resume_from_checkpoint="last-checkpoint")`. The final repository won't have that checkpoint folder, but you can find it in the second-to-last commit.
- `"all_checkpoints"`: like `"checkpoint"` but all checkpoints are pushed like they appear in the output folder (so you will get one checkpoint folder per checkpoint in your final repository)

The default value is `"every_save"`, which slightly changes the behavior when a user passes along `--push_to_hub`.

This PR relies on [this PR](https://github.com/huggingface/huggingface_hub/pull/315) on the huggingface_hub part, so checking out that branch is necessary for testing it. Tests won't pass before a new release of huggingface_hub with this PR is out.

cc @julien-c 

Comments:
How can I convert a `checkpoint.pth` (a model trained with pytorch-pretrained-bert) to huggingface model with `config.json` and `pytorch_model.bin` file?
Author: AmitChaulwar
Pull request: no
Labels: name: Migration

# 📚 Migration

## Information

<!-- Important information -->

Model I am using (Bert, XLNet ...):

Language I am using the model on (English, Chinese ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## Details

<!-- A clear and concise description of the migration issue.
    If you have code snippets, please provide it here as well.
    Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
    Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.
    -->

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
 
- `transformers` version:
- Platform:
- Python version:
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

<!-- IMPORTANT: which version of the former library do you use? -->
* `pytorch-transformers` or `pytorch-pretrained-bert` version (or branch):


## Checklist

- [x] I have read the migration guide in the readme.
 ([pytorch-transformers](https://github.com/huggingface/transformers#migrating-from-pytorch-transformers-to-transformers);
  [pytorch-pretrained-bert](https://github.com/huggingface/transformers#migrating-from-pytorch-pretrained-bert-to-transformers))
- [ ] I checked if a related official extension example runs on my machine.


Comments:
Hello! I believe that `pytorch_pretrained_BERT` followed the same approach of having a pytorch checkpoint and a configuration. Do you have no configuration accompanying your `checkpoint.pth`? What is contained in that file once you load it with `torch.load`?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Generate text until condition
Author: StellaAthena
Pull request: no
Labels: name: WIP

Is there a simple way to have the model generate text until a condition is met? I'm interested in data memorization and want to prompt the model with some tokens from the training data and then have it generate text until it makes a mistake (aka deviates from the training data). The naive approach with a while loop has significant overhead, and I was wondering if there was something smarter I can be doing.

Comments:
Maybe of interest to @patrickvonplaten @patil-suraj 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Related PR: https://github.com/huggingface/transformers/pull/12219
[Flax] Addition of FlaxPegasus
Author: bhadreshpsavani
Pull request: yes

# What does this PR do?

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. [link of PR](https://github.com/huggingface/transformers/pull/12402)
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?
@patrickvonplaten
@patil-suraj

Comments:
Hi @patil-suraj  and @patrickvonplaten,
I was not able to figure out how to add `PegasusSinusoidalPositionalEmbedding` in the Flax Version and `QuestionAnswering` and `Classification` classes are not added yet since the original torch version don't have the classes. Shall we add it?
Please let me know your review on this PR. 

You could find the flax version of `SinusoidalPositionalEmbedding` in the `FlaxMarian`

https://github.com/huggingface/transformers/blob/76c4d8bf26de3e4ab23b8afeed68479c2bbd9cbd/src/transformers/models/marian/modeling_flax_marian.py#L752

Also, Pegasus isn't really intended for QA and classification so it's okay to not add those heads yet.

Thanks a lot for more or less completing the PR - great job @bhadreshpsavani !

It seems like there are some small differences between the PyTorch & Flax Model. This could be due to slightly different activation functions or small differences with the position ids.... 

It would be awesome if you could try to debug layer by layer what might be the problem there @bhadreshpsavani 

Another possibility is that there is no difference and it's just the framework that causes the difference. In this case, we'll just have to accept it and change the tolerance.

Sure @patrickvonplaten,
I will compare the code and debug it :)

Hi @patil-suraj and @patrickvonplaten,
Thanks for the review and suggestions. Please let me know if anything missing in the PR.

Thanks a lot for fixing the issues, looks good now. If you could give me access to this branch I would like to update the slow tests.

Done!
Please go ahead with the fix for slow tests.
Once this is merged, I will create another PR for that Typo Fix in BART and PEGASUS that I come across
Add checks to build cleaner model cards
Author: sgugger
Pull request: yes

# What does this PR do?

To avoid model cards being rejected by the metadata validation on the Hub, this PR cleans up a few things:
- removing None values from lists so that a [None] passed for a field is ignored (this fixes #13528 )
- ignoring results that don't have the three keys task, datasat and metrics since those three are mandatory.



Comments:
separate model card git push from the rest
Author: elishowk
Pull request: yes
Labels: name: model card

name: work in progress

name: trainer

# What does this PR do?

- After model card metadata contents validation was deployed to the Hub, we need to ensure transformer's trainer git push are not blocked because of an invalid README.mld yaml.
- as discussed with @julien-c @Pierrci @sgugger and @LysandreJik the first step to match Hub's model card validation system is to avoid failing a whole git push after training, for the only reason that README.md metadata is not valid.
- therefore, I tried in this PR to git push the training result independently from the modelcard update, so that the modelcard update failing does not fail the rest, keeping only logging for README.Md push failures.

- Relates to https://github.com/huggingface/huggingface_hub/pull/326

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?



Comments:
For tracking purposes, do we have another issue for `datasets: - null` @elishowk? (which will still fail with this, AFAICT)

> For tracking purposes, do we have another issue for `datasets: - null` @elishowk? (which will still fail with this, AFAICT)

Yep : https://github.com/huggingface/transformers/issues/13528

🥳 first PR on transformers, thanks for your help you all !
QUESTION: How to perform 2D interpolation of pre-trained position embeddings for fine-tuning on Vision Transformers?
Author: clrkwng
Pull request: no

Hi all,

I know that in examples for fine-tuning on the pre-trained ViT, there is a ViTFeatureExtractor that can upsample images from CIFAR10 -> 224x224 to match the image resolution that that model is pre-trained on. However in the paper, the authors suggest to perform 2D Interpolation on the pre-trained position embeddings. I saw here in the TIMM source code (https://github.com/rwightman/pytorch-image-models/blob/54e90e82a5a6367d468e4f6dd5982715e4e20a72/timm/models/vision_transformer.py#L481) there is a method to handle this interpolation. Is there a similar method / parameter that I can use in the HuggingFace ViT to enable this interpolation behavior, in lieu of upsampling the CIFAR10 32x32 to match 224x224?

Comments:
Hi,

See #12167
Loading custom model
Author: saksham-s
Pull request: no

I had changed the definition of a token classification model and added another output head. Now when I try to load the model using AutoModelForTokenClassification it does not load the weights of the modified final layer which I had added. Is there any other class I can use to load this custom model.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Same. I run into several problems. I figured that I would need to call something like `CustomModel.from_pretrained("PATH_TO_CHECKPOINT")` to load the model. But that did not work for me either. If I find a solution I will give an update.
[Flax] Fixes typo in Bart based Flax Models
Author: bhadreshpsavani
Pull request: yes

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@patil-suraj @patrickvonplaten 

Comments:
3 sentences as input for BertForSequenceClassification?
Author: mikelkl
Pull request: no

Hi there,

Thanks for releasing this awesome repo, it does lots people like me a great favor.

So far I've tried sentence-pair BertForSequenceClassification task, and it indeed work. I'd like to know if it is possible to use BertForSequenceClassification to model triple sentences classification problem and its input can be described as below:

    **[CLS]A[SEP]B[SEP]C[SEP]**

Expecting for your reply!

Thanks & Regards

Comments:
Technically it is possible but BERT was not pretrained to handle multiple SEP tokens between sentences and does not have a third token_type, so I think it won't be easy to make it work. You may also want to use a new token for the second separation.

> Technically it is possible but BERT was not pretrained to handle multiple SEP tokens between sentences and does not have a third token_type, so I think it won't be easy to make it work. You may also want to use a new token for the second separation.

Hi artemisart,

Thanks for your reply.

So, if someone wanna take multiple sentences as input of BertForSequenceClassification, let's say a whole passage, an alternative way is to concatenate them into a single "sentence" and then fit it in, right?

I you don't have a separation (like question/answer) then yes you can just concatenate them (but you are still limited to 512 tokens).

@mikelkl I would also go with the solution and answer of @artemisart.

@artemisart  hi, if i have a single sentence classification task, should the max length of sentence limited to half of 512,  that is to say 256?

No, it will be better if you use the full 512 tokens.

wouldn't concatenating the whole passage into a single sentence mean losing context of each sentence? @artemisart 

No it shouldn't 

What if I want to check on a huge corpus, that even concatenating into one sentence exceeds the 512 token limit? @artemisart

@thedrowsywinger maybe u should try Transformer-XL

> I you don't have a separation (like question/answer) then yes you can just concatenate them (but you are still limited to 512 tokens).

I have 3 inputs, 1 of the input contains conversation (QUERY, ANSWER). 
QUERY: I want to ask a question.

> ANSWER: Sure, ask away.
> QUERY: How is the weather today?
> ANSWER: It is nice and sunny.
> QUERY: Okay, nice to know.
> ANSWER: Would you like to know anything else?

How can I tell the model to separate the turns of conversation? Model is classification model.
I was thinking to add a new special token <EOT> between the turns but could not get it work.
Fix GPTNeo onnx export
Author: patil-suraj
Pull request: yes

# What does this PR do?

#13491 simplified the GPTNeo's local attention by removing the complicated operations, as a result, the `GPTNeoAttentionMixin` class is no longer needed and was removed. This breaks `GPTNeoOnnxConfig` as it was importing the `GPTNeoAttentionMixin` class.

This PR (tries to!) fixes `GPTNeoOnnxConfig` by removing the `PatchingSpec` and updates the `input`, `output` accordingly.

I'm an onnx noob so it would be awesome if you could please take a deeper look @michaelbenayoun :) 

Thanks a lot, @tianleiwu for reporting the bug!

Comments:
@patil-suraj, thanks for providing a quick fix. 

For dynamic axes mapping for inputs and outputs, I suggest the following:
```
"input_ids": {0: "batch", 1: "sequence"}
f"past_key_values.{i}.key" = {0: "batch", 2: "past_sequence"}
f"present.{i}.key" : {0: "batch", 2: "present_sequence"}      #or {0: "batch", 2: "past_sequence+sequence"}
"attention_mask" : {0: "batch", 1: "present_sequence"}    #or {0: "batch", 1: "past_sequence+sequence"}
```




I have updated the onnx config input and output names to support dynamic axes as @tianleiwu suggested.
The simplification brought by #13491 along with this PR also solves the issue #13175.

@sgugger @LysandreJik I have left the custom implementations `custom_unfold` and `custom_get_block_length_and_num_blocks` which are no longer needed thanks to #13491: should we keep them there as they might be useful one day for other models or get rid of them to keep things clean?

I think we can keep those two functions for now. We can look again in a few months and remove them if they are still not used anywhere.
sentencepiece version need upgrade
Author: elishowk
Pull request: no

## Environment info

- `transformers` version: 4.11.0.dev0
- Platform: Debian Gnu/Linux
- Python version: 3.9
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help

- setup: @LysandreJik may be ?

## Information

sentencepiece==0.1.91 does not exist anymore

pip install fails:

```
ERROR: Could not find a version that satisfies the requirement sentencepiece==0.1.91
ERROR: No matching distribution found for sentencepiece==0.1.91
```

## To reproduce

Steps to reproduce the behavior:

1. create a new venv
2. pip install -e .


## Expected behavior

- Need to bump to the latest but minor version change 0.1.96 for transformer's install to work


Comments:
It just worked fine on a new env. Maybe you need a `pip install --upgrade pip`?

> It just worked fine on a new env. Maybe you need a `pip install --upgrade pip`?

Yeah indeed, it worked for me too.
Trainer error when finetuning Pegasus - cannot import name 'amp' from 'apex' 
Author: Oxi84
Pull request: no

I get this error when i try to finetune Pegasus.

`ImportError: cannot import name 'amp' from 'apex' (unknown location) `

Is it possible to pun trainer without apex installed? I tried to specify, fp16=False in agruments but i still get this error.

the finetuning code is here  - https://gist.github.com/jiahao87/50cec29725824da7ff6dd9314b53c4b3

Thanks



Comments:
I disabled it in trainer.py and now it works, but I do get memory error unless i freeze encoder.
if is_apex_available():
	pass
    #from apex import amp