noamaanMulla-03/transformers-issues huggingface dataset:
Layoutlm onnx support
Author: nishprabhu
Pull request: yes

# What does this PR do?

This PR extends ONNX support to LayoutLM as explained in https://huggingface.co/transformers/serialization.html?highlight=onnx#converting-an-onnx-model-using-the-transformers-onnx-package

Fixes Issue  #13300


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [X] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@mfuntowicz @NielsRogge 


Comments:
@nishprabhu would you mind closing this PR and opening a new one, without any additional changes?

Github has a weird issue making the diff almost impossible to review.

Sorry for the inconvenience, please let us know if you need any help.

Sure, @mfuntowicz 
I'll open a new PR with the changes.
Tests fetcher tests
Author: sgugger
Pull request: yes

# What does this PR do?

If you received a notification for this PR and are not a reviewer, I apologize: I clicked the button create PR to early :grimacing: 

This PR fixes the tests_fetcher utils for the test files dependencies: currently modifying `tests_modeling_common.py` won't trigger any other tests than `tests_modeling_common.py`  when we would like it to run all the modeling tests. To fix this, the same logic as in the modules is applied to the tests files: they are screened for dependencies to other tests files and this is all added before we compute the reverse dependency map.

As an example this is properly working, this PR has a diff in `tests_modeling_common.py`  and you can check the triggered tests [here](https://circle-production-customer-artifacts.s3.amazonaws.com/picard/5bdabdd888af1f000130874a/612d399a732e644e043dbe7e-0-build/artifacts/~/transformers/test_preparation.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20210830T200953Z&X-Amz-SignedHeaders=host&X-Amz-Expires=59&X-Amz-Credential=AKIAJR3Q6CR467H7Z55A%2F20210830%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=7b5b35edd60b676c46bc3bd0715fadd1089ed4d44f68c2bf987bbcd905faef3d).

Comments:
Will be working on tests in a few hours and would like to test this out - will merge this into `master` and check everything runs smoothly.
Missing on_predict event in TrainerCallback
Author: rpowalski
Pull request: no

# 🚀 Feature request

Can we add `on_predict` event support in the `Trainer` and `TrainerCallback`?

## Motivation

I was in need to use it already in multiple projects. I think it makes sense since `Trainer` already support `on_evaluate` event inside `evaluate()` method. Corresponding event handler is missing in `predict()` method which is part of the `Trainer` class. Some of the training support libraries are supporting such events, so I guess there are no strong reasons against that ([link](https://pytorch-lightning.readthedocs.io/en/latest/extensions/generated/pytorch_lightning.callbacks.Callback.html#pytorch_lightning.callbacks.Callback.on_test_end)).

## Your contribution

I am willing to make a PR that will implement that.


Comments:
cc @sgugger 

The reason why there is no `on_predict` event is that the `predict` method is never called during the training loop. You can add the code you want to be run after `Trainer.predict` just after calling that method.

Ok, I see. For my cases having this custom code in the callback feels cleaner and more consistent with the post-processing done during `evaluate()` method, but will do it as you suggest.
TF/Numpy variants for all DataCollator classes
Author: Rocketknight1
Pull request: yes

This is a draft PR again - I've written an example of what a TF variant of one of our data collators would look like. If we're happy with this format, it should be easy to expand it to support Numpy/JAX as well, and to do the same for other data collators, and I'll probably add most of the other data collators to this PR before merging it. Let me know what you think!

Comments:
More updates done - please note that tests will fail until all of the data collators are updated, because I removed the top-level imports. I definitely won't be merging this until that's done, don't worry!

All the classes are in! Thank you to @aromans and @sdwalker62, whose PR #12199 I cannibalized for MLM and its variants. Next step is finishing tests and making sure all of this actually works.

Hi @aromans and @sdwalker62, we're ready to merge now. I just realized I'll need your Github no-reply e-mail addresses to add you though - see the docs [here](https://docs.github.com/en/github/committing-changes-to-your-project/creating-and-editing-commits/creating-a-commit-with-multiple-authors#required-co-author-information). 

dalton_walker@icloud.com

Thanks!

andrew.romans@hotmail.com

It's in, and all authors have been properly credited! If you want to delete the messages with your e-mails (in case of spambot harvesting), feel free.
Add generate kwargs to Seq2SeqTrainingArguments
Author: sgugger
Pull request: yes

# What does this PR do?

This PR adds two new `Seq2SeqTrainingArguments` to control which `max_length` and `num_beams` are used during the intermediate evaluations of the `Seq2SeqTrainer`. This feature has been requested multiple times, the last in date being #13252.

Comments:
🌟 New model addition - GPT-J-6B
Author: Xirider
Pull request: no
Labels: name: New model

# 🌟 New model addition - GPT-J-6B

## Model description

The GPT-J-6B model (GPT-NEO model in Jax with 6B parameters trained on the Pile)

Repo: https://github.com/kingoflolz/mesh-transformer-jax

Weights:
[Slim weights (bf16 weights only, for inference, 9GB)](https://the-eye.eu/public/AI/GPT-J-6B/step_383500_slim.tar.zstd)
[Full weights (including optimizer params, 61GB)](https://the-eye.eu/public/AI/GPT-J-6B/step_383500.tar.zstd)

## Open source status

* [x] the model implementation is available: (give details)
* [x] the model weights are available: (give details)
* [ ] who are the authors: (mention them, if possible by @gh-username)


Comments:
Hello @patrickvonplaten! would you be able to give an estimation for the timeline of the implementation of the this model in huggingface? 

I have a PR adding support for this model here: #12098

> I have a PR adding support for this model here: #12098

You probably wanted to link this PR: https://github.com/huggingface/transformers/pull/12106

Yeah, copied the wrong thing somehow.

@finetuneanon great! Do you know when it will be ready to use from the transformer library? Thnx for the work. 

Depends on when it will be merged. Until then you can install my branch like this:

```
pip install git+https://github.com/finetuneanon/transformers@gpt-j
```

Convert the weights with the conversion script linked from the PR.

@finetuneanon I did pip install the transformers@gpt-j and I managed to convert the weights through the script you referenced but only thing I'm now struggling with is making the config file. I uploaded the gpt-j-6b.json file to colab but I don't how to make the config variable via AutoConfig class(don't know if that is how it is made). So If you could let me know how to make the config file, I would appreciate it a lot. 
this [colab](https://colab.research.google.com/drive/1xl5tRYTiVISn6FMfhgyB70LZ-Biwj43E#scrollTo=QI_zE5QA8ycF) file containts all the code. 

Rename it into config.json, put it into a folder and you should be able to `AutoConfig.from_pretrained("whatever-folder")`
Padding labels is wrong when using `pad_to_multiple_of`
Author: dirkgr
Pull request: no

I haven't tried it myself, but it looks like this line is wrong:

https://github.com/huggingface/transformers/blob/42f359d015aee3835490bdcfa20df657a4d97049/src/transformers/data/data_collator.py#L285

If `self.pad_to_multiple_of` is set to anything but 1, then the length of the labels and the length of the input won't match anymore.

Comments:
cc @sgugger @Rocketknight1 

The length of the label and the length of the input rarely match for seq2seq problem, so this is not an issue.

🤦‍♂️You're right. I was looking for MLM and grabbed the wrong class.
Unexpected weights were not initialized from the model checkpoint error 
Author: sivakhno
Pull request: no

## Environment info

- `transformers` version: 4.9.2
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.11
- PyTorch version (GPU?): 1.9.0+cu102 (True)
- Tensorflow version (GPU?): 2.6.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no 

### Who can help
@LysandreJik @sgugger

## Information

I am defining a simple multi-class BERT classification model and then training it using pytorch-lightning. The code is below under class `BertForMulticlassSequenceClassification(BertPreTrainedModel)`. The issue is that after training when I am loading the classifier model `model = ClassTaggerModel.load_from_checkpoint(checkpoint_file)` I get 
```
Some weights of BertForMulticlassSequenceClassification were not initialized from the model checkpoint at bert-base-cased and are newly initialized: ['classifiers.0.weight', 'classifiers.1.bias', 'classifiers.0.bias', 'classifiers.1.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
```
This is quite strange as I am loading BertForMulticlassSequenceClassification checkpoint. In fact if I inspect state_dict with torch I can see `model.classifiers.0.weight` etc, so they are in there. Any suggestions on why I am still getting this error would be most appreciated! Thank you! 

## Reproduce 
Extract notebook https://colab.research.google.com/drive/1os9mz7w7gmLBL_ZDvZ9K1saz9UA3rmD7?usp=sharing and minimal data files attached here and "run all". 
[classes.txt](https://github.com/huggingface/transformers/files/7085157/classes.txt)
[train.csv](https://github.com/huggingface/transformers/files/7085158/train.csv)
[val.csv](https://github.com/huggingface/transformers/files/7085159/val.csv)


Comments:
You are not providing the code necessary to reproduce the error so there is little we can do to help. Also note that code like `model = ClassTaggerModel.load_from_checkpoint(checkpoint_file)` does not execute anything from the Transformers library as it uses PyTorch Lightning, so it's unlikely you are getting the warning printed above with it.

@sgugger thanks for fast reply - I have added minimal example in my original report above - the link is to colab. It does a simple one epoch training on a subset of data (seconds) and saves checkpoint. Then it loads checkpoint leading to reported error. 

I'm sorry but I'm really confused as to why you are opening an issue here: you are saving a checkpoint with PyTorch ligthning and reloading it with PyTorch lightning, not Transformers.
cannot import name AddedToken
Author: nrjvarshney
Pull request: no

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Albert

Language I am using the model on (English, Chinese ...): English

     27 from typing import List, Optional, Sequence, Tuple, Union
     28 
---> 29 from tokenizers import AddedToken, Encoding
     30 from tokenizers.decoders import Decoder
     31 from tokenizers.implementations import BaseTokenizer

ImportError: cannot import name 'AddedToken'


The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)


The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name): 
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:
- Platform:
- Python version:
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:


Comments:
Created another environment from scratch and it got resolved.

you need install transformers this way:

```
git clone https://github.com/huggingface/transformers
cd transformers
pip install .
```

> you need install transformers this way:
> 
> ```
> git clone https://github.com/huggingface/transformers
> cd transformers
> pip install .
> ```

This worked for me thanks!

> you need install transformers this way:
> 
> ```
> git clone https://github.com/huggingface/transformers
> cd transformers
> pip install .
> ```

I was having a similar issue on colab:
```
can't pickle AddedToken objects
```
This solution also worked for me. Thanks!

> you need install transformers this way:
> 
> ```
> git clone https://github.com/huggingface/transformers
> cd transformers
> pip install .
> ```

Thanks much, this worked for me!

> you need install transformers this way:
> 
> ```
> git clone https://github.com/huggingface/transformers
> cd transformers
> pip install .
> ```

This solution doesn't work for me! I don't know maybe there is a conflict with `pip` an `conda`. I guess after I installed a package `bert_score` by conda, this error appeared and won't go
Fix minor typo in parallelism doc
Author: jaketae
Pull request: yes

# What does this PR do?

This PR addresses a very minor typo in `parallelism.md`. 


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?



Comments:
model(**batch) returns loss dictionary that cant be divided by gradient_accu
Author: samarth-b
Pull request: no


Running summarization example with T5 small with the following command produces a dict divided by int error. 

to reproduce run: 

```
export TASK_NAME=mrpc

accelerate launch run_summarization_no_trainer.py \
    --model_name_or_path t5-small \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --source_prefix "summarize: " \
    --output_dir ~/tmp/tst-summarization
```

https://github.com/huggingface/transformers/blob/8be921f9de012d6f82f1cf4b2dcd4bdf2262071b/examples/pytorch/summarization/run_summarization_no_trainer.py#L521

```
----> 1 loss = loss / args.gradient_accumulation_steps
TypeError: unsupported operand type(s) for /: 'dict' and 'int'
```

**transformer version==4.9.2**
**GPU: Telsa V100 X 2**
**accelerate config:** 

```
compute_environment: LOCAL_MACHINE
deepspeed_config: {}
distributed_type: MULTI_GPU
fp16: true
machine_rank: 0
main_process_ip: null
main_process_port: null
main_training_function: main
num_machines: 1
num_processes: 1
``` 

should it be?: 
```       
outputs = model(**batch)
loss = outputs.loss['loss']
```
happy to do the PR, if confirmed.  

Comments:
when using a single GPU the script works. 
ie, outputs.loss returns a shape (1,) that can be divided by int. 

what is the best way to use `outputs.loss` for multiple GPU settings? 

 This PR https://github.com/huggingface/accelerate/pull/149 will help you.
cookiecutter template for adding flax model
Author: changjonathanc
Pull request: no

# 🚀 Feature request

Add cookiecutter template for adding flax model.

## Motivation

There is no cookiecutter template for adding flax model.

## Your contribution

I am trying to add a flax model (#12411), I think it's a good opportunity to create a template at the same time. Can I work on this? Any suggestions?

@patil-suraj @patrickvonplaten 


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Fix for the issue of device-id getting hardcoded for position-ids during Tracing for Flaubert
Author: HamidShojanazeri
Pull request: yes
Labels: name: WIP

# What does this PR do?

This PR is part of a series of PRs that follows PR #11252 and applies similar changes to Flaubert.
Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.issues #5664 and #976
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests? Does not apply.


## Who can review?
@LysandreJik 

Comments:
Fix for the issue of device-id getting hardcoded for position-ids during Tracing for Distillbert
Author: HamidShojanazeri
Pull request: yes
Labels: name: WIP

# What does this PR do?
This PR is part of a series of PRs that follows PR #11252 and applies similar changes to Distillbert.

Fixes # (issue)
Registering a buffer for position_ids in the constructor and then resizing it in the forward method based on input-shape.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.  issues #5664 and #976
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests? does not apply.

## Who can review?
@LysandreJik .

Comments:
Fix for the issue of device-id getting hardcoded for token_type_ids during Tracing for ConvBert
Author: HamidShojanazeri
Pull request: yes
Labels: name: WIP

# What does this PR do?

This PR is part of a series of PRs that follows PR #11252 and applies similar changes to ConvBert.

Fixes # (issue) 
Registering a buffer for token_type_ids in the constructor and then resizing it in the forward method based on input-shape.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. issues #5664 and #976
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests? Not required.


## Who can review?
@LysandreJik


Comments:
Fall back to `observed_batch_size` when the `dataloader` does not know the `batch_size`.
Author: mbforbes
Pull request: yes

# What does this PR do?

Motivated by #12995, this adds support for users to provide a `batch_sampler` to the DataLoader instead of a (single index) `sampler`. (The [pytorch docs](https://pytorch.org/docs/stable/data.html) has more info on these two sampler types.)

When we provide a `batch_sampler`, a DataLoader doesn't know the batch size, so it's set to `None`. Currently, the Trainer retrieves the batch size from the data loader:

https://github.com/huggingface/transformers/blob/1fec32adc6a4840123d5ec5ff5cf419c02342b5a/src/transformers/trainer.py#L2172

... leading to a crash a few lines later when it tries to use it:

https://github.com/huggingface/transformers/blob/1fec32adc6a4840123d5ec5ff5cf419c02342b5a/src/transformers/trainer.py#L2217

```txt
TypeError: repeat(): argument 'repeats' (position 1) must be tuple of ints, not NoneType
```

Fortunately, the observed batch size is calculated between those two spots, so this change simply uses it instead if the batch size wasn't found on the data loader.

I added the `None` check just to ensure this does not change existing behavior, though I would imagine it would not even without the check.

_Re: testing: I was not sure how much code you want surrounding this fix / added support, as I don't think Transformers includes any batch samplers itself yet, so I didn't include a test. Let me know otherwise and I can take a stab at it!_

## Who can review?

I suggest @sgugger due to issue context and Trainer :-)


Comments:
Apologies for the ping @sgugger, but might you be able to take a look at this? tl;dr with this 2-line change, users can now provide batch samplers without evaluation crashing 🥳 

Thanks, Sylvain! Gaah, sorry, your inbox must have been crazy by the time you came back 😅  — I hope you had a nice break!



It was great, thanks for asking!