noamaanMulla-03/transformers-issues huggingface dataset:
ValueError in predict function for ClassificationModel
Author: GoyalMansi
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.6.1
- Platform: Linux-4.15.0-122-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.2
- PyTorch version (GPU?): 1.8.1+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
@LysandreJik 


## Information

Model I am using (Bert, XLNet ...): bert

The problem arises when using:
* [ ] my own modified scripts: 
 I noticed then when I decrease my train batch size from 32 to 16,  i get the following bug:
 
 
![image](https://user-images.githubusercontent.com/47935154/121743373-41728f80-cacf-11eb-9dd0-5c07fdba9010.png)

 (Please note that training for 10 epochs happens successfully)

The tasks I am working on is:
* [ ] my own task or dataset: 
My own dataset for binary classification of text documents.


## Expected behavior
Evaluation should happen as expected. I am not sure what to fix/how to investigate. Could not find much about it online. 
ValueError: could not broadcast input array from shape (16,2) into shape (4,2)

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Why attention mask is -10000 but not * 0?
Author: chenxran
Pull request: no

I am reading the code in Roberta and I found that the implementation of avoiding padding token to be calculated in self-attention is to minus its value by 10000 in function `get_extended_attention_mask`.

 I am wondering that why implement the mask by directly multiplying the value of padding tokens with zero?

Comments:
Hi,

searching previous Github issues, I found this one, which might help you: #542

Thanks for the reference! While I consider that maybe implementing a softmax function that allows masking after e^x may also be an approach to implement `attention_mask`?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
[trainer] sanity checks for `save_steps=0|None` and `logging_steps=0`
Author: stas00
Pull request: yes

This PR deals with combinations of training args that lead to:


```
# --do_eval --evaluation_strategy=steps --logging_steps 0 (no --eval_args)
Traceback (most recent call last):
  File "examples/pytorch/translation/run_translation.py", line 617, in <module>
    main()
  File "examples/pytorch/translation/run_translation.py", line 537, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/trainer.py", line 1340, in train
    self.control = self.callback_handler.on_step_end(args, self.state, self.control)
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/trainer_callback.py", line 359, in on_step_end
    return self.call_event("on_step_end", args, state, control)
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/trainer_callback.py", line 378, in call_event
    result = getattr(callback, event)(
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/trainer_callback.py", line 414, in on_step_end
    if args.evaluation_strategy == IntervalStrategy.STEPS and state.global_step % args.eval_steps == 0:
ZeroDivisionError: integer division or modulo by zero
```

and:

```
# --do_eval --evaluation_strategy=steps --logging_steps 0 --eval_args 5
Traceback (most recent call last):
  File "examples/pytorch/translation/run_translation.py", line 617, in <module>
    main()
  File "examples/pytorch/translation/run_translation.py", line 537, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/trainer.py", line 1343, in train
    self.control = self.callback_handler.on_step_end(args, self.state, self.control)
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/trainer_callback.py", line 359, in on_step_end
    return self.call_event("on_step_end", args, state, control)
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/trainer_callback.py", line 378, in call_event
    result = getattr(callback, event)(
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/trainer_callback.py", line 406, in on_step_end
    if args.logging_strategy == IntervalStrategy.STEPS and state.global_step % args.logging_steps == 0:
ZeroDivisionError: integer division or modulo by zero
```
(this last one doesn't happen with master, but I reworked things to do the same enforcement in init as for `eval_steps`)

@sgugger 


Comments:
All good to merge, thanks again!
Expose get_config() on ModelTesters
Author: LysandreJik
Pull request: yes

This exposes the `get_config()`method on `XXXModelTester`s. It makes it accessible in a platform-agnostic way, so that utilities may use that value to obtain a tiny configuration for tests even if `torch` isn't installed.

Part of the pipeline refactor, which needs to have tiny configurations for all models.

Comments:
Refactor slow sentencepiece tokenizers.
Author: PhilipMay
Pull request: yes

PR for #11646

## ToDo

- [x] `AlbertTokenizer`
- [x] `BarthezTokenizer`
- [x] `BertGenerationTokenizer`
- [x] `BigBirdTokenizer`
- [x] `CamembertTokenizer`
- [x] `DebertaV2Tokenizer`
- [x] `M2M100Tokenizer`
- [x] `MarianTokenizer`
- [x] `MBart50Tokenizer`
- [x] `PegasusTokenizer`
- [x] `ReformerTokenizer`
- [x] `Speech2TextTokenizer`
- [x] `T5Tokenizer`
- [x] `XLMProphetNetTokenizer`
- [x] `XLM RoBERTa`
- [x] `XLNetTokenizer`

Comments:
`SentencePieceProcessor.decode` is doing "the same but more than `SentencePieceProcessor.decode_pieces`. 
That is why we replace `SentencePieceProcessor.decode_pieces` with `SentencePieceProcessor.decode` in this PR.

See here:

https://github.com/google/sentencepiece/blob/6256ef243844e5848499cf519eb2a7e2755e75a1/python/src/sentencepiece/__init__.py#L307

rebased on upstrem/master

We need to rebase on master after PR #11737 has been merged.

Rebased on master - CI is green again. :-) 

Rebased on master to get integration tests - see #11737

Rebased on master

> I think generally speaking we'd like to have methods that are common to all tokenizers in the base class - but not methods that are common to some of them only. I'd also like to keep the number of abstraction layers to a minimum, tokenizers are already quite tough to understand.

@LysandreJik 
Yes. I also prefer a low number of abstraction layers. At the same time I like dry code. There is 100% duplicate code in the tokenizers impl. that has just been duplicated by copy & paste. IMO that should be removed by an refactoring. That is what I try to introduce here.

The general approach of the library is to keep the number of abstractions as low as possible, and to keep implementations as separate as possible from each other, hence the high amount of copy-pasted code.

We want users to be able to experiment with single models/tokenizers without their changes impacting other models or tokenizers - and we want them to be able to understand how a model or tokenizer behaves by simply checking a single file, rather than having to hop around multiple files.

We are failing at this with tokenizers as there are already two levels of abstraction, but adding a third one isn't really the direction we want to head to :)

Does that make sense?

> Does that make sense?

Yes. Sure. Your project, your call.
I will revert my changes and keep it as simple as possible as discussed in the beginning.


@LysandreJik I have redone the PR. Everything is green and the changes are as simple as planned in the issue.
This is ready for review.

Averything is tested by setting `test_sentencepiece = True` in the tokenizer test classes and by the following 
testfunction: `TokenizerTesterMixin.test_sentencepiece_tokenize_and_convert_tokens_to_string`
Strange implementation of `convert_tokens_to_string` in albert tokenizer.
Author: PhilipMay
Pull request: no
Labels: name: WIP

Hi,

the albert tokenizer implements the `convert_tokens_to_string` function:

https://github.com/huggingface/transformers/blob/ba0d50f2148f0db0e04a80cddb1f57ce0c91c182/src/transformers/models/albert/tokenization_albert.py#L222-L223

While the deberta v2 and some other tokenizer just delegate this to the sentencepiece tokenizer:

https://github.com/huggingface/transformers/blob/ba0d50f2148f0db0e04a80cddb1f57ce0c91c182/src/transformers/models/deberta_v2/tokenization_deberta_v2.py#L146

IMO it would be better to always delegate to the sentencepiece tokenizer. What do you think?

## PS:
Some more examples here 

https://github.com/huggingface/transformers/blob/ba0d50f2148f0db0e04a80cddb1f57ce0c91c182/src/transformers/models/barthez/tokenization_barthez.py#L251-L252

https://github.com/huggingface/transformers/blob/ba0d50f2148f0db0e04a80cddb1f57ce0c91c182/src/transformers/models/camembert/tokenization_camembert.py#L251-L252

https://github.com/huggingface/transformers/blob/ba0d50f2148f0db0e04a80cddb1f57ce0c91c182/src/transformers/models/m2m_100/tokenization_m2m_100.py#L187-L188

https://github.com/huggingface/transformers/blob/ba0d50f2148f0db0e04a80cddb1f57ce0c91c182/src/transformers/models/mbart/tokenization_mbart50.py#L208-L209

https://github.com/huggingface/transformers/blob/ba0d50f2148f0db0e04a80cddb1f57ce0c91c182/src/transformers/models/speech_to_text/tokenization_speech_to_text.py#L169-L173

https://github.com/huggingface/transformers/blob/ba0d50f2148f0db0e04a80cddb1f57ce0c91c182/src/transformers/models/xlm_prophetnet/tokenization_xlm_prophetnet.py#L264-L265

Comments:
Indeed, you're probably right! When updating the ALBERT tokenizer to use the `sentencepiece.decode` instead of the manual handling - do all tests pass? Even the integration test?

Makes me think we really should have integration tests for all tokenizers, as scenarios like this one are bound to happen.

Well yes. While "adding subword regularization in more tokenizers": #11417
I recognized that the tokenizers could benefit from some bigger refactoring.
Pulling commen functions into a base class would be nice. And while doing this adding tests....
There is  lot of duplicate code there...

I might do this as a PR the next days (weeks) - we will see.

PR with a fix started: #11716

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

I am still working on this...

Fixed in #11716 closing here.
Add _CHECKPOINT_FOR_DOC to all models
Author: LysandreJik
Pull request: yes

This adds the `_CHECKPOINT_FOR_DOC` variable for all models that do not have it. This value is used in the pipeline tests refactoring, where all models/tokenizers compatible with a given pipeline are tested. To that end, it is important to have a source of truth as to which checkpoint is well maintained, which should be the case of the `_CHECKPOINT_FOR_DOC` that is used in the documentation.

Comments:
Using checkpoints in gpt neo xl
Author: MK096
Pull request: no

Hi,
I downloaded gpt neo xl pretrained model from theeye.eye on my pc.
It downloaded  various checkpoints.
How do i use them? ... Because in order to load and use model I'd need encoder. Json, pytorch. Bin, etc..


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Adding fastseq support to more recent version of HF transformers
Author: tingofurro
Pull request: no

# 🚀 Feature request

Would it be possible to integrate [fastseq](https://github.com/microsoft/fastseq) with the later version of HuggingFace transformer models.

## Motivation

fastseq is a library that gives speedup on Transformers for text generation. They claim to have pretty large speedups (3-8x) for various Transformer architectures (GPT2, Bart, etc.).
The only caveat is they only support an older version of HF transformers (3.0.2).

Has anyone already looked into making it compatible with the latest API of HuggingFace models?

## Your contribution

I am willing to discuss and can contribute if no one has planned to do so already.


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
[Deepspeed] [performance] inefficient load with `from_pretrained` w/ zero3
Author: stas00
Pull request: no
Labels: name: DeepSpeed

name: WIP

# 🚀 Feature request

Currently under Deepspeed stage3 with `from_pretrained` we:

a. loop over each sub-module in zero.Init
1. init the sub-module
2. shard and scatter the shards

b. then to load pre-trained weights we loop over each sub-module:
1. gather the shards
2. `load_state_dict` for the one layer layer
3. shard and scatter the shards

c. any sub-module params that weren't in the pretrained state_dict
1. run the postponed `module_init` as it was done in https://github.com/huggingface/transformers/pull/11471
2. shard and scatter the shards XXX: I actually don't think `deepspeed.zero.GatheredParameters` was handled here. so these params don't get ZeRO'ed - need to fix that https://github.com/huggingface/transformers/issues/12272

Because we unnecessarily do scatter/gather/scatter, this takes much longer than just:

a. init the modules w/o allocating any storage as it has been implemented in pt-1.9.0/1.9.1 https://pytorch.org/tutorials/prototype/skip_param_init.html#implementation-details

b. for each sub-module with pretrained weights
1. load_state_dict 
2. shard and scatter the shards

c. any sub-module params that weren't in the pretrained state_dict
1. materialize and module_init
2. shard and scatter the shards

Solving this will most likely require support from Deepspeed, https://github.com/microsoft/DeepSpeed/issues/1142 or perhaps we can just try to remove `zero.Init` if the weights aren't materialized during model creation. So the very first sharding will get postponed to the `load_state_dict` stage (and `module_init` for the sub-modules that don't have pre-trained weights).

Comments:
[Deepspeed zero3] lazy weights init 
Author: stas00
Pull request: no
Labels: name: DeepSpeed

name: WIP

I'm pretty sure we need to follow up to the lazy weights init feature https://github.com/huggingface/transformers/pull/11471
and add under zero3 `deepspeed.zero.GatheredParameters` here (or inside `_init_weights`):

https://github.com/huggingface/transformers/pull/11471/files#diff-6b72b98c4c2dcfc6cc606843917733f5d858374fbc22a735ff483bbc0c1e63eaR1275-R1276

plus need a test.

Comments:
[debug] DebugUnderflowOverflow doesn't work with DP
Author: stas00
Pull request: yes

As reported in https://github.com/huggingface/transformers/issues/12815 `DebugUnderflowOverflow` breaks under DP since the model gets new references to model sub-modules/params on replication and the old references are needed to track the model layer names. 

It might be possible to think of some workaround, most likely overriding `torch.nn.parallel.data_parallel.replicate` to refresh the model references after the replication, but at the moment this is not required, since DDP works just fine. (or single GPU).

So this PR adds a clean assert when DP is used, instead of a confusing exception. Update docs.

Fixes: https://github.com/huggingface/transformers/issues/12815

@sgugger 

Comments:
Raise warning in HP search when hp is not in args
Author: sgugger
Pull request: yes

# What does this PR do?

As seen on the [forums](https://discuss.huggingface.co/t/using-hyperparameter-search-in-trainer/785), an HP search using the `Trainer` will error if someone tries to include model parameters like dropout in the search. This PR addresses that by changing an error to a warning in that case.

Comments:
Tokenizer Encode More than 2 inputs
Author: pertschuk
Pull request: no
Labels: name: wontfix

# 🚀 Feature request

Increasingly I'm seeing more than 2 inputs in some cases to BERT model, separated by [SEP] tokens. Often this helps by including context or for pairwise search ranking. 

## Motivation

Right now I have to manually add the [SEP] token and concat the output to two tokenizer.encode_plus calls. Seems like it would be simple to just grab all the positional args and treat them as additional fields. 

Also seems like this is more expected behavior than arbitrarily limiting encode_plus to single or pairs of text.

## Your contribution

I could submit a PR. 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Had this problem been solved?

@Zhylkaaa  It seems that the problem is still unsolved. Are there any easier ways to encode more than 2 inputs with tokenizer?

@skpig I think the best way is to use `' <sep_token> '.join(inputs)`. For roberta that would be `' 
 '.join(inputs)`.
But keep in mind that some models are designed to have [EOS] token at the and (for roberta it's `
`, bert doesn't have one I think)
EDIT: actually I realised that first 
 and 
 will be added automatically after you pass resulting string to `tokenizer(' <sep_token> '.join(inputs))`

@Zhylkaaa That works. Thanks a lot.
Setting global tokens in BigBirdModel
Author: MFajcik
Pull request: no

## Environment info
- `transformers` version: 4.6.1
- Platform: Linux-5.8.0-55-generic-x86_64-with-glibc2.31
- Python version: 3.9.5
- PyTorch version (GPU?): 1.8.1+cu111 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


### Who can help
@vasudevgupta7 , @patrickvonplaten, @patil-suraj


## Information

Model I am using: BigBirdModel

Hi, I cannot find the way to set the global tokens for BigBirdModel as in BigBird-ITC. I hope I only missed something. The official paper says (Appendix E.2, paragraph "Natural Questions") "For BIGBIRD-ITC, we make the first 128 tokens as global.".

For e.g. longformer, setting global tokens to attend to is possible via `global_attention_mask` model parameter. However I found no option how to enforce the global attention on certain tokens for bigbird. This is crucial for e.g. all QA tasks. 
Is the explicit setting of global tokens supported?







Comments:
So after closely inspecting the code I ran into following comment in method `bigbird_block_sparse_attention`.

There I found
```
       # BigBird block-sparse attention as suggested in paper

        # ITC:
        #     global tokens: 2 x block_size
        #     window tokens: 3 x block_size
        #     random tokens: num_rand_tokens x block_size

        # Note:
        #     1) Currently, ETC is not supported.
        #     2) Window size is fixed to 3 blocks & it can be changed only by
        #     changing `block_size`.
        #     3) Number of global blocks are fixed (2 blocks here) & global tokens can be
        #     controlled only by `block_size`.
```

I suppose that answers my 'issue'.
How can we modify the MM-IMDB model for sequence to sequence generation tasks?
Author: abhaygargab
Pull request: no

Hi all, thank you so much for the wonderful service.

I have some doubts regarding the training details for MM-Imdb dataset.

Are the image encoder's and tokenizer's embeddings fine-tuned during training on MM-Imdb dataset? If not, can you suggest a way to do it or refer any material for help?

Is there a way to modify the code so that the model’s pre-trained weights can be used for sequence-to-sequence generations tasks instead of classification?

Thank You..

Comments:
Hi,

could you please ask this question on the [forum](https://discuss.huggingface.co/) rather than here? We like to keep Github issues for bugs/feature requests.

Thanks!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
🤗 The Hugging Face Course is out!
Author: LysandreJik
Pull request: no

The first part of the Hugging Face Course is finally out!

Come learn how the :hugs:  Ecosystem works :partying_face: : Transformers, Tokenizers, Datasets, Accelerate, the Model Hub! 

Share with your friends who want to learn NLP, it's free!

Come join us at https://hf.co/course

Students following this course will understand how to approach (almost) any NLP problem and benefit from all the past experiences of the community.

Come register for the live sessions, ask any questions, and organize study groups on the Hugging Face forums: 

https://discuss.huggingface.co/c/course/20

![image](https://user-images.githubusercontent.com/30755778/121926625-22ad0c80-cd0c-11eb-9fd9-930831a0149a.png)


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.