noamaanMulla-03/transformers-issues huggingface dataset:
Update Loss calculation in prediction_step
Author: ZHANG-GuiGui
Pull request: yes

In `predition_step` function, loss is calculated using `_compute_loss` function. If the `compute_loss` function is overridden by a customized loss function, the evaluation will still calculate the original loss. I think it is more interesting to use `compute_loss` function rather than `_compute_loss` function in `prediction_step`.

# What does this PR do?


## Before submitting
- [ x ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Did you make sure to update the documentation with your changes? I think that no need to update documentation.


## Who can review?

Library:

- text generation: @patrickvonplaten
- trainer: @sgugger
- pipelines: @LysandreJik

Comments:
No this won't work with label smoothing, so we can't accept that change.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
GPT2 (117M) results mismatch on PTB, enwik8 and text8 metrics
Author: rfzahid
Pull request: no

Model used: GPT2 117M parameter model
Platform: Google colab with GPU

I am trying to reproduce the results of GPT2 (117M) on various datasets but getting different values of various metrics. I have used this script from [Huggingace platform](https://huggingface.co/transformers/perplexity.html).

1. On PTB, I have used one from the source: https://github.com/wojzaremba/lstm/tree/master/data
Hopefully this is the original dataset used in GPT2 paper. **Reported value in GPT2 paper: 65.85 PPL**. I am getting the following results:
52.53 PPL (on test set using token count)
79.65 PPL (on test set using word count)
60.08 PPL (on validation set using token count)
**Can anyone clarify whether token or word count is used while calculating PPL on PTB dataset?**

2. On enwik8 dataset, **reported metric BPB value is 1.16** while I am getting a value of 2.23 using stride count of 1024. Changing stride to 512 is not having much of an affect.

3. On text8 dataset, **reported metric BPC value is 1.17** while I am getting a value of 2.29 using stride count of 1024. Again, changing stride to 512 is not improving the result.

**Is there some pre-processing that needs to be done which is not mentioned in the paper for all of the above?**

Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co) instead?

Thanks!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Squeeze and Excitation Network
Author: AdityaDas-IITM
Pull request: yes

# What does this PR do?
This PR implements an optional Squeeze and Excitation Block in Bert and the copied modules (RoBerta, Electra, splinter and layoutlm) in pytorch. 
Fixes #11998 
Additional tests have been added to the corresponding test scripts and the docs updated.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?
Anyone in the community is free to review the PR once the tests have passed. 
@LysandreJik 

Comments:
Hi,

Thanks for your PR! However, I don't think that we want to add this block to files of other models. It's more appropriate to add a new SesameBERT model (if pretrained weights are available), or add it under the `research_projects` directory.

cc @LysandreJik 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Hi @AdityaDas-IITM, are you interested in working on this PR (making it a research project instead)?

Hey @NielsRogge, Yes I'll get started on it soon

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Changed relative imports to absolute to allow convert_graph_to_onnx.py to run as a script.
Author: nbertagnolli
Pull request: yes

# What does this PR do?
Fixes relative imports in transformers.onnx.convert.  This allows convert_graph_to_onnx.py to run as a script again from outside the transformers repository.  This PR fixes issue #14314.


## Who can review?
@sgugger



Comments:
Expand dynamic supported objects to configs and tokenizers
Author: sgugger
Pull request: yes

# What does this PR do?

This PR expands support for dynamic objects (from just models to models, configurations and tokenizers). The API to enable this is still a bit hack-ish (see tests) but this will be put together with the new `register` API of the Auto classes in a followup PR, which should make everything all fit together.

Comments:
[deepspeed] Enable multiple test runs on single box, defer to DS_TEST_PORT if set
Author: jeffra
Pull request: yes

# What does this PR do?
DeepSpeed currently runs the HF/DS integration tests in our CI for every PR we get. We are attempting to co-locate some of our test runners on single long running nodes. This PR will help us run multiple tests on the same node in parallel by allowing the torch.distributed port to be defined in an environment variable. This PR should work as it did previously if the variable is not set.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@stas00 


Comments:
Pretrained model download slowly，It will speed up when I delete the folder $USER/.cache/huggingface
Author: mokeyish
Pull request: no

I don't know why, It's very slow, But when I delete the folder `$USER/.cache/huggingface`, It will be `6.2MB/s`. 

![image](https://user-images.githubusercontent.com/16131917/136638557-70625b19-ac0f-48f4-aaf0-119289cc62e8.png)


Comments:
The speed of manual download.

![image](https://user-images.githubusercontent.com/16131917/136638962-455ef2da-99df-45f2-9eac-2736ee26ae5e.png)


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Why GPT2ForTokenClassification doesn't shift logits and labels?
Author: JaheimLee
Pull request: no

Why GPT2ForTokenClassification doesn't shift logits and labels? I think it's loss is similar to  GPT2LMHeadModel. By the way, the docstring of GPT2ForTokenClassification in forward function seems not right!

Comments:
Hi,

`GPT2ForTokenClassification` behaves exactly as `BertForTokenClassification`. There's no need to shift the labels; it just needs to predict a label for every token.

`GPT2LMHeadModel` on the other hand needs to predict the next token given the previous ones, which is why the logits and labels are shifted.

Got it! Thanks for your reply.
Small change to Wav2Vec2 model to support Tensor-Parallelism with DeepSpeed
Author: RezaYazdaniAminabadi
Pull request: yes

# What does this PR do?

This PR adds a minor modification to BartAttention and its copies to support tensor-parallelism with DeepSpeed. This relates to this [PR](https://github.com/microsoft/DeepSpeed/pull/1512) on DeepSpeed side.

Please see the added comments in the code that explain the change.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@stas00 


Comments:
> Could you also add a test to make sure the feature works (we might not be able to run it on our 2 GPUs machine, but a 4 GPUs one is coming).

We will have a full battery of tests for Deepspeed Inference. I will take care of this, Sylvain. The plan is to have a model zoo style test - identical to Deepspeed ZeRO tests, so to cover as many models as possible. (there will be also Deepspeed ZeRO Inference tests https://github.com/huggingface/transformers/pull/14253, which is different from Deepspeed Inference)

We didn't feel a test was needed for this particular PR since it doesn't change anything for a normal application.

You now need to run `make style` on your branch to fix the code quality issue :-)

Thanks @sgugger and @stas00
Request for advanced documentation on the text generation pipeline.
Author: Jcwscience
Pull request: no

# 🚀 Feature request
Detailed information on the various arguments that the pipeline accepts.

Explanation of the use cases described, eg. How to provide examples to prime the model for a task. 
## Motivation
I have hit a wall in several of my projects, and being relatively new to this field of computer science I can’t seem to get my foot in the door. Each model or example project I attempt to use seems to assume I already have certain knowledge. I am not sure what this knowledge is but since I find it hard to believe you could make serious progress with just the information available in the documentation, I feel there must be something I’m missing. If I’ve missed some obvious piece of information or documentation I apologize for the bother, but if not I could really use a bit of help. Thanks!

## Your contribution
I’m more than willing to help in any way I can within the realm of my abilities, however at the moment that realm is fairly small.


Comments:
Hello! The text generation pipeline takes as argument all of the `generate` method's argument, to tweak the generation. The documentation for that method is available here: https://huggingface.co/transformers/main_classes/model.html?highlight=generate#transformers.generation_utils.GenerationMixin.generate

Does that help?
cc @patrickvonplaten

@LysandreJik There it is! That’s what I have been missing! Right in front of me. Thank you! I can’t believe I haven’t made it to that page before somehow.

That's helpful feedback! The page should not be hard to find, we'll see what we can do to make it clearer. Anything in particular would have helped you find it? Where would you have expected to see a mention of it, for example?

@LysandreJik Well the first place I landed when looking for info was the [Pipelines description page.](https://huggingface.co/transformers/main_classes/pipelines.html#transformers.TextGenerationPipeline) 

I think the confusing part was the fact that it has *some* parameters listed. If it didn’t have any I would probably have realized I needed to check somewhere else. But since it has a few I just assumed that was most of the info.

Maybe if there was just a link saying something like “more information” or “for a full list of parameters visit this page”? Just a thought. I know the models != the pipelines exactly, so I’m not sure what needs to be linked where for sure.


That's very valuable feedback indeed!
Multilingual T5 is not adding new tokens
Author: sasidharreddy25
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.11.2
- Platform:google colab
- Python version:3.7.12
- PyTorch version (GPU?):
- Tensorflow version (GPU?): 2.6.0
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?:No

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 I am using mT5 model. The problem arised when I am trying to resize the token embeddings. I added new special tokens to my tokenizer's vocabulary, when I resized the model's token embeddings according to the length of the tokenizer, I am facing the issue.
Link: https://colab.research.google.com/drive/1ooKa5aQ_FAEnicxBL8bJnNAO4H9vFuv-?usp=sharing

Code from the notebook:
!pip install transformers
!pip install sentencepiece
from transformers import TFMT5ForConditionalGeneration, T5Tokenizer
model = TFMT5ForConditionalGeneration.from_pretrained("google/mt5-base")
tokenizer = T5Tokenizer.from_pretrained("google/mt5-base")
tokenizer.add_special_tokens({'bos_token':'','eos_token':''})
model._resize_token_embeddings(len(tokenizer))
The error message is:
ValueError: Attempt to convert a value (None) with an unsupported type (<class 'NoneType'>) to a Tensor.


 -->

## Information


The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1.You can directly run the code & check the error coming from the last line.
2.Also you can go the colab notebook & have a look at the error as well.



Comments:
related #13840

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
AttributeError: 'TFDistilBertForSequenceClassification' object has no attribute 'to'
Author: kovkev
Pull request: no

## Environment info
- `transformers` version: 4.11.2
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyTorch version (GPU?): 1.9.0+cu102 (False)
- Tensorflow version (GPU?): 2.6.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: YES
- Using distributed or parallel set-up in script?: PARALLEL

- https://huggingface.co/transformers/custom_datasets.html

### Who can help
- trainer: @sgugger

Examples:


## Information

Model I am using: TFDistilBertForSequenceClassification

The problem arises when using:
* [ ] the official example scripts: https://colab.research.google.com/github/huggingface/notebooks/blob/master/transformers_doc/custom_datasets.ipynb

## To reproduce

Steps to reproduce the behavior:

1. Run the tensorflow cells
2. See `AttributeError: 'TFDistilBertForSequenceClassification' object has no attribute 'to'` when training

## Expected behavior

I expect `model.to()` to work in that cell doing the training.


Excuse me for not providing a stack trace. I am trying to re-run the colab but stanford is not serving the imdb dataset

Comments:
That notebook has a PyTorch and a TensorFlow version. If you take the mixed one, you will need to execute only the relevant cells.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
remove an irrelevant test from test_modeling_tf_layoutlm
Author: ydshieh
Pull request: yes

# What does this PR do?

This PR remove `test_model_various_embeddings` from `test_modeling_tf_layoutlm.py`.

## More Information

Current version of `test_modeling_tf_layoutlm.py` has 

```
def test_model_various_embeddings(self):
    config_and_inputs = self.model_tester.prepare_config_and_inputs()
    for type in ["absolute", "relative_key", "relative_key_query"]:
        config_and_inputs[0].position_embedding_type = type
        self.model_tester.create_and_check_model(*config_and_inputs)
```

After a quick search inside the repo, I believe `position_embedding_type` is only for (some) PyTorch models, and `test_model_various_embeddings` is not necessary here. Furthermore, 

- `test_modeling_tf_layoutlm.py` is the only TF test script containing `position_embedding_type`
- `modeling_tf_layoutlm.py` has no usage of `config.position_embedding_type`.


Therefore, this PR remove `test_model_various_embeddings` from `test_modeling_tf_layoutlm.py`.

Comments:
bump flax version
Author: patil-suraj
Pull request: yes

# What does this PR do?

jax `v0.2.21` introduced a slight breaking change, `jnp.ndarray` now only returns `True` for `jax` arrays and not `numpy` arrays. so for standard `numpy` array x, `isinstance(x, jnp.ndarray)` will
now return `False` . (cf [here](https://github.com/google/jax/releases/tag/jax-v0.2.21), the last point under breaking changes)

And the `jax.random.split` method returns numpy array instead of jax. We use this method during random init of the model to create the `rngs`  `dict`.

Flax has a check for valid rngs where it expects `jnp.ndarray` , but since `split` returns np array and jax now does not treat np arrays as `jnp.ndarray` the check returns `False` and raises an error.

This has been fixed in `flax==0.3.5` (cf [here](https://github.com/google/flax/blob/v0.3.5/flax/core/scope.py#L753)).  So all of our models fail with `jax>=0.2.21` and `flax<0.3.5`.

To fix this, this PR bumps the flax version to `0.3.5`.

Comments:
Rewrite guides for fine-tuning with Datasets
Author: stevhliu
Pull request: yes
Labels: name: Documentation

# What does this PR do?

This PR updates the current documentation for [Fine-tuning with custom datasets](https://huggingface.co/transformers/custom_datasets.html#). It removes the custom code for loading a dataset in favor of using the Datasets library for loading and preprocessing a dataset. The new guide also introduces the Keras method for compiling and fitting a model instead of using `TFTrainingArguments` and `TFTrainer`.

⚠️ The TensorFlow example for Question Answering still needs a little work. It returns `ValueError: No gradients provided for any variable` when calling `model.fit`.  @Rocketknight1 has provided a temporary solution where we set `dummy_labels=True` in `tf_to_dataset`.

This [notebook](https://drive.google.com/file/d/1irNIGeRtfUb-4x79ZBloNy49m_FW-sOy/view?usp=sharing) contains all the code examples shown in the guide.

# To do:

- [ ] Update the TensorFlow code example for question answering once we have a better solution.

Comments:
Thank you, @stevhliu!