noamaanMulla-03/transformers-issues huggingface dataset:
[torch.fx] Abstract dynamic tracer
Author: thomasw21
Pull request: yes

# What does this PR do?

We hope to abstract the dynamic features of tracer from the `HFTracer` class. The idea is to leverage the new feature for non HF modules, for example having a torch like tracer that support the dynamic allocation mechanisms.

If this PR is approved, we can move inheritance so that HFTracer implements DynamicTracer.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@michaelbenayoun 

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.


Comments:
Closing as @michaelbenayoun is making substantial changes in fx. Will create the abstraction again after his PR is merged.
Fix RNG reload in resume training from epoch checkpoint
Author: sgugger
Pull request: yes

# What does this PR do?

This PR fixes the reproducibility in training when checkpoints are saved every epoch. The main reason it was failing (as pointed out in #17032) is that the RNG states were never reloaded. They need to be reloaded exactly before iterating through the new epoch, as the call to this will change the global PyTorch RNG (even if the dataloader uses its own generator...) The new test added makes sure this reproducibility is fully tested.

While debugging this, two issues occurred, which this PR also fixes.
1. There are multiple warnings for the computation of flos when the model is not an NLP model. This PR reduces it to one.
2. The test of this reproducibility is flaky on multiple GPUs because it relies on some randomness inside the model, but the PyTorch RNG will be called in random order between the two "copies" of the model executed by `DataParallel` (an issue that wouldn't be the case with `DistributedDataParallel` but we would need to execute the test via  a launcher in that case). So in the test, we only do PyTorch randomness on one or zero GPU to fix this flakiness.

Fixes #17032 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
PerceiverIO Attention maps
Author: ooguz166
Pull request: no

Hi, I am quite amazed with the usage and the implementation of PerceiverIO so far.
I am trying to visualize the attention maps for an arbitrary image using "PerceiverForImageClassificationConvProcessing".

I want to visualize the attention maps like in  https://arxiv.org/pdf/2103.03206.pdf Figure 3.

I can hook into the layers to get attention vectors of the decoder and most probably I need a pixel-wise multiplication with the query vector to achieve such images. But if there are any other fast paced approach I really like to use it for this purpose.

Thanks

Comments:
Hi,

You can get the attention scores by passing `output_attentions=True` to the forward method of any model in the Transformers library. See an example of how you can visualize the attention maps of DETR in [this notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/DETR/DETR_minimal_example_(with_DetrFeatureExtractor).ipynb).

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
How to calculate GPT-2 sentence loss (for each sentence) if batch has 2 or more sentences?
Author: Oxi84
Pull request: no

I need to calculate loss for batch of sentence but when i do this I get only average loss for all the sentences not all the loses.   
 
    from transformers import GPT2LMHeadModel, GPT2Tokenizer
    model = GPT2LMHeadModel.from_pretrained('distilgpt2')
   loss_list = model(input_ids, labels=input_ids)

Is it possible to somehow calculate the loss of each sentence separatelly in one batch or I have to use only batch-size=1 which will be much slower.   

  

Comments:
Hello,

[Not an official answer, I'm not part of the the team]
The `GPT2LMHeadModel` uses the `CrossEntropyLoss` with its default parameters. The default for `reduction` is 'mean', so the loss is averaged. One thing that you can do is to define your custom `GPT2LMHeadModel` and replace this:

```
loss_fct = CrossEntropyLoss()
loss = loss_fct(shift_logits.view(-1, shift_logits.size(-1)), shift_labels.view(-1))
```

with:

```
loss_fct = CrossEntropyLoss(reduction='none')
loss = loss_fct(shift_logits.view(-1, shift_logits.size(-1)), shift_labels.view(-1))
```

this gives you the loss for every token. If you want the average by sequence, you need to undo the flatten first:
```
loss_by_sequence = loss.view(logits.size(0), -1).mean(dim=1)
```

Be aware that I say "by sequence" and not "by sentence" because that depends on how you structure the inputs and nothing prevents a sequence from containing more than 1 sentence.

Awesome. Thanks very much. This should speed up things considerably.

I need to change this in file: transformers/models/gpt2/modeling_gpt2.py and I get loss for every token. So if i have 2 sequences Do I get this [ [loss_word1a,loss_word2a ...], [loss_word1b,loss_word2b,...] ?






Yes, that's what you'd get in `loss`.

In `loss_by_sequence` you'd get [loss_seq1, loss_seq2, ...]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Set Experiment Name in `MLflowCallback`
Author: PhilipMay
Pull request: no

In the `MLflowCallback` it is not possible to set the Experiment Name. 
As a consequence the run is logged to the Default experiment.

I would like to suggest the following feature:

It is possible to set an env. variable called `ML_FLOW_CALLBACK_EXPERIMENT_NAME`
which then calls `mlflow.set_experiment(experiment_name)`.

What do you think? As always I can provide a PR if wanted.


Comments:
PS: additionaly to the env. var we might want to add a constructore parameter - what do you think?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

This issue should be reopened as it is intended for mlflow experiment name which is different from run_name.
Seq2Seq: decoder hidden_states shape not tested
Author: sshleifer
Pull request: no

[this line] points at encoder hidden states for Bart.

https://github.com/huggingface/transformers/blob/2ee410560e45ae3c619dc1e0b0fc4d257c48e18a/tests/test_modeling_common.py#L464

Comments:
same for `T5`. I thought a bit about how to correctly sort the `encoder_hidden_states`, `decoder_hidden_states` output (if the user wants to output it). I think it's not easy at all to implement this cleanly...namedtuples would make this so much easier, so maybe just wait until we add those? 

Makes sense. 
Are namedtuples on anybody's roadmap?

Not really as far as I know! It would be a big change (lots of code has to be adapted). In my opinion it would be best to start with the outer-most outputs (the returned `outputs` of the models) and see how that goes:
- How easy is it to have everything backwards compatible?
- How much cleaner does the code get in `generate()` this way?
- How much code has to be added for named tuples? 

(Just assigning myself so that I can easily find our discussion again)

Hi @patrickvonplaten,
for a distillation purpose of T5, I want to return the `deocder_hidden_states`. using this:

```
from transformers import T5Tokenizer, T5ForConditionalGeneration
tokenizer = T5Tokenizer.from_pretrained("t5-small")
model = T5ForConditionalGeneration.from_pretrained("t5-small")
# training
input_ids = tokenizer("The <extra_id_0> walks in <extra_id_1> park", return_tensors="pt").input_ids
labels = tokenizer("<extra_id_0> cute dog <extra_id_1> the <extra_id_2>", return_tensors="pt").input_ids
outputs = model(input_ids=input_ids, labels=labels)
loss = outputs.loss
logits = outputs.logits
print(outputs.decoder_hidden_states) # None!!
```


Can you add `output_hidden_states=True` to `model(...)`?
Added new Spanish translation of autoclass_tutorial
Author: duedme
Pull request: yes

# Translation of autoclass_tutorial.mdx into spanish

I made the translation of autoclass_tutorial.mdx into Spanish (fixes #15947). The document is located in the transformers/docs/source_es folder.

This PR is linked to #16348, which originally had some merge conflicts.

FYI @omarespejel 

## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?


Comments:
_The documentation is not available anymore as the PR was closed or merged._

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi @Duedme thank you for your new commits. Could you please add `autoclass_tutorial` to `../transformers/docs/source/es/_toctree.yml`. According to the [new Translation](https://github.com/huggingface/transformers/blob/main/docs/TRANSLATING.md) guide? This would allow the tests to pass.
Fx with meta
Author: michaelbenayoun
Pull request: yes

# What does this PR do?

This PR simplies and improves the way tracing works with torch.fx.

Instead of recording concrete values via a forward pass on the original model, metadata is attached to the proxies, either tensors on the `meta` device (which saves us from making actual computations, only shape inference is performed) or any other type such as `torch.Size` and builtin types. On top of allowing to trace very big models, this gives much more flexibility and should allow to support many new architectures.

A big thanks to @jamesr66a as he was the one who provided the basis for tracing with meta tensors, and I simply extended what was already done to our purposes.

@jamesr66a @pbelevich I would love your review and feedbacks!


Comments:
_The documentation is not available anymore as the PR was closed or merged._

@sgugger is it ready now?

Since it does not change the API, and does not provide any new feature, the current set of test is enough.
I also ran the test with `torch=1.10.2` (we have `TORCH_FX_REQUIRED_VERSION = version.parse("1.10")`), and they pass. The torch version could have been a limitation since we use the meta device, but everything is okay.
Fix code examples for doctests
Author: stevhliu
Pull request: yes

This PR fixes some code examples to pass the doctests for the pipeline and `AutoClass` tutorials. 

I was unable to pass the audio code examples on my local machine because soundfile is not supported on M1 yet. I was able to run and reproduce the code snippets in Colab though so I think they should also pass on the CI.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks for the work.

Other than the `>>>` things, there are 2 failures when I ran it.

For audio pipeline:
```
Expected:
    [{'label': 'calm', 'score': 0.1315},
     {'label': 'neutral', 'score': 0.1307},
     {'label': 'sad', 'score': 0.1274},
     {'label': 'fearful', 'score': 0.1261},
     {'label': 'happy', 'score': 0.1242}]
Got:
    [{'score': 0.1315, 'label': 'calm'}, {'score': 0.1307, 'label': 'neutral'}, {'score': 0.1274, 'label': 'sad'}, {'score': 0.1261, 'label': 'fearful'}, {'score': 0.1242, 'label': 'happy'}]

```
(this is just a format issue I think)

For vision pipeline:
```
Expected:
    [{'score': 0.4403, 'label': 'lynx, catamount'}, {'score': 0.0343, 'label': 'cougar, puma, catamount, mountain lion, painter, panther, Felis concolor'}, {'score': 0.0321, 'label': 'snow leopard, ounce, Panthera uncia'}, {'score': 0.0235, 'label': 'Egyptian cat'}, {'score': 0.023, 'label': 'tiger cat'}]
Got:
    [{'score': 0.4335, 'label': 'lynx, catamount'}, {'score': 0.0348, 'label': 'cougar, puma, catamount, mountain lion, painter, panther, Felis concolor'}, {'score': 0.0324, 'label': 'snow leopard, ounce, Panthera uncia'}, {'score': 0.0239, 'label': 'Egyptian cat'}, {'score': 0.0229, 'label': 'tiger cat'}]
```
~~(This might be due to some random ops. I remembered I have similar situations before. I can take a look too.)~~
I get deterministic results, which is on Ubuntu 20.04. It's not very clear why the result is different than the previous one in the doc. I also get the same results on my local Windows machine. Maybe we could just update the values, cc @sgugger?



I don't know why you ask me @ydshieh this is not my PR ;-) 

> I don't know why you ask me @ydshieh this is not my PR ;-)

I know. Just to make sure you are also fine with my suggestion about `just update the values`. But I guess I should be more confident 😄
Fix hashing for deduplication in CodeParrot
Author: thomasw21
Pull request: yes

# What does this PR do?

Fix hashing mechanism to be process independent. Typically `hash` doesn't generate the same hash when using different process. So this makes the maximum number of occurence of a text to be `num_proc` instead of `1` when deduplicating.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@lvwerra @loubnabnl 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Skip RoFormer ONNX test if rjieba not installed
Author: lewtun
Pull request: yes

# What does this PR do?

This PR adds the `@require_rjieba` decorator to the slow ONNX tests to deal with the following error in our daily CI runs:

```
(line 164)  ImportError: You need to install rjieba to use RoFormerTokenizer. See https://pypi.org/project/rjieba/ for installation.
```

~~I wasn't sure if `rjieba` should actually be installed in the GitHub workflow, but it doesn't seem to be the case for the RoFormer tests and so I omitted that for now.~~

Edit: I've added `rjieba` to the `"tests"` extras and also tested that the slow ONNX test passes when this dep is installed:

```
RUN_SLOW=1 pytest tests/onnx/test_onnx_v2.py -s -k "roformer"
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._

So this test is then currently just skipped on our ONNX tests? 

Should we maybe not better add `rjieba` to the test package to the test RoFormer (I couldn't find it in the `setup.py` or in the Docker file) cc @LysandreJik @sgugger 

It looks like RoFormer tokenization is completely untested yes, so this package should be added in the `"testing"` extra.

I agree we should include `rjieba` for the tests, unless there are reasons for not adding specific packages.

### Further remark
If we could not add `rjieba`, it is not a good idea to add `@require_rjieba` for `test_pytorch_export`, otherwise this test won't be run for other models neither. In this case, I think we might need a specific way to skip this test for `RoFormer` (and others that require `rjieba`).

Thanks for the feedback - I'll add `rjieba` to our testing suite as well :)

Hey @sgugger @patrickvonplaten I'm hitting some peculiar issues with 2 of the slow tests of the RoFormer tokenizer. Would you mind taking a look and seeing if my decision to skip them is valid?

I'll let @patrickvonplaten decide as I know nothing on that model too :-)

There is a test dedicated to custom tokenizers with specific dependencies: https://github.com/huggingface/transformers/blob/main/.circleci/config.yml#L538

It installs `jieba` but not `rjieba`. Would it make sense to add it there? If you're testing for ONNX, it's very likely that it does not make sense as it's limited to tokenizer tests right now.

> There is a test dedicated to custom tokenizers with specific dependencies: https://github.com/huggingface/transformers/blob/main/.circleci/config.yml#L538
> 
> It installs `jieba` but not `rjieba`. Would it make sense to add it there? If you're testing for ONNX, it's very likely that it does not make sense as it's limited to tokenizer tests right now.

Thanks for the tip! Done in [3cafcb2](https://github.com/huggingface/transformers/pull/16981/commits/3cafcb2e06bc7caf0eba2e03e817fedcf0cfe073)

Hey @patrickvonplaten @LysandreJik I think this PR is ready for a final pass :)

The failing test is unrelated to the PR itself (a failing Pegasus generate test)
AutoConfig.from_pretrained("model", return_unused_kwargs=True) returns `"_from_auto": True` field against specification
Author: GabrielKP
Pull request: no
Labels: name: bug

### System Info

```shell
- `transformers` version: 4.17.0
- Platform: Linux-5.13.0-40-generic-x86_64-with-glibc2.34
- Python version: 3.9.7
- PyTorch version (GPU?): 1.11.0+cu102 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
```


### Who can help?

@sgugger

Small bug in which for some cases `{"_from_auto": True}` is returned [against specification](https://github.com/huggingface/transformers/blob/31ec2cb2badfbdd4c1ac9c6c9b8a74e974984206/src/transformers/models/auto/configuration_auto.py#L646).

Seems to originate [here](https://github.com/huggingface/transformers/blob/31ec2cb2badfbdd4c1ac9c6c9b8a74e974984206/src/transformers/models/auto/configuration_auto.py#L671) and/or [here](https://github.com/huggingface/transformers/blob/31ec2cb2badfbdd4c1ac9c6c9b8a74e974984206/src/transformers/configuration_utils.py#L659)




### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Replicable with
```py
>>> from transformers import AutoConfig
>>> config, kwargs = AutoConfig.from_pretrained("bert-base-uncased", return_unused_kwargs=True)
>>> kwargs
{'_from_auto': True}
```

### Expected behavior


There should be no `"_from_auto": True` field in returned dict.
```py
>>> from transformers import AutoConfig
>>> config, kwargs = AutoConfig.from_pretrained("bert-base-uncased", return_unused_kwargs=True)
>>> kwargs
{}
```


Comments:
I can reproduce, thanks for flagging and for taking the time to give us a clear example of code that fails!
I will try to dive into this and have a fix ready later today.
Make sure telemetry arguments are not returned as unused kwargs
Author: sgugger
Pull request: yes

# What does this PR do?

As pointed out in #17056, the telemetry arguments are sometimes returned as unused kwargs. This is because `AutoConfig.from_pretrained` ends up using the `from_dict` method and not the `from_pretrained` method in most cases, and that `from_dict` method does not treat those kwargs.

Fixes #17056

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Bump notebook from 6.4.1 to 6.4.10 in /examples/research_projects/visual_bert
Author: dependabot[bot]
Pull request: yes
Labels: name: dependencies

Bumps [notebook](http://jupyter.org) from 6.4.1 to 6.4.10.


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=notebook&package-manager=pip&previous-version=6.4.1&new-version=6.4.10)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
- `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
- `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
- `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
- `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/huggingface/transformers/network/alerts).

</details>

Comments:
_The documentation is not available anymore as the PR was closed or merged._