noamaanMulla-03/transformers-issues huggingface dataset:
Reproducing Bart Xsum from Bart Large
Author: swethmandava
Pull request: no

 Summarization: @sshleifer
 Bart: @sshleifer

## Information

I'm trying to finetune bart large for Xsum and unable to reproduce the results from the paper.

When I try eval with facebook/bart-large-xsum, I get R1=45.3595, RLSum=37.1717 so I assume my eval script is working ok. For finetuning bart large, I use the same config as bart-large-xsum with vocab size=50265 to enable starting from bart-large. However, I am unable to reach the same scores. The best I have is R1=45.4188, RLSum=36.6986 with LR=1.2e-4, gbs=128 and --max_target_length=60 --max_source_length=1024 --val_check_interval 0.1 --val_max_target_length=60 --warmup_steps 50 --max_steps 5000. 

How can I reproduce the results?


Comments:
I don't know the answer to this question. Your numbers are close enough that all I can suggest is to either try fairseq's [command](https://github.com/pytorch/fairseq/blob/master/examples/bart/README.summarization.md#4-fine-tuning-on-cnn-dm-summarization-task) or look at differences between our command and fairseq.


> Summarization: @sshleifer Bart: @sshleifer
> 
> ## Information
> I'm trying to finetune bart large for Xsum and unable to reproduce the results from the paper.
> 
> When I try eval with facebook/bart-large-xsum, I get R1=45.3595, RLSum=37.1717 so I assume my eval script is working ok. For finetuning bart large, I use the same config as bart-large-xsum with vocab size=50265 to enable starting from bart-large. However, I am unable to reach the same scores. The best I have is R1=45.4188, RLSum=36.6986 with LR=1.2e-4, gbs=128 and --max_target_length=60 --max_source_length=1024 --val_check_interval 0.1 --val_max_target_length=60 --warmup_steps 50 --max_steps 5000.
> 
> How can I reproduce the results?

Hi @swethmandava . I'm trying to reproduce the result by Transformres. Would you mind sharing your fine-tuning script?
[Benchmark]
Author: Yousse7888887
Pull request: no

# 🖥 Benchmarking `transformers`

## Benchmark

Which part of `transformers` did you benchmark?

## Set-up

What did you run your benchmarks on? Please include details, such as: CPU, GPU? If using multiple GPUs, which parallelization did you use?

## Results

Put your results here!


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
use scale=1.0 in floats_tensor called in speech model testers
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix the failure of `Speech2TextModelTest.test_pt_tf_model_equivalence`. This is caused by 

https://github.com/huggingface/transformers/blob/e6f00a11d7fa34215184e3c797e19e6c7debe0fe/tests/speech_to_text/test_modeling_speech_to_text.py#L134-L136

where the `input_features` get a large magnitude of `1e2` (from `self.vocab_size=99`).

(probably this happens because we just copied the `input_ids = ids_tensor([self.batch_size, self.seq_length], self.vocab_size)` from NLP models?)

I changed it to `scale=1.0`, but need @patrickvonplaten's expertise **to make sure there was no particular reason to use `self.vocab_size`.**

### Details
Current speech model testers have

```
def prepare_config_and_inputs(self):
    input_values = floats_tensor([self.batch_size, self.seq_length], self.vocab_size)
```

The ` self.vocab_size` argument is the `scale`, so the generated dummy `input_values` has the magnitude of  `self.vocab_size`.
For `Speech2TextModelTester`, we have `vocab_size=99`.

Furthermore, `Speech2TextEncoder` has

https://github.com/huggingface/transformers/blob/e6f00a11d7fa34215184e3c797e19e6c7debe0fe/src/transformers/models/speech_to_text/modeling_speech_to_text.py#L705

and from the tester's `hidden_size=16,` we get `embed_scale=4`.

The `input_features` goes through the conv layer(s) and being scaled:
https://github.com/huggingface/transformers/blob/e6f00a11d7fa34215184e3c797e19e6c7debe0fe/src/transformers/models/speech_to_text/modeling_speech_to_text.py#L767-L768

On `CPU` however, the conv layers of PT/TF gives diff. with a magnitude of `1e-7` for input values with 1s. So with the above 2 scalings, this error becomes `4e-5`, and the PT/TF equiv. test fails.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks for fixing all the tests!
Refactor all require decorators to use skipUnless when possible
Author: muellerzr
Pull request: yes
Labels: name: Tests

name: cleanup

# What does this PR do?

I was refactoring the Accelerate tests today and I noticed we use `if .. else..` as a conditional for skipping tests based on imports. Unittest has `skipUnless` and `skipIf`, letting us simplify those decorators to be one line. 

E.g.:

```python
    if not _run_slow_tests:
        return unittest.skip("test is slow")(test_case)
    else:
        return test_case
```
Can be:
```python
    return unittest.skipUnless(_run_slow_tests, "test is slow")(test_case)
```

(Adding you as a reviewer for this Sylvain, unsure who else should be added)

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thank you 🚀 for this PR, and also for pinning me so that I can learn!
BertEmbeddings import missing for Torch in __init__ file
Author: seanbenhur
Pull request: no
Labels: name: bug

### System Info

```shell
- `transformers` version: 4.18.0
- Platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- Huggingface_hub version: 0.5.1
- PyTorch version (GPU?): 1.11.0+cu113 (True)
- Tensorflow version (GPU?): 2.8.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No
```


### Who can help?

@LysandreJik 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

`from transformers import BertEmbeddings`

raises

```
----> 1 from transformers import BertEmbeddings

ImportError: cannot import name 'BertEmbeddings' from 'transformers' (/usr/local/lib/python3.7/dist-packages/transformers/__init__.py)
```

### Expected behavior


[BertEmbeddings](https://github.com/huggingface/transformers/blob/e6f00a11d7fa34215184e3c797e19e6c7debe0fe/src/transformers/models/bert/modeling_bert.py#L182) is a class in Bert Model file that can be used for creating BertEmbeddings, the class is not imported in[ __init__ ](https://github.com/huggingface/transformers/blob/main/src/transformers/models/bert/__init__.py)file, so the user may not be able to import it. While the TF and jax version are imported it
```


It should be imported



Comments:
Feel free to open a PR to fix this :)

I think in that case it's the TF and Jax versions which shouldn't add the embeddings to the main init. Those are not meant to be accessed from the main init, but to be accessed from their respective modules:

```py
>>> from transformers.models.bert.modeling_bert import BertEmbeddings
```

We took that decision so that the internal may be modified without breaking to public root API. These have very rarely been updated, however.

Removing the Flax and TF imports from the init isn't an option either, unfortunately, as it would result in a breaking change for users that do use it.

Agree with you, Can I close this issue then?

Yes, thanks for opening it in the first place!
[Data2Vec] Incompatibility with the original implementation
Author: arxyzan
Pull request: no

Hello dear HuggingFace team!
According to the original paper, data2vec is not an actual model but more of a self-distilling training strategy. It takes an encoder model as backbone (RoBERTa for text, BEiT for vision, wav2vec for audio as mentioned in the paper) and pre-trains the encoder (student) to predict representations extracted from the EMA instance of the encoder (teacher), meaning the encoder can be any Transformer-based encoder model.
After pretraining, in order to finetune or get predictions, the encoder itself is what matters and data2vec is of no use! (as seen [here](https://github.com/pytorch/fairseq/tree/main/examples/data2vec#finetuning-data2vec-text-on-glue))
I reviewed data2vec implementation in HF transformers and noticed that you decided to use static encoders (BERT for text, BEiT for vision and wav2vec2 for audio) so for example, using Data2VecVisionModel for any task would be the same as using BEiTModel. 
Also I noticed that the encoders used for HF Data2Vec are not exactly the same models I mentioned above and there are some minor differences. The reason I'm wondering this, is because I was trying to copy the weights from your models to apply them to my own models in [my own repo](https://github.com/AryanShekarlaban/data2vec-pytorch) and found out that I can't due to those incompatibilities. 
So my question is, what was the purpose behind all this? and did you train all those models or copied the weights from the original checkpoints in fairseq?

Regards,
Aryan

Comments:
Replace dict/BatchEncoding instance checks by Mapping
Author: sgugger
Pull request: yes

# What does this PR do?

We have several instance checks in the code base for `(dict, BatchEncoding)` (because `BatchEncoding` is a `UserDict` which is not an instance of `dict`). Those all miss the newer `BatchFeatures` (which is another `UserDict`) as was pointed out in #16983 

In Accelerate we use the more general `Mapping` from `collections.abc` for those checks (which catches any kind of `dict`), this PR suggest to do this here too.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Result of new doc style with fixes
Author: sgugger
Pull request: yes

# What does this PR do?

This PR shows the changes in Transformers that will be occasioned by the new release of `doc-builder` with some fixes in the style command.

Code quality will fail util the next release of `hf-doc-builder` this PR will be merged just after.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add a check on config classes docstring checkpoints
Author: ydshieh
Pull request: yes

# What does this PR do?

A follow-up for #16900: add a test to make sure all config classes have at least one valid checkpoint (unless explicitly specified to ignore).

By `valid`, it only means the format is valid, i.e. of the form `[XXX](https://huggingface.co/XXX)` with `XXX` being any string.
A more strict verification could be implemented by trying to load the config. But maybe it is a bit too much?

Also fix 2 more config classes without valid checkpoint.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Arg to begin `Trainer` evaluation on eval data after n steps/epochs
Author: OllieBroadhurst
Pull request: no

# 🚀 Feature request

It would be great to be able to have the option to delay evaluation on the eval set until after a set number of steps/epochs.

## Motivation

Evaluation metrics can take a while to calculate and are arguably often not as meaningful in the early stages of training (like during warmup). This change would free up time that would be better spent training. Increasing `eval_steps` works until you end up waiting a while at the business end of training so this would not be an ideal compromise.

## Your contribution

First maybe have a training argument such as `evaluation_delay`.

Then:

(Adapted from [here](https://github.com/huggingface/transformers/blob/v4.17.0/src/transformers/trainer.py#L1578).)

```python

if self.args.evaluation_delay is not None:
    if self.args.save_strategy == IntervalStrategy.STEPS:
        is_delayed = self.state.global_step <= self.args.evaluation_delay
    else:
        is_delayed = epoch <= self.args.evaluation_delay

if self.control.should_save and not is_delayed:
    self._save_checkpoint(model, trial, metrics=metrics)
    self.control = self.callback_handler.on_save(self.args, self.state, self.control)
```

Maybe get some input from @sgugger?

Thanks!

Comments:
That sounds like a very interesting new feature, and I agree with the proposed implementation. Would you like to work on a PR for this?

Sure thing! @sgugger 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Can we support the trace of ViT and Swin-Transformer based on torch.fx?
Author: maggiez0138
Pull request: no

# 🚀 Feature request

Can we support the trace of ViT and Swin-Transformer based on torch.fx()? If not, what's the difficulty for that?

## Motivation

ViT and Swin-Transformation is widely used in CV scenarios. Hope we could support the trace of torch.fx, so we can do the quantization based on the work.

## Your contribution

I'm not sure right now. Currently I wanda whether we could support the ViT and Swin-Transformer. If not, the reason. 


Comments:
@michaelbenayoun   May I know the current situation or your schedule first? Thanks.

Hi, 
The plan is to work on a nicer solution for tracing the model as soon as possible.
I will take a look to add those models in the mean time as they might not be hard to add in the current setting.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
ibert seems to be quite slow in quant_mode = True
Author: huu4ontocord
Pull request: no

Hi, for iBert - 

I found it is MUCH slower in quant_mode = True. here's a notebook with a slightly modified version of the HF code to allow dynamically switching quant_mode. You can see the timing difference.

https://colab.research.google.com/drive/1DkYFGc18oPvAn5nyGEL1aIFHmD_aNlXW

 @patrickvonplaten

https://github.com/kssteven418/I-BERT/issues/21


Comments:
Hey @ontocord,

I don't have much experience with iBERT. @kssteven418 any ideas here? Is such a slowdown expected? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Enable the use of private models in example scripts using use_auth_token
Author: KMFODA
Pull request: no

# 🚀 Feature request

Currently private models can be used with the examples scripts by setting the `use_auth_token argument` as `True` however this doesn't seem to be an option for private datasets.

## Motivation

Enabling the use of private datasets using the example scripts

## Your contribution

I'm happy to work on a PR to add this to all example scripts if this is indeed the case and I haven't missed another method for using private datasets.


Comments:
cc @sgugger 

Sure, I believe it's just a matter of passing the token in the right places :-)

Great thanks @sgugger. Will work on a PR adding those in :)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
The current equivalent of transformers.models.bert.modeling_bert.gelu
Author: Pzoom522
Pull request: no

### System Info

```shell
- `transformers` version: 4.5.0
- Platform: Linux-5.13.0-40-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyTorch version (GPU?): 1.11.0+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
```


### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Hi, I believe it's an issue relevant to version migration. The code I'm using contains
```
x = transformers.models.bert.modeling_bert.gelu(x)
```
which seems to be no longer usable.

Similar problem was discussed in https://stackoverflow.com/questions/66133626 but with no good answer.

### Expected behavior


Please let me know what is the current API that can be a replacement of `transformers.models.bert.modeling_bert.gelu`, or it's safe to directly use `x = x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))` instead.

Many thanks!

PS: the link to the migration guide should be changed into https://huggingface.co/docs/transformers/migration.



### Checklist

- [X] I have read the migration guide in the readme. ([pytorch-transformers](https://github.com/huggingface/transformers#migrating-from-pytorch-transformers-to-transformers); [pytorch-pretrained-bert](https://github.com/huggingface/transformers#migrating-from-pytorch-pretrained-bert-to-transformers))
- [X] I checked if a related official extension example runs on my machine.

Comments:
All of those have been refactored in a `ACT2FN` dictionary which is used across the codebase. You can import it as such:

```py
from transformers.activations import ACT2FN

gelu_function = ACT2FN['gelu']
```

Hope that helps!

Many thx. Closed : )
Add translating guide
Author: omarespejel
Pull request: yes

# What does this PR do?
Add a translation guide so users have all the information they need to (1) contribute to a language that's already being translated, or (2) start their own issue for translating into a new language.

# Next step
Create a Translation Template for new issues (for example, [this template for Portuguese](https://github.com/huggingface/transformers/issues/16824) with all the docs that should be translated). I can do this.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks for the comments @sgugger! 

I moved the `_toctree.yml` tip to a part where it would be more relevant. Please let me know if you would prefer it in another part 🤗

LGTM!
Add CANINE and TAPAS to doc tests
Author: NielsRogge
Pull request: yes

# What does this PR do?

Adds CANINE and TAPAS to the doc tests.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_16354). All of your documentation changes will be reflected on that endpoint.

Thank you working on this, @NielsRogge 

Do you think it's possible to:
- Add more realistic examples for `TapasForQuestionAnswering`, `TapasForSequenceClassification`, etc.
- Add expected output shape for `TapasModel` and make its docstring output the shape instead of the logits.

@patrickvonplaten Do you think it is good to (always) ask the contributors to change the docstrings of `ModelForXXX` to something more meaningful (i.e. instead of the logits), as my comments here above?

Or should I just review only the changes done in their PRs?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[FlaxGenerate] Fix bug in `decoder_start_token_id`
Author: sanchit-gandhi
Pull request: yes

In Python, `bool` is a subclass of `int`, and `False` has the value `0`. We observe this by calling the `__bool__` method of `0`:
```python
print((0).__bool__())
print((1).__bool__())
```
```
False
True
```
https://github.com/huggingface/transformers/blob/da47c264f9a881f5db5f6fbb59a30c95e428571f/src/transformers/generation_flax_utils.py#L266-L268
In the preceding lines of code, if `decoder_start_token_id` has the value `0` (valid): 

- `if decoder_start_token_id` will be `False`
- `decoder_start_token_id` will be set to `self.config.decoder_start_token_id`

The correct behaviour should be that if `decoder_start_token_id` has the value `0`, it remains set to `0`, and not changed to `self.config.decoder_start_token_id`.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[Flax(Speech)EncoderDecoder] Fix bug in `decoder_module`
Author: sanchit-gandhi
Pull request: yes

The current use of `decoder_module` assumes that `encoder_hidden_states` is the fourth positional argument of the decoder's call method. We see that this is indeed true of the two current Flax decoder models: [`FlaxGPT2LMHeadModel`](https://github.com/huggingface/transformers/blob/da47c264f9a881f5db5f6fbb59a30c95e428571f/src/transformers/models/gpt2/modeling_flax_gpt2.py#L691) and [`FlaxBartForCausalLM`](https://github.com/huggingface/transformers/blob/da47c264f9a881f5db5f6fbb59a30c95e428571f/src/transformers/models/bart/modeling_flax_bart.py#L1911). However, for other possible decoder models, such as the work-in-progress [`FlaxBertForCausalLM`](https://github.com/huggingface/transformers/blob/9c9e49bd3aeb3f84c0d61b7f0fdca8ea853ac5a1/src/transformers/models/bert/modeling_flax_bert.py#L1545), there may be additional positional arguments (such as `token_type_ids` or `head_mask`) **prior** to `encoder_hidden_states`. To handle this more general case, we should not assume `encoder_hidden_states` is necessarily the fourth positional argument, and should instead pass it as a _key-word argument_.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix typo in RetriBertTokenizer docstring
Author: mpoemsl
Pull request: yes

# What does this PR do?

Fixes typo in RetriBertTokenizer docstring.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sgugger

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Disable Flax GPU tests on push
Author: sgugger
Pull request: yes

# What does this PR do?

The Flax GPU tests have been failing for more than a month on every commit they are run in master (error changed 20 days ago to an install error). This is making CI checks hard to read, so disabling those tests until someone really fixes them.

Comments:
_The documentation is not available anymore as the PR was closed or merged._