noamaanMulla-03/transformers-issues huggingface dataset:
[Trainer] Very different loss reported during training vs. at the end.
Author: aqred1
Pull request: no

I have trained GPT from scratch. During training, I got logs outputting evaluation and 'loss', like shown below:

`{'loss': 6.5349, 'learning_rate': 0.00014698625308424393, 'epoch': 42.65}`

But after the last step I get this log, different to the previous ones:

`{'train_runtime': 751.251, 'train_samples_per_second': 580108.554, 'train_steps_per_second': 566.455, 'train_loss': 0.008420203693772214, 'epoch': 50.0}`

Besides, I get this metrics:

```
 ***** train metrics *****
epoch                    =       50.0
train_loss               =     0.0084
train_runtime            = 0:12:31.27
train_samples            =    8716143
train_samples_per_second = 580087.323
train_steps_per_second   =    566.434
```

The values of `train_loss` and `loss` are remarkably different, so what's each of them referring to? And what is the real training loss? If it's the last one, what is the point of plotting the training curves using the variable `loss`?



Comments:
cc @sgugger 

It looks like the reported `"train_loss"` is averaged twice, so it's not accurate. Will fix this week.

I was actually wrong, and the training loss is properly computed (see #15096 that adds a test which is passing). Are you sure you are using the most recent version of Transformers?

I trained this model using transformers==4.14.1. Was it fixed after that?  If so, would I need to train again?
Training takes several weeks so updating to every new version is not possible in my case.

I think the value of 'train_loss' is meant to be different to 'loss'. If you follow the code: https://github.com/huggingface/transformers/blob/master/src/transformers/trainer.py

logs["loss"] = round(tr_loss_scalar / (self.state.global_step - self._globalstep_last_logged), 4)
'loss': loss on the batch data within one step only

train_loss = self._total_loss_scalar / self.state.global_step
'train_loss' average on all steps




This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Fix broken code blocks in README.md
Author: upura
Pull request: yes

# What does this PR do?

Fix a broken code blocks in `README.md` at [`transformers/examples/pytorch/contrastive-image-text`](https://github.com/huggingface/transformers/tree/master/examples/pytorch/contrastive-image-text).

## Current screenshot

<img width="1030" alt="Screen Shot 2022-03-07 at 22 45 57" src="https://user-images.githubusercontent.com/31459778/157046074-24163384-b132-4e98-b179-40137e382d9a.png">

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests), Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Use tiny models for get_pretrained_model in TFEncoderDecoderModelTest
Author: ydshieh
Pull request: yes

# What does this PR do?

Use tiny models for `get_pretrained_model` in `TFEncoderDecoderModelTest`.

This is originally for avoiding **GPU OOM** for `TFRembertEncoderDecoderModelTest` on CI daily testing.
But @patrickvonplaten suggests that we should actually use the small model in the following quote:

_... think we can rename it to test_model_save_loaf_from_pretrained(...) :wink: I think this "real" name was propragated since the first encoder-decoder tests existed in PyTorch. **Since the test does no integration testing** (e.g. checking if the output corresponds to something reasonable) **it makes 0 difference whether we use dummy weights or no** dummy weights here ..._

Comments:
_The documentation is not available anymore as the PR was closed or merged._
how can we use the model output to do predict?
Author: wenlai-lavine
Pull request: no

Hi, I am using transformers to do generation now. especially in ```m2m100``` model. We can easily call ```model.generate()``` to generate the sentences, but can not do predict using the model outputs. can anyone help me? @patrickvonplaten @Narsil 
For example, in the following code, what should we do after we have ```outputs``` from the model to generate the translation text?
```
from transformers import M2M100Config, M2M100ForConditionalGeneration, M2M100Tokenizer

model = M2M100ForConditionalGeneration.from_pretrained("facebook/m2m100_418M")
tokenizer = M2M100Tokenizer.from_pretrained("facebook/m2m100_418M", src_lang="en", tgt_lang="fr")

src_text = "Life is like a box of chocolates."
tgt_text = "La vie est comme une boîte de chocolat."

model_inputs = tokenizer(src_text, return_tensors="pt")
with tokenizer.as_target_tokenizer():
    labels = tokenizer(tgt_text, return_tensors="pt").input_ids

outputs = model(**model_inputs, labels=labels)
```
I mean, whether we can use the ```logits``` to generate the text? Any suggestions will be welcome.

Comments:
Hi

I've answered this question on our forum here: https://discuss.huggingface.co/t/generate-without-using-the-generate-method/11379?u=nielsr

For such questions, please use the forum rather than GitHub issues, which are meant for bugs/feature requests.

Thanks!

@NielsRogge Thanks for the reply, this really help me and solve my problems.
what us the difference between Trainer and Seq2SeqTrainer ?
Author: Darshan2104
Pull request: no

Anybody?

Comments:
I advise you to use Google, it's pretty great. 
<img width="892" alt="Screenshot 2022-03-10 at 09 41 21" src="https://user-images.githubusercontent.com/48327001/157622136-0a1d5ced-20dd-4d41-bbad-a1d0088f02f3.png">


=> answer explained here: https://discuss.huggingface.co/t/trainer-vs-seq2seqtrainer/3145
Freeze Feature Encoder in FlaxSpeechEncoderDecoder
Author: sanchit-gandhi
Pull request: yes

This PR builds on #15873 by enabling the feature encoder of a Flax Wav2Vec2 model to be frozen when used in the FlaxSpeechEncoderDecoder framework.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Add one quick test here maybe as well? :-)
Learning rate finder for the trainer 
Author: franz101
Pull request: no

Hi,

is it possible to integrate a learning rate finder with the trainer api?
Something like this:
https://github.com/davidtvs/pytorch-lr-finder

It's probably has been done before, but couldn't find it yet.

Comments:
The LR finder does not give reliable results for Transformers models (usually indicates one in the 1e-3 when the actual best value is in the 1e-5 range). That's why we don't have support for it.

@sgugger thanks for the explanation. Any other best practices?
Add Document Image Transformer (DiT)
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR adds the conversion script used to convert DiT checkpoints from the [original repo](https://github.com/microsoft/unilm/tree/master/dit) to the hub.

It also adds a dedicated docs page (referring to the docs of BEiT).

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Failing test is unrelated, therefore merging.
[Tests] Add attentions_option to ModelTesterMixin
Author: NielsRogge
Pull request: yes

# What does this PR do?

The library is called HuggingFace Transformers, I know.. but we recently have additions of non-Transformer based models, that don't use attention (namely ConvNeXT and PoolFormer). Soon, we'll also have ResNet in the library (which will back Transformer-based models such as DETR).

As these models don't use attention, they need to overwrite 3 tests:
* `test_attention_outputs`
* `test_retain_grad_hidden_states_attentions`
* `test_model_outputs_equivalence`

This PR adds a `has_attentions` attribute to `ModelTesterMixin` which can be set to False for such models. It will then make sure these tests are properly tested, without taking into account attention.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

You'll need to fix the failing tests as well.

Thanks for this work @NielsRogge !

I left a review comment (question) about `test_forward_signature`.

And then a minor question here:

https://github.com/huggingface/transformers/blob/114b62cc9927d85804540d46a6f85e57ba13ea51/tests/test_modeling_common.py#L1046-L1049

It is a bit strange (to me) to have `config.output_attentions = True` without any condition like `if self.has_attentions:`. WDYT?

@LysandreJik I've fixed the failing tests, failing test seems unrelated

Feel free to merge if every test passes after a rebase
Fix  TFDebertaV2ConvLayer in TFDebertaV2Model
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix a [CI failure for TFDebertaV2Model](https://github.com/huggingface/transformers/runs/5474315111?check_suite_focus=true), caused by the mistake in `TFDebertaV2ConvLayer`.

## Remark

This test `test_inference_no_head` also fails with the version in #13120. I think this slow test was not run manually to ensure it works before being merged to master.

## Code to demonstrate the issue and the effect of this PR

This is adapted from [test_inference_no_head](https://github.com/huggingface/transformers/blob/a69e185074fff529ed60d936c6afe05580aee8ac/tests/deberta_v2/test_modeling_tf_deberta_v2.py#L269)

```python
########## Prep ########## 

import numpy as np
import torch
import tensorflow as tf
from transformers import DebertaV2Model, TFDebertaV2Model

input_ids = np.array([[0, 31414, 232, 328, 740, 1140, 12695, 69, 46078, 1588, 2]], dtype=np.int32)
attention_mask = np.array([[0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]], dtype=np.int32)

########## PT ########## 

pt_model = DebertaV2Model.from_pretrained("microsoft/deberta-v2-xlarge")

input_ids_pt = torch.from_numpy(input_ids)
attention_mask_pt = torch.from_numpy(attention_mask)
pt_output = pt_model(input_ids_pt, attention_mask=attention_mask_pt)[0]

# compare the actual values for a slice.
pt_expected_slice = torch.tensor(
    [[[0.2356, 0.1948, 0.0369], [-0.1063, 0.3586, -0.5152], [-0.6399, -0.0259, -0.2525]]]
)
pt_output_slice = pt_output[:, 1:4, 1:4]

pt_slice_diff = np.abs(pt_expected_slice.detach().to("cpu").numpy() - pt_output_slice.detach().to("cpu").numpy())
max_pt_slice_diff = np.amax(pt_slice_diff)

print(f"max_pt_slice_diff = {max_pt_slice_diff}")

########## TF ##########

tf_model = TFDebertaV2Model.from_pretrained("microsoft/deberta-v2-xlarge")

input_ids_tf = tf.constant(input_ids)
attention_mask_tf = tf.constant(attention_mask)
tf_output = tf_model(input_ids_tf, attention_mask=attention_mask_tf)[0]

# compare the actual values for a slice.
tf_expected_slice = tf.constant(
    [[[0.2356, 0.1948, 0.0369], [-0.1063, 0.3586, -0.5152], [-0.6399, -0.0259, -0.2525]]]
)
tf_output_slice = tf_output[:, 1:4, 1:4]

tf_slice_diff = tf_expected_slice.numpy() - tf_output_slice.numpy()
max_tf_slice_diff = np.amax(tf_slice_diff)

print(f"max_tf_slice_diff = {max_tf_slice_diff}")

########## PT-TF ########## 

max_pt_tf_diff = np.amax(np.abs(pt_output.detach().to("cpu").numpy() - tf_output.numpy()))
print(f"maximal pt_tf_diff = {max_pt_tf_diff}")
```

This scripts gives

Before this PR

```python
max_pt_slice_diff = 5.037523806095123e-05
max_tf_slice_diff = 0.5608187317848206
maximal pt_tf_diff = 5.981985092163086
```

With this PR:

```python
max_pt_slice_diff = 5.037523806095123e-05
max_tf_slice_diff = 4.8374757170677185e-05
maximal pt_tf_diff = 0.000133514404296875
```


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Build the doc in a seperate folder then move it
Author: sgugger
Pull request: yes

# What does this PR do?

To fix the issue pointed out in #16019 this removes the need for stashing by:
- building the doc in a folder outside any repos
- pull the distant repos once the doc build is finished
- then move other the built doc to those repos and push

TODO: the dev job as well once the main job has been tested

Comments:
Don't compute metrics in LM examples on TPU
Author: sgugger
Pull request: yes

# What does this PR do?

This PR removes the accuracy computation in the LM examples on TPU, as TPUs can't handle the accumulation of the model logits.

Patches #16005 A better fix would be to be able to gather those logits on TPU but waiting for advice from the pytorch XLA team on this.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thank you for the fix!
How to accelerate the inference speed when using pipeline
Author: yanan1116
Pull request: no

> transformers version: 4.6.0.dev0
> torch version:1.8.1+cu102

I am using the simple API - pipeline, to do inference, where the input are tens of thousands of sentences.

```
nlp  = pipeline("text-generation", model= 'gpt2', device=0, return_full_text=False)
results = nlp(df_train['content'].tolist(), max_length=250, do_sample=True, top_p=0.9, top_k=0, \
                    repetition_penalty=1, num_return_sequences=64)
```
take the generation, for instance, I want to generate new synthesized samples from each sentence from the df_train.
The codes work well, but it is not fast enough. I mean the GPU usage is just 76% ~ 85%.
Is there any trick or parameters I can tune to speed up ?

Another question is that how can I eliminate the info:

> Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.





Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Hi @yananchen1989 did you get a solution yet
TF: Unpack model inputs through a decorator 
Author: gante
Pull request: yes

# What does this PR do?

Unpacks TF model inputs through a decorator, improving code clarity. To be read with issue https://github.com/huggingface/transformers/issues/15908, which holds the description of the problem, the proposed solution, and future plan.

Closes https://github.com/huggingface/transformers/issues/15908

How to review: start with `modeling_tf_utils.py`, and then check the changes in the models. I've run `RUN_SLOW=1 py.test -vv tests/model_name/test_modeling_tf_model_name.py` for all involved models, and the changes were applied to BERT (plus its copy-linked architectures) and Speech2Text, showing that it works for multiple modalities.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

As suggested by @sgugger in #15908, I've opened the PR :)

This looks great! Also pinging @patrickvonplaten for review as it's quite central

Thanks for running the tests on it. It is way more readable like that indeed.

Looks very nice! Thanks for cleaning this up

Looks very nice! Thanks for cleaning this up

It seems like everyone is in agreement :) @Rocketknight1 can you have a final look, and approve if you agree with the change?

> Is the plan to wait and see how things go with BERT, and assuming no problems to repeat those edits across all model classes?

Yeah! I'm also going to open an issue with the `good first issue` tag and try to get contributors in to replicate the change over other models. After a month or so, I will update any missing model.
How to use multi-node multi-gpu cluster for large batch beam-search decoding
Author: whaowhao
Pull request: no

I have a self-implemented-from-scratch torch.nn.Module GPT2 model (self-implemented for custom tweaks to the model) and now I have a 4-node-32-GPU cluster which I like to use to beam-search generate continuation for millions of prefixes. Is there a way to use hugginface as a wrapper to do this in a multi-node-multi-gpu fashion? I know it can be done by manually splitting the file into small chunks and individually beam-search generate,but it is messy.

Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co) instead?

Thanks!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Add FlaxBartForCausalLM
Author: sanchit-gandhi
Pull request: yes

This PR contributes the Flax Bart model for causal language modelling (Causal LM). It adds:

- `FlaxBartPreTrainedDecoderModel`: a subclass of `FlaxPreTrainedModel` designed specifically for leveraging pre-trained Bart decoder checkpoints. This is as opposed to `FlaxBartPreTrainedModel`, which treats pre-trained Bart encoder-decoder models.
- `FlaxBartDecoderWrapper`: a wrapper class required to correctly load pre-trained checkpoints when the causal language model is used in combination with the `(Speech)EncoderDecoderModel` framework.
- `FlaxBartForCausalLM`: a Bart Model with a language modelling (LM) head on top. It facilitates encoder-decoder cross-attention layers, thus enabling it to be used in the `(Speech)EncoderDecoderModel` for seq2seq tasks.

The implementation of the Causal LM model is validated through use as a decoder module in two key `(Speech)EncoderDecoderModel` tests:

- `FlaxBartEncoderDecoderModelTest`: The Flax Bart Causal LM model is used as a decoder module in a `Flax-Bert-2-Bart` seq2seq configuration. The PyTorch-Flax cross-tests assert that all Flax hidden-states are to within a `1e-5` tolerance of their PyTorch equivalents.
- `FlaxWav2Vec2BartModelTest`: The Flax Bart Causal LM model is used as a decoder module in a` Flax-Wav2Vec2-2-Bart` speech-encoder-decoder configuration. The PyTorch-Flax cross-tests assert that all Flax output logits are to within a `4e-2` tolerance of their PyTorch equivalents.

The PyTorch-Flax cross-tests verify that the CausalLM model has been correctly implemented and that it works as expected in its use case as a decoder module in an encoder-decoder framework.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix Bug in Flax Seq2Seq Models
Author: sanchit-gandhi
Pull request: yes

This PR makes two changes to each of `FlaxEncoderDecoderModel` and `FlaxSpeechEncoderDecoderModel`:

1. Amends the input docstrings to remove incorrect information about the model "_shifting tokens right for denoising_". In Flax, `decoder_input_ids` are obtained by shifting the target labels right **outside** of the seq2seq model, not **within** as stated in the docstrings.
2. Raises a `ValueError` if `decoder_input_ids` are not provided. The current behaviour allows for `decoder_input_ids` to be omitted, in which case they default to `None`. This causes errors when `decoder_input_ids=None` is manipulated with JAX functions to build the `decoder_attention_mask` and `decoder_position_ids` should they be omitted from the arguments too.

The following code snippet throws the error aforementioned in 2:
```python
from transformers import FlaxSpeechEncoderDecoderModel
import jax.numpy as jnp
model = FlaxSpeechEncoderDecoderModel.from_encoder_decoder_pretrained('hf-internal-testing/tiny-random-wav2vec2', 'hf-internal-testing/tiny-random-gpt2', encoder_from_pt=True, decoder_from_pt=True)
inputs = jnp.ones((2, 5000), dtype=jnp.float32)
outputs = model(inputs)
```

Output:
```python
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/sanchitgandhi/transformers/src/transformers/models/speech_encoder_decoder/modeling_flax_speech_encoder_decoder.py", line 688, in __call__
    decoder_attention_mask = jnp.ones_like(decoder_input_ids)
  File "/Users/sanchitgandhi/venv/lib/python3.8/site-packages/jax/_src/numpy/lax_numpy.py", line 3706, in ones_like
    _check_arraylike("ones_like", a)
  File "/Users/sanchitgandhi/venv/lib/python3.8/site-packages/jax/_src/numpy/lax_numpy.py", line 570, in _check_arraylike
    raise TypeError(msg.format(fun_name, type(arg), pos))
TypeError: ones_like requires ndarray or scalar arguments, got <class 'NoneType'> at position 0.
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix Bug in Flax-Speech-Encoder-Decoder Test
Author: sanchit-gandhi
Pull request: yes

This PR fixes a bug in a Flax-Speech-Encoder-Decoder test that was failing after push (https://github.com/huggingface/transformers/runs/5493110105?check_suite_focus=true). Specifically, it amends the `test_freeze_feature_encoder` test to **omit** the random `decoder_attention_mask` from the input arguments of the speech-encoder-decoder model. This random `decoder_attention_mask` was resulting in `nan` values on the output logits of the `FlaxWav2Vec2BartModelTest`. Removing it as an input results in real valued output logits, and the test of concern passes following this change. The behaviour of the FlaxBartForCausalLM model to output `nan` values given a random `decoder_attention_mask` has been noted.

Comments:
_The documentation is not available anymore as the PR was closed or merged._