noamaanMulla-03/transformers-issues huggingface dataset:
Generate: validate `model_kwargs` (and catch typos in generate arguments)
Author: gante
Pull request: yes

# What does this PR do?

A common cause for issues in generate is around it not behaving as expected, as arguments can be silently ignored as part of the selected generation submethod (greedy_search, sample, ...). Typos also often fly under the radar, as the method accepts **model_kwargs, which in turn are passed to models that also accept **kwargs.

This PR solves the low-hanging fruit (derived from https://github.com/huggingface/transformers/pull/18218): validates `model_kwargs`, which notifies the users about problems in model arguments AND about typos, as they will fall in `model_kwargs`.

Will open a PR with TF and FLAX equivalent after this one gets merged. A solution for the other generation arguments is also on the way :) 

Fixes https://github.com/huggingface/transformers/issues/18130

___________________

Here is an example of the output for

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("distilgpt2")
model = AutoModelForCausalLM.from_pretrained("distilgpt2")

prompt = tokenizer(["hello world"], return_tensors="pt")
model.generate(**prompt, do_samples=True, foo="bar")
```

![image](https://user-images.githubusercontent.com/12240844/180473721-1e216e12-b25a-470e-b4cd-2d3236f8e8ff.png)


Comments:
_The documentation is not available anymore as the PR was closed or merged._

@sgugger ready for a final review :) [equivalent TF and FLAX changes will come in separate PRs, as they might need test corrections like this one]

Super-useful, thank you, @gante!
[doc] fix anchors
Author: stas00
Pull request: yes

the manual anchors end up being duplicated with automatically added anchors and no longer work.

Examples:
https://huggingface.co/docs/transformers/v4.21.1/en/glossary#input-ids
https://huggingface.co/docs/transformers/v4.21.1/en/glossary#position-ids

I confirmed that this fix works via the auto-generated docs link.

@sgugger 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[fsmt] deal with -100 indices in decoder ids
Author: stas00
Pull request: yes

Fixes: https://github.com/huggingface/transformers/issues/17945

decoder input ids get the default index -100, which breaks the model - like t5 and many other models add a hardcoded fix to replace -100 with the correct pad index. 

For some reason this use case hasn't been used with this model until recently - so this issue was there since the beginning it seems.

Any suggestions to how to add a simple test here? or perhaps we have something similar already? user's script is quite massive. I think it's Trainer's collate that leads to that padding since it uses -100.

@sgugger 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[bnb] Fix non passing trainer tests
Author: younesbelkada
Pull request: yes

# What does this PR do? 

It fixes a small slow test that was not passing due a very small typo when designing the tests in https://github.com/huggingface/transformers/pull/15622 

cc @ydshieh 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thank you all!
Cannot get WER during WavVec2 fine-tuning
Author: changyeli
Pull request: no
Labels: name: bug

### System Info


- `transformers` version: 4.21.1
- Platform: Linux-5.15.0-43-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.8.1
- PyTorch version (GPU?): 1.12.0+cu116 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

@patrickvonplaten, @anton-l

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I followed the [blog](https://huggingface.co/blog/fine-tune-wav2vec2-english) and revisited one of my old project, but I couldn't get WER during the fine-tuning. Surprisingly it worked perfectly early this year. I did get the following evaluation metrics during the first round:

```
{'eval_loss': -0.26198065280914307, 'eval_runtime': 41.4222, 'eval_samples_per_second': 25.904, 'eval_steps_per_second': 6.494, 'epoch': 0.27}
```

As you can see, there was no `eval_wer` in this entry.  Tried the following, still not seeing `eval_wer`

```python
def compute_metrics(pred):
    """
    batchfy and compute the WER metrics

    :param pred: _description_
    :type pred: _type_
    :return: _description_
    :rtype: _type_
    """
    wer_metric = load_metric("wer")
    pred_logits = pred.predictions # change to pred.logits did not help
    pred_ids = np.argmax(pred_logits, axis=-1)
    pred.label_ids[pred.label_ids == -100] = processor.tokenizer.pad_token_id
    pred_str = processor.batch_decode(pred_ids)
    # we do not want to group tokens when computing the metrics
    label_str = processor.batch_decode(pred.label_ids, group_tokens=False)
    wer = wer_metric.compute(predictions=pred_str, references=label_str)
    return {"wer": wer}



for batch in trainer.get_eval_dataloader():
    print(batch.keys())
    # return dict_keys(['input_values', 'labels'])
    batch = {k: v.to("cuda") for k, v in batch.items()}
    print(trainer.evaluate())
    # return {'eval_loss': 0.29788103699684143, 'eval_runtime': 44.4312, 'eval_samples_per_second': 24.15, 'eval_steps_per_second': 6.054}
    break
```

Any suggestions? Thanks!



### Expected behavior

Should return `eval_wer` in the evaluation step.

Comments:
Add Donut
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR adds Donut to the library. Donut is to LayoutLM what T5 is to BERT. :D 

The model is implemented as an instance of our existing `VisionEncoderDecoderModel`.

See also https://github.com/clovaai/donut/issues/10#issue-1324734927

To do:

- [x] move repos to appropriate organization in
- [x] update niels to appropriate organization in `test_modeling_vision_encoder_decoder.py`

Comments:
_The documentation is not available anymore as the PR was closed or merged._

I have implemented a new `DonutSwinModel`, that copies everything of `SwinModel`, except the final layer norm. I've added it in a file called `modeling_donut_swin.py` (and implemented a corresponding `DonutSwinConfig` in `configuration_donut_swin.py`). 

I went with `modeling_donut_swin.py` (and `configuration_donut_swin.py`) in the "donut" folder rather than `modeling_donut.py` (and `configuration_donut.py`) since it only implements the model and configuration of the encoder part (Swin Transformer). For the decoder, BART is leveraged. Let me know if this is ok.

Hi @NielsRogge , do you plan on supporting: `Document Parsing` modality?
Trainer dataloader_drop_last=True looks can not work with distributed setting
Author: yiwc
Pull request: no
Labels: name: bug

### System Info

 dataloader_drop_last=True , works fine in the single GPU setting. 

But fails while using distributed launch
`python -m torch.distributed.launch --nproc_per_node $NUM_GPU --master_port $PORT_ID train_model.py`

it fails when compute metrics, I checked the shape, it's not as expected.

Thank you team,
@sgugger 

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

define a custom compute_metrics function, and set trainer with  dataloader_drop_last=True, and use distributed launch on single machine multiple GPU `python -m torch.distributed.launch --nproc_per_node $NUM_GPU --master_port $PORT_ID train_model.py`.

you can check the input of pred shape in the compute_metrics function, and found it is not correct. (doesn't drop the last)

### Expected behavior

 dataloader_drop_last=True should works fine while using `python -m torch.distributed.launch --nproc_per_node $NUM_GPU --master_port $PORT_ID train_model.py`

Comments:
I have trouble understanding the bug you are reporting. If you set `dataloader_drop_last=True`, the dataloader you use for evaluation will have its last batch truncated and yes, you won't have the right number of examples.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Segformer ouput size
Author: joihn
Pull request: no
Labels: name: bug

### System Info

thanks for this repo,

Segformer `output size` is `input_size/4,` as mentionned here
https://github.com/huggingface/transformers/blob/main/src/transformers/models/segformer/modeling_tf_segformer.py#L780

However, this line of documentation is wrong:
https://github.com/huggingface/transformers/blob/main/src/transformers/models/segformer/modeling_tf_segformer.py#L850

By the way, what would be the easiest way to augment the ouput size, adding upsampling layers at the end ? 

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

read the code :) there is an inconsitency in the doc :)

### Expected behavior

the doc. should be consistent

Comments:
cc @NielsRogge -- is the right logits shape `(batch_size, num_labels, height, width)` or `(batch_size, num_labels, height/4, width/4)`?

@joihn depending on @NielsRogge answer, would you like to open a PR to fix the documentation? :) The PyTorch model has the same comments, that may need to be fixed.

Hi,

Yes it should be `(batch_size, num_labels, height/4, width/4)`.

PR merged
Post-processing for HumanEval code generations not working properly
Author: loubnabnl
Pull request: no
Labels: name: bug

### System Info

Not system-dependent

### Who can help?


### Information

- [X]  The official example scripts
- [ ] My own modified scripts

### Tasks

- [X]  An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The post-processing function for HumanEval code generation in CodeParrot doesn't work as expected, precisely this function 
https://github.com/huggingface/transformers/blob/d6eeb871706db0d64ab9ffd79f9545d95286b536/examples/research_projects/codeparrot/scripts/human_eval.py#L67
It returns an empty string if no EOF_STRING is present:
```python
EOF_STRINGS = ["\nprint"]
def remove_last_block(string):
    """Remove the last block of the code containing EOF_STRINGS"""
    string_list = re.split("(%s)" % "|".join(EOF_STRINGS), string)
    # last string should be ""
    return "".join(string_list[:-2])

example = "def somme(x,y)\n  return x+y"
print(f"example :\n{remove_last_block(example)}")
```

```
example :

```
Going to an old version of the repo, I found we had this function instead which works properly so I'm wondering why we changed it, it would also be more practical in case no stopping criteria at EOF_STRINGs was used during the generation as it only keeps the first block
```python
def first_block(string):
    """Split off first block of code by scanning for class, def etc. on newlines."""
    return re.split("|".join(EOF_STRINGS), string)[0].rstrip()

print(f"example :\n{first_block(example)}")
```
```
example :
def somme(x,y)
  return x+y
```

Comments:
Update: the case I mentioned would never occur with the stoppingcriteria in codeparrot as all generation should include an eof_string so no bug https://github.com/huggingface/transformers/blob/d6eeb871706db0d64ab9ffd79f9545d95286b536/examples/research_projects/codeparrot/scripts/human_eval.py#L59
Warnings for unexpected parameters when resuming training.
Author: Nilabhra
Pull request: no

### Feature request

The `Trainer` class should report warning messages if the provided training arguments for the resumed run, don't match with the stored configuration in the given checkpoint.

### Motivation

I started an MLM training with 9 epochs and then tried to resume the training til 15 epochs. I didn't realise at the time that the learning rate had already decayed to zero in the previous run. In the new run, the LR was re-computed and no warning was given, making me think the training is going on as expected. 

What would have helped is a warning message saying that the previous run was for 9 epochs and that cannot change in this run.

### Your contribution

N/A

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
NLP HydraNet
Author: sam-h-bean
Pull request: no

### Feature request

I think it would be awesome to be able to easily train a Tesla style HydraNet but using a transformer backbone. The model would take a model_id and a series of tasks. The dataset would need to supply labels for both the task heads and the loss would be controllable via coefficients in the configuration. 

### Motivation

This would allow people to train multi-task learners using the same neural backbone and co-learn a number of related tasks.

### Your contribution

I have some code working for this. Unsure if there is already a pattern for this type of thing or if this is novel enough to contribute back to OSS.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Include `timeout` attribute (related to DDP) to TrainingArguments
Author: gugarosa
Pull request: no

### Feature request

Would it be possible to include a `timeout` attribute to the `TrainingArguments` dataclass, such as it will be used as an argument of the `torch.distributed.init_process_group` calls?

Reference: https://pytorch.org/docs/stable/distributed.html#torch.distributed.init_process_group

### Motivation

Essentially, if a process uses DDP and performs a set of operations prior to using the GPUs, such as tokenization/mapping, for more than `timeout` seconds (defaults to 30 minutes), the process will stop and get killed due to a `Socket Timeout`. (issue #17106).

By adding a `timeout` argument to the `TrainingArguments` class, we will be able to let users decide in overriding the default timeout defined by PyTorch and hopefully introduce a way to prevent `Socket Timeouts` when mapping large datasets.

### Your contribution

I could definitely submit a PR, seems to be pretty straightforward to add a new attribute to the `TrainingArguments` class.

Comments:
Hi @gugarosa 

I have started the work on it. Will create PR for the same by tomorrow.

That's amazing @dvlshah! Thank you so much for doing it!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Configurations should support id2label as a list
Author: sijunhe
Pull request: no

### Feature request

This is a rather minor/trivial feature request.

Currently `id2label` is of type `Dict[int, str]` in `PretrainedConfig`. Since this map is used to map class ids to labels, the keys is set to `range(0, num_labels)`. Essentially, a list would also work here. I can even argue that a list is better in this case, because it avoids the potential error of the out-of-range keys and the weird serialization as
```
"id2label": {
    "0": "O",
    "1": "B-address",
    "2": "I-address",
    ...
}
```
 
Given that this attribute is mostly used for lookup `id2label[i]`, it should be pretty easy to support this field both as a `list` and a `map`. We just need to fix a few places where we assumed `.items()` API exists. 

I'll be happy to contribute a small PR for this.

### Motivation

described above

### Your contribution

Happy to contribute a PR for this.

Comments:
I would personally advocate for a single way of doing things, rather than many different ways of doing the same thing. In this situation, while the dict is more complex, it is also more complete. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
A basic NLP question regarding NER task
Author: liususan091219
Pull request: no

Hi @patrickvonplaten, I have one basic conceptual NLP question regarding the evaluation for NER.

According to [run_ner.py](https://github.com/huggingface/transformers/blob/main/examples/pytorch/token-classification/run_ner.py), the ground truth label is truncated to max_seq_length during prediction. However, this means the ground truth label will be changed. 

My question is: is the prediction still valid? For example, if the ground truth has 150 tokens, when max_seq_length = 128, both the prediction and label are truncated to 128, isn't the prediction required to contain 150 tokens for consistency of evaluation? 

Thank you very much in advance for your help, apologize if this is the wrong place for posting. 

Comments:
hello @liususan091219 . 
I believe the logic you are looking for is the [tokenize_and_align_labels](https://github.com/huggingface/transformers/blob/main/examples/pytorch/token-classification/run_ner.py#L420) function. Looks like the logic is as following:

- if `len(labels) > len(tokens)`, then the extra labels are thrown out to ensure the same length
- if `len(labels) < len(tokens)` (due to padding), then the extra labels with value of `-100` are added to ensure they have the same length

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Different sentiment class probabilities for sequential processing vs batch processing
Author: Kayne88
Pull request: no
Labels: name: bug

### System Info

platform: windows
python: 3.7
transformers: latest
Model: finetuned BERT (from cardiffnlp/twitter-roberta-base-sentiment)

I have posted the issue here [https://discuss.huggingface.co/t/different-sentiments-when-texts-processed-in-batches-vs-singles/19462](url), but didn't receive any answer. The behavior is not really explainable and might look like a bug.

Cheers

@LysandreJik 

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

see description under the above link.

### Expected behavior

I would expect the class probabilities to be equal for a given text, no matter if the classification is done sequentially or in batches.

Comments:
Hi @Kayne88 -- could you please share a complete script for reproducibility? In your [original script](https://discuss.huggingface.co/t/different-sentiments-when-texts-processed-in-batches-vs-singles/19462) you were missing the definition of `tokenizer` and `model` :)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Distributed training for streaming dataset
Author: cyk1337
Pull request: no

### Feature request

Any documentations for the the `load_dataset(streaming=True)` for (multi-node multi-GPU) DDP training? 

### Motivation

Given a bunch of data files, it is expected to split them onto different GPUs. Is there a guide or documentation?

### Your contribution

Does it requires manually split on data files for each worker in `DatasetBuilder._split_generator()`?

Comments:
Hi @cyk1337 -- this seems like a question for the [datasets repo](https://github.com/huggingface/datasets) :)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add EfficientNet model for PyTorch
Author: tanaymeh
Pull request: yes

# What does this PR do?
This PR adds the EfficientNet model family to HuggingFace transformers proposed in #15759 (PyTorch only for this PR).
The implementation is based on that of Ross Wightman's pytorch-image-models [implementation](https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/efficientnet.py).

## Who can review?
@NielsRogge 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Working on this.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.