noamaanMulla-03/transformers-issues huggingface dataset:
improving efficiency of mlflow metric logging
Author: wamartin-aml
Pull request: yes

Signed-off-by: Walter Martin <wamartin@microsoft.com>

# What does this PR do?

This PR switches MLflow metric logging over to the more efficient log_metrics API, which logs in a batch.



## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
@sgugger , @jingyanwangms , @harshithapv , I've moved the PR to the original repository as suggested. A review would be great!
Add BeitForSemanticSegmentation
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR is a follow-up of #12994, and adds the semantic segmentation head of [BEiT](https://arxiv.org/abs/2106.08254). It's the state-of-the-art model currently for semantic segmentation (i.e. the task of labeling each pixel of an image), on datasets like [ADE20k](https://groups.csail.mit.edu/vision/datasets/ADE20K/) and [CityScapes](https://www.cityscapes-dataset.com/) (see [this chart](https://paperswithcode.com/sota/semantic-segmentation-on-ade20k-val) on paperswithcode).

Now it's easily available with a HuggingFace API! :)

Models are on the hub: https://huggingface.co/models?search=ade-640

Here's a notebook for quick inference: https://colab.research.google.com/drive/1AS3z0plOhWWibBvDgsQkRbfJB73vSsR3?usp=sharing

Comments:
@NielsRogge 
Do you have plans to add an example fine-tuning script for the segmentation task?

Hi @kamalkraj,

Yes I do plan to add that. However, this will become easier once the Image feature will be available in the Datasets library.

Okay @NielsRogge 

I can update the flax version of beit, after this PR merge
Fix generation docstring
Author: qqaatw
Pull request: yes

# What does this PR do?

As mentioned in this [comment](https://github.com/huggingface/transformers/issues/14206#issuecomment-955607275), the `add_prefix_space` argument is not available in `GPT2TokenizerFast.__call__` but `GPT2Tokenizer.__call__`. 

Therefore, this PR fixes this error by switching the fast tokenizer to the slow one.


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@patrickvonplaten, @LysandreJik

Comments:
Thank you!
Can deepspeed and gradient checkpointing be used at the same time?
Author: lockon-n
Pull request: no

I'm finetuning roberta-large for some downstream tasks, and i'm wondering if deepspeed and gradient checkpointing can be used at the same for much more GPU memory reduction? Is there any conflict between these two?

Comments:
Yes.

See the guide: https://huggingface.co/transformers/master/performance.html
Update Transformers to huggingface_hub >= 0.1.0
Author: sgugger
Pull request: yes

# What does this PR do?

This PR updates the minimum version of huggingface_hub to 0.1.0 and makes all the simplifications related to `HfApi` methods being exposed as functions.

Note that some of the commands that relied on the old HfApi were failing since the switch a few versions ago but no issue was opened. This PR removes the code and just returns a somewhat helpful error message.

This PR also fixes one push_to_hub test for the Trainer, currently failing if there are several GPUs.

Comments:
Add an example of exporting BartModel + BeamSearch to ONNX module.
Author: fatcat-z
Pull request: yes

This PR is aimed to deliver an example to describe how to export BartModel + BeamSearch to ONNX model and inference it by ONNXRuntime.

These new example files were added without updates to existing Huggingface model code. It mainly focuses on:

1. Providing an entry file run_onnx_exporter.py to run this example.
2. In generation_onnx.py file, adding a BARTGenerator model to wrap the native BartModel code. To support Beam Search, we need to convert the whole model to a TorchScript module finally. This new BARTGenerator model will trace the Encoder and Decoder of BartModel to make them be compatible with torch.jit.script() method.
3. In generation_onnx.py file, copying Beam Search code from generation_utils.py and updating these code so that beam_search function could be converted to TorchScript by calling torch.jit.script method.



Comments:
Think we just need to run `make style` once and then we can merge - lgtm!

Thanks for adding this! Could you perhaps add a README to this example, to explain what the example is about and how to use it?

Hey @fatcat-z,

kindly pinging you for taking into account the comments on this PR (perhaps in a follow-up PR), if possible! 

> Thanks for adding this! Could you perhaps add a README to this example, to explain what the example is about and how to use it?

I'm working on a new PR to address your and Gary's comments.
Deprecate prepare_seq2seq_batch
Author: sgugger
Pull request: yes

# What does this PR do?

This PR officially deprecates `prepare_seq2seq_batch` to prepare its removal in Transformers v5. As discussed before, the proper way to prepare data for sequence-to-sequence tasks is to:
- call the tokenizer on the inputs
- call the tokenizers on the targets inside the context manager `as_target_tokenizer`

When only dealing with input texts without targets, just using the tokenizer call works perfectly well.

For `mBART` and `mBART50` tokenizers the source and target language can be specified at init or changed at any time by setting the attributes `.src_lang` and `.tgt_lang`.

Here is a full example showing how to port old code using `prepare_seq2seq_batch` to the new way in the case of an mBART tokenizer (remove the mentiones of `src_lang` and `tgt_lang` for other tokenizers:
```
tokenizer = MBartTokenizer.from_pretrained('facebook/mbart-large-en-ro')
batch = tokenizer.prepare_seq2seq_batch(src_texts, tgt_texts, padding=True, truncation=True, src_lang="en_XX", tgt_lang="ro_RO", return_tensors="pt")
``` 
becomes
```
tokenizer = MBartTokenizer.from_pretrained('facebook/mbart-large-en-ro', src_lang="en_XX", tgt_lang="ro_RO")
batch = tokenizer(src_texts, padding=True, truncation=True, return_tensors="pt")
with tokenizer.as_target_tokenizer():
    targets = tokenizer(tgt_texts, padding=True, truncation=True, return_tensors="pt")
batch["labels"] = targets["input_ids"]
```
The languages can be changed at any time with
```
tokenizer.src_lang = new_src_code
tokenizer.tgt_lang = new_tgt_code
```

This PR fixes a few things in `MBartTokenizer` and `MBartTokenizerFast` for the new API to work completely and removes all mentions of `prepare_seq2seq_batch` from the documentation and tests (except the test of that method in the common tests). It was already not used anymore in the seq2seq example `run_seq2seq`.

Comments:
Hi all! Sorry, but this seems to be cleaner:  (Some feature request: #14255)
```python
encoded_train_dataset = train_dataset.map(
    lambda batch: tokenizer.prepare_seq2seq_batch(
        batch['text'], batch['summary'], padding='max_length', truncation=True, max_length=256, max_target_length=64
    ),
    batched=True,
    remove_columns=train_dataset.column_names,
)
```
Add LayoutXLMProcessor (and LayoutXLMTokenizer, LayoutXLMTokenizerFast)
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR implements `LayoutXLMProcessor`, which can be used to prepare all data for [LayoutXLM](https://huggingface.co/transformers/master/model_doc/layoutxlm.html). LayoutXLM is a multilingual version of LayoutLMv2. It uses the same vocabulary as XLMRoBERTa.

Big thanks to @kingyiusuen for setting up a first draft. This PR is built on his work: #14030 

- [x] To do: it might make sense to make a new "layoutxlm" folder in the models directory, where the following files can be added:

* `tokenization_layoutxlm.py`
* `tokenization_layoutxlm_fast.py`
* `processor_layoutxlm.py`

Comments:
have you merged it into master?
Add cross attentions to TFGPT2Model
Author: ydshieh
Pull request: yes

# What does this PR do?

Add cross attention to TFGPT2.

This was previously done in #13222, but we decided to move this to a new PR.
I also added `TFGPT2EncoderDecoderModelTest` with `test_bert2gpt2_summarization`.

Comments:
Just correct the line about 

```
"""Not working, because pt checkpoint has `encoder.encoder.layer...` while tf model has `encoder.bert.layer...`.
```

For tf here, it has `encoder.bert.encoder.layer...` instead.

Sorry, I tried fixing some merge conflicts, but I think this introduced a new error. @ydshieh could you maybe quickly go into the PR again and fix those last tests? :-) The PR looks good for me otherwise!

> 
> 
> Sorry, I tried fixing some merge conflicts, but I think this introduced a new error. @ydshieh could you maybe quickly go into the PR again and fix those last tests? :-) The PR looks good for me otherwise!

It's OK now :)
Wrong max_position_embeddings for roberta-large
Author: Jabb0
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.11.3
- Platform: Linux-5.13.0-7614-generic-x86_64-with-glibc2.33
- Python version: 3.8.12
- PyTorch version (GPU?): 1.9.1+cu102 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
@patrickvonplaten 

## Information

Model I am using (Bert, XLNet ...): roberta-large

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

I am using roberta-large and giving it texts that are as large as max_position_embeddings (514).
This causes an error during forward pass:
```
...venv/lib/python3.8/site-packages/torch/nn/functional.py", line 2043, in embedding
return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
IndexError: index out of range in self
```

https://huggingface.co/roberta-large#preprocessing states that the number of max position embeddings are 512.
The config, however, sets them to 514. Only using 512 token large texts does work without the error.

## To reproduce

Steps to reproduce the behavior:

1. Load roberta-large using `AutoTokenizer.from_pretrained('roberta-large')` and `model = AutoModel.from_pretrained('roberta-large')`
2. Get `em = model.config.max_position_embeddings` which is 514 (given by https://huggingface.co/roberta-large/blob/main/config.json)
3. Input texts with size `em`  (514) into the model.

## Expected behavior
The `max_position_embeddings` in the config should be correct and not cause the transformer to crash.

I can provide further information if necessary.
Thank you!

Comments:
Hey @Jabb0, 

While `model.config.max_position_embeddings` says 514 is the maximum allowed # of position embeddings, the actual number is 512. One should rather look at:

```python
from transformers import RobertaTokenizer
tok = RobertaTokenizer.from_pretrained("roberta-large")
tok.model_max_length  # is 512
```

cc @Narsil 

Hi @patrickvonplaten,
thank you.
I am using `tokenizer.model_max_length` now.
minimal fixes to run DataCollatorForWholeWordMask with return_tensors="np" and return_tensors="tf"
Author: dwyatte
Pull request: yes

# What does this PR do?

This PR addresses https://github.com/huggingface/transformers/issues/13890 with the minimal fixes to run `DataCollatorForWholeWordMask` with `return_tensors="np"` and `return_tensors="tf"`

Specific problems addressed:
* Renamed `np_call` -> `numpy_call`
* Call `_numpy_collate_batch` instead of` _tf_collate_batch` when returning numpy tensors
* Fix size of `random_words` in `numpy_mask_tokens`
* Clone tensorflow tensors with `tf.identity(tensor)`
* Use tensorflow tensors’ built-in iteration instead of attempting to convert to list
* Change calls to `tf.convert_to_tensor` with `dtype=tf.bool` to `tf.cast` with `dtype=tf.bool`

I've only added a simple test to check for regressions vs all of the padded/unpadded cases as is done for `DataCollatorForLanguageModeling`

Fixes https://github.com/huggingface/transformers/issues/13890


## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

CC @Rocketknight1 (looks like you did the initial Numpy/TF implementation of these data collators)

Comments:
Bump @Rocketknight1 @LysandreJik @sgugger since `transformers` releases frequently and it would be nice to have this functionality.

This gives a vetted source for masked language modeling that is not https://github.com/google-research/bert/blob/master/create_pretraining_data.py or https://github.com/tensorflow/models/blob/master/official/nlp/data/create_pretraining_data.py in a package many people are going to be using for modeling (e.g., I could delete my code that is a copy/paste of the former and integrate against this instead for creating a large offline dataset).

Sorry for the delay, and thank you for the ping! This looks good to me. I've got one question about the `convert_to_tensor` versus `cast` call, but other than that this is a solid and necessary PR.
DataCollatorForWholeWordMask does not work with return_tensors = "np" and return_tensors = "tf"
Author: dwyatte
Pull request: no

## Environment info
- `transformers` version: 4.11.2
- Platform: Linux-4.19.0-14-cloud-amd64-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.5
- PyTorch version (GPU?): 1.9.1+cu102 (False)
- Tensorflow version (GPU?): 2.6.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
@Rocketknight1 

## Information
The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

I'd like to use `DataCollatorForWholeWordMask` for masked language modeling in TensorFlow.

## To reproduce

##### `return_tensors = "np"`

```
from transformers.data import DataCollatorForWholeWordMask
from transformers import BertTokenizerFast

return_tensors = "np"
tokenizer = BertTokenizerFast.from_pretrained("bert-base-uncased")
collate_fn = DataCollatorForWholeWordMask(tokenizer, return_tensors=return_tensors, mlm=True)
collate_fn([{"input_ids": list(range(10))}, {"input_ids": list(range(10))}])
```
```TypeError: numpy_mask_tokens() got an unexpected keyword argument 'special_tokens_mask'```

This is at least partly due to not implementing `DataCollatorMixin.numpy_call` in `DataCollatorForWholeWordMask` (`np_call` is implemented instead). There are also some issues with using TensorFlow tensors in the `np_call` function.

##### `return_tensors = "tf"`

```
from transformers.data import DataCollatorForWholeWordMask
from transformers import BertTokenizerFast

return_tensors = "tf"
tokenizer = BertTokenizerFast.from_pretrained("bert-base-uncased")
collate_fn = DataCollatorForWholeWordMask(tokenizer, return_tensors=return_tensors, mlm=True)
collate_fn([{"input_ids": list(range(10))}, {"input_ids": list(range(10))}])
```

```AttributeError: 'tensorflow.python.framework.ops.EagerTensor' object has no attribute 'clone'```

`inputs.clone()` appears to have been copied from `torch_call` which is not valid in TensorFlow.



Comments:
Fix for the issue of device-id getting hardcoded for token_type-ids during Tracing for iBert 
Author: HamidShojanazeri
Pull request: yes

# What does this PR do?
This PR is part of a series of PRs that follows PR #11252 and applies similar changes to Flaubert.
Fixes # (issue)
issue #5664

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@LysandreJik 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Sorry for taking so long to get back to it, this one really fell through the cracks. Would you mind implementing a test for this like it was done with other models, for example in https://github.com/huggingface/transformers/pull/13350?

Thanks a lot!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Fix of issue #13327: Wrong weight initialization for TF t5 model
Author: dshirron
Pull request: yes

# What does this PR do?

Fixes # 13327

## Who can review?
@patrickvonplaten 


Comments:
This looks great! The only change I'd suggest is that we don't usually sign names on comments - there are a lot of contributors to this repo and it could get out of hand.

Looks good! Let me know when you're happy and I'll merge it.

You can merge it. I only saw your comment and didnt see you already pushed a fix that removed all my comments in code, this is why i pushed a fix. Any solution is ok with me...

We're leaving the comments in, just with the author attributions removed. Thanks for your contribution, I'm merging now!
Unexpected output from bart-large
Author: jc-hou
Pull request: no

I am looking this thread about generation, https://stackoverflow.com/questions/64904840/why-we-need-a-decoder-start-token-id-during-generation-in-huggingface-bart

I re-run his code,
Use ```facebook/bart-base``` model,
 ```
from transformers import *

import torch
model = BartForConditionalGeneration.from_pretrained('facebook/bart-base')
tokenizer = BartTokenizer.from_pretrained('facebook/bart-base')
input_ids = torch.LongTensor([[0, 894, 213, 7, 334, 479, 2]])
res = model.generate(input_ids, num_beams=1, max_length=100)

print(res)

preds = [tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=True).strip() for g in res]
print(preds)
```
I get output:
```
tensor([[  2,   0, 894, 213,   7, 334, 479,   2]])
['He go to school.']
```
Then I just simply change the model to ```facebook/bart-large```  with everything kept the same, i.e.
 ```
from transformers import *

import torch
model = BartForConditionalGeneration.from_pretrained('facebook/bart-large')
tokenizer = BartTokenizer.from_pretrained('facebook/bart-large')
input_ids = torch.LongTensor([[0, 894, 213, 7, 334, 479, 2]])
res = model.generate(input_ids, num_beams=1, max_length=100)

print(res)

preds = [tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=True).strip() for g in res]
print(preds)
```
Then I get output:
```
tensor([[  2, 894, 894, 213,   7, 334, 479,   2]])
['HeHe go to school.']
```
Is this normal?
Thanks.


Comments:
@jc-hou: I just tested the above script and I get the same output as you. Why did you close the issue?

`res = model.generate(input_ids, num_beams=1, max_length=100, forced_bos_token_id=0)` solves the issue