noamaanMulla-03/transformers-issues huggingface dataset:
Fix dummy creation for multi-frameworks objects
Author: sgugger
Pull request: yes

# What does this PR do?

Follow-up from #17304, this finishes fixing the dummy creation script when there is more than one framework on which the object depends.

The problem is that it does not recognize the lines in the init like:
```
    if not (is_sentencepiece_available() and is_tokenizers_available()):
```

To check the change, after the PR the following passes:
```py
import sys

# Adapt to where the repo is
sys.path.append("../git/transformers/utils")
from check_dummies import find_backend

assert find_backend("    if not is_tokenizers_available():") == "tokenizers"
assert find_backend("    if not (is_sentencepiece_available() and is_tokenizers_available()):") == "sentencepiece_and_tokenizers"
```

Before this PR only the first test passes.

I will work on adding unit tests such as the one above for all of our quality scripts.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_19144). All of your documentation changes will be reflected on that endpoint.
Add some tests for check_dummies
Author: sgugger
Pull request: yes

# What does this PR do?

This PR adds a new subfolder in the `tests` folder for the tests of the quality scripts used in the CI (so not inside the Transformers lib). As seen recently with the `check_dummies` script, when people apply too many modifications to our repo and update those scripts, there might be some breaking changes that slip through the cracks. Hopefully such new tests will help catch the failures early.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_19146). All of your documentation changes will be reflected on that endpoint.
`is_torch_tpu_available() got an unexpected keyword argument 'check_device'`
Author: Sakurakdx
Pull request: no
Labels: name: bug

### System Info

transformers==4.18.0
pytorch==1.12.0
pytorch-lightning==1.6.0

### Who can help?

@patrickvonplaten @sgugger @SaulLu
When I run `examples/pytorch/question-answering/run_qa.py`, it reports a bug that  `is_torch_tpu_available() got an unexpected keyword argument 'check_device'`. I don't know why this problem happens. Could you help me solve this problem?

![image](https://user-images.githubusercontent.com/48399040/191486285-5a4bbec6-c6e5-4033-83b0-7f5407d090e9.png)


### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

python run_qa.py \
  --model_name_or_path xlm-roberta-base \
  --dataset_name squad \
  --do_train \
  --do_eval \
  --per_device_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2 \
  --max_seq_length 384 \
  --version_2_with_negative \
  --doc_stride 128 \
  --output_dir /tmp/debug_squad/
  

### Expected behavior

 Could you help me solve this problem?

Comments:
You are trying to run the examples of the main branch along with an older version of Transformers. Either upgrade your Transformers or use the [examples matching v4.18.0](https://github.com/huggingface/transformers/tree/v4.18.0/examples).

Thanks for your response.
[fix] Add DeformableDetrFeatureExtractor
Author: NielsRogge
Pull request: yes

# What does this PR do?

As pointed out by @Deppmeng who is adding Conditional DETR in #18948, the postprocessing of Deformable DETR is actually different compared to regular DETR. Namely, a sigmoid activation function is used rather than softmax, and the no-object class is included, whereas DETR discards this class.

Hence, we'll need a new `DeformableDetrFeatureExtractor` which includes this custom postprocessing logic. As only the postprocessing of object detection is different, I'm using `Copied from` statements wherever possible.

To do:

- [x] update `preprocessor_config.json` of all repos on the hub
- [x] use `from_pretrained` in the code snippets for the feature extractor

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add MSN checkpoints to ViT
Author: sayakpaul
Pull request: no

### Feature request

It's been a while Meta released the [MSN](https://github.com/facebookresearch/msn) (Masked Siamese Networks) checkpoints. MSN is known to perform pretty well in few-shot settings. Few-shot regime is important for real-world applications. They have only released the encoder weights.

I have worked on a [Colab Notebook](https://colab.research.google.com/gist/sayakpaul/51556ce57facea02041cf0c0f1c3fed3/scratchpad.ipynb) that converts the released MSN checkpoints into HF format and performs assertions to ensure the converted weights can be loaded into HF's ViT classes.

This issue proposes adding the MSN checkpoints to be compatible with HF's ViT classes. 

### Motivation

MSN is particularly useful in few-shot learning regimes.

### Your contribution

Can contribute to the checkpoints and other related utilities.

@NielsRogge could you assign this issue to me?

Comments:
TF: check embeddings range
Author: gante
Pull request: yes

# What does this PR do?

Adds the same [check that was recently added to TFBart](https://github.com/huggingface/transformers/blob/ba7f2173cc578fe6d9f1cdb900d5af609f195cf6/src/transformers/models/bart/modeling_tf_bart.py#L751), which asserts that the inputs are within the embedding input range, in all models with token embeddings. As a reminder: TF doesn't enforce this check by default on `tf.gather`-dependent operations on GPU, returning a vector of `0.0` when out of bounds.

After this change, all `test_embeddings_out_of_bounds_raise_exception` tests pass (36 failures in the previous scheduled CI).

To simplify the review, there are 3 models you should check. All others are copy/paste from these.
1. Bert (Encoder)
2. GPT2 (Decoder)
3. Pegasus (Encoder-Decoder with `TFSharedEmbeddings` or `TFWrappedEmbeddings`. Encoder-Decoder models that only use the embeddings at the decoder, like Speech2Text, also follow the same code pattern)

Comments:
_The documentation is not available anymore as the PR was closed or merged._

(cc @ydshieh -- this fixes a large number of scheduled CI failures)

@gante Is this PR ready to merge? I guess so, but would like to wait your confirmation (or better for you to merge).

> @gante Is this PR ready to merge? I guess so, but would like to wait your confirmation (or better for you to merge).

@ydshieh It was ready -- merged now :D
Reduce LR for TF MLM example test
Author: Rocketknight1
Pull request: yes

The TF MLM example test was a little flakey, depending on the exact shuffling order of the dataset. I reduced the LR to 1e-4 and it seems to be consistently around 36 final perplexity now (compared to the test threshold of 42).

Cc @sgugger 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Parallelization for OPT model
Author: lifan-yuan
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.20.1
- Platform: Linux-5.4.0-113-generic-x86_64-with-glibc2.17
- Python version: 3.8.13
- Huggingface_hub version: 0.8.1
- PyTorch version (GPU?): 1.10.1+cu111 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@Lyandrejik

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

from transformers import OPTForCausalLM

model = OPTForCausalLM.from_pretrained("facebook/opt-6.7b").cuda()
model.parallelize()

### Expected behavior

Hi,
I am loading the OPT model. Due to many parameters, I cannot put the whole model on one CUDA device. Therefore, I tried to parallelize the OPT like T5, using "model.parallelize()". However, I find that there is no parallelization mechanism for OPT model now.
Can you please implement the parallelization mechanism for OPT models? Or what can I do to load and run a large OPT model?

Comments:
I have the same question since directly tuning models with this size (OPT) on one single device is infeasible. It would be great to have the parallelization mechanism implemented for OPT models. Many thanks! 

See this conversation which might help out: https://huggingface.co/facebook/opt-66b/discussions/6

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
How to load multiple TXT training files when pre-train RoBERTa from scratch
Author: skye95git
Pull request: no

https://github.com/huggingface/transformers/blob/d6eeb871706db0d64ab9ffd79f9545d95286b536/examples/pytorch/language-modeling/run_mlm.py#L308

Hi, I want to pre-train RoBERTa from scratch on my dataset. But in example run_mlm.py:
```
        data_files = {}
        if data_args.train_file is not None:
            data_files["train"] = data_args.train_file
            extension = data_args.train_file.split(".")[-1]
        if data_args.validation_file is not None:
            data_files["validation"] = data_args.validation_file
            extension = data_args.validation_file.split(".")[-1]
        if extension == "txt":
            extension = "text"
        raw_datasets = load_dataset(
            extension,
            data_files=data_files,
            cache_dir=model_args.cache_dir,
            use_auth_token=True if model_args.use_auth_token else None,
        )
```
The train file argument `train_file` seems to support only one file:
```
train_file: Optional[str] = field(default=None, metadata={"help": "The input training data file (a text file)."})

data_files["train"] = data_args.train_file
extension = data_args.train_file.split(".")[-1]
```
Because we have too much training data, it is inconvenient to store it in a file. Can train_file support multiple text files?

Comments:
I would recommend using `datasets` in order to store all of your data; you can then pass it either as a local dataset or as a dataset stored on the Hub to that same script.

> I would recommend using `datasets` in order to store all of your data; you can then pass it either as a local dataset or as a dataset stored on the Hub to that same script.

Thanks for your reply! How to build a dataset with our data? Is there a tutorial?

This part of the datasets documentation should likely help out: https://huggingface.co/docs/datasets/loading#local-and-remote-files

> This part of the datasets documentation should likely help out: https://huggingface.co/docs/datasets/loading#local-and-remote-files

Thanks! It works.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Behavior of shift_tokens_right on padded input_ids
Author: duongna21
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.20.1


### Who can help?

@patrickvonplaten @patil-suraj @sgugger 

### Reproduction

When I applied shift_tokens_right on a padded input_ids, I get this 
```python
from transformers import AutoTokenizer
from transformers.models.bart.modeling_flax_bart import shift_tokens_right

tokenizer = AutoTokenizer.from_pretrained("roberta-base")
labels = tokenizer("My dog is cute", padding='max_length', max_length=8, return_tensors='np').input_ids
decoder_input_ids = shift_tokens_right(labels, tokenizer.pad_token_id, tokenizer.eos_token_id)

print(tokenizer.batch_decode(labels))
# ['
My dog is cute
<pad><pad>']

print(tokenizer.batch_decode(decoder_input_ids))
# ['

My dog is cute
<pad>']
```


### Expected behavior

Should the desired behavior of shift_token_right be the following?

```python
print(tokenizer.batch_decode(labels))
# ['
My dog is cute
<pad><pad>']

print(tokenizer.batch_decode(decoder_input_ids))
# ['

My dog is cute<pad><pad>']
```

Comments:
Hey @duongna21! This method isn't exposed in the main init so we consider it to be private (this should really be written somewhere if we haven't done so yet).

It's used internally by the BART model, but we don't validate it to work for any other purpose.

@LysandreJik Yeah, I specifically raise above question in the context of BART training. IMO `shift_tokens_right` should not take the `<pad>` token into account (this is actually the behavior of [fairseq's original code](https://github.com/facebookresearch/fairseq/blob/8e804cb38a1575c65a1fc981d75ae5a97c24dd5b/fairseq/data/data_utils.py#L69)). 
Also, I believe this issue also applies to other models using `shift_tokens_right`, such as T5.

Any comment? I'm happy to create a PR if my assumption is correct.

Pinging @patil-suraj and @patrickvonplaten regarding the `shift_tokens_right` method and its purpose.

Hey @

Hey @duongna21,

Good question! Note however that it doesn't really matter whether you pass

```py
['

My dog is cute<pad><pad>']
```

or

```python
['

My dog is cute
<pad>']
```

to the model if the labels are:

```py
['
My dog is cute
<pad><pad>']
```

because every loss token that gets mapped to the `<pad>` token is ignored. More specifically this means that during training the following happens:
- The model learns that `
` should predict `
`
- then `

` should predict `My`
- then `

My` should predict `dog`
- ... until
- `

My dog is cute` should predict `
`
**Now** it doesn't matter whether `

My dog is cute
` or `

My dog is cute<pad>` is passed because both will be ignored as the predicted token is `<pad>` and the model should never learn to predict pad tokens -> so this loss will be ignored. Does this make sense?


@patrickvonplaten Thanks for elaborating on it. Totally agree with you that it doesn't matter with seq2seq training. I just tried to make sure it doesn't create any side-effect somewhere :D.
load_in_8bit=True crashes GPT-J when running model.generate()
Author: petertjmills
Pull request: no
Labels: name: bug

### System Info

Colab Pro, NVIDIA P100
Transformers: 4.22.1
Accelerate: 0.12.0

- `transformers` version: 4.22.1
- Platform: Linux-5.10.133+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.14
- Huggingface_hub version: 0.9.1
- PyTorch version (GPU?): 1.12.1+cu113 (True)
- Tensorflow version (GPU?): 2.8.2 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Y
- Using distributed or parallel set-up in script?: N



### Who can help?

@patil-suraj

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

This works fine, loads the model using 6GB GPU memory, 10GB free
```
from transformers import GPTJForCausalLM, AutoTokenizer
import torch

model =  GPTJForCausalLM.from_pretrained(
    "EleutherAI/gpt-j-6B", 
    revision="float16", 
    torch_dtype=torch.float16, 
    load_in_8bit=True, 
    device_map='auto', 
    low_cpu_mem_usage=True
).to("cuda")
tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-j-6B")
```
However when I run:
```
prompt = "I "
input_ids = tokenizer(prompt, return_tensors="pt").input_ids

gen_tokens = model.generate(input_ids, do_sample=True, temperature=0.9, max_length=5,)
gen_text = tokenizer.batch_decode(gen_tokens)[0]
print(gen_text)
```
The colab environment crashes with an unknown error, during model.generate()

`max_length=5` and a short prompt were tested to lower mem requirements, but this happens with any length and prompt.
Presumbaly it's not a memory issue as I can just about use inference without load_in_8bit, on the same arch.

### Expected behavior

Model generates exciting tokens, decodes, and prints!

Comments:
Hey @petertjmills -- can you share with us the error you're seeing or, better yet, a link to a colab with the error?

Thank you for the reply! I threw together a quick notebook to send and it worked 🙄

I've found the source of the issue. In the bitsandbytes readme they specify:
```
Hardware requirements:

    LLM.int8(): NVIDIA Turing (RTX 20xx; T4) or Ampere GPU (RTX 30xx; A4-A100); (a GPU from 2018 or older).
    8-bit optimizers and quantization: NVIDIA Maxwell GPU or newer (>=GTX 9XX).
```

The first time I got a P100 (Pascal arch, 2016)
The second time I got a T4, go figure. 
May be worth documenting?

> May be worth documenting?

Definitely! Would you like to open a PR with a more informative exception? :D
Add semantic segmentation post-processing method to MobileViT
Author: alaradirik
Pull request: yes

# What does this PR do?
Adds post_process_semantic_segmentation method to `MobileViTFeatureExtractor`.  

I will open an issue and separate PRs to make sure that
- Segmentation models (DETR, MaskFormer, SegFormer, etc.) have consistently named post-processing methods, arguments and outputs
- ImageSegmentationPipeline works with all available segmentation models

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?




Comments:
_The documentation is not available anymore as the PR was closed or merged._
Token type ids generation func. for Bert-like models
Author: wavy-jung
Pull request: yes

# What does this PR do?

For a text sequence input (separated with comma), Bert tokenizer nicely create an input_ids with [SEP] token inside and proper token_type_ids as well. However for input_ids with already separated with [SEP] token id inside, couldn't find any function to generate proper token_type_ids.
So, I made a simple function which create proper token_type_ids with padding.
Input : list or tensor / pad_to_multiple_of (target length) / tokenizer (to recognize sep token id and add proper pad token id)
output : tensor of token_type_ids with pad id
Hope this might be used for those who want to override DataCollator class.

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [V] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
- new function for Bert-like models collator : @LysandreJik

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_18879). All of your documentation changes will be reflected on that endpoint.
Detr preprocessor fix
Author: alaradirik
Pull request: yes

# What does this PR do?
Ensures that `DetrFeatureExtractor` doesn't preprocess input `images` and `annotations` in-place by creating deep copies of inputs within `DetrFeatureExtractor.__call__()`.

`YolosFeatureExtractor` has the same issue and will be fixed in a separate PR.

Fixes #[18987](https://github.com/huggingface/transformers/issues/18987)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [X ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
    This issue is documented over [here](https://github.com/huggingface/transformers/issues/18987).
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?



Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix TrainingArguments documentation
Author: sgugger
Pull request: yes

# What does this PR do?

@Rocketknight1 added a new class variable for `TrainingArguments` but put it before the docstring, which erased all doc for this class :grimacing: This PR fixes that.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

In retrospect I really should have foreseen that one, I'm sorry!
Add offline runners info in the Slack report
Author: ydshieh
Pull request: yes

# What does this PR do?

So we see which runners are offline directly in the report.

Currently, this information is added only if the check is run through `check_runner_status.yml`, where all runners are checked but reported to scheduled CI channel. Adding this information avoids confusion in the case where push/doctest CI runners are offline but reported in the scheduled CI channel.



Comments:
_The documentation is not available anymore as the PR was closed or merged._