noamaanMulla-03/transformers-issues huggingface dataset:
[CookieCutter] Clarify questions
Author: NielsRogge
Pull request: yes

# What does this PR do?

I've seen quite some mistakes in terms of people answering the questions of the CookieCutter in the wrong way. This is because the questions are sometimes a bit vague, i.e. it's not clear whether one should provide Roberta, RoBERTa or roberta. This was actually not clear for me either.

This PR aims to clarify the questions, making sure contributors understand better what to answer.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix CI for `PegasusX`
Author: ydshieh
Pull request: yes

# What does this PR do?

- `PegasusXGlobalLocalAttention` returns attentions as dictionary:

  ```python
        if output_attentions:
            attn_probs = {"global": global_attn_probs, "local": local_attn_probs}
        else:
            attn_probs = None
  ```
  Skip this test to avoid failure [here](https://github.com/huggingface/transformers/actions/runs/2986602848/jobs/4788460628).

- Fix `PegasusXModelIntegrationTests` by updating the expected values + proper checkpoint

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Usage of Ġ in BPE tokenizer
Author: maschasap
Pull request: no
Labels: name: wontfix

Hello, 
I want to add new words to my BPE tokenizer. I know the symbol Ġ means the end of a new token and the majority of tokens in vocabs of pre-trained tokenizers start with Ġ. Assume I want to add the word **Salah** to my tokenizer. I tried to add both **Salah** token and **ĠSalah**:
`tokenizer.add_tokens(['Salah', 'ĠSalah'])` # they get 50265 and 50266 values respectively.
However, when I tokenize a sentence where **Salah** appears, the tokenizer will never return me the second number (neither when using `.tokenize` nor `.encode`), for instance:
`tokenizer.tokenize('I love Salah and salad')` returns `['I', 'Ġlove', 'Salah', 'Ġand', 'Ġsalad']`.
The question is: should I use the symbol `Ġ` when adding new tokens or the tokenizer does it itself? Or, probably, it must be specified manually?
Thanks in advance!

Comments:
@maschasap 
AFAIK, you won't need to add `Ġ ` when adding a new token. 
And you can use  the `convert_tokens_to_string` method to convert these tokens to their respective strings.

tagging @mfuntowicz  for more info

@patil-suraj thanks for your response! But still why doesn't the tokenizer add `Ġ` symbol when returning the tokenized sentence in the example? Does it mean it still needs to be tuned or is it OK?

@patil-suraj btw using the method `convert_tokens_to_string` the whitespace between the words **love** and  **Salah** really disappears :(
`tokenizer.convert_tokens_to_string(tokenizer.tokenize('I love Salah and salad'))` outputs `'I loveSalah and salad`

@LysandreJik @mfuntowicz may I ask you for help?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Can someone help me understand the symbol Ġ ? I am running into the same thing , want to make sure I am not breaking things down the line

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


There is also a difference here between the fast and slow tokenisers:
```python
import transformers
tokeniser1 = transformers.RobertaTokenizer.from_pretrained('roberta-base')
tokeniser2 = transformers.RobertaTokenizerFast.from_pretrained('roberta-base')
tokeniser1.add_tokens(['Salah'])
tokeniser2.add_tokens(['Salah'])
print(tokeniser1.tokenize('I love Salah and salad'))
print(tokeniser2.tokenize('I love Salah and salad'))
```
Outputs:
```
['I', 'Ġlove', 'Salah', 'and', 'Ġsalad']
['I', 'Ġlove', 'Ġ', 'Salah', 'Ġand', 'Ġsalad']
```

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


same here

(in case someone comes across this issue, have a look at [this post in our forum](https://discuss.huggingface.co/t/bpe-tokenizers-and-spaces-before-words/475/2?u=joaogante))
Mark right save_load test as slow
Author: sgugger
Pull request: yes

# What does this PR do?

Tried to mark the `test_save_load` as slow for TFVitMAE with a simple commit on main, but it looks to be too complicated a task :sweat_smile: 

Sooo putting the decorator on the right tests ;-)

Comments:
_The documentation is not available anymore as the PR was closed or merged._
LayoutLMV3 Tokenizer Inserts Odd Characters
Author: logan-markewich
Pull request: no
Labels: name: bug

### System Info

(This is on a fresh google colab instance)
- `transformers` version: 4.21.2
- Platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- Huggingface_hub version: 0.9.1
- PyTorch version (GPU?): 1.12.1+cu113 (True)
- Tensorflow version (GPU?): 2.8.2 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


### Who can help?

@NielsRogge 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers import LayoutLMv3TokenizerFast

tok = LayoutLMv3TokenizerFast.from_pretrained('microsoft/layoutlmv3-base')

tok.tokenize('Hello world')
>>> ['ĠHello', 'Ġworld']
```

### Expected behavior

The special characters are not expected when calling tokenize(). This doesn't happen when using the LayoutLMV2 tokenizer.

Comments:
@logan-markewich -- have a look at this [entry in our forum](https://discuss.huggingface.co/t/bpe-tokenizers-and-spaces-before-words/475/2?u=joaogante)

@gante Yea ok, that gives some context for what is actually going on

I guess I will just need to remove those characters myself? A little annoying, but oh well

Hi,

No that's just the way tokenization happens. The tokenization is based on Byte Pair Encoding (BPE) algorithm. This is also used by RoBERTa for instance. 

You can check out the Huggingface course if you want to learn about this: https://huggingface.co/course/chapter6/5?fw=pt
TF: tf.debugging assertions without tf.running_eagerly() protection
Author: gante
Pull request: yes

# What does this PR do?

In most cases, `tf.debugging` assertions (which exist as normal asserts in PyTorch models) are protected by a check for eager execution. In some places, we have stated in comments that these ops do not work with XLA, hence the check for eager execution.

The actual state of `tf.debugging` is the following:
- They never cause crashes, not even in XLA;
- In XLA, they are not executed.

Since they are innocuous, this PR removes the checks for eager execution. Non-XLA graph-mode model calls will now benefit from these checks (e.g. when a user calls `model.fit()`)

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add safeguards for CUDA kernel load in Deformable DETR
Author: sgugger
Pull request: yes

# What does this PR do?

Transformers has become unusable on GPU when doing some imports since:
- deformable DETR compiles some custom CUDA kernels at init
- this require ninja which is not in the dependencies

This PR adds some additional checks before compiling the CUDA kernels, and also adds a warning instead of a hard error when that load fails.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Wrong `special_tokens_mask` when using `facebook/opt-350m`
Author: ArthurZucker
Pull request: no
Labels: name: bug

### System Info

The logic behind the computation of `special_tokens_mask` seems a bit broken when using the OPT model (might also be true for other models but I have not been able to reproduce the behaviour yet). 

```python 
tokenizer = transformers.AutoTokenizer.from_pretrained("facebook/opt-125m")
result = tokenizer(
    [
        ("Two radio engineers got married.", "The reception was fantastic."),
        ("Atheism is", "a non-prophet organization.")
    ],
    padding=True,
    return_tensors='pt',
    is_split_into_words=False,
    return_special_tokens_mask=True,
    return_token_type_ids=True
)
```


### Who can help?

@LysandreJik this is related to the #allenai-colab issue mentioned by `Dirk Groeneveld`.


The part of the `tokenizer` code that is interacted with is the following : 

```python 
        # Add special tokens
        if add_special_tokens:
            sequence = self.build_inputs_with_special_tokens(ids, pair_ids)
            token_type_ids = self.create_token_type_ids_from_sequences(ids, pair_ids)
        else:
            sequence = ids + pair_ids if pair else ids
            token_type_ids = [0] * len(ids) + ([0] * len(pair_ids) if pair else [])

        # Build output dictionary
        encoded_inputs["input_ids"] = sequence
        if return_token_type_ids:
            encoded_inputs["token_type_ids"] = token_type_ids
        if return_special_tokens_mask:
            if add_special_tokens:
                encoded_inputs["special_tokens_mask"] = self.get_special_tokens_mask(ids, pair_ids)
            else:
                encoded_inputs["special_tokens_mask"] = [0] * len(sequence)
```

 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The problem is that `get_special_tokens_mask` will only return ` [0] * ((len(token_ids_1) if token_ids_1 else 0) + len(token_ids_0))` at this point. 


### Expected behavior


In order to get the expected behaviour, the call should be : `encoded_inputs["special_tokens_mask"] = self.get_special_tokens_mask(sequence, already_has_special_tokens = True)` which makes sense as at this point, `sequence` should contain special tokens. 

Comments:
Automate check for new pipelines and metadata update
Author: sgugger
Pull request: yes

# What does this PR do?

This PR adds a script to check that new pipelines are properly added in `update_metadata.py` so that we properly update the table that is used by the frontend to determine which pipeline tag to use by default for a given model.
This check is added as part of the `repo-consistency` job.

Of course there were some new pipeline tags missing, this PR adds them too.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix a broken link for deepspeed ZeRO inference in the docs
Author: nijkah
Pull request: yes

# What does this PR do?

Fix a broken link for deepspeed ZeRO inference in the documentation

Comments:
In addition to this, 
Line 84 does not have an appropriate link. How should I change it?
`
If you're still struggling with the build, first make sure to read [zero-install-notes](#zero-install-notes).
`

_The documentation is not available anymore as the PR was closed or merged._
How to set different LOCAL_RANK env variable values for multiple GPUs of a single node machine with Accelerate
Author: chris-opendata
Pull request: no

This is more of a question than a bug report. 

I am adopting the script with Accelerate from https://github.com/huggingface/transformers/blob/master/examples/pytorch/summarization/run_summarization_no_trainer.py.

I intend to parallel on two gpus of a single machine, one on GPU 0 and the other on GPU 1. As I notice that 
`Line 203:  self.local_process_index = int(os.environ.get("LOCAL_RANK", -1))`
(https://github.com/huggingface/accelerate/blob/209db19dc885887682b07ff88fb6c840cbeb3c1c/src/accelerate/state.py#L128).

Since the LOCAL_RANK env variable is a single integer value (say, I set it to 0), I am just wondering how Accelerate to pick up a second GPU rank (i.e. 1). I don't see that from the above code.

I must have not fully understood the mechanism quite right yet, and haven't found an FAQ of it. Could anyone explain me a bit more.

Thanks

Comments:
Hi, 

Same problem here, did you find any solution to this problem? @chris-opendata 

Here is the solution:

1. Activate your venv environment.
2. Install accelerate package.
3. Type "accelerate config" at terminal to go through multigpu configuration.
4. In your running bash script, use sth like "accelerate launch main_train.py ...." instead of "python -m main_train.py ..."
    Note that you may not need '&' at the end in order to run it at the background depending on computing env. I am on Slurm.
That is it. Accelerate will work out the rest itself.
ImportError: cannot import name 'SAVE_STATE_WARNING' from 'torch.optim.lr_scheduler' - SAVE_STATE_WARNING has been removed from pytorch
Author: dbonner
Pull request: no

ERROR:
..../my37/lib/python3.7/site-packages/transformers/trainer_pt_utils.py", line 40, in <module>
    from torch.optim.lr_scheduler import SAVE_STATE_WARNING
ImportError: cannot import name 'SAVE_STATE_WARNING' from 'torch.optim.lr_scheduler' (......./my37/lib/python3.7/site-packages/torch/optim/lr_scheduler.py)

Please update transformers to be compatible with the latest pytorch source code (build from master branch:
'SAVE_STATE_WARNING' was removed from pytorch a few days ago.

Comments:
.I can see you have fixed this in the source code.

I just upgraded to torch 1.8 and I got this error. 

```ImportError while importing test module '/home/dwalter/Documents/projects/lm/lm_ml/modules/quantization/tests/test_quantize.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/home/dwalter/anaconda3/envs/lm-torch1.8/lib/python3.6/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_quantize.py:7: in <module>
    import quantization.fused_nn as qnni
quantization/__init__.py:2: in <module>
    from .fused_nn import ConvNL2d
quantization/fused_nn.py:4: in <module>
    from .nn import Conv2d
quantization/nn/__init__.py:8: in <module>
    import transformers.modeling_bert as bert
/home/dwalter/anaconda3/envs/lm-torch1.8/lib/python3.6/site-packages/transformers/__init__.py:626: in <module>
    from .trainer import Trainer
/home/dwalter/anaconda3/envs/lm-torch1.8/lib/python3.6/site-packages/transformers/trainer.py:69: in <module>
    from .trainer_pt_utils import (
/home/dwalter/anaconda3/envs/lm-torch1.8/lib/python3.6/site-packages/transformers/trainer_pt_utils.py:40: in <module>
    from torch.optim.lr_scheduler import SAVE_STATE_WARNING
E   ImportError: cannot import name 'SAVE_STATE_WARNING'
```
Is there something I need to fix in my code or did I not upgrade correctly?
upgraded with `pip install --upgrade torch`

@dwalterlm you're probably on an older Transformers version. This was fixed in https://github.com/huggingface/transformers/pull/8979, could you try upgrading to a more recent version, like `v4.3.0`?

> 

the version of torch is too high,try use : torch 1.7.1
[bnb] Small improvements on utils
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes a small typo in `bitsandbytes.py`, should address https://github.com/huggingface/blog/pull/463#discussion_r946067141 
I will have to test it first and mark it as ready for review!

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Can confirm the tests pass!

so will there always be just one module not to convert?

won't it be safer to have modules instead and work with the list?

I have proposed a small refactoring that includes:

- checking the list of modules to not convert instead of a single value. 
- changing an error message as it confused some user. Check: https://github.com/TimDettmers/bitsandbytes/issues/10 

The bnb slow tests are passing with this fix!

From https://github.com/huggingface/transformers/issues/18660 I also just added a commit to support having a custom list of the keys to ignore 

Thanks a lot @stas00 !
There is no rush at all for this PR, we can definitely wait for @sgugger before moving forward with it 

Can confirm the bnb slow tests are passing with the proposed fixes! Would love to have a final round of review 💪  
cc @sgugger @stas00 

Can confirm the slow tests pass after rebasing on `main`, will merge once it's green! 🟢
Fix `test_save_load` for `TFViTMAEModelTest`
Author: ydshieh
Pull request: yes

# What does this PR do?

`test_save_load` means to test with `saved_model=False`, which is very fast.

The test for `saved_model=True` is done in `test_saved_model_creation` (slow test)


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Pin minimum PyTorch version for BLOOM ONNX export
Author: lewtun
Pull request: yes

# What does this PR do?

Due to the deprecation of `position_ids` in https://github.com/huggingface/transformers/pull/18342, the BLOOM ONNX export fails unless the `torch` version is >= 1.12

This PR fixes that by pinning the minimum required version in the ONNX configuration (the user gets a warning now).


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Update serving signatures and make sure we actually use them
Author: Rocketknight1
Pull request: yes

This PR does some stuff to lay the groundwork for my plan to focus on TF model deployment:

- Updates the signatures on all our `model.serving` methods to use `int64` instead of `int32`
- Overrides `model.save()` so that the default signature for our models is now `model.serving`
- Casts all `int32` inputs to our TF models to `int64` in `input_processing`

The net effect is to standardize the int dtypes we use, and also make `model.save()` actually save a usable trace (although users can of course still override it with their own signatures if needed).

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Top_P sampling samples an extra token when the cum sum of probabilities is exactly equal to top_p
Author: ekagra-ranjan
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.20.1
- Platform: Linux-5.10.133+-x86_64-with-debian-bullseye-sid
- Python version: 3.7.12
- Huggingface_hub version: 0.8.1
- PyTorch version (GPU?): 1.11.0+cpu (False)
- Tensorflow version (GPU?): 2.6.4 (False)
- Flax version (CPU?/GPU?/TPU?): 0.6.0 (cpu)
- Jax version: 0.3.16
- JaxLib version: 0.3.15

### Who can help?

@patrickvonplaten @Narsil @gante

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Top p sampling samples an extra token when the cumulative sum of probabilities of token is exactly equal to the given top p. E.g., if the input probabilities is `[0.3, 0.1, 0.1, 0.5]` and top_p = `0.8` then only 2 tokens with probability `0.5` and `0.3` should be sampled as their sum would exactly be equal to `0.8`. I believe this is the expected behavior of Top P sampling according to the [definition](https://huggingface.co/docs/transformers/main_classes/text_generation) which states that: 
top_p (float, optional, defaults to 1.0) — If set to float < 1, only the most probable tokens with probabilities that add **up to top_p** or higher are kept for generation. 

I have created a notebook which reproduces this behavior. The notebook also has a proposed implementation which will fix this with an added optimization of not needing to clone tensor and shifting to left or right. https://www.kaggle.com/ekagra/hf-contrib-topp

I have checked locally that the proposed implementation passes the existing [unittest ](https://github.com/huggingface/transformers/blob/f7196f2e63b14e9fbb4ad664e71912aab3b484cf/tests/generation/test_generation_logits_process.py#L162).

### Your contribution
If this makes sense then I would be happy to raise a PR for this.

Comments:
Hey @ekagra-ranjan 👋 

EDIT: I've checked the [original paper](https://arxiv.org/pdf/1904.09751.pdf) and you are correct -- in your example, only two tokens should be up to consideration. Adding a quick fix for it.



@gante  I can raise a PR right away for this. Should I go ahead?


Oh, my bad, already opened a PR 🙈 

@gante Actually, I wanted to raise a PR with my implementation because it has an optimization of not requiring to clone an intermediate tensor and shifting things to right (as done in current implementation). I have raised the [PR](https://github.com/huggingface/transformers/pull/18984). Could you please review it?

@ekagra-ranjan that is fine, as long as you also edit the test for FLAX and TF (as in my PR), to ensure the three frameworks have the same behavior
Fix custom tokenizers test
Author: sgugger
Pull request: yes

# What does this PR do?

The custom tokenizers tests were never run because the test fetcher was not run before we look at its output to decide whether or not to run the tests. This PR fixes that and also adds missing tests to the nightly suite.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@ydshieh Those three files contain tests that require some specific dependencies to be installed (ftfy for openai and clip). So in the other test jobs, those tests are never run.
AMOS
Author: jpcorb20
Pull request: no
Labels: name: New model

### Model description

Abstract

"We present a new framework AMOS that pretrains text encoders with an Adversarial learning curriculum via a Mixture Of Signals from multiple auxiliary generators. Following ELECTRA-style pretraining, the main encoder is trained as a discriminator to detect replaced tokens generated by auxiliary masked language models (MLMs). Different from ELECTRA which trains one MLM as the generator, we jointly train multiple MLMs of different sizes to provide training signals at various levels of difficulty. To push the discriminator to learn better with challenging replaced tokens, we learn mixture weights over the auxiliary MLMs’ outputs to maximize the discriminator loss by backpropagating the gradient from the discriminator via Gumbel-Softmax. For better pretraining efficiency, we propose a way to assemble multiple MLMs into one unified auxiliary model. AMOS outperforms ELECTRA and recent state-of-the-art pretrained models by about 1 point on the GLUE benchmark for BERT base-sized models."

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

HF Hub : https://huggingface.co/microsoft/amos
GitHUB : https://github.com/microsoft/AMOS
Paper : https://arxiv.org/pdf/2204.03243.pdf

Authors : @yumeng5 @xiongchenyan

Comments: