noamaanMulla-03/transformers-issues huggingface dataset:
Unpin numba
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

Numba was pinned to <0.57.0 in #23118 - this is because it forced an update of the numpy package to >= 1.24. From numpy >= 1.24, converting a ragged list to a numpy array requires the user to **explicitly** set `dtype=object` (before this happened automatically, but threw a deprecation warning).

This PR updates the feature extraction and tokenisation utils to explicitly specify `dtype=object` when converting ragged lists to numpy arrays.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Failing test is unrelated ([tf compile test](https://circleci-tasks-prod.s3.us-east-1.amazonaws.com/forks/storage/artifacts/d5b57382-7f67-4274-9623-7f238ef4fb6f/457033993/0fb5a370-4bdb-43e1-a1a2-c242ca17c8d3/0/~/transformers/reports/tests_tf/failures_short.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAQVFQINEONDE666BB%2F20230522%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230522T165404Z&X-Amz-Expires=60&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEJn%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMSJIMEYCIQCSbOdyE07pYZr8Irw7BBJgEnLQCNFnrj3zK2%2FcTGV9jwIhAM3cUGClvwCTfkqdiYJKR5orc0gzQPRB5KHXpXqX00XzKrQCCML%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEQAxoMMDQ1NDY2ODA2NTU2Igw2mZInyce2C0JoQbsqiALBk31dgOmIJwAV%2FwHQoioyIo8GaTDckrk0%2BW1lfxXFVCAB8YxcUGOvSBFIIijvkMpBa6jlob4IQ9dAZf%2FvMKH1aXfOEXU284URRx6VEYoapQELm8CUc35O3YEeF%2BOIyQojAI9e0oBTYlhqzn%2Fg2bzV3mvyzvKtHWN17wvOWHBAKoba%2Bt2FdxkJBKq%2BnmQHHyQYq7JxDJ1L0Jd6xKa6uT85hjmkPNFiZr8iAvicJtcSM4UVdY5o9yI2Wty8lc8IDykk9%2BS9HeJ2mONGAowSBJz33LKjZDnRe4oOWkKt%2F3kaRm%2BTVzchn4Hy6poGKG6wr%2FqAHv0Kyd%2FJ09FviDzMvppV4tFzoOXkgicwm7auowY6nAHbz06ploAc04Toucr8X%2BlicPoUNiKWQBpolxtbSGpfvxKsTlSge8HaMvGqx6EZSjDG0JlC3MCgKfLfg5WhwkX0MMBQnv4UhP65R%2B7aEUNv%2B9yZOC6NCvvu8Bv9Hj0ml4fRCcggNYtqwNQKZRmshd59IK0ZqBCdWIfbQ5x8uvmJubBnBR7kmfmexwxiUOZ%2BbVMZBqDpSuzXGhXlH%2B0%3D&X-Amz-SignedHeaders=host&x-id=GetObject&X-Amz-Signature=5c8e53587d9a3023543892b06988a69ef02ebd3d2ccff9bfdac9b8dbf7587786))
[ASR pipeline] Check for torchaudio
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/pull/23445#discussion_r1213368268: checks whether `torchaudio` is available to import and throws a useful error message to the user if not


Comments:
_The documentation is not available anymore as the PR was closed or merged._
[EnCodec] Changes for 32kHz ckpt
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

Updates the EnCodec config and modelling code to allow two options for the residual connection in the Resnet block:
1. Pass the residual through a Conv1d
2. Apply the residual directly (identity)

=> this change is required to use the latest 32kHz EnCodec model in the Music Gen model (#24109). It is tested for with a fast test, and confirmed to match the original implementation.


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Leverage Langchain Prompt Templates
Author: surajsharan
Pull request: no

By leveraging the Prompt Templates and the Conversation Memory we can induce ReAct ( Thought, Observation ) approach from Langchain

Comments:
cc @LysandreJik @sgugger 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fix gradient checkpointing bugs in freezing part of models (requires_grad=False)
Author: IrisRainbowNeko
Pull request: yes

# What does this PR do?

Same as the [PR I opened](https://github.com/huggingface/diffusers/pull/3404) in diffusers. 

Using ```torch.utils.checkpoint.checkpoint``` directly will cause the parameters in the checkpoint section to not be learned when part of the model parameters are freezed. As these discussions state:

https://discuss.pytorch.org/t/use-of-torch-utils-checkpoint-checkpoint-causes-simple-model-to-diverge/116271

https://discuss.pytorch.org/t/checkpoint-with-no-grad-requiring-inputs-problem/19117/19

In pytroch versions larger than 1.11.0, the ```use_reentrant=False``` can be added to fix this bug.

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_23422). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
"MPTForCausalLM not supported" error when using pipeline, but not when using from_pretrained
Author: leondz
Pull request: no

### System Info

Python 3.8.10 (default, Nov 14 2022, 12:59:47) 

transformers.__version__ is '4.30.2'

lambda labs 1xA100

invoking `generator = transformers.pipeline(task="text-generation", model="mosaicml/mpt-7b", trust_remote_code=True)` ends with this exception:

```
You are using config.init_device='cpu', but you can also use config.init_device="meta" with Composer + FSDP for fast initialization.
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:05<00:00,  2.73s/it]
Xformers is not installed correctly. If you want to use memory_efficient_attention to accelerate training use the following command to install Xformers
pip install xformers.
The model 'MPTForCausalLM' is not supported for text-generation. Supported models are ['BartForCausalLM', 'BertLMHeadModel', 'BertGenerationDecoder', 'BigBirdForCausalLM', 'BigBirdPegasusForCausalLM', 'BioGptForCausalLM', 'BlenderbotForCausalLM', 'BlenderbotSmallForCausalLM', 'BloomForCausalLM', 'CamembertForCausalLM', 'CodeGenForCausalLM', 'CpmAntForCausalLM', 'CTRLLMHeadModel', 'Data2VecTextForCausalLM', 'ElectraForCausalLM', 'ErnieForCausalLM', 'GitForCausalLM', 'GPT2LMHeadModel', 'GPT2LMHeadModel', 'GPTBigCodeForCausalLM', 'GPTNeoForCausalLM', 'GPTNeoXForCausalLM', 'GPTNeoXJapaneseForCausalLM', 'GPTJForCausalLM', 'LlamaForCausalLM', 'MarianForCausalLM', 'MBartForCausalLM', 'MegaForCausalLM', 'MegatronBertForCausalLM', 'MvpForCausalLM', 'OpenLlamaForCausalLM', 'OpenAIGPTLMHeadModel', 'OPTForCausalLM', 'PegasusForCausalLM', 'PLBartForCausalLM', 'ProphetNetForCausalLM', 'QDQBertLMHeadModel', 'ReformerModelWithLMHead', 'RemBertForCausalLM', 'RobertaForCausalLM', 'RobertaPreLayerNormForCausalLM', 'RoCBertForCausalLM', 'RoFormerForCausalLM', 'RwkvForCausalLM', 'Speech2Text2ForCausalLM', 'TransfoXLLMHeadModel', 'TrOCRForCausalLM', 'XGLMForCausalLM', 'XLMWithLMHeadModel', 'XLMProphetNetForCausalLM', 'XLMRobertaForCausalLM', 'XLMRobertaXLForCausalLM', 'XLNetLMHeadModel', 'XmodForCausalLM'].
```

However, loading using:

```
model = transformers.AutoModelForCausalLM.from_pretrained(
  'mosaicml/mpt-7b',
  trust_remote_code=True
)
```

works fine.

How can I load this model in a `pipeline`?

### Who can help?

@Narsil @ArthurZucker @younesbelkada 



### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. `generator = transformers.pipeline(task="text-generation", model="mosaicml/mpt-7b", trust_remote_code=True)`


### Expected behavior

The pipeline would load OK, just as .from_pretrained works

Comments:
Hey, I suggest you to open this issue on the repo, this is because the `auto_map ` attribute in the `config.json` file is not properly set. We are probably going to add this model to transformers soon too!
add missing alignment_heads to Whisper integration test
Author: hollance
Pull request: yes


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @sgugger

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam

Documentation: @sgugger, @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: @sgugger
- TensorFlow: @Rocketknight1

 -->


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix tpu_metrics_debug
Author: cowanmeg
Pull request: yes

# What does this PR do?
Adding the `--tpu_metrics_debug` argument causes an error. Quick fix before the argument is deprecated. 

In `training_args.py` check if `self.debug` is None before appending the string since `self.debug` is now initialized to None instead of the empty string. Related to https://github.com/huggingface/transformers/pull/24033.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@amyeroberts 



Comments:
_The documentation is not available anymore as the PR was closed or merged._
Update AlbertModel type annotation
Author: amyeroberts
Pull request: yes

# What does this PR do?

Fixes type annotations.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
_The documentation is not available anymore as the PR was closed or merged._
deepspeed z1/z2 state dict fix
Author: pacman100
Pull request: yes

# What does this PR do?
1. Fixes https://github.com/huggingface/transformers/issues/22822
2. Should be merged after https://github.com/huggingface/accelerate/pull/1638

The fix in accelerate uses the `deepspeed.checkpoint.utils.clone_tensors_for_torch_save` which removes the bloated state_dict. In Trainer, we use `accelerator.get_state_dict` to get the resultant lean state dict when using DeepSpeed and the ZeRO stage is not 3. 

Also, a typo fix.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hello Sylvain, updated the description of the PR, Thank you!
when resume from peft checkpoint, the model should be trainable
Author: sywangyi
Pull request: yes

model.training is false after model.load_adapter of peft. see https://github.com/huggingface/peft/blob/main/src/peft/peft_model.py#L402, default value of is_trainable is False

- trainer: @sgugger



Comments:
@younesbelkada please help review

_The documentation is not available anymore as the PR was closed or merged._
For loop support in python interpreter of Transformer agent
Author: dcy0577
Pull request: no

### Feature request

Hello, I would like to add for loop support in https://github.com/huggingface/transformers/blame/c2393cad085e3875ee2206d917d46d15e50602a3/src/transformers/tools/python_interpreter.py
Any idea about how to implement this?

### Motivation

For loop is quite common in generated code. Usually it will not cause infinite loop.

### Your contribution

An additional 'elif' for 'ast.For' before the 'else' will be nice: https://github.com/huggingface/transformers/blame/c2393cad085e3875ee2206d917d46d15e50602a3/src/transformers/tools/python_interpreter.py#L132 

Comments:
cc @sgugger 

Would you like to open a PR for this?

I would like to, but I don't know how to implement it yet. Do you have any suggestions?

Should be added with the PR mentioned above :-)
LlamaTokenizer should follow signature of PreTrainedTokenizer
Author: zplizzi
Pull request: no

### Feature request

PreTrainedTokenizer has a signature such that if the eos/bos tokens shouldn't be applied, they're set to None in the constructor. LlamaTokenizer (which is a subclass of PreTrainedTokenizer) instead always sets these, and adds a `add_eos/bos_token` field to enable/disable them. This breaks code that depended on the behavior of the base class in order to detect how to form sequences, eg when doing custom tokenizations using `tokenizer(stuff, add_special_tokens=False)` to build pieces of a sequence and then manually adding the EOS/BOS tokens.

cc @zphang @ArthurZucker from git blame

### Motivation

-

### Your contribution

-

Comments:
The default `eos_token` and `bos_tokens` are there because the `sentence piece` model has these set, which means we are following the `llama` implementation. Having `add_eos` and `add_beo` gives the flexibility of enabling the addition, while not having to set the tokens to do so. 
This might not fit your specific usage, but most of our tokenizer work that way! 
I am not sure I understand why it would break in your case, but you can easily set the `eos` and `bos` to `None`, the same goes with the `add_eos` and `add_bos` that you can set to `False`

This is even more confusing after I was told that the normal transformers tokenizers add CLS and SEP to sequences by default when `add_special_tokens=True`, but in this class, you're (optionally) instead adding BOS and EOS instead.

Simply setting eos/bos to None works, but a user isn't expecting to have to do this to get behavior that's compatible with the base class. And tokenization bugs tend to be very subtle and take a lot of time to track down - this issue didn't crash my code, it just silently inserted an extra token (which caused havoc downstream). The point of inheritance is that a subclass should have the same public interface as the parent, so that a user just has to conform with the interface of the parent and can expect all subclasses to just work. This isn't the case here. 

Thanks for educating me on inheritance, I understand your use-case and how this can be confusing. The problem is that in order to keep the information of the content of these tokens, while not necessarily adding them prevents us from setting them to `None`. Indeed this breaks your code, but it also allows people to use the tokenizer in another way. They can decide whether to add or not the eos and bos depending on the usage. 

Overall it's a design choice and lots of other tokenizer don't respect this either. I am sorry that it broke your pipeline.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Wrong special tokens using XLM-RoBERTa's tokenizer for question answering
Author: severinsimmler
Pull request: no

### System Info

- `transformers` version: 4.30.2
- Platform: Linux-6.2.0-20-generic-x86_64-with-glibc2.37
- Python version: 3.11.3
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.0+cu117 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

@ArthurZucker 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Tokenizing a question and its context with XLM-RoBERTa's tokenizer:

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("xlm-roberta-base")

question = "This is a question?"
context = "This is the context."

inputs = tokenizer(question, context)

tokenizer.decode(inputs["input_ids"])
```

returns something like this:

```

 This is a question?

 This is the context.

```

i.e. with _two_ SEP tokens between question and context. Is this expected behavior? Shouldn't it be separated by only one `</sep>` or even `</sep><sep>`?

### Expected behavior

I'd expect the tokenizer to return:

```

 This is a question?

 This is the context.

```

Comments:
Hey, I am not sure where your expectations come from, but in the task of `sequence_classification`, you are not simply concatenating two sentences. If you have a look [here](https://github.com/facebookresearch/XLM/blob/cd281d32612d145c6742b4d3f048f80df8669c30/generate-embeddings.ipynb#L130), an original colab formats the sentences the same way. 

No that's indeed expected behaviour, RoBERTa models use 2 special tokens in between context and question, unlike BERT.

See here: https://github.com/huggingface/transformers/blob/8e164c5400b7b413c7b8fb32e35132001effc970/src/transformers/models/roberta/tokenization_roberta.py#L346

Got it, thanks a lot for clarifying :) 🙏
[`InstructBlip`] Add accelerate support for instructblip
Author: younesbelkada
Pull request: yes

# What does this PR do?

As per title, let's make users benefit from 8bit / 4bit loading of instructblip models

cc @amyeroberts @sgugger @NielsRogge 

all `accelerate` tests pass for this model

As a side note, as instruct blip relies on flan-t5 as backbone for some models, therefore it is important to add
```python
_keep_in_fp32_modules = ["wo"]
```
To ensure inference stability in fp16 / int8 / fp4 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thank you :) could you add an integration test?

Hey @NielsRogge !
Let's maybe add it together with: https://github.com/huggingface/transformers/pull/24490/files#r1242095062 so we should probably  merge this first :D
Compute `dropout_probability` only in training mode
Author: ydshieh
Pull request: yes

# What does this PR do?

Same issue as in #24483 (caused by #24434 ), but with a different fix. For core maintainers to decide which one is better.

If we decide to go this way, I will do fix-copies.

Comments:
_The documentation is not available anymore as the PR was closed or merged._