noamaanMulla-03/transformers-issues huggingface dataset:
[`Blip`] Fix blip doctest
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes the current Blip Failing doctest:

```python
from PIL import Image
import requests
from transformers import AutoProcessor, BlipForQuestionAnswering

model = BlipForQuestionAnswering.from_pretrained("Salesforce/blip-vqa-base")
processor = AutoProcessor.from_pretrained("Salesforce/blip-vqa-base")

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

# training
text = "How many cats are in the picture?"
label = "2"
inputs = processor(images=image, text=text, return_tensors="pt")
labels = processor(text=label, return_tensors="pt").input_ids

inputs["labels"] = labels
outputs = model(**inputs)
loss = outputs.loss
loss.backward()
```

In https://github.com/huggingface/transformers/pull/23153 I removed the redundant tokens shifting but also removed the assignment of `decoder_input_ids` if they are set to `None`, which is needed for training 

This PR applies also the same fix on TF Blip

cc @sgugger @ydshieh  

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[WIP] Flax EfficientNet
Author: Shubhamai
Pull request: yes

# What does this PR do?

Following the PR https://github.com/huggingface/transformers/pull/21563 by [alaradirik](https://github.com/alaradirik) to add the corresponding flax model. 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

- Flax: sanchit-gandhi


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_21876). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@sanchit-gandhi was not properly tagged.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Feel free to re-open if you want to finish this one off @Shubhamai! Otherwise leaving closed for now.
is_batched fix for remaining 2-D numpy arrays
Author: LWprogramming
Pull request: yes

# What does this PR do?

Fix `is_batched` logic for 2-D numpy arrays, as described in https://github.com/huggingface/transformers/pull/23223#pullrequestreview-1423033751

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [X] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sanchit-gandhi


Comments:
Hmm, this is odd-- I just noticed manually (locally) running `pytest` on the tests fixed here just shows they're all automatically skipped. Looking into configuration now...

EDIT: oh, due to https://github.com/huggingface/transformers/issues/18355#issuecomment-1543277694 I hadn't had torch installed, torchaudio, a whole bunch of libraries. should work now

_The documentation is not available anymore as the PR was closed or merged._

Thanks for the quick changes @LWprogramming! Let's wait until #23223 is finalised before getting this merged so that we can update this PR with any amendments from there

> Thanks for the quick changes @LWprogramming! Let's wait until #23223 is finalised before getting this merged so that we can update this PR with any amendments from there

Ok, updated the code with comments from that PR, and ran tests + linters
Fix some docs what layerdrop does
Author: zspo
Pull request: yes

# What does this PR do?

Fix some configuration docs what layerdrop does. Copy from `configuration_opt.py` and reset defaults which is same as init values.

Fixes # (issue)
https://github.com/huggingface/transformers/issues/23351

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sgugger, @stevhliu, @MKhalusova and @sanchit-gandhi 



Comments:
_The documentation is not available anymore as the PR was closed or merged._

> Thanks for your fix! Just have one syntax comment to propagate on all docs.

Thanks for your suggestion! I will add more commits later.
Splitting the transformers dependencies
Author: gokceneraslan
Pull request: no

### Feature request

Right now this Python packages has a lot of dependencies including major DL frameworks like PyTorch, TensorFlow and Jax. This causes some complexity in the downstream packages that use `transformers` e.g. CI/CD environments get gigantic and it likely creates dependency issues e.g.

![image](https://github.com/huggingface/transformers/assets/1140359/b9651a1a-bf67-4674-ba0f-866cea465c91)

and

![image](https://github.com/huggingface/transformers/assets/1140359/a6a14ba7-bee4-40f7-9b62-08552ec551d6)


This is a CI exception of a Torch code using enformer-pytorch, which depends on transformers. Although there is nothing using Jax either in the Torch code or in the enformer-pytorch, we have to solve this Jax-related issue now.

I was wondering if you can somehow split the dependencies into groups e.g. `'pip install transformers[jax]'` or `'pip install transformers[pytorch]'`. Let me know what you think.

### Motivation

I think splitting the dependencies into reasonable groups would improve installation or testing of the downstream packages using transformers.

### Your contribution

I am not able to contribute, but I think my suggestions is relatively simple to implement.

Comments:
This is already the case.

Oh I didn't know! So `'pip install transformers[torch]'` doesn't install jax or tensorflow?

No. You may have it in your environment from other installs, but `pip install transformers[torch]` will only install Transformers and its core dependencies (very light) and torch.

Thanks so much!
ValueError: Attempting to unscale FP16 gradients.
Author: lucasjinreal
Pull request: no

### System Info

V100 torch2./0

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

When i train a LLMs with this

```
# data_collator = DataCollatorForSupervisedDataset(tokenizer=tokenizer)
    data_collator = DataCollatorForSeq2Seq(
        tokenizer, pad_to_multiple_of=8, return_tensors="pt", padding=True
    )
```

it throws above error, but thwn I using self-defined one, it works OK. why?

```
@dataclass
class DataCollatorForSupervisedDataset(object):
    """Collate examples for supervised fine-tuning."""

    tokenizer: transformers.PreTrainedTokenizer

    def __call__(self, instances: Sequence[Dict]) -> Dict[str, torch.Tensor]:
        input_ids, labels = tuple(
            [instance[key] for instance in instances] for key in ("input_ids", "labels")
        )
        input_ids = torch.nn.utils.rnn.pad_sequence(
            input_ids, batch_first=True, padding_value=self.tokenizer.pad_token_id
        )
        labels = torch.nn.utils.rnn.pad_sequence(
            labels, batch_first=True, padding_value=-100
        )
        return dict(
            input_ids=input_ids,
            labels=labels,
            attention_mask=input_ids.ne(self.tokenizer.pad_token_id),
        )
```

the only differences here is the data collator

### Expected behavior

should trainable with fp16

Comments:
Llama Tokenizer uses incorrect indices for PAD
Author: michaelroyzen
Pull request: no

### System Info

latest transformer main

### Who can help?

@gante

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

configuration_llama.py sets 
```
pad_token_id=0,
bos_token_id=1,
eos_token_id=2
```

but this is wrong. After checking the original tokenizer from FB,

```python
sp_model = SentencePieceProcessor(model_file="/home/ubuntu/llama/tokenizer.model")

print("bos_id: ", sp_model.bos_id())
print("eos_id: ", sp_model.eos_id())
print("pad_id: ", sp_model.pad_id())
```

we see that 

```
bos_id:  1
eos_id:  2
pad_id:  -1
```



### Expected behavior

```
bos_id:  1
eos_id:  2
pad_id:  -1
```

instead of 

```
pad_token_id=0,
bos_token_id=1,
eos_token_id=2
```

Comments:
cc @ArthurZucker -- is this fixed in #22402 ?

This is probably not gonna be fixed with regard to the `configuration_llama`. 
However note that having the `sp_model` sending `-1` as a pad token means that it does not have any indices. Llama does not use a padding token. 
The fix that we provide is that in the `tokenization_llama` the `pad_token` is set to `None`. 

The config should be fixed to ensure that `pad_token=None` rather than `pad_token = 0`

Thanks @gante @ArthurZucker 

What about the mismatch between the eos and bos tokens? Or is HF's tokenizer zero-indexed while Meta's native tokenizer is one-indexed?

As you said and showed using the `sp_model`, 
> bos_id:  1
eos_id:  2

If you instantiate the tokenizer using [this](https://huggingface.co/hf-internal-testing/llama-tokenizer/tree/main) for example, it has the same ids so I am not sure I follow the problem? 

@ArthurZucker if I want to batching then I have to manually add a pad_token. In this case how do I ensure that the pad_token_id is actually correct? I.e how do I get the tokenizer to set pad_tokens to 0 instead of 32000  that I am getting now, by using add_special_tokens like `add_special_tokens({'pad_token': '[PAD]'})`

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

The problem is that `0` is already the `unk` token. The easiest way is to set the pad token to the unk token.
Export to ONNX doc refocused on using optimum, added tflite
Author: MKhalusova
Pull request: yes

Currently, despite the notes saying that the recommended way is to use Optimum, the main focus of the doc is on using `transformers.onnx`, which is no longer maintained (according to optimum team). 
This PR restructures the doc to put the optimum examples forward as the primary way for exporting models to ONNX. 
The example of using `transformers.onnx` is kept for potential compatibility reasons, however, I have removed the examples of adding new architectures to transformers.onnx, as I believe this should be done in Optimum instead (links provided).
As suggested by the team, I have also added an example for TFLite, and for that reason, renamed the page to "Export to ONNX, TFLite" 

UPD: since we now link to Optimum docs for the list of supported architectures, I have also removed the ONNX list check & update from the check_table script.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Better TF docstring types
Author: Rocketknight1
Pull request: yes

cc @sgugger - this is the PR based on the conversation we had on Slack! I just scanned our TF files and replaced some `Optional[]` and `Union[]` patterns with `|` instead. The doc builder now writes much cleaner docstrings, instead of the `tensorflow.python.framework.ops.Tensor` stuff it was writing before.

Some tests are failing because the test files also need `from __future__ import annotations` I think - will make sure everything passes before merging this.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

The tests seem to indicate we use those type annotations for something (auto-detecting the labels would be my guess).

We use them in one of the TF tests, in a way that causes issues with the `from __future__` import. I think there's an easy workaround for the affected test, though - working on it!

@sgugger issues have been resolved. What happened was two of the tests were trying to figure out which models were trainable using the type annotations if they were there, which was kind of flaky anyway and broke when we did this. I refactored the relevant code, which enabled some tests that were previously being skipped, and that surfaced a couple of small issues in the tests. The models are all fine, though, and everything passes now!
Minor awesome-transformers.md fixes
Author: pagarsky
Pull request: yes

# What does this PR do?

Fixes minor typos and updates link to Nebuly (renamed from nebullvm)

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

@LysandreJik

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @sgugger

Integrations:

- deepspeed: HF Trainer: @stas00, Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam

Documentation: @sgugger, @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: @sgugger
- TensorFlow: @Rocketknight1

 -->


Comments:
_The documentation is not available anymore as the PR was closed or merged._

@LysandreJik sorry for the ping, but can you check the PR?
QuestionAnsweringPipeline is never able to truncate the question
Author: Marcusntnu
Pull request: no

### System Info

I was trying out feeding some in-context examples question side to the pipeline, and based on the design of the QuestionAnsweringPipeline it's basically impossible to get the truncation done at the question side and not the context. 

question_first = bool(self.tokenizer.padding_side == "right") in the [docs](https://huggingface.co/transformers/v4.6.0/_modules/transformers/pipelines/question_answering.html) makes sure that whatever you try it's not possible to get actual question truncation (unless you just write out a whole new pipeline basically). 

Think this should be easily fixable if you made it possible to "force" truncation=longest.


tagging @Narsil as it's pipeline related.


### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Setup whatever QAPipeline

1. Send a long question together with whatever context that would require truncation.
2. See: Exception: Truncation error: Sequence to truncate too short to respect the provided max_length


### Expected behavior

It should be possible to pass very long questions, at least if specifying truncation=longest (this just get's overrides now).

Comments:
Are you sure it' s OK to truncate questions ? For the actual results of the model ?

We can definitely add more control over the truncation process, currently it's quite hardcoded because of all the Squad specific controls: https://github.com/huggingface/transformers/blob/main/src/transformers/pipelines/question_answering.py#L409

We could switch to `tokenizer_kwargs` to allow any parameters to be passed.

@sgugger for confirmation it' s a good idea ?

I don't think it is a good idea. This seems like a specific use-case for which you can use the tokenizer and model directly, instead of using the `pipeline`.

Understandable 👍
[docs] duplicate sentences in `document_question_answering.mdx`
Author: jungnerd
Pull request: no

### Description
There are duplicate sentences in `document_question_answering.mdx` from line number 40 to 45.

### Document / Language
`document_question_answering.mdx` / [en](https://huggingface.co/docs/transformers/tasks/document_question_answering)

### Suggestion
![Capture 20230524_1541@2x](https://github.com/huggingface/transformers/assets/46880056/899f6de9-3730-44f7-95eb-32f3f7d34be7)
 should be either:

<table>
<tr>
<td> candidate 1 </td> <td> candidate 2 </td>
</tr>
<tr>
<td>

(...), to predict the positions of the start and end tokens of the answer. (...)


</td>
<td>

(...), in order to predict which token is at the start of the answer and which token is at the end of the answer. (...)

</td>
</tr>
</table>

Please let me know which of the two candidates you would prefer. 


Comments:
I'd prefer candidate 1 personally. Thanks for reporting and looking forward to your PR with a fix!

Thanks for your feedback and I agree that candidate 1 is better. I opened PR #23735.
Imports in multiline try blocks are not properly ignored when determining the necessary packages for a modeling file
Author: dakinggg
Pull request: no

### System Info

- `transformers` version: 4.29.2
- Platform: macOS-13.3.1-arm64-arm-64bit
- Python version: 3.10.11
- Huggingface_hub version: 0.14.1
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: N/A
- Using distributed or parallel set-up in script?: N/A

### Who can help?

@sgugger 

### Reproduction

The minimal repro is to just call https://github.com/huggingface/transformers/blob/v4.29.2/src/transformers/dynamic_module_utils.py#L118-L134 on a file that has a multiline try import, e.g.

```
try:
    from package import function
    from package2 import function
except:
    pass
```

and run the `get_imports` function on it. The output will be `['package', 'package2']`, when it should be `[]`

### Expected behavior

Imports in multiline try blocks should be ignored when determining what packages a modeling file requires. I believe https://github.com/huggingface/transformers/blob/ba7054533fa455e8b2dd35feb077e0c7aae646b3/src/transformers/dynamic_module_utils.py#L126 just needs to be modified to include `flags=re.MULTILINE | re.DOTALL`.

Comments:
I'd make the change myself, but I can't tell where this codepath is tested (implicitly everywhere?), so will leave to a dev more familiar with `transformers` dev process and tests.

This is the right fix indeed. I don't think that function is tested yet, but you can add a new test file in `tests/utils/` named `test_dynamic_module_utils` with your failing test if you want to go the extra mile.
Fix the regex in `get_imports` to support multiline try blocks and excepts with specific exception types
Author: dakinggg
Pull request: yes

# What does this PR do?
Fixes the regex in `get_imports` to support multiline try blocks and excepts with specific exception types, by
1. adding `re.DOTALL` so that new lines are matched in the try block
2. adding `.*?` after the except so that it will match things like `except ImportError`

Fixes #23667


## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?
@sgugger 


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Done!