noamaanMulla-03/transformers-issues huggingface dataset:
fix blip2config int8 error to serialize json
Author: Andrechang
Pull request: yes



# What does this PR do?

Running the following using transformers=4.30.0.dev0
```
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)
model = AutoModelForVision2Seq.from_pretrained("Salesforce/blip2-opt-2.7b", quantization_config=bnb_config, device_map='auto')
print(model.config)
```
Will give the following error:
```
TypeError: Object of type BitsAndBytesConfig is not JSON serializable
```
Solution add convert BitsAndBytesConfig to json in Blip2Config



## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@amyeroberts
@NielsRogge 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @sgugger

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam

Documentation: @sgugger, @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: @sgugger
- TensorFlow: @Rocketknight1

 -->


Comments:
cc @younesbelkada 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_24094). All of your documentation changes will be reflected on that endpoint.

I ran `pip install -e .` on transformer main branch 8b169142f8b5735f25ad81313ee382350161d993
Then ran the code snippet and gave the error: 
`TypeError: Object of type BitsAndBytesConfig is not JSON serializable`

Adding my commit will fix this

print screen attached
![install0](https://github.com/huggingface/transformers/assets/9553458/1a95e026-fbfb-4eac-8454-bcdb35b86888)

![error0](https://github.com/huggingface/transformers/assets/9553458/f77f56ce-341e-4fff-a2c4-fb263539c718)




Hi @Andrechang 
Again thanks very much for flagging the issue, I realized that you have flagged an issue that is quite important to fix, therefore I made https://github.com/huggingface/transformers/pull/24137 and added you as a co-author, will merge that PR and I will close this one. Feel free to re-open it if you think that the issue is not resolved. Again thanks a lot!

Thank you for checking and for fix
Flaky Whisper PT-TF & PT-Flax Equivalence Test
Author: amyeroberts
Pull request: no
Labels: name: WIP

### System Info

transformers 4.29.0 dev

### Who can help?

@ArthurZucker @sanchit-gandhi @ydshieh 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Flaky test, so not reproducible. Example run where the error occurred: 

* https://app.circleci.com/pipelines/github/huggingface/transformers/64100/workflows/b4463c5d-b3dc-4b00-a7cd-19acd096cb07/jobs/792381
* https://app.circleci.com/pipelines/github/huggingface/transformers/64111/workflows/dc9092c4-0673-46c7-b89f-f805bc20128c/jobs/792557
* https://app.circleci.com/pipelines/github/huggingface/transformers/64230/workflows/e2d42ca4-f367-4a85-9054-a0ea99e49849/jobs/794534

Occasionally, the PT-TF and PT-Flax whisper equivalence test fails. The tolerance was increased in #23257 and #23288 but the reason for recent failures has not yet been found. 
 
### Expected behaviour

Equivalence tests reliably pass. 

Comments:
If this started happening recently, it might be related to https://github.com/huggingface/transformers/pull/21998

It's possible the feature extraction for PyTorch now gives different results than the TF / Flax versions. It shouldn't, but it's possible that a small difference in the preprocessed inputs is causing this.
Transformers can not load dependency of tensorflow - `cannot import name 'call_context' from 'tensorflow.***.keras.engine'`
Author: MaximeChurin
Pull request: no

### System Info

- `transformers` version: 4.30.0
- Platform: Linux-5.19.0-43-generic-x86_64-with-glibc2.34
- Python version: 3.8.16
- Tensorflow version (GPU?): 2.8.2 (False)

### Who can help?

@gante and @Rocketknight1

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The problem arises due to this [PR](https://github.com/huggingface/transformers/pull/23760) and the [case](https://github.com/huggingface/transformers/blame/main/src/transformers/modeling_tf_utils.py#L84) for tensorflow import of `call_context` according to minor version. In tf < 2.11, the import should be `from tensorflow.python.keras.engine.base_layer_utils import call_context`
Steps to reproduce:
1. Install the System Info dependencies mentionned above
2. `from transformers import DistilBertTokenizerFast, TFDistilBertModel`
3. Then you should see
```bash
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/***3.8/site-packages/transformers/models/distilbert/modeling_tf_distilbert.py", line 37, in <module>
    from ...modeling_tf_utils import (
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/***3.8/site-packages/transformers/modeling_tf_utils.py", line 84, in <module>
    from tensorflow.***.keras.engine import call_context
ImportError: cannot import name 'call_context' from 'tensorflow.***.keras.engine' (/opt/hostedtoolcache/Python/3.8.16/x64/lib/***3.8/site-packages/tensorflow/***/keras/engine/__init__.py)
```

### Expected behavior

Update the case where the import is [located](https://github.com/huggingface/transformers/blame/main/src/transformers/modeling_tf_utils.py#L84) to the correct naming inside tensorflow `from tensorflow.python.keras.engine.base_layer_utils import call_context`

Comments:
Agh, sorry about that. PR to fix it is open at #24138 

thanks feel free to close it when it is merged, if the PR does not close it automatically

Fixed on `main`, but we'll need to organize a patch release before you can unpin. Sorry for the trouble!

@MaximeChurin This should be resolved by the 4.30.1 hotfix release, along with a couple of other release bugs. You should be able to unpin now!
Tool types
Author: LysandreJik
Pull request: yes

WIP PR to have specific types for tool outputs, which should clarify interaction to and from Agents.

Left to do:
- [x] Remove or complete the video integration
- [x] Add support for remote tools
- [x] Complete documentation
- [x] Test it out with real world use-cases
- [x] Add a test to ensure that the inputs are cast correctly (so far only the outputs are tested)
- [x] Arrange the dependencies so that they don't make all the tests fail

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Provide a different API solution instead of offline mode
Author: rbavery
Pull request: no

### Feature request

First off, thanks for the stellar lib and for all the work to get state of the art models in a consumable and documented state! I'm somewhat new to transformers, so this feedback is coming from a place of heavily using the library for a week.

I find offline mode unintuitive. I don't see it referenced on a variety of tutorials so I spent a day trying to figure out how to load local files with transformers, see https://github.com/huggingface/transformers/issues/23116

The docs are somewhat buried in a section I wouldn't expect: Installation https://huggingface.co/docs/transformers/installation#offline-mode

### Motivation

Offline mode makes it more difficult to work with local files, instead, methods like `from_pretrained` could pay attention to if a file path is locally sourced (starts with / ) or a url (https) or huggingface repo (no / or url prefix, has a username/repo pattern)

### Your contribution

I'm happy to provide feedback.

Comments:
Hi there. `from_pretrained` already accepts a path to a folder, I'm not sure what it is you are requesting.

The relevant docs to load from local data can be found here: https://huggingface.co/docs/transformers/v4.28.1/en/main_classes/model#transformers.PreTrainedModel.from_pretrained.

The `from_pretrained` method accepts either a `repo_id` to a repo on the 🤗 hub or a local path to a folder.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
After completion of Trainer.hyperparameter_search() attribute trainer.state.best_model_checkpoint references the last trained model instead of the best one
Author: fantauzzi
Pull request: no

### System Info

- `transformers` version: 4.28.1
- Platform: Linux-5.19.0-41-generic-x86_64-with-glibc2.35
- Python version: 3.11.3
- Huggingface_hub version: 0.14.1
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.0+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no


### Who can help?

@sgugger

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

https://colab.research.google.com/drive/1Ht14ntTQy96-_zO-iVlwvAgkyY8t6vKc?usp=sharing

Call `Trainer.hyperparameter_search()`; when it completes, attribute `Trainer.state.best_model_checkpoint` and other `Trainer.state` attributes reference the last trained model, in the sequence of models trained by `Trainer.hyperparameter_search()`.

Note: to speed-up reproduction of the issue, I have limited the training dataset size in the provided code, line #49; that's why the evaluation metrics at the end of the hyperparameters search are poor.

### Expected behavior

After `Trainer.hyperparameter_search()` completes, attribute `Trainer.state.best_model_checkpoint` should contain the filename of the checkpoint with the **best** model among all the models trained during hyperparameters search, not the **last** model; that is, the model trained during the run indicated in the `BestRun` instance returned by `hyperparameter_search()`

Likewise, other `Trainer.state` attributes should relate to the same model, e.g:
`Trainer.state.best_metric`
`Trainer.state.epoch`
`Trainer.state.global_step`


Comments:
Hyperparameter search does not play well with the best model indeed. That's not something in our roadmap for fixing, but we are happy to look at any PR!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[tests] fix bitsandbytes import issue
Author: stas00
Pull request: yes

this test has been failing when `peft` library was installed since it tries to access `bitsandbytes.nn`
```
$ pip install accelerate peft bitsandbytes==0.39.0
$ RUN_SLOW="yes" pytest -sv tests/extended/test_trainer_ext.py::TestTrainerExt::test_run_seq2seq_bnb
[...]
RuntimeError: Failed to import transformers.trainer_seq2seq because of the following error (look up to see its traceback):
E           module 'bitsandbytes' has no attribute 'nn'
```

here is why it's happening:

1. We push `transformers/tests` into `sys.path` when running the subprocess-based tests [here](https://github.com/huggingface/transformers/blob/deff5979fee1f989d26e4946c92a5c35ce695af8/src/transformers/testing_utils.py#L1226)
2. but we have `transformers/tests/bitsandbytes` dir under `transformers/tests`
3. so when you do import `bitsandbytes.nn` it finds the wrong `bitsandbytes` dir, which is not the bnb library and it breaks

So this PR renames `transformers/tests/bitsandbytes` to `transformers/tests/bnb` which removes the conflicts and the failing test.



Comments:
_The documentation is not available anymore as the PR was closed or merged._
Avoid OOM in doctest CI
Author: ydshieh
Pull request: yes

# What does this PR do?

This is done by inject `gc.collect()` to the source code during pytest/doctest collecting the test to run.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Update: after a correction, only 30 minutes longer or a bit more.

The job runs 2x slower ...

Running the whole doctest: No more OOM. Only one test failure (lucky!)

Tag @sgugger so we can have doctest run until @Rocketknight1 have a complete solution in #24146.
(We haven't received any doctest report for 2 weeks)
Fix `MobileViTV2` checkpoint name
Author: ydshieh
Pull request: yes

# What does this PR do?

For `tests/models/mobilevitv2/test_modeling_mobilevitv2.py::MobileViTV2ModelTest::test_model_from_pretrained`,  we get
```bash
(line 433)  OSError: apple/mobilevitv2-1.0-imagenet1k-256 is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
```

This PR changes the checkpoint name to avoid this failure.

**Question**: Should we upload the model to `apple/mobilevitv2-1.0-imagenet1k-256` and change all checkpoint names in the code to `apple/mobilevitv2-1.0-imagenet1k-256`?

cc @shehanmunasinghe 


Comments:
_The documentation is not available anymore as the PR was closed or merged._

@ydshieh Sorry, my bad, I'd assumed because the checkpoints in the PR were under the org they were already uploaded and didn't verify. 

> Question: Should we upload the model to apple/mobilevitv2-1.0-imagenet1k-256 and change all checkpoint names in the code to apple/mobilevitv2-1.0-imagenet1k-256?

Yep! 

OK, so I should change everything to `apple` in this PR instead. I am not a member of `apple` on `Hub`. Maybe I can ask @hollance to help me on this?

Also, I'd like to suggest moving the following checkpoints under `apple` org as well.

```
shehan97/mobilevitv2-1.0-voc-deeplabv3

shehan97/mobilevitv2-2.0-imagenet1k-256

shehan97/mobilevitv2-1.5-voc-deeplabv3
```


@amyeroberts checkpoint is uploaded to `apple`. All tests pass now 🙏 

Thanks. I missed the other 3 ones 😭 

> Also, I'd like to suggest moving the following checkpoints under `apple` org as well.
> 
> ```
> shehan97/mobilevitv2-1.0-voc-deeplabv3
> 
> shehan97/mobilevitv2-2.0-imagenet1k-256
> 
> shehan97/mobilevitv2-1.5-voc-deeplabv3
> ```

Thank you @shehanmunasinghe for the heads up 🤗
Fix `Wav2Vec2` CI OOM 
Author: ydshieh
Pull request: yes

# What does this PR do?

After #23813, we get OOM for `tests/models/wav2vec2/test_modeling_wav2vec2.py::Wav2Vec2ModelIntegrationTest::test_wav2vec2_with_lm_invalid_pool` (when running the whole `wav2vec2` test suite)

Just doing some cleaning up and no more OOM.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
ONNX conversion of deberta_v2 models
Author: kobiche
Pull request: no

### System Info

transformers: 4.22.2
platform: Ubuntu 20.04.2
python: 3.8.10

### Who can help?

DeBERTa-v2: @LysandreJik

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The following command fails:
python -m transformers.onnx --model=microsoft/deberta-v3-small onnx/

The model can be microsoft/{mdeberta-v3-base, deberta-v3-small, deberta-v3-base, etc.}. 
The main problem lies in the `symbolic(...)` method of the class `XSoftmax` class, that is called while tracing the model. However, there are several other warnings while converting the model (Are they important?)

### Expected behavior

Model saved as ONNX format.

Comments:
I believe I solved it (rather 'I forced it to work')
(line: 159)
```python
        t = self.type().dtype() if hasattr(self.type(), 'dtype') else self.type().scalarType()
        TYPE_CAST = {
            'float': torch.float64,
            'int': torch.int64
        }
        t = TYPE_CAST[t.lower()]
        output = masked_fill(
            g, self, r_mask, g.op("Constant", value_t=torch.tensor(torch.finfo(t).min))
        )
```

This is not tested with all models, but at least works for now. Hopefully you'll find a nicer solution.

cc @michaelbenayoun @lewtun 

[UPDATE]

Remember when I said that there were warnings while tracing the model? Well, I know now they are important.
I cannot use the traced model with an input different from the one I used to trace the model.
This is the error I get:
```python
onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Reshape node. Name:'Reshape_389' Status Message: /onnxruntime_src/onnxruntime/core/providers/cpu/tensor/reshape_helper.h:35 onnxruntime::ReshapeHelper::ReshapeHelper(const onnxruntime::TensorShape&, onnxruntime::TensorShapeVector&, bool) size != 0 && (input_shape.Size() % size) == 0 was false. The input tensor cannot be reshaped to the requested shape. Input shape:{12,15,15}, requested shape:{-1,12,136,136}
```

Please can you fix this?

Hey @kobiche thanks for reporting this error, we also noticed this in #19255 when validating the `deberta` models with different batch size / seq len compared to the one used to trace the model.

We'll aim to patch this ASAP

https://github.com/huggingface/transformers/blob/94b3f544a1f5e04b78d87a2ae32a7ac252e22e31/src/transformers/models/deberta_v2/modeling_deberta_v2.py#L577
And ONNXRuntime dosen't support torch.where op now.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[Trainer] Why not use `tqdm`'s `dynamic_ncols=True` option?
Author: gmlwns2000
Pull request: no

### Feature request

# Problem

Tqdm progress bar is getting ugly when the width of the terminal is shrunk!

![image](https://github.com/huggingface/transformers/assets/4879345/b60f232f-41a5-40de-b759-8bb2710d3b5f)

It progress bar makes the new line on every update! It is very ugly...

# Solution

Simply add the `dynamic_ncols=True` option to `tqdm`. It is located in `trainer_callbacks.ProgressCallback`.

![image](https://github.com/huggingface/transformers/assets/4879345/6741eb00-7430-48db-acc8-4c3a0eb00217)

You can check the progress bar is now dynamically resized when the terminal size is updated.

### Motivation

When I connect `tmux` session with different widths of the terminal, then the `tqdm` printing is getting ugly.

### Your contribution

Please check the PR #24101

Comments:
Change ProgressCallback to use dynamic_ncols=True
Author: gmlwns2000
Pull request: yes

# What does this PR do?

Fixes #24100 

## Who can review?

This is about the trainer module, therefore I will tag,

@sgugger

Thank you

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hi @sgugger,

I merge from `upstream/main`, and then run `make style`!

I hope this will be fine.

Mm the diff now shows 43 files. COuld you limit your changes to the file you are actually modifying?

@sgugger 

Sorry for the late reply, and it looks fine now!

Thanks! Failures are unrelated and due to a down time on the Hub.
[i18n]Translated "attention.mdx" to korean
Author: kihoon71
Pull request: yes

# What does this PR do?

Translated attention.mdx file of the documentation to Korean.

Thank you in advance for your review.

Part of https://github.com/huggingface/transformers/issues/20179

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. [[lowercased-header]])
- [x] Check live-preview for gotchas (live-preview로 정상작동 확인)


## Who can review?(Initial)

Team PseudoLab, may you please review this PR? @0525hhgus, @KIHOON71, @sim-so, @gabrielwithappy, @HanNayeoniee, @wonhyeongseo, @jungnerd

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?(Final)
@sgugger, @ArthurZucker, @eunseojo May you please review this PR?


<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @sgugger

Integrations:

- deepspeed: HF Trainer: @stas00, Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam

Documentation: @sgugger, @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: @sgugger
- TensorFlow: @Rocketknight1

 -->


Comments:
_The documentation is not available anymore as the PR was closed or merged._