noamaanMulla-03/transformers-issues huggingface dataset:
TF port of Convnextv2
Author: IMvision12
Pull request: yes

# What does this PR do?

TF port of convnextv2

@amyeroberts 


Comments:
While converting pt weights to TensorFlow I am getting this error:
how to solve this?
```
All PyTorch model weights were used when initializing TFConvNextV2ForImageClassification.

All the weights of TFConvNextV2ForImageClassification were initialized from the PyTorch model.
If your task is similar to the task the model of the checkpoint was trained on, you can already use TFConvNextV2ForImageClassification for predictions without further training.
Traceback (most recent call last):
  File "/usr/local/bin/transformers-cli", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/dist-packages/transformers/commands/transformers_cli.py", line 55, in main
    service.run()
  File "/usr/local/lib/python3.10/dist-packages/transformers/commands/pt_to_tf.py", line 344, in run
    raise ValueError(
ValueError: The cross-loaded TensorFlow model has different outputs, something went wrong!

List of maximum output differences above the threshold (5e-05):
logits: 3.871e+00

List of maximum hidden layer differences above the threshold (5e-05):
hidden_states[1]: 3.463e-01
hidden_states[2]: 1.682e+00
hidden_states[3]: 2.259e+01
hidden_states[4]: 6.839e-01
```

Code used:

```
!transformers-cli pt-to-tf --model-name facebook/convnextv2-nano-1k-224 --no-pr --local-dir /content/convnextv2-nano-1k-224
```

_The documentation is not available anymore as the PR was closed or merged._
Load model from cloud storage
Author: Dref360
Pull request: no

### Feature request

I would like to load models directly from GCS/AWS

Example if your model is on Google Cloud:

```python
   model = AutoModelForSequenceClassification.from_pretrained('gs://my_bucket/my_model')
   tokenizer = AutoTokenizer.from_pretrained('gs://my_bucket/my_model')
   pipe = pipeline('text-classification', model='gs://my_bucket/my_model')
```

The `datasets` library supports this use case through `fsspec`. I propose to also use this library.

This could also simplify the code of `PretrainedConfig` as everything would use `fsspec` except if it's on the Hub.

### Motivation

I train my models in the cloud and then I load them into another tool such as Azimuth or a local Jupyter notebook for error analysis. 

I could simply upload them to the Hub, but I don't want to upload all my models to the Hub or manually upload them.

### Your contribution

I would be happy to submit a PR, might require some help.

Thanks!

Comments:
Thanks for opening an issue. There is no plan to support any other platform than the Hugging Face Hub for remote models.

That's fair.

Just saw the notice around the deprecation of remote model using HTTP as well.

Closing :)
Generate: skip left-padding tests on old models
Author: gante
Pull request: yes

# What does this PR do?

We have a few left-padding tests polluting our daily CI -- these are from old models, where it is not worth committing >1hr per model to add support for left padding.

This PR skips the test in those models, so we can focus our energy where it matters :)

Comments:
_The documentation is not available anymore as the PR was closed or merged._

It seems like `GPTBigCode` need to be fixed! Adding back the `@slow` to defer back fixing

@gante it all passes now 😄
remove unnecessary print in gpt neox sequence classifier
Author: cfhammill
Pull request: yes

# What does this PR do?

Removes an unnecessary print from gpt neox's sequence classifier output which muffles unnecessary output that was likely used for debugging at one point. Before, training gpt neox as a sequence classifier would `print` the logit/label sizes every training step which is hard to muffle and generally not useful.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation). **NA, original behaviour not documented**
- [ ] Did you write any new necessary tests? **NA, change too simple to require test**


## Who can review?

- text models: @ArthurZucker and @younesbelkada




Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix some `is_xxx_available`
Author: ydshieh
Pull request: yes

# What does this PR do?

FYI, after #23163,  `is_bs4_available()` and `is_faiss_available()` gives `False` even if they are actually available. This causes some CI errors, in particularly, `MarkupLM`.

This PR fixes the issue in a quick way. It's better to discuss if we want to enhance the function `_is_package_available` to be able to handle such edge cases.

(cc. @apbard FYI)

Comments:
@ydshieh thanks for fixing this and sorry for having introduced these bugs in first place.

_The documentation is not available anymore as the PR was closed or merged._
Where does hugging face's transformers save models?
Author: pure-rgb
Pull request: no

### Feature request

A clear instruction of receving the downloaded weight in system. 

### Motivation

Running the below code downloads a model - does anyone know what folder it downloads it to?

```
!pip install -q transformers
from transformers import pipeline
model = pipeline('fill-mask')
```

### Your contribution

Same issue with diffusers.

Comments:
This is described in the [installation page of the doc](https://huggingface.co/docs/transformers/installation#cache-setup).

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Make `RwkvModel` accept `attention_mask` but discard it internally
Author: ydshieh
Pull request: yes

# What does this PR do?

`RwkvModel` doesn't accept `attention_mask` but the tokenizer (it is `GPTNeoXTokenizer` from the checkpoints) prepares this input. When I try to run the pipeline tests for this model, I get 
```bash
TypeError: forward() got an unexpected keyword argument 'attention_mask'
```

I see it would be quite annoying for people using this model with the default tokenizer. So it might be good to accept it then discard it internally with a warning.
(If `RwkvModel` had its own tokenizer class `RwkvTokenizer`, we could do this inside tokenizer class. But here is not the case)


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Generate: add test to check KV format
Author: gante
Pull request: yes

# What does this PR do?

This PR adds a test to ensure our `.generate()`-compatible models have a standard KV cache format. Advanced generation methods (e.g. contrastive search or assisted generation) rely on cache manipulation, so it quickly becomes unmanageable if we don't stick to a conventional format (or a set of conventional formats). 

I expect that future non-standard KV formats will have to be well justified in PRs, since it will now imply skipping this test.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Generate: work around PT `multinomial` sampling 0 probability tokens
Author: gante
Pull request: yes

# What does this PR do?

Fixes #22979 

As raised in [this `transformers` issue](https://github.com/huggingface/transformers/issues/22979) and [this `pytorch` issue](https://github.com/pytorch/pytorch/issues/48841), `multinomial` can erroneously pick `0` probability tokens. According to the reports and my own observations, the error is much more likely on CPU.

There is a high chance that a token with `-inf` logits is selected: in this [simple example with `top_k=40`](https://github.com/huggingface/transformers/issues/22979#issuecomment-1529770291), it happens 0.158% of the times on CPU -- or ~50% chance that a sequence with 500 newly generated tokens to have at least one token that shouldn't be there. 

This PR adds a quick-and-dirty workaround, while the PT team works in the issue: at each sample step, pick 5 candidates, and keep the first valid one. Assuming independence, the probability of having one or more forbidden token in the example above drops to ~5e-10 %.

Runtime overhead: considering `distilgpt2`, a small model where operations outside the model have some weight, it got 2% slower on GPU (RTX3090) and 1% slower on CPU (Ryzen 9 5950X). On larger models, the slowdown becomes negligible.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

As discussed internally, this is a regression on the PyTorch side for 2.0, so this should be fixed by PyTorch and not by us adding some overload to `generate`.

(closing because of the comment above)
TF: embeddings out of bounds check factored into function
Author: gante
Pull request: yes

# What does this PR do?

Cosmetic PR: the check to confirm that `input_ids` is within bounds, added by me, is annoyingly verbose. Since it is not part of the model code, this PR factors it out into a separate function -- the model code becomes more readable 🤗 

The corresponding test passes after these changes (`RUN_SLOW=1 py.test tests/models/ -k test_embeddings_out_of_bounds_raise_exception -vv`)

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Generate: correctly detect default max length
Author: gante
Pull request: yes

# What does this PR do?

Fixes #20894.

Now that we are using the generation config, we can detect the use of a default `max_length` and a potential clash with `max_new_tokens` with values other than `max_length=20` :)

After this change, the example in the issue linked above works correctly.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Generate: delete unused TF `_reorder_cache`
Author: gante
Pull request: yes

# What does this PR do?

Starts off the year with my favorite task: deleting unused code 🥳 The deleted private function (`_reorder_cache`) is no longer used due to the removal on non-XLA-compatible generate functions (#20927) 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Generate: FLAX infers pad token in its absence and has functional example
Author: gante
Pull request: yes

# What does this PR do?

Some bug fixing in advance of #21007 (PR that adds generation config to Flax), to ensure we start from a functional flax generate codebase. 

In particular:
1. Flax now assumes the value `pad_token_id` when it is `None` and `eos_token_id` is not `None`, like TF and PT do. This is very helpful for open text generation examples, like with GPT2, was an open request (https://github.com/huggingface/transformers/issues/18884), and was one of the causes for failure in the existing example. This also includes the recent changes of #20727, where `eos_token_id` can be a list of tokens.
2. An `int32` type specification was missing in the special tokens -- when converted to JAX variables, JAX assumed they were `float32`;
3. The existing flax generate example is now part of our doctests, and runs.


Comments:
_The documentation is not available anymore as the PR was closed or merged._
CI: skip failing TF hubert test
Author: gante
Pull request: yes

# What does this PR do?



Comments:
cc @amyeroberts 

_The documentation is not available anymore as the PR was closed or merged._

@gante is this issue fixed? I am asking because I am still getting this error [here](https://app.circleci.com/pipelines/github/huggingface/transformers/57650/workflows/dcdbb1bd-f61c-445a-b276-4f649416931e/jobs/699951?invite=true#step-111-4137) in this [PR](https://github.com/huggingface/transformers/pull/21349) . I did rebase to `upstream/main` .

@susnato see #21606 (I skipped the wrong one here)

Hi @gante thanks for the fix but now there seems to be a new issue popping up regarding that same model, 
```
tests/models/hubert/test_modeling_tf_hubert.py::TFHubertRobustModelTest::test_keras_fit
```
This test is failing in the same PR I mentioned above. I did rebase before pushing.
Generate: input expansion for any model input
Author: gante
Pull request: yes

# What does this PR do?

Fixes #21599 

In line with #21603, this PR aims at generalizing `.generate()` for any-to-text models. In particular, it rewrites the function that expands the inputs when `num_beams>1` or `num_return_sequences>1` -- instead of expanding certain keywords within `model_kwargs`, expands any tensor therein. This assumes that all tensors in `model_kwargs` are per-row inputs, but that seems to be the case so far 👍 

The TF case had a more complex change, as we had two functions to expand the inputs (depending on whether we wanted a new dimension or not). This PR also standardizes that distinction.

Slow tests were run for:
- [x] GPT2 (both frameworks)
- [x] T5 (both frameworks)
- [x] BLIP2 (PT)

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Generate: Fix GIT batched captioning
Author: gante
Pull request: yes

# What does this PR do?

Fixes #21714 (batched image captioning with GIT not working)

The problem, at a higher level, boils down to a previously incomplete `batch_size` inference when no `input_ids` nor `inputs_embeds` was being passed to `.generate()` in decoder-only models -- we were always assuming `batch_size=1`, which is not correct in some multimodal models like GIT. 

If the `.generate()` doesn't receive `input_ids`, then some input tensor must live in `model_kwargs`. Now, we look for tensors in `model_kwargs` and use them as a source of information to determine the `batch_size`, which is then used to initialize `input_ids` with the correct shape.

👉 changes also made on the TF side
👉 a test was added to ensure we don't regress

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Generate - Fix broken documentation links
Author: gante
Pull request: yes

# What does this PR do?

Fixes #22077

`main_classes` is not on the same level as `generation_strategies`, hence the broken link.

EDIT: confirmed that it works in the CI docs.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

_The documentation is not available anymore as the PR was closed or merged._
Update tiny models and pipeline tests
Author: ydshieh
Pull request: yes

# What does this PR do?

Update tiny models and pipeline tests.

The two failing pipeline tests for `rwkv` is addressed in #23442 and #23444 (and need to merge them before this one)

Comments:
_The documentation is not available anymore as the PR was closed or merged._

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_23446). All of your documentation changes will be reflected on that endpoint.
add cleanlab to awesome-transformers tools list
Author: jwmueller
Pull request: yes

This PR adds the cleanlab package to the awesome-transformers list.  

cleanlab uses **transformers** & Hugging Face in all sorts of ways. Here's a couple of them:
- [tutorial for text data with **transformers**](https://docs.cleanlab.ai/stable/tutorials/datalab/text.html)
- [active learning with **transformers**](https://github.com/cleanlab/examples/blob/master/active_learning_transformers/active_learning.ipynb)
- [auditing data in the  **datasets** format](https://github.com/cleanlab/examples/blob/master/datalab_image_classification/datalab.ipynb), which is now the [primary data format](https://docs.cleanlab.ai/stable/cleanlab/datalab/datalab.html) supported by cleanlab
- [how to wrap a **transformers** model to be sklearn compatible](https://github.com/cleanlab/examples/blob/master/transformer_sklearn/transformer_sklearn.ipynb)

I'm not sure which reviewer is most appropriate, but perhaps one of the listed documentation reviewers: 
@sgugger, @stevhliu or @MKhalusova



Comments:
Also ccing @LysandreJik who compiled the list :)

_The documentation is not available anymore as the PR was closed or merged._
group_texts function produces batches with length shorter than block_size
Author: bodasadallah
Pull request: no

### System Info

- `transformers` version: 4.28.1
- Platform: Linux-3.10.0-1160.76.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.10.0
- Huggingface_hub version: 0.14.1
- Safetensors version: not installed
- PyTorch version (GPU?): 1.12.1+cu102 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

There's a  bug in the PyTorch language_modeling examples: 
for example: [run_clm.py](https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_clm.py)

The `group_texts` function is supposed to group the data into batches of equal sizes to `block_size`, and to ignore the small remainder chunk. There's a bug and instead, it inserts the last batch with a smaller size, which creates an error, as the model takes in data with equal sequence length. 




## Error: ValueError: expected sequence of length 128 at dim 1 (got 94)

### Expected behavior

The expected behavior is for the function to return data with equal sequence lengths for all batches.

Comments:
Train mobileBERT from scratch for other languages
Author: fabianbrandscheid
Pull request: no
Labels: name: New model

### Model description

Hi,
I am thinking of training a mobileBERT model from scratch for the German language. Can I use the [English mobileBERT model from HuggingFace](https://huggingface.co/google/mobilebert-uncased) to apply it to a dataset in another language? It makes sense that I would have to adapt the teacher model of mobileBERT to a BERT model of the corresponding language. Unfortunately, I could not find a parameter to adapt the teacher model.

Are there any other ideas on how best to train a mobileBERT model for another language?

Many greetings and many thanks!

### Open source status

- [ ] The model implementation is available
- [ ] The model weights are available

### Provide useful links for the implementation

_No response_

Comments:
Hi. I am interested in working on this.
TF port of the Segment Anything Model (SAM)
Author: Rocketknight1
Pull request: yes

This is a first draft of the SAM port - will update this PR as I port tests and make sure everything is working okay. It's also a first proof-of-concept for full GPT-4 auto-translation from PyTorch: The entire `modeling_tf_sam.py` file was converted from PyTorch by GPT-4 with the exception of the imports at the top, because I haven't written a prompt for those yet.

Update: I checked over all of the code and fixed the issues in the GPT port. Equivalence tests all look good! This is almost ready to merge, but there are a few small issues left:

- [x] Get saved model creation working and re-enable tests (problem with the serving signature)
- [x] Check for duplication in the processor files - I can probably refactor and simplify things a bit
- [x] Refactor convolutions - `channels_first` doesn't actually work on CPU in TF

Comments:
_The documentation is not available anymore as the PR was closed or merged._

This is now almost ready to go and the code should be ready for review! Remaining issues:

- I added a `tol` parameter to the TF-PT equivalence test - 1e-5 is too low for SAM (errors are more like 1e-4, but I used 5e-4 in the test to avoid flakiness). This will require a couple of minor tweaks in other models that are calling that test.
- Cleanup/refactor in the processor, there's probably some code duplication that I can remove.

Thanks for the review - about half of the comments relate to the processor code, which is definitely in need of a refactor, yes. Working on that now!

@amyeroberts @sgugger I refactored all the changes to the common tests, and just overrode `check_pt_tf_outputs` to change the `tol` in the tests instead - this is much cleaner and resolves most of the issues there. I also refactored the processor, removing the duplicated files and merging methods where appropriate. I think all comments have now been addressed!

@gante I think all comments are now addressed, and I added `training` wherever it touched a layer that had training-specific behaviour (which is literally one dropout call)

All comments from @amyeroberts and @sgugger should be addressed too - are you okay with going ahead and merging now once tests pass?

I think comments are addressed now - are we okay to merge?

I'm treating silence as agreement, merging!
[`RWKV`] Rwkv fix for 8bit inference
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes #23467

RWKV architecture scales down some linear layer weights with a certain factor at various stages for inference and training.
In the case of 8bit models, this leads to an error because the weight matrix is now an `int8` matrix. Therefore to apply the scaling one needs to scale the quantization statistics that are stored inside the `SCB` attribute.

cc @amyeroberts 

```python
from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig
import torch

model_id = "RWKV/rwkv-4-1b5-pile"

model = AutoModelForCausalLM.from_pretrained(model_id, load_in_8bit=True, device_map={"":0})
tokenizer = AutoTokenizer.from_pretrained(model_id)

generation_config = GenerationConfig(max_new_tokens=20, pad_token_id=tokenizer.eos_token_id)
question = "Hello my name is"
inputs = tokenizer(question, return_tensors="pt").to(0)
output_int8 = model.generate((inputs["input_ids"]), generation_config=generation_config)
print(tokenizer.decode(output_int8[0], skip_special_tokens=True))
>>> Hello my name is John and I am a student at the University of Texas at Austin. I am a member of the

model_fp16 = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16, device_map={"":1})
output_fp16 = model_fp16.generate((inputs["input_ids"]), generation_config=generation_config)
print(tokenizer.decode(output_fp16[0], skip_special_tokens=True))
>>> Hello my name is John and I am a student at the University of South Florida. I am a member of the Alpha
```


Comments:
^ for the snippet in the description, could you add what's printed out please? :) 

_The documentation is not available anymore as the PR was closed or merged._

Waiting For This PR to Merge...

Merge this PR ASAP!

@amyeroberts  Please Review this PR Fast!

Thanks! Just updated the comment! 

Thanks You So Much @younesbelkada  @amyeroberts  For Your Work...

Hope it Works without getting into another problem now :)
Use config to set name and description if not present
Author: sgugger
Pull request: yes

# What does this PR do?

This PR makes sure we read the name and description of a donwloaded tool in the config to set them in the tool class if they were not set by the user. We also perform a consistency check and override values by using the tool config when they are different, as the tool config should be the source of truth.

Fies #23469 

Comments:
_The documentation is not available anymore as the PR was closed or merged._