noamaanMulla-03/transformers-issues huggingface dataset:
clean up vision/text config dict arguments
Author: ydshieh
Pull request: yes

# What does this PR do?

Remove `vision_config_dict` and `text_config_dict`: just use `vision_config` and `text_config`.
- Make code base cleaner
- Avoid surprising behavior (see the comment)


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Without this PR, we have somehow surprising/confusing results

```python
from transformers import CLIPConfig, CLIPModel

config = CLIPConfig.from_pretrained("openai/clip-vit-base-patch16")
print(config.vision_config.patch_size)
print(config.vision_config_dict["patch_size"])

config.vision_config.patch_size = 32
config.save_pretrained("v2")

config_v2 = CLIPConfig.from_pretrained("v2")
# This is not `32` which is unexpected!
# In fact, it is `vision_config_dict` is being used during loading to set `vision_config`
print(config_v2.vision_config.patch_size)
# This is 32 - unexpected!
print(config_v2.vision_config_dict["patch_size"])

config.vision_config_dict["patch_size"] = 32
config.save_pretrained("v3")

config_v3 = CLIPConfig.from_pretrained("v3")
# This is 32 - unexpected!
print(config_v3.vision_config.patch_size)
# This is 32 - OK
print(config_v3.vision_config_dict["patch_size"])
```

@sgugger If you are happy with the current change, I will apply the changes to some other models, and the testing files.
So far it is good even if I don't change `to_dict`. It has already

```python
output["text_config"] = self.text_config.to_dict()
output["vision_config"] = self.vision_config.to_dict()
```

Awesome that you are working on fixing this!

Encountered the same issue with a new model I'm working on called CLIPSeg.

Also, could we update `GroupViT` as well? This is also a CLIP-like model.
chore: remove inference code, add pt framework.
Author: sayakpaul
Pull request: yes

To keep it consistent with the other docs ([image classification](https://huggingface.co/docs/transformers/tasks/image_classification), [audio classification](https://huggingface.co/docs/transformers/tasks/audio_classification), etc.), this PR:

* removes inference code 
* adds separation for PT as a framework 

in the [semantic segmentation](https://huggingface.co/docs/transformers/tasks/semantic_segmentation) guide. 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Sorry, I missed that PR. I thought this was outside the scope of task guides. Sounds good to not go ahead with this change. Let's still add the pt framework block

Agreed.
fix gradient checkpoint tests in encoder-decoder
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix the test `test_training_gradient_checkpointing` in `test_modeling_encoder_decoder.py`.

The current error is
```python
RuntimeError: Expected all tensors to be on the same device, but found at least two device
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix the test for corrupted checkpoints in from_pretrained
Author: sgugger
Pull request: yes

# What does this PR do?

As pointed out in #19974, there is a bug in `from_pretrained` when the model with head contains the same key as the base model, the checkpoint is then detected as corrupted. This PR fixes it and introduces a test to make sure there is no regression.

Fixes #19974
cc @NielsRogge 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
RuntimeError: Expected object of scalar type Long but got scalar type Float for argument #2 'target' in call to _thnn_nll_loss_forward
Author: 14H034160212
Pull request: no
Labels: name: bug

### System Info

Hi,

I got the following issue `RuntimeError: Expected object of scalar type Long but got scalar type Float for argument #2 'target' in call to _thnn_nll_loss_forward` when I was finetuning the stsb by using a finetuned roberta. I use [this code](https://github.com/huggingface/transformers/blob/main/examples/pytorch/text-classification/run_glue_no_trainer.py) to finetune the stsb task.

![image](https://user-images.githubusercontent.com/23516191/192100348-2fba2ff9-db30-44ea-af32-7d24b2d7f364.png)

Does anyone have idea to fix the bug? The label for stsb is float type, so the code is using regression to train the model. 

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Download the finetuned roberta model from [here](https://drive.google.com/drive/folders/1S03HHFKDDQA9mZfS-jMxtsmaJy6-M3A9?usp=sharing). The finetuned roberta was finetuned on a sentence binary classification task. (The issue may caused by the first stage finetuning which I was using a binary classification task, but the downstream task stsb is a regression task.)
2. Load the finetuned roberta model and use [this code](https://github.com/huggingface/transformers/blob/main/examples/pytorch/text-classification/run_glue_no_trainer.py) to finetune the stsb task and I got the above issue.

### Expected behavior

Please give me any advice to solve the issue. Many Thanks.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Connection error even though files are already downloaded.
Author: Drake53
Pull request: no
Labels: name: bug

### System Info

transformers version: 4.16.2

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Run demo.py (from https://github.com/bes-dev/stable_diffusion.openvino) without having an internet connection, but the models are already downloaded.

### Expected behavior

When TRANSFORMERS_OFFLINE is not used, I get the following error:
```
Stacktrace:
  File "stable_diffusion_engine.py", line 26, in __init__
    self.tokenizer = CLIPTokenizer.from_pretrained(tokenizer)
  File "transformers/tokenization_utils_basepy", line 1707, in from_pretrained
    resolved_vocab_files[file_id] = cached_path(
  File "transformers/file_utils.py", line 1846, in cached_path
    output_path = get_from_cache(
  File "transformers/file_utils.py", line 2102, in get_from_cache
    raise ValueError(
ValueError: Connection error, and we cannot find the requested files in the cached path. Please try again or make sure your Internet connection is on.
```

When TRANSFORMERS_OFFLINE is set to 'true', it works as expected.
I expect it to work even if I don't use this environment variable.

Not sure if related, but I have also set the TRANSFORMERS_CACHE environment variable.

Comments:
This should be fixed by https://github.com/huggingface/transformers/pull/19206, which we just released in a patch. Could you upgrade your transformers version and let me know if the issue is fixed? Thank you!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Update auto processor to check image processor created
Author: amyeroberts
Pull request: yes

# What does this PR do?

Fixes failing test which was checking if a feature extractor was loaded. This is now updated to reflect that an image processor is now loaded. 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Improve model tester
Author: ydshieh
Pull request: yes

# What does this PR do?

Some model testers have `__init__` like
```python
    def __init__(
        self,
        parent,
    )
```
and others accept many more arguments to customize them.

- This PR makes them accept arguments, so we have uniform style.
- This is also necessary to make `tiny model creation` to give **more correct** outputs (i.e. config/model/processor files), where `vocab_size` need to be sync between the tiny config (via model testers) and the converted (smaller) tokenizers.

I think for the review, you can just look the change in a single model test file :-)

#### TODO (in another PR 🙏 ): same change for some TF/Flax model testers

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@sgugger Just FYI

I updated this PR for `CANINE` https://github.com/huggingface/transformers/pull/19984/commits/bc83147623cb6441fed78d5ca9c46b114791706b and `ESMFold` https://github.com/huggingface/transformers/pull/19984/commits/0f8e4061f884fee859e63bf4b5d4bd0b8365906a

Don't really think you will reject these changes, but just in case!

Still LGTM!
When using GPT2，CPU  usage is high
Author: mazzzystar
Pull request: no

### System Info

transformers: 4.23.1

### Who can help?

Models:
- GPT-2 @patil-suraj, @patrickvonplaten, @LysandreJik

Library
- Pipelines @Narsil 

### Reproduction

Questions: When using GPT-2 as:
```python
from transformers import pipeline
gpt2_pipe = pipeline('text-generation', model='XXX', tokenizer='gpt2')
starting_text = "a young boy"
response = gpt2_pipe(starting_text, max_length=60, num_return_sequences=1)
```
The CPU  usage keep being > 90% for a few seconds, the generation is also slow.

However, when I manually change the `transformers` package in:
https://github.com/huggingface/transformers/blob/49b77b89ea1e89a9940f2b84da1bcc0696ecb07a/src/transformers/pipelines/text_generation.py#L229

to 
```python
self.model = self.model.to('cuda')
input_ids = input_ids.to('cuda')
attention_mask = attention_mask.to('cuda')
generated_sequence = self.model.generate(input_ids=input_ids, attention_mask=attention_mask, **generate_kwargs)
```
namely changing the variables to CUDA version, the CPU usage was much lower, and the speed is faster.

However, I could not find a good way to use cuda in `gpt2_pipline` such as `sd_pipeline` where you only need to add a `.to('cuda')`.




### Expected behavior

Can anyone give some advices? It's not an elegant way to fix the problem by changing the `transformers` package source code.

Comments:
Please use the [forums](https://discuss.huggingface.co/) for such questions as we keep the issues for bugs and feature requests only.

Try

```python
gpt2_pipe = pipeline('text-generation', model='XXX', tokenizer='gpt2', device=0) # or `cuda:0`
```
Include better versions of a model when they are available in the model doc pages
Author: sayakpaul
Pull request: no

@NielsRogge provided a great suggestion.

We could add a banner on top of Swin's docs (and other models where this is applicable), to indicate we have a better model now, SwinV2. The same could be done for DETR, as Conditional DETR and Deformable DETR greatly improve the convergence and AP metrics.

We can start with the following:

- [ ] Swin
- [ ] DETR

And then expand to the other models. Or else, we can start with five such models (feel free to suggest more). When reviewing model PRs to `transformers`, we would just need to be mindful about this a bit so that we can suggest it accordingly to the contributors. 

Cc: @osanseviero @nateraw 

Comments:
I am very wary about this. If you start telling on the page of a model that have been developed by Org 1 that there is a better model developed by Org 2, they will get mad and we will then have unnecessary conflicts to handle.

For the same reason we stay away from benchmarks between frameworks/hardware, I would stay away from this.

Point noted. 

But what if the same org comes up with a better version? But I understand this creates a weird distinction which is not desirable. 

Leaving it open for today in case anyone has any inputs. 

Yeah it was just a suggestion, I wouldn't have opened an issue for this actually. 

I agree that this could become very opinionated (it's very subjective which model is better). We could just do it for papers that come from the same team (Swin => Swinv2), to promote upcoming work. But for models that originate from different teams, this might be hard. That's where "evaluate on the hub" will come into play, where people can see which models perform best on a given task. 



Closing it for now. Should there be a need, it can easily be reopened. 

>We could just do it for papers that come from the same team (Swin => Swinv2), to promote upcoming work. 

Sure, why not :) for those scenarios I think it's ok to open small PRs updating the corresponding docstrings
Fix doctest
Author: ydshieh
Pull request: yes

# What does this PR do?

We just need `# doctest: +IGNORE_RESULT` after `>>> dataset = load_dataset` as usual



Comments:
Thanks for fixing!

Thanks @ydshieh! Sorry for breaking our streak of 6 days of no failures! 

Oh, I haven't request yet and you already approved! Thanks a lot, so impressive your speed of response!

![image](https://user-images.githubusercontent.com/35901082/199562025-6426b986-744e-4858-b7e4-3f5b3c341bd2.png)


_The documentation is not available anymore as the PR was closed or merged._

I am so bad in doctest ... What we need here is actually `# doctest: +IGNORE_RESULT`. Sorry.
Show installed libraries and their versions in CI jobs
Author: ydshieh
Pull request: yes

# What does this PR do?

Whenever there is a need to check
  - if the versions of install libraries change
  - and/or find out which ones change

it is not super easy to get this information.

This PR adds `pip freeze | tee installed.txt` to
  - show the results
  - save to a file
  - and upload as an artifact.

It makes the access to this information easier, and potentially make the process to **get the difference** between previous/current runs easier too.

Example run job (and the artifact): [here](https://app.circleci.com/pipelines/github/huggingface/transformers/50778/workflows/cf542f91-cc42-4942-bac8-100436555dda/jobs/606945)

I plan to do the same for GH actions jobs, but maybe in another PR :-)

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Transformers scheduler doesn't alter LR of added param group after model unfreeze
Author: rbracco
Pull request: no

### System Info

python 3.8.10 on ubuntu 20.04
pytorch 1.12.1
transformers 4.20.1

### Who can help?

@sgugger 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. Add a transformers scheduler, such as `transformers.get_linear_schedule_with_warmup` to any training run that begins with a frozen model and unfreezes. 
2. Monitor the learning rate, LR for param groups that are unfrozen after the 0th epoch are constant and unaffected by the scheduler

Sorry I don't have a better reproduction section. I mainly use transformers as a dependency of another library, and I tried making a reproducible script [here on Colab](https://colab.research.google.com/drive/1LXkIUP_vcVV3Vmc0sLInpIUEWU2H8K6P?usp=sharing), but Colab is crashing due to one of the imports. 

### Expected behavior

I believe this is a bug but it could be expected behavior. I would expect the unfrozen param groups (param-group 2) should also be controlled by the scheduler, as they are when using pytorch schedulers such as `torch.optim.lr_scheduler.LinearLR`. 

### LR of param groups after unfreeze when using `torch.optim.lr_scheduler.LinearLR`.
![image](https://user-images.githubusercontent.com/47190785/199602432-e47433b3-9a09-4d89-b379-dc6f8a0e9877.png)

### LR of param groups after unfreeze when using `transformers.get_linear_LR_with_warmup`.
![image](https://user-images.githubusercontent.com/47190785/199602498-0bde2ea8-ca12-439c-8add-35d92d35e7e5.png)

### LR of param groups after unfreeze when using `transformers.get_cosine_schedule_with_warmup`
![image](https://user-images.githubusercontent.com/47190785/199602591-1ca3e075-1600-46df-9bf1-ae3b99daca4b.png)



Comments:
The Trainer by itself does not support several parameter groups, you will need to subclass and overwrite the methods that create schedulers/optimizers.

Thank you! Closing for now, but I'll reopen if there is any further issue.
Add Flan-T5 Checkpoints
Author: chujiezheng
Pull request: no
Labels: name: New model

name: Good First Issue

### Model description

Flan-T5 models are instruction-finetuned from the T5 v1.1 LM-adapted checkpoints. They can be directly used for few-shot prompting as well as standard fine-tuning. Here is the [paper](https://arxiv.org/abs/2210.11416).

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

* Paper: https://arxiv.org/abs/2210.11416
* Model implementation: same as T5 v1.1
* Model weights: https://github.com/google-research/t5x/blob/main/docs/models.md#flan-t5-checkpoints

Comments:
Cool! cc @patrickvonplaten 

FYI the conversion script can be found here: https://github.com/huggingface/transformers/blob/main/src/transformers/models/t5/convert_t5x_checkpoint_to_flax.py

cc @younesbelkada @ArthurZucker one of you interested in taking it? :-) 

Yes! On it 🤗 

They are available now: https://huggingface.co/models?search=flan-t5

@NielsRogge: Quick question: when I try to load any of the FLAN models:

`model = AutoModelForSeq2SeqLM.from_pretrained("google/flan-t5-small")`

I receive the following error: `AttributeError: 'T5LayerFF' object has no attribute 'config'`

I have transformers==4.18.0.

Hey @BalazsFeherUK feel free to open a new issue with the full trace of the error and ping me 🤗
`Pathlike` objects are treated as `AutoModel` objects in `pipeline` initialization
Author: harsh8398
Pull request: no

### System Info

- `transformers` version: 4.22.2
- Platform: macOS-13.0-arm64-arm-64bit
- Python version: 3.9.13
- Huggingface_hub version: 0.10.0
- PyTorch version (GPU?): 1.12.1 (False)
- Tensorflow version (GPU?): 2.10.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

@Narsil 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Steps to reproduce the behaviour:
1. Run the following code snippet
```python
from transformers import pipeline
from pathlib import Path

# store pipeline locally
gen = pipeline("image-classification", "google/vit-base-patch16-224")
gen.save_pretrained(Path("./models") / "google/vit-base-patch16-224")

# load pipeline locally
new_gen = pipeline(
    "image-classification", Path("./models") / "google/vit-base-patch16-224"
)
```

Loading pipeline fails with following error:
<img width="1010" alt="image" src="https://user-images.githubusercontent.com/20420308/199565245-190f8291-d691-48f4-bd43-3a4500bc225d.png">

This works file if I pass string path:
```
new_gen = pipeline(
    "image-classification", str(Path("./models") / "google/vit-base-patch16-224")
)
```

### Expected behavior

The pipeline model argument should check for Pathlike objects and not treat them the same as AutoModel instances.

Comments:
Excellent suggestion! 

I opened a PR https://github.com/huggingface/transformers/pull/20030
Add **kwargs to preprocess method 
Author: amyeroberts
Pull request: yes

# What does this PR do?

Fixes failing doctests with (and real life usage of) processors which contain two processing objects: one image processor + one tokenizer/feature extractor. 

When the processor is called, all kwargs are passed to both processing objects e.g. in CLIPProcessor: https://github.com/huggingface/transformers/blob/main/src/transformers/models/clip/processing_clip.py#L81-L85

Image processors therefore have to be able to accept arguments they will not use when called. 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Agreed - I'll add splitting up the kwargs on the TODO list!
Fix some doctests after PR 15775
Author: ydshieh
Pull request: yes

# What does this PR do?

After PR #15775, we need to either update some expected values or specify `skip_special_tokens=True`.

I am not very comfortable to use `skip_special_tokens=True` for `PT_QUESTION_ANSWERING_SAMPLE` in `doc.py`, as it might fail other tests. We will have to run the doctest manually to see if everything is fine.

(A lazy way is not to use this argument, but just to update the expected values)

#### update
I launched doctest [here](https://github.com/huggingface/transformers/actions/runs/3384706275). The tests with `ForQuestionAnswering` all pass, so we are good!


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Sorry, I always missed the last piece. Added the final commit to fix `docs/source/en/model_doc/speech_to_text.mdx`.


https://github.com/huggingface/transformers/pull/20036/commits/e6c4bc5f45fee4a8d3997bc4c04896fac1c25284