noamaanMulla-03/transformers-issues huggingface dataset:
add: task guide on video classification model fine-tuning.
Author: sayakpaul
Pull request: yes

This PR adds a task guide on fine-tuning video classification models. 

Comments:
@nateraw I leveraged the video classification pipeline with the custom fine-tuned model and a URL from the [UCF-101 subset](https://huggingface.co/datasets/sayakpaul/ucf101-subset) (`avi` format). It worked like a charm! 🔥

_The documentation is not available anymore as the PR was closed or merged._

@MKhalusova pushed a few changes. Will add another commit adding you as a co-author when you're back online. 

@amyeroberts thank you! Addressed all your comments. 

@sgugger one small pending part is https://github.com/huggingface/transformers/pull/20827#discussion_r1061285987. Once it's resolved, we should be good to merge. 

Updated the links. Will wait for the tests to pass and will merge afterward.
Fix (DeepSpeed) docker image build issue
Author: ydshieh
Pull request: yes

# What does this PR do?

Currently, the docker image build job [Latest PyTorch + DeepSpeed](https://github.com/huggingface/transformers/actions/runs/3834393836/jobs/6526754769) from time to time. The issue occurs after #20788 where `apex` is recompiled during the build. It seems a resource issue (most likely the memory issue) due to the parallel build (multiple worker). So set `MAX_JOB=1` to avoid the failure.

This will increase the build time to `1h30m`, but we have to build 2 same image (for daily CI and push CI), therefore 3h, and this is way too long. Previously those 2 images are built sequentially due to some issue, but now it seems the issue is gone and we can build them in parallel.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Improve Mlflow Callbacks documentation.
Author: y1450
Pull request: no

I recently followed https://julsimon.medium.com/using-mlflow-with-hugging-face-transformers-4f69093a6c04 , transformers documentation on mlflow callback is not formatted properly.

![image](https://user-images.githubusercontent.com/107429941/209551361-6b546fba-d1fc-4273-b84a-e15057873857.png)

It is better to read from the source code https://github.com/huggingface/transformers/blob/accad48e5b4a98302ea396b9f15c5f1c987b6f7f/src/transformers/integrations.py#L894 than the documentation site. 
![image](https://user-images.githubusercontent.com/107429941/209551549-e7b2729f-894f-4c82-841d-4ecfe7c8f70a.png)

I am sure that in the past, I might have seen such examples over other classes. I thought may be I  should report it.

Comments:
[`BLIP`] Fix daily CI failing test
Author: younesbelkada
Pull request: yes

# What does this PR do?

This PR fixes: https://github.com/huggingface/transformers/actions/runs/3754402958/jobs/6378634199

## Why this fix is relevant? 

The reference logits for this test were obtained under pytorch==1.13.1+cu116 and the daily CI uses pytorch==1.13.0+cu116. Setting the tolerance slightly higher (`4e-2`) fixes the test to make it cross-versions compatible. 

cc @LysandreJik @sgugger @ydshieh 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hmm at the beginning I thought that the `Softmax` was causing the issue, leading to large round errors but the test pass locally with `torch+cu116==1.13.0` but does not pass on the docker image that uses the same version. Will investigate more!

On GCP (my own/ CI runners), all torch versions give

(torch 1.13.x)
```python
[[0.97982633 0.02017363]]
[[0.50528485]]
```
or (torch 1.12.1)
```
[[0.97982633 0.02017365]]
[[0.5052849]]
```

so
```python
[[0.9798, 0.0202]]
[[0.5053]]
```
will work. Not sure why you got larger differ though, but it is likely an env issue.

Thanks so much @ydshieh 💯 , the tests seem to pass now on the CI docker image with your suggested values!
Seems that something was wrong with my env indeed
Make sure dynamic objects can be saved and reloaded
Author: sgugger
Pull request: yes

# What does this PR do?

Fixes #20884 

This PR makes sure that models that use the code on the Hub feature can be saved and repushed while still including the necessary code files. as reported in #20884, this was not the case previously. The fix is simple enough and the tests have been extended to test this use case.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[CLIPSeg] Fix integration test
Author: NielsRogge
Pull request: yes

# What does this PR do?

A user reported at https://github.com/timojl/clipseg/issues/18 that CLIPSeg uses the ImageNet mean + std instead of the [-1, 1] range for normalization. This PR updates the integration test as repos on the hub were fixed.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Yes, it uses `ViTImageProcessor` with different settings.
Remove cuda dependency from Deformable-DETR
Author: alaradirik
Pull request: yes

# What does this PR do?
Removes the CUDA dependency from Deformable DETR, [OneFormer](https://github.com/huggingface/transformers/pull/20577) and [Mask2Former](https://github.com/huggingface/transformers/pull/20792) PRs also use the same multi-scale deformable attention function and eliminate the CUDA dependency.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?



Comments:
_The documentation is not available anymore as the PR was closed or merged._
Standardize expected input shapes for audio models
Author: fxmarty
Pull request: no

### Feature request

Hi, it seems some audio models expect `(batch_size, feature_size, n_frames)` (e.g. whisper) while others expect `(batch_size, sequence_length)` (e.g. wavlm, wav2vec2). Could this be standardized? It seems like the naming follows `input_features` vs `input_values` (but wav2vec2 feature extractor returns input_values!)

It seems to be the difference between models taking raw audio as an input vs MEL-spectrogram / STFT.

### Motivation

I ask because I am wondering if for models handling stereo I should pass inputs as `(batch_size, 2, n_frames)`. And if so, why not be able to pass `(batch_size, 1, n_frames)` for mono? Is not raw audio a feature as well?

Related https://github.com/huggingface/transformers/issues/16564 

Preliminar internal discussion at https://huggingface.slack.com/archives/C02G13FEMDH/p1669637602417809

### Your contribution

/

Comments:
It's not possible to standardize this across all models, since some models naturally work on raw audio while others use various types of spectrograms.

However, I feel that we should make the FeatureExtractors more flexible in the type of inputs they accept. 

For example, right now, the Wav2Vec2FeatureExtractor expects the input to be mono and 16 kHz. It would be nicer if you could pass in a tensor of shape `(batch_size, num_channels, num_samples)` and an arbitrary sampling rate. 

If the model requires mono input, the FeatureExtractor can automatically downmix stereo to mono. If the sampling rate does not match what the model expects, the FeatureExtractor can automatically resample the data. It's more convenient to do this in the FeatureExtractor than having the user do this themselves.

This is analogous to what the ImageProcessors do in vision models: if the image is not the expected size, it will be resized (= resampled). If the number of color channels is wrong, the ImageProcessor fixes this. 

These operations are optional, so if the user already has a pipeline where they put the data in the correct format, they can choose to skip the resampling / downmixing stages.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Generate: FLAX uses `GenerationConfig` as the basis for `.generate()` parametrization
Author: gante
Pull request: yes

# What does this PR do?

Changes the FLAX side of `.generate()` such that it relies on the `GenerationConfig`. This is the FLAX equivalent of https://github.com/huggingface/transformers/pull/20388

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Added mask_time_prob and mask_time_length arguments to wav2vec2 pretraining script
Author: mpierrau
Pull request: yes

This PR relates to [PR 19997](https://github.com/huggingface/transformers/pull/19997), in which I messed up the PR by forgetting the --force flag when pushing. Hopefully this PR is correctly performed.

@sanchit-gandhi @sgugger @patrickvonplaten 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@sanchit-gandhi
How to fine-tune a pre-trained protein language model on the protein folding task ？
Author: pengshuang
Pull request: no

### Feature request


@Rocketknight1, Thanks for releasing the notebook and example of [How to fine-tune a pre-trained protein model](https://github.com/huggingface/notebooks/blob/main/examples/protein_language_modeling.ipynb) and [How to generate protein folds](https://github.com/huggingface/notebooks/blob/main/examples/protein_folding.ipynb). These codes help me quickly apply the SOTA protein structure prediction model. 

However, I wonder what is the best way to train the ESMFold from the scratch. To be specific, I want to finetune the protein language model ESM-2 on the large-scale protein sequence database (e.g. UniRef 90), And I can get a new model for  the downstream protein folding task. 

But I don't know whether it can use the Transformers [Trainer](https://huggingface.co/docs/transformers/training) to implement this function. Hope for your suggestions. 

Thanks in advance!



### Motivation

[HuggingFace Trainer](https://huggingface.co/docs/transformers/training) has provided very convenient functions (e.g training on many GPUs) to fine-tune a pre-trained model. 

Providing a notebook or example to fine-tune the protein language model ESM-2 for the protein folding task may be very helpful for the engineer that works on protein structure prediction. 



### Your contribution

I can work with you to contribute the notebook or example of fine-tune the protein language model ESM-2 for the protein folding task if necessary.

Comments:
Hi @pengshuang - right now our port of ESMFold is only really usable for inference, and is lacking some of the training code. This happened because we were rushing to launch simultaneously with the release by FAIR, and so we had to launch with a couple of bits missing!

We're working with the team at FAIR to add this, though, and I'll let you know when we have anything to report.

@Rocketknight1 Thanks for your quick reply. Looking forward to your future work.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@Rocketknight1  Following up on @pengshuang request.  Was hoping to finetune ESM 2 for Antibodies. How can we plug in the finetuned ESM-2  into ESMFold structure prediction
TypeError: TextInputSequence must be str
Author: suppathak
Pull request: no

### System Info

- `transformers` version: 4.23.1
- Platform: Linux-4.18.0-305.62.1.el8_4.x86_64-x86_64-with-glibc2.2.5
- Python version: 3.8.8
- Huggingface_hub version: 0.11.1
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: YEs
- Using distributed or parallel set-up in script?: 

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

Steps to Reproduce:

1. Trying the re-run this notebook: https://github.com/suppathak/aicoe-osc-demo/blob/teach-stu/notebooks/demo2/teacher_student_exp.ipynb (Reference: https://github.com/neuralmagic/sparseml/blob/main/integrations/huggingface-transformers/tutorials/sparsifying_bert_using_recipes.md)


### Expected behavior

It should run without any error. The notebook was running perfectly until last Friday(3 days ago). When I try to re-run it again. It is failing and giving me this error. Has there been any kind of update in the system? 

Any helpful feedback is appreciated. Thanks! 

![Screenshot from 2022-12-19 05-27-58](https://user-images.githubusercontent.com/30439457/208504300-62a1aa36-23d5-4a5a-bd0c-150cd97acd93.png)


Comments:
Please use the [forums](https://discuss.huggingface.co/) to help debug your code or provide us with a short reproducer we can run. The notebook relies on credentials we do not have.

Closing the Issue. Resolved the error by replacing the dataset. There was some manually added extra values in the old dataset.
santacoder: saved checkpoints after fine-tuning do not have required .py files
Author: arjunguha
Pull request: no

### System Info

- `transformers` version: 4.25.1
- Platform: Linux-3.10.0-1160.49.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.9.12
- Huggingface_hub version: 0.11.1
- PyTorch version (GPU?): 1.12.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

The santacoder model uses `trust_remote_code=True` to load Python files from the model repository. However, when I fine-tune a model and save a checkpoint, these Python files are not placed in the repository. Thus I get an error when trying to load the saved checkpoint. Here is the smallest program that shows the problem:

```python
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("bigcode/santacoder", revision="dedup-alt-comments", trust_remote_code=True)
model.save_pretrained("./silly-checkpoint")
model = AutoModelForCausalLM.from_pretrained(f"./silly-checkpoint", trust_remote_code=True, revision="dedup-alt-comments")
```

This produces the error `Could not locate the configuration_gpt2_mq.py inside ./silly-checkpoint.`

I can work around it by manually downloading the two Python files from the model repository:

https://huggingface.co/bigcode/santacoder/tree/dedup-alt-comments

But, this should probably not be necessary.

### Expected behavior

I think my script should work as-is, and should not require copy-pasta Python code.

Comments:
Thanks for reporting! I'll have a look into it after the holidays, the first week of January.

Thanks for your patience. Could you try the PR linked above?

I'm away this week. But, I'll check it out next week. Thanks!
Add GIT (GenerativeImage2Text)
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR implements GIT, short for GenerativeImage2Text. The model is a decoder-only Transformer conditioned on CLIP patch tokens + text tokens for tasks like image captioning and VQA.

To do:

- [x] add support for user-provided attention_mask
- [x] fix model_input_names of processor, see #20549 
- [x] fix issue where GitProcessor seems to instantiate a `VideoMAEImageProcessor` by default
- [x] transfer checkpoints to `microsoft`

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@sgugger PR is ready for a final review :) only remaining issue is that we need to update `from_pretrained` in `ProcessorMixin` to handle multiple image processors

Hi, is it possible to finetune git with transformers?
Update tokenizer_summary.mdx
Author: bofenghuang
Pull request: yes

# What does this PR do?

Hi, thanks for this document. But I think the headings here are a little bit misunderstanding. I changed them to:

```
- Introduction
- Subword tokenization
    - Byte-Pair Encoding (BPE)
        - Byte-level BPE
    - WordPiece
    - Unigram
    - SentencePiece
```

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

cc @LysandreJik 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_20135). All of your documentation changes will be reflected on that endpoint.
[NumPy] Remove references to deprecated NumPy type aliases
Author: hvaara
Pull request: yes

This change replaces references to a number of deprecated NumPy type aliases (np.bool, np.int, np.float, np.complex, np.object, np.str) with their recommended replacement (bool, int, float, complex, object, str).

NumPy 1.24 drops the deprecated aliases, so we must remove uses before updating NumPy.

See huggingface/diffusers#1810 for a similar issue in diffusers.

Co-authored-by: Peter Hawkins <phawkins@google.com>

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@sgugger I heard you might be the right person to review this. Please let me know if you have any questions 🤗
Disable ClearML automatic model uploading
Author: david1542
Pull request: no

### Feature request

Allow users of `transformers` to disable the automatic model uploading of ClearML. Perhaps we can also allow users to write their own integration callbacks in case they want to configure some more stuff.

The place where the saving happens is `src/transformers/integrations.py`:
```
    def on_save(self, args, state, control, **kwargs):
        if self._clearml_task and state.is_world_process_zero:
            ckpt_dir = f"checkpoint-{state.global_step}"
            artifact_path = os.path.join(args.output_dir, ckpt_dir)
            logger.info(f"Logging checkpoint artifacts in {ckpt_dir}. This may take time.")
            self._clearml_task.update_output_model(artifact_path, iteration=state.global_step, auto_delete_file=False)
```

We should add a condition to the main `if`, similar to what `NeptuneCallback`  or `MLflowCallback` is doing.

### Motivation

Several experiments that I ran were interrupted because I reached the max limit of model uploading. However, I was not interested in uploading my models in the first place. Hence, a configuration would be appropriate in this case.

### Your contribution

I can submit a PR if the contributors would help figure out the correct way of handling it :)

Comments:
Would you like to make a PR using the same kind of environment variable as WandB and CometML to control model logging through clearML?

(PS: There is no limit to the number of uploaded models on the Hugging Face Hub when you set `push_to_hub=True` ;-) )

Yes, I'll try to create a PR :)

I'm having the same issue!

@sgugger I created a [PR](https://github.com/huggingface/transformers/pull/20969). Can you please review? :)
Fix arguments passed to predict function in QA Seq2seq training script
Author: Observer46
Pull request: yes

# What does this PR do?

I used the script for training Seq2seq QA and realized that `--do_predict` contains a bug - kwarg `outputs` should be an instance of class `EvalLoopOutput`, but NumPy array is passed instead. Together with extracting predictions in the body of the method `post_processing_function`  (line 610 in run_seq2seq_qa.py) this results in an error every time you try to run tests with this script.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@karthikrangasai @sgugger



Comments:
_The documentation is not available anymore as the PR was closed or merged._