noamaanMulla-03/transformers-issues huggingface dataset:
resume train
Author: MikeDean2367
Pull request: no

### System Info

- `transformers` version: 4.28.0.dev0
- Platform: Linux-4.15.0-175-generic-x86_64-with-glibc2.31
- Python version: 3.10.10
- Huggingface_hub version: 0.13.3
- Safetensors version: not installed
- PyTorch version (GPU?): 1.12.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@sgugger

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Currently, I am using 8 GPUs to train GPT. I used the `Trainer` API provided by huggingface for training. Due to the large amount of data, it is expected to train only one epoch. When I was halfway through my training, I stopped training and increased the number of GPU to 24. When I resumed training, there was no change in the number of steps trained. However, due to the increase in the number of GPUs, the global batch size will also increase, so theoretically, the overall training step should change.

### Expected behavior

Is this a displayed bug? In other words, the steps required to complete an epoch during training will be less than the displayed number of steps.
If not, how should I skip data that has already been trained?

Comments:
Resuming training with a different setup than the one that begun it is at your own peril and is definitely not recommended or officially supported.

Ok, thank you for your reply. I will attempt to manually skip these already trained data.
Tests: disable `accelerate_tests` mark warnings
Author: gante
Pull request: yes

# What does this PR do?

Adds the `accelerate_tests` mark to `conftest.py`, so we don't get related warnings at test time. 

Here's a print screen before the fix:
<img width="1512" alt="Screenshot 2023-04-05 at 11 27 32" src="https://user-images.githubusercontent.com/12240844/230054803-cc95b93d-aab4-4133-baa8-7115760cd3ee.png">

And after the fix:
<img width="1512" alt="Screenshot 2023-04-05 at 11 27 53" src="https://user-images.githubusercontent.com/12240844/230054894-46a9435a-43be-4ef9-b111-92d664017d13.png">


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix PT-TF equivalence test for GPT1
Author: Rocketknight1
Pull request: yes

This PR fixes the hidden states output from `TFOpenAIGPTDoubleHeadsModel` to have the same shapes as the PT version, and re-enables the relevant test.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

I may have tagged you slightly too early, please be slower to get to your notifications

I will do my best to take 72 hours to get to your PR next time you ask for a review ;-)

:handshake:
Add thousands separator in training summary
Author: qmeeus
Pull request: yes

# What does this PR do?

The logger prints a summary at the beginning of training that displays some info such as number of examples, number of parameters, total number of steps, etc. Those numbers can be quite large and difficult to read. I added a thousand separator to improve readability for the following:
- num_examples
- num_train_epochs
- per_device_train_batch_size
- total_train_batch_size
- max_steps
- num_trainable_params

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sgugger 


Comments:
_The documentation is not available anymore as the PR was closed or merged._
nn.Transformer
Author: bilal2vec
Pull request: no
Labels: name: wontfix

## 🚀 

Use Pytorch's own attention and transformer modules.

## Motivation

Pytorch now offers modules like [nn.MultiheadAttention](https://pytorch.org/docs/stable/nn.html?highlight=attention#torch.nn.MultiheadAttention) and [nn.Transformer](https://pytorch.org/docs/stable/nn.html#transformer-layers). It would be nice to use the official Pytorch implementations in `transformers` now that they are available.

## Additional context

There is an offical Pytorch [tutorial](https://pytorch.org/tutorials/beginner/transformer_tutorial.html) that shows how nn.Transformer can be used and customized.

These modules are only available in Pytorch 1.1 (`nn.MultiHeadAttention`) and 1.2 (`nn.Transformer`). Using them would mean that anyone with Pytorch 1.0 would have to update their own version.


Comments:
Even though I am in favour of using as many built-ins as possible, I wonder whether it is not too early to do this. You will end up with a lot of pseudo-duplicate code: for those who are on 1.0 (no transformer-like support), 1.1 (only nn.*Attention), and 1.2 (full transformer). I don't know any statistics about people using `transformers` but I can imagine that many are still on PyTorch 1.0. 

We have a small codebase on the side where we use `nn.Transformer` to build both a BERT-style and a GPT2-style model that are compatible with our pretrained weights, but we still think it's a bit too early to refactor/freeze the lib's internals. A lot of research is still going to focus on the models' internals so we don't want to overfit to the current architecture.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@julien-c @BramVanroy just curious, do you guys still think it’s too early to use nn.Transformer?

A friendly ping to the maintainers to implement the in-built modules.

That's not something we can do because it will break all existing checkpoints.
fairseq-preprocess, Why are the number of rows in src and trg different?
Author: DamonCC
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->
After I used subword_nmt for bpe, when using preprocess, I found that the number of rows read src and trg are not equal, but I checked the data after bpe, the number of rows of src and trg are equal, although trg has dozens of Blank lines, the results here show that the gap between the number of lines in src and trg is much larger than the number of blank lines.
![image](https://user-images.githubusercontent.com/35246891/80357931-431ecb00-88ae-11ea-8cce-5fd4cb9b75f9.png)
![image](https://user-images.githubusercontent.com/35246891/80358019-60ec3000-88ae-11ea-9f0f-5ae6181292c3.png)

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**: 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


have you fixed this problem?
How to prune a transformer?
Author: jyotiyadav94
Pull request: no

### System Info

Hi, I am trying to reduce memory and speed up my own fine-tuned transformer. I came across the [tutorial ](https://huggingface.co/docs/optimum/intel/optimization_inc) for pruning on the huggingface site. I am referring to the following snippet. The trainer.train() is missing, so I added it. It ran without error, however, there is no reduction in memory (I used model.get_memory_footprint() and before and after pruning it was Model memory footprint: 503695916 bytes). Same for inference speed. I also tried out different pruning configurations (global pruning, different pruning types or target sparsities) but it did not help. Can someone help me?

```
from optimum.intel.neural_compressor import INCTrainer
from neural_compressor import WeightPruningConfig
from transformers import TrainingArguments, Trainer
from transformers.data.data_collator import default_data_collator

pruning_config = WeightPruningConfig(
    pruning_type="magnitude",
    start_step=0,
    end_step=15,
    target_sparsity=0.2,
    pruning_scope="local",
)

from transformers import TrainingArguments, Trainer
save_dir="prunedModel"

trainer = INCTrainer(
    model=model,
    pruning_config=pruning_config,
    args=TrainingArguments(save_dir, max_steps=500,num_train_epochs=1.0, do_train=True, do_eval=True,metric_for_best_model="f1",greater_is_better=True),
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    compute_metrics=compute_metrics,
    tokenizer=processor,
    data_collator=default_data_collator,
)
train_result = trainer.train() # <-- Added by me
trainer.save_model(save_dir) # <-- Added by me
optimized_model = AutoModelForSequenceClassification.from_pretrained(save_dir)

memory_footprint = optimized_model.get_memory_footprint()
print(f"Model memory footprint: {memory_footprint} bytes")
```


### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)


### Expected behavior

As per the model should be pruned and the actual model without pruned and the pruned model should have different sizes but they have the Model memory footprint:

Comments:
Please do not tag so many people, especially for an issue which is linked to the optimum repo (where you found this tutorial) and not the Transformers one.

okay Thank you so much for the suggestions I will remove from the task. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
`pipeline` does not load from local folder, instead, it always downloads models from the internet.
Author: z7ye
Pull request: no

### System Info

I create `pipeline` and called `save_pretrained(...)` to save to some local directory. However, when I load it back using `pipeline(model="local_folder")`, it either load from cache or try to start downloading from the internet.

However, if I do the following, it works. I am using the latest `transformers`. am I misused it or misunderstood something?
```
mypipeline.save_pretrained(save_directory=model_path)
mypipeline.model.config.use_pretrained_backbone = False
mypipeline.model.config.save_pretrained(save_directory=model_path)
```

### Who can help?

_No response_

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. run the code below
```
from transformers import pipeline

vision_classifier = pipeline(task="image-classification", model="google/vit-base-patch16-224")
vision_classifier.save_pretrained("./huggingface")
```

2. now delete the cache
3. load the model now
```
vision_classifier = pipeline("./huggingface")

```
and it will start download the pretrained model again.

### Expected behavior

I expect it loads the model from the local folder 

Comments:
cc @Narsil 

Seems to be working fine on my end, but there needs to be a few modifications (I'm super suprises it *can* download anything, your included code just crashes normally).

```python
from transformers import pipeline

pipe = pipeline(task="image-classification", model="google/vit-base-patch16-224")
pipe.save_pretrained("./local_vit")

pipe = pipeline(task="image-classification", model="./local_vit")
```

You need to specify the `task` for local, since that information is contained in the HUB, not in the config directly. So if you specify it, it works with everything locally.

On what version of transformers are you on ?

Hi, sorry I specified the task for local, and it still does not work. I will check the version. 

the version is 4.24.0

I cannot reproduce even on 4.24.0 Can you include the full script + error ? 

sure. I am still seeing the error. I will post the code later.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add DePlot + MatCha on `transformers`
Author: younesbelkada
Pull request: yes

# What does this PR do?

Adds [MatCha](https://arxiv.org/pdf/2303.18223.pdf) and [DePlot](https://arxiv.org/pdf/2212.10505.pdf) on `transformers`. 
Those are two different papers from Google AI but fully based on `Pix2Struct`. 

Model weights: 

- https://huggingface.co/ybelkada/deplot
- https://huggingface.co/ybelkada/matcha-base
- https://huggingface.co/ybelkada/matcha-chart2text-pew
- https://huggingface.co/ybelkada/matcha-chart2text-statista
- https://huggingface.co/ybelkada/matcha-plotqa-v1
- https://huggingface.co/ybelkada/matcha-plotqa-v2

I will move them to Google org once I will double check the model card contents with the authors

EDIT: all the weights have been moved



Comments:
_The documentation is not available anymore as the PR was closed or merged._

All models have been moved to Google org and model cards updated correctly! This PR is ready for review cc @sgugger
Minimum set of requirements
Author: markdjwilliams
Pull request: no

### Feature request

Separate base requirements from development dependencies, as the existing `requirements.txt` file conflates them.

For example, `accelerate` already does this via setuptools' "extras" [[link](https://github.com/huggingface/accelerate/blob/3cb9d5fd9c78c1da9fbc3127d6e63679a2475c6a/setup.py)]

### Motivation

I am a package maintainer who would like to install `transformers` from source. The current `requirements.txt` lists multiple dependencies, covering various use cases: testing, linting, formatting, MLOps, etc. Not all of these would be required for basic usage of the package, such as loading and inference of pre-trained models. Installation could be streamlined by allowing users to only install the dependencies necessary for their workflow.




### Your contribution

This would best be tackled by somebody more familiar with the extent of the functionality available in `transformers` and consequences of any changes here.

Comments:
Transformers also makes use of `extras` and the base requirements are limited to what's strictly necessary (see [here](https://github.com/huggingface/transformers/blob/176ceff91f5e5ff15922715e5a4a4d9f66b92d14/setup.py#L412)).

Thank you. I shouldn't have assumed that the presence of `requirements.txt` meant that `setup.py` was using it.

Which requirements.txt are you talking about? We only have some for the examples, but there isn't one at the root of the repo.
Backbone add mixin tests
Author: amyeroberts
Pull request: yes

# What does this PR do?

Adds a set of tests specifically for the Backbone class

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

Comments:
_The documentation is not available anymore as the PR was closed or merged._
allow separate relative attention bias
Author: agemagician
Pull request: yes

# What does this PR do?

It supports umt5 models, which need separate relative attention biases for each layer.
The current code will have backward compatibility with previous T5 and MT5 checkpoints.

Fixes # (issue)
https://github.com/huggingface/transformers/issues/22573

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Models:

- text models: @ArthurZucker and @younesbelkada 
- @stefan-it

Comments:
_The documentation is not available anymore as the PR was closed or merged._

> Thanks for your PR. Transformers is not a modular toolbox so we never add functionality to existing models like this. If you need this parameter for a new model, you should create a copy of T5 with the add-new-model-like command and just adapt the modeling code.
> 
> Or you can jsut host your slightly modified T5 model on the Hub with the [code on the Hub API](https://huggingface.co/docs/transformers/custom_models).

Thanks for your feedback. No problem, I will close this PR and add another one that have a separate model.
Make tiny model creation + pipeline testing more robust
Author: ydshieh
Pull request: yes

(After this, we/I can start to look the current skipped failing pipeline tests. I will also start to write the documentation on how some CIs are performed on Notion)

# What does this PR do?

- make pipeline testing **could also** work against **local** tiny models
- make tiny model creation script work with multi-processes
- A (renamed) workflow to:
  - (new steps) create **all** tiny models locally + run pipeline tests against local tiny models
  - (steps already on `main`) create + upload tiny models for new model **architecture** to Hub and generate new summary file

### Motivation
- make sure any modification to the tiny model creation script don't break things
- to ease the process of updating the summary file and test new tiny models on Hub)

Comments:
_The documentation is not available anymore as the PR was closed or merged._
update_pip_test_mapping
Author: ydshieh
Pull request: yes

# What does this PR do?

#22180 added a new script to add/update the attribute `pipeline_model_mapping` (for pipeline testing) in a systematic way.
This PR uses that script to update this attributes for new and existing model test files.

It turns out that `translation` was missing when I first time adding this attribute in #21516. Fortunately, I am so persistent to continuously improve things, and find+fix this problem as a consequence 🚀 🐛 .


Comments:
_The documentation is not available anymore as the PR was closed or merged._
[`bnb`] 8bit models should not be converted to `DDP`
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fix issues that users can encounter on multi-GPU such as: https://github.com/huggingface/peft/issues/269#issuecomment-1498776567 

In fact 8bit models should not be converted to DDP

cc @sgugger @pacman100 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
BLIP coco base default num_attention_heads should be 12
Author: ghost
Pull request: no

### System Info

I was trying to replicate BLIP results on VG Relation dataset as in https://github.com/mertyg/vision-language-models-are-bows. I was using `BlipForImageTextRetrieval.from_pretrained("Salesforce/blip-itm-base-coco")` 
following this example https://huggingface.co/docs/transformers/model_doc/blip#transformers.BlipForImageTextRetrieval 

The results were much poorer, and after a lot of digging it turned out that this is because the num_attention_head in the current HuggingFace implementation is 8 instead of 12 as in https://github.com/mertyg/vision-language-models-are-bows (and as in common in BLIP).

I manually changed it via 
```
model = BlipForImageTextRetrieval.from_pretrained("Salesforce/blip-itm-base-coco")
processor = AutoProcessor.from_pretrained("Salesforce/blip-itm-base-coco")

tmp_config=model.config 
tmp_config.text_config.num_attention_heads = 12
model = BlipForImageTextRetrieval.from_pretrained(
    "Salesforce/blip-itm-base-coco", config=tmp_config
)   
```
and matched the results on VGR.  For example, if you run the doc example (with a slightly changed caption):
```
from PIL import Image
import requests
from transformers import AutoProcessor, BlipForImageTextRetrieval

model = BlipForImageTextRetrieval.from_pretrained("Salesforce/blip-itm-base-coco")
processor = AutoProcessor.from_pretrained("Salesforce/blip-itm-base-coco")

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
text = "two cats on a couch"

inputs = processor(images=image, text=text, return_tensors="pt")
outputs = model(**inputs)
print(torch.softmax(outputs['itm_score'],dim=-1))
```
![image](https://user-images.githubusercontent.com/13796686/230425156-008d7ae2-d947-4f63-bfa1-8688f1e870c4.png)

**with 8 heads you get scores of `[[0.9489, 0.0511]]` for the above image i.e. the caption ` "two cats on a couch"` is not relevant, whereas with 12 heads you get scores of `[[0.1727, 0.8273]]` i.e. the caption is very relevant (which is the correct answer)**

Proposed fix:
**I suggest we switch it to 12 by default.**


Comments:
cc @amyeroberts and @younesbelkada 

This seems to be the right fix, I will check the original repo again and get back to you


The attention head is indeed seems to be 12 instead of 8: https://github.com/salesforce/BLIP/blob/3a29b7410476bf5f2ba0955827390eb6ea1f4f9d/configs/bert_config.json#L14 / https://github.com/salesforce/BLIP/blob/3a29b7410476bf5f2ba0955827390eb6ea1f4f9d/configs/med_config.json#L14 
We'll need to modify the `num_attention_heads` of all blip models as they all use the same base text model. 
Doctests and slow tests might need to be adapted accordingly. 

Thanks a lot @DianeBouchacourt for the great catch
Everything should have been updated correctly, now if you load again your model, you should see the correct results!