noamaanMulla-03/transformers-issues huggingface dataset:
LayoutLM model only able to classify individual words instead of entire sections
Author: ghost
Pull request: no
Labels: name: New model

### Model description

Model I am using (LayoutLM ...):

Here, I would like to develop a custom resume parser model that can accurately predict the sections for *EDUCATION*, *SKILLS*, and *EXPERIENCE* based on the resume. I have fine-tuned the *LayoutLMv3* model on a custom dataset that is similar to the *FUNSD* dataset.

Although the LayoutLM model can predict education keywords, it only does so at the word level. For instance, if the resume states "My education is in computer engineering from LD College Ahmedabad," the model will label "computer" and "engineering" as *EDUCATION*. However, I aim to have all classified words in a single section rather than in individual word sections.

Therefore, here are some random screenshots of the LayoutLM model output.
![Screenshot from 2023-03-13 18-36-47](https://user-images.githubusercontent.com/105478351/225013023-a1ea58b0-4e26-49f2-9cd5-cdebc8365d55.png)


And here, I would like the output to include box coordinates for the EDUCATION section as well as the SKILLS section, identified by their respective keywords.
![Screenshot from 2023-03-13 18-32-05](https://user-images.githubusercontent.com/105478351/225013056-31073b3f-605d-4728-8058-0300bb8fd977.png)



Note: I have attempted to use the *Layout Parser* model with the *PublayNet* dataset. However, this model was unable to accurately predict and classify the sections for *EDUCATION*, *SKILLS,* *EXPERIENCE*, etc.

If there are any other models that would be suitable for my use case, please kindly suggest them.
*Thank you all for your help.*


### Open source status

- [ ] The model implementation is available
- [ ] The model weights are available

### Provide useful links for the implementation

_No response_

Comments:
Hi @keval2415  - thanks for opening an issue. 

Can you please use the [forum](https://discuss.huggingface.co/) for questions like this. We try to keep the github issues reserved for bugs or feature requests.
TypeError: zero_grad() got an unexpected keyword argument 'set_to_none'
Author: savitamittal1
Pull request: no

### System Info

Getting below error while runing Bert pretraining using Huggingface trainer with deepspeed 0.8.2 version and pytorch 2.0RC

Traceback (most recent call last):
  File "pretrain_glue.py", line 124, in <module>
    result = trainer.train()
  File "/opt/conda/envs/env/lib/python3.8/site-packages/transformers/trainer.py", line 1631, in train
    return inner_training_loop(
  File "/opt/conda/envs/env/lib/python3.8/site-packages/transformers/trainer.py", line 1814, in _inner_training_loop
    model.zero_grad(set_to_none=True)
TypeError: zero_grad() got an unexpected keyword argument 'set_to_none'

@sgugger , could this be due to below change?
Enforce same behavior as PyTorch 2.0 for older versions (https://github.com/huggingface/transformers/pull/22136)

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Run Bert pretrain with HF trainer and deepspeed 0.8.2 version on Pytorch 2.0 RC build

### Expected behavior

run with no errors

Comments:
Yes, this has been fixed already on main. You just need to pull from source :-)

ah, thankyou for quick fix. :)
CUDA OOM when loading optimizer state dict
Author: willxie
Pull request: no

### System Info

I am finetuning GPT NEO 1.3B with 1 GPU and 24GB VRAM. 

From scratch, model weights load and train fine with <1GB of memory to spare.

When loading from checkpoint, I run into issues loading optimizer states.
https://github.com/huggingface/transformers/blob/04bfac83b793b757e7b33188f88eebe21ac65ef7/src/transformers/trainer.py#L2434-L2436

If I hard-code `map_location ='cpu'`, the OOM goes away and I am able to resume training. After some reading, this was caused by the optimizer state dict being associated with 'gpu:0' at the time of checkpoint saving. I also found another ticket adding device logic to `map_location=self.args.device` in the first place due to CPU OOM for the PR author. I was wondering if the interface needs to be change in order to accommodate both use cases.

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

see above. 

Otherwise from using run_clm.py from the repo without deepspeed. 

### Expected behavior

Resume from checkpoint should fit in the same vram footprint as finetune from scratch. 

Comments:
cc @sgugger 

I tried to find the reference to the issue mentioning a CPU OOM but couldn't find it. Do you have a link handY?
We could always load the weights on the CPU then move them which would be a tiny bit slower, but this just happens once. With models getting bigger and bigger it's probably makes sens to have this as a default behavior though.

Sorry I lost track of the issue with the CPU OOM but it was related to GPU memory in aggregate (deepspeed iirc) is greater than CPU. 

https://github.com/huggingface/transformers/issues/3730#issuecomment-629563466 is also relevant. My understanding is that `self.args.device` is where the model lives which is usually `gpu` or `tpu`. This puts optimizer state state_dict loading also to device with limited memory. Is there any reason to not always put this into CPU? My understanding is that if the model is trained using GPU they will be implicitly copied over during optimization but avoids the setup OOM.
Load optimizer state on CPU to avoid CUDA OOM
Author: sgugger
Pull request: yes

# What does this PR do?

As reported in #22123 resuming from a checkpoint can get a user out of memory if the optimizer state is loaded directly on GPU. This PR loads it on CPU by default and it will be copied over to the proper device by PyTorch in `load_state_dict`. This might be a bit slower at the checkpoint loading time (so just once) but will benefit users training large models.

Fixes #22123 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Run all tests by default
Author: sgugger
Pull request: yes

# What does this PR do?

This PR makes the default value for the crosstests PT<>TF and PT<>FLAX true by default. This way when a user runs tests locally, all tests are run (the only exception being the hub staging tests, which require setting an env variable anyway to use moon-staging instead of moon-landing). In the CI however each job runs the same tests as before since the env variables are set at False by default.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Specify metric aggregation strategy when evaluating on multiple validation datasets using `Trainer` class
Author: larrylawl
Pull request: no

### Feature request

Specify metric aggregation strategy when evaluating on multiple validation datasets using `Trainer` class. This metric aggregation strategy will output a `metric: Dict[str, float]` by aggregating the metrics computed from the multiple validation sets.



### Motivation

When evaluating on multiple validation datasets using `Trainer`, the metrics used to compute the best checkpoint is the last metric.  However, the user will likely want an aggregation of metrics from the multiple validation datasets. 

https://github.com/huggingface/transformers/blob/ff8870350151091d3d8b2af4c1c0fa3ebcc1052a/src/transformers/trainer.py#L2224-L2235



### Your contribution

I would be happy to make a PR. My idea is as follows:
1. Collate all metrics from evaluation dataset
2. Aggregate the metrics based on a user-specified strategy (e.g. average a user-specified common metric between all evaluation dataset; here, I can leverage the TrainingArgument `metric_for_best_model `). This step should return a `Dict[str, float]` so as to be compatible with the `metric` type. 

Comments:
This is a bit too niche to be added in the Trainer, it might be best to write your own subclass for this.
Create MaskedImageCompletionOutput and fix ViT docs
Author: alaradirik
Pull request: yes

# What does this PR do?
Fixes the output class and docs of `ViTForMaskedImageModeling `which returns images of shape (batch_size, num_channels, height, width) whereas MaskedLMOutput logits are of shape (batch_size, seq_length, vocab_size).

**Notes:**
- We have no checkpoints for `ViTForMaskedImageModeling` and the docstrings use the pretrained ViTModel combined with random head parameters. I will open a separate PR for the other affected model - `DeiTForMaskedImageModeling`, for which no checkpoints are available either. 
- Swin has its own MaskedImageOutput class but only has base model checkpoints (trained on the masked image modeling task) and no task-specific checkpoints.
- I'm planning to create a masked-image-completion pipeline and add Swin and ICT (once Sheon's PR is merged).

CC: @sheonhan

## Before submitting
- [X ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [X ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [X ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?



Comments:
_The documentation is not available anymore as the PR was closed or merged._

Gently pinging @amyeroberts for the final approval

masked-image-completion pipeline sounds awesome 🙌
Fix ViTForMaskedImageModeling example in documentation
Author: fxmarty
Pull request: yes

Following https://github.com/huggingface/transformers/pull/22152, `logits` is not the right output key name.

cc @alaradirik @amyeroberts 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@fxmarty I am just going to open a PR, but you are too fast! Thanks.

@ydshieh It could be I missed it elsewhere, so feel free to push here / do an other PR!

You have to check your CircleCI setting however. The CI has issue to run in this PR.

OK, nothing missed, but as the CI has some problem, I am going to open PR.

I will make sure you are a contributor in #22186 @fxmarty . Close this one as mentioned above.

Ah yes I did not configure SSO with CircleCI, maybe that's the issue.
Revert 22152 MaskedImageCompletionOutput changes
Author: amyeroberts
Pull request: yes

# What does this PR do?

Reverts the breaking changes introduced by #22152. Temporary fix until it's decided how to change the model output. 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


cc @alaradirik 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Will do a patch asap
Regression pipeline device
Author: sgugger
Pull request: yes

# What does this PR do?

Fixes the regression introduced by #21479. Basically doing
```py
from transformers import pipeline

classifier = pipeline("text-classification", device=-1)
```
now fails on v4.27.0 whereas it used to work on 4.26.1

This PR fixes this and adds a test to avoid future regression.

Fixes #22189

Comments:
Tests pipelines pass for both PyTorch and TF so merging!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_22190). All of your documentation changes will be reflected on that endpoint.
t5 remove data dependency
Author: prathikr
Pull request: yes

# What does this PR do?

Prevents clamping logic from being skipped by torch.onnx tracer by moving data-dependent inf check into fp16-specific training code. Unclamped inf results in NaN being returned during loss calculation and eventually will crash with the following error:

```bash
Traceback (most recent call last):
  File "/home/prathikrao/optimum/examples/onnxruntime/training/translation/run_translation.py", line 680, in <module>
    main()
  File "/home/prathikrao/optimum/examples/onnxruntime/training/translation/run_translation.py", line 588, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/opt/conda/envs/ptca/lib/python3.8/site-packages/optimum/onnxruntime/trainer.py", line 373, in train
    return inner_training_loop(
  File "/opt/conda/envs/ptca/lib/python3.8/site-packages/optimum/onnxruntime/trainer.py", line 658, in _inner_training_loop
    self.deepspeed.step()
  File "/opt/conda/envs/ptca/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 2169, in step
    self._take_model_step(lr_kwargs)
  File "/opt/conda/envs/ptca/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 2071, in _take_model_step
    self.optimizer.step()
  File "/opt/conda/envs/ptca/lib/python3.8/site-packages/deepspeed/runtime/zero/stage_1_and_2.py", line 1759, in step
    self._update_scale(self.overflow)
  File "/opt/conda/envs/ptca/lib/python3.8/site-packages/deepspeed/runtime/zero/stage_1_and_2.py", line 2016, in _update_scale
    self.loss_scaler.update_scale(has_overflow)
  File "/opt/conda/envs/ptca/lib/python3.8/site-packages/deepspeed/runtime/fp16/loss_scaler.py", line 156, in update_scale
    raise Exception(
Exception: Current loss scale already at minimum - cannot decrease scale anymore. Exiting run.
```

## Who can review?

Models:
- text models: @ArthurZucker and @younesbelkada

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Cool! Thanks for this contribution! Pretty sure that this can also be applied to `SwitchTransformers` (as it implements the similar procedure) and ~`MT5`~ LongT5

Let's maybe address this in a follow up PR no? Btw this PR includes the changes for `mt5` (EDIT: you meant `LongT5`)

@prathikr Would you mind changing the two other models in this PR or would you prefer we followup in a separate PR?

@sgugger I think a separate PR would be best. Thank you
Fix DeepSpeed CI
Author: ydshieh
Pull request: yes

# What does this PR do?

Since 2 days, the daily CI runs with torch 2.0.0. In DeepSpeed CI job, there is an issue regarding `torch-tensorrt` (currently `v1.3.0`).
The installation was already disabled in #22135, but there was a version shipped with the base image, and I forgot to uninstall it in our docker image during building.

Remark: The failure is our `undefined symbo` friend
```python
E   ImportError: /opt/conda/lib/python3.8/site-packages/torch_tensorrt/lib/libtorchtrt.so: undefined symbol: _ZN2at11show_configB5cxx11Ev
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._

ouch, i checked it took tensor-rt 1 month to release a version supporting pt-1.13 after the latter was released. 

so I won't expect any coming updates quickly.

do we need tensor-rt for anything?

Thank you for fixing this, @ydshieh! Glad to have you on top of things as always!

Hi @stas00 It's probably my bad.

In #20758, it (the one shipped with the base image) was uninstalled (same reason as this PR) 
In #20758 (next day), I found a way to install it - and just updated the docker file without thinking if we need it.

A quick search gives me
```python
def is_torch_tensorrt_fx_available():
    if importlib.util.find_spec("torch_tensorrt") is None:
        return False
    return importlib.util.find_spec("torch_tensorrt.fx") is not None
```
But I think it's irrelevant to DeepSpeed CI job.

**I can actually remove these 2 lines**
```bash
# This installation instruction will uninstall torch 2.0.0
# TODO: uncomment and update the following line once `torch-tensorrt` is ready for `torch 2.0.0`
```
~~**if you are also OK with this.**~~

Well, let's remove the installation line, as it was never there originally - it was just shipped with the base image 

(BTW, thank you for checking the release time of  `torch-tensorrt` ❤️ )
Update expected values in `MgpstrModelIntegrationTest`
Author: ydshieh
Pull request: yes

# What does this PR do?

[CI](https://github.com/huggingface/transformers/actions/runs/4422120632/jobs/7753698495) failed: the expected values provided by the contributor didn't match the one given in CI runner, and we just need to update it.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Update tiny model creation script
Author: ydshieh
Pull request: yes

# What does this PR do?

- Update `UNCONVERTIBLE_MODEL_ARCHITECTURES` with a few recent models: they don't have processor class (or not included in the `XXX_MAPPING_NAMES`)
- Make the detection of model test class more robust.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_22202). All of your documentation changes will be reflected on that endpoint.
Add TF GPTNeoX 
Author: JIPHF
Pull request: no

### Feature request

Add the GPTNeoX model in TensorFlow.

### Motivation

Having GPTNeoX in TensorFlow would benefit the community.

### Your contribution

@gante is it possible to assign this to me?  

Comments:
(@Rocketknight1 FYI)

@JIPHF If you're happy to make a PR for this, then please do! Let us know if you need any help with that.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Datasets performance :( 
Author: webshared
Pull request: no

I am using https://huggingface.co/docs/transformers/model_doc/time_series_transformer as an example to setup time series transformers for my use case. 

I have modified the training script to just traverse all batches and dump a summary - essentially max and min value of first & last time feature elements for each of 366 static categories. 

40 epochs, 100 batches per epoch, 256 train sequences per batch
= 1_024_000 training sequences in total

**Elapsed time: 452.257402 seconds**

Iterating over 1M elements with really simple logic took over 7 minutes on M2 macbook :( 

I am new to python - is it kinda expected?


Comments:
Hi there. Please use the [forums](https://discuss.huggingface.co/) for questions like this as we keep issues for bugs and feature requests only. Also make sure to include the code you are running or no one will be able to help.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[`XGLM`] Add `accelerate` support for XGLM
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes: https://github.com/huggingface/transformers/issues/22188
With this PR users will be able to load XGLM models in 8bit

cc @amyeroberts 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
LLAMA model won't release VRAM when deleted
Author: devilismyfriend
Pull request: no

### System Info

latest git, windows (tested on WSL as well), pytorch 1.13

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Run this code

tokenizer = LlamaTokenizer.from_pretrained("decapoda-research/llama-7b-hf")

model = LlamaForCausalLM.from_pretrained(
    "decapoda-research/llama-7b-hf",
    load_in_8bit=True,
    torch_dtype=torch.float16,
    device_map={'':0},
)
import time
#try to unload the model from GPU memory
del model
torch.cuda.empty_cache()
time.sleep(5)


### Expected behavior

After del the memory should be freed from VRAM.

Comments:
You also need to call the garbage collector:
```python
import gc

gc.collect()
```

It worked thanks!, I did try it before and it didn't but checked it again now and it did lol
🔥py38 + torch 2 🔥🔥🔥🚀
Author: ydshieh
Pull request: yes

# What does this PR do?

Title is all we need.

There is one line in `setup.py` I don't know if I need to change and if yes how.

```python3
deps["importlib_metadata"] + ";python_version<'3.8'",  # importlib_metadata for Python versions that don't have it
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._

The whole suite of tests is run and passed in this [run](https://app.circleci.com/pipelines/github/huggingface/transformers/60009/workflows/73d754f8-017d-458f-8dc2-c6166d30e1de)