noamaanMulla-03/transformers-issues huggingface dataset:
How to use LayoutLMv3 for Document Layout Detection  task?
Author: matthew-wei
Pull request: no
Labels: name: bug

### System Info

```shell
trasformers = 4.20.1
Models: layoutlmv3

How to use LayoutLMv3 for Document Layout Detection, for example microsoft unilm(https://github.com/microsoft/unilm/tree/master/layoutlmv3)?
I do not find Document Layout Detection task infomation in example and modeling.
The typical dataset for Document Layout Detection is call PubLayNet (https://github.com/ibm-aur-nlp/PubLayNet).


example :https://github.com/huggingface/transformers/tree/main/examples/research_projects/layoutlmv3
modeling: https://github.com/huggingface/transformers/tree/main/src/transformers/models/layoutlmv3
```


### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

How to use LayoutLMv3 for Document Layout Detection, for example microsoft unilm(https://github.com/microsoft/unilm/tree/master/layoutlmv3)?
I do not find Document Layout Detection task infomation in example and modeling.
The typical dataset for Document Layout Detection is call PubLayNet (https://github.com/ibm-aur-nlp/PubLayNet).

### Expected behavior

```shell
I can know how to use layoutlmv3 model in transformers for  Document Layout Detection?
I would really appreciate it if you can give me an example code.
```


Comments:
Hi,

If you read a bit more closely: https://github.com/microsoft/unilm/tree/master/layoutlmv3#document-layout-analysis-on-publaynet

You'll see they provide a guide regarding fine-tuning LayoutLMv3 on PubLayNet. The Mask R-CNN framework is leveraged. This framework currently is not available in Huggingface Transformers, so you'll need to use the unilm repo for that.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Unable to Load the Pre-Trained Model using Spark-Submit 
Author: paresmi
Pull request: no

Hi Team, 

I am trying to trigger spark-submit and passing the pre-trained model with --archives ; however Spark is not able to locate the model while executing the BertTokenizer.from_pretrained(BERT_MODEL_NAME).

Please advise how to I get the spark to point to the BERT Model ; I tried placing in HDFS as well as Local Linux path - but same error. 

`22/06/20 17:09:23 INFO e: Traceback (most recent call last):
  File "etl.py", line 509, in run
    self.ingest()
  File "etl.py", line 315, in ingest
    BERT_MODEL_NAME="score/bert-base-cased/",
  File "/data-12/hadoop/yarn/local/usercache/svc/appcache/application_1641/container_e397_000001/e_process-0.0.1-py3.7.egg/classification/pipelinescore.py", line 30, in __init__
    self.tokenizer = BertTokenizer.from_pretrained(BERT_MODEL_NAME,cache_dir="/cache")
  File "/data-12/hadoop/yarn/local/usercache/svc/appcache/application_1641/container_e397_00001/dep.tar.gz/transformers/tokenization_utils_base.py", line 1773, in from_pretrained
    f"Can't load tokenizer for '{pretrained_model_name_or_path}'. If you were trying to load it from "
OSError: Can't load tokenizer for 'classification/score/bert-base-cased/'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'classification/score/bert-base-cased/' is the correct path to a directory containing all relevant files for a BertTokenizerFast tokenizer.
`

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
XLM-Roberta offset mapping is off by one in case of whitespace-subwords
Author: robvanderg
Pull request: no
Labels: name: bug

### System Info

```shell
- `transformers` version: 4.19.2
- Platform: Linux-5.4.0-94-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.6.0
- PyTorch version (GPU?): 1.7.1+cu110 (True)
- Tensorflow version (GPU?): 2.9.1 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```


### Who can help?

@LysandreJik @SaulLu

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained('xlm-roberta-large', use_fast=True)
>>> tokenizer.tokenize('Quality of work is sufficient')
['▁Quality', '▁of', '▁work', '▁is', '▁', 'sufficient']
>>> tokenizer.encode_plus('Quality of work is sufficient', return_offsets_mapping=True)
{'input_ids': [0, 124604, 111, 4488, 83, 6, 129980, 2], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1], 'offset_mapping': [(0, 0), (0, 7), (8, 10), (11, 15), (16, 18), (19, 20), (19, 29), (0, 0)]}


### Expected behavior

```shell
The third-last offset-tuple (19,20) overlaps with the second-last offset-tuple(19,29). I believe that this should be (18,19), and thus refer to the whitespace.
```


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I think @robvanderg is right (with my intuitive understanding). Since `return_offsets_mapping` is only for fast tokenizer, let me kindly tag @Narsil here

The results should be like
`... (18, 19), (19, 29) ...`
I think.

I agree with you @ydshieh, to solve it it might be best to open an issue on the [tokenizers](https://github.com/huggingface/tokenizers) library as this is where the offsets are computed :blush: 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Converting batch into jax arrays during training is inefficient
Author: sunitharavi9
Pull request: no
Labels: name: bug

### System Info

The Flax Transformers examples use a custom dataloader which yields batches. This includes converting the batch into jax arrays: 

https://github.com/huggingface/transformers/blob/main/examples/flax/summarization/run_summarization_flax.py#L355

This operation is inefficient, it can take about 700ms to generate a batch during training. On A100 GPUs, this bottleneck leads to lower GPU utilization. 

The solution would be to set ds.set_format("jax") before loading batches for training. 

### Who can help?

@patil-suraj 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

All examples in flax use this operation for batch generation:
https://github.com/huggingface/transformers/blob/main/examples/flax/

### Expected behavior

The fix gives better GPU utilization and faster training (Tested locally on A100 GPUs). 

Comments:
Hi @isunitha98selvan. In the example linked, we actually force the batch indices to `numpy` arrays:
https://github.com/huggingface/transformers/blob/286a18fa0080dd39bd373008d11d831fbb1a77f1/examples/flax/summarization/run_summarization_flax.py#L345-L346

And then slice the dataset according to these `numpy` arrays.

When using JAX, we want to keep everything that's outside of the jit'd function as a `numpy` array, and everything that's inside as `jax.numpy` arrays. The reason being if everything outside of the jit'd function is a `numpy` array, we can make maximal use of JAX's asynchronous dispatch (https://jax.readthedocs.io/en/latest/async_dispatch.html): the CPU can run ahead and compute any `numpy` arrays ahead of time, whilst the GPUs/TPUs tackle the `jax.numpy` arrays inside the jit'd function.

In the forward pass of the model, any `numpy` arrays are converted to `jax.numpy` arrays, putting the data on the accelerator device.

So whilst it might be slower computing the batches on a per-batch basis like this, when we combine it with a jit'd training function we'll actually train much faster overall 🚀

Hey @isunitha98selvan, feel free to ask if you have any questions! Otherwise, closing this for now.
GFT: Generative Fundamental Training
Author: wangyi-fudan
Pull request: no
Labels: name: New model

### Model description

Hi,
It is just an ad of our GFT.
It has a novel attention head by fusing attention and mlp layer. It has an ultra-thin, ultra-deep architecture that maximizing model performance with minimal parameters. More interestingly,  it equips a novel decoder call top-E(entropy) algorithm. 
Our model has 81 layers but only 300M parameters. 
Both Chinese and English(Biomedical) model are available.
It is coded from scratch by cuda and C++, no DL framework is needed.
Enjoy!

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

https://github.com/wangyi-fudan/GFT

Chinese Model Download

链接: https://pan.baidu.com/s/1HKw83YWttCIPdnZCZ-hdOA

密码: 0r1j

English Medical Model Download

链接: https://pan.baidu.com/s/1yazmdB8xFzLwXbKmXWslRA

密码: rqdc


Comments:
Migrate metrics used in all examples from Datasets to Evaluate
Author: sgugger
Pull request: no
Labels: name: Good First Issue

The metrics are slowly leaving [Datasets](https://github.com/huggingface/datasets) (they are being deprecated as we speak) to move to the [Evaluate](https://github.com/huggingface/evaluate) library. We are looking for contributors to help us with the move.

Normally, the migration should be as easy as replacing the import of `load_metric` from Datasets to the `load` function in Evaluate. See a use in this [Accelerate example](https://github.com/huggingface/accelerate/blob/1486fa35b19abc788ddb609401118a601e68ff5d/examples/nlp_example.py#L104). To fix all tests, a dependency to evaluate will need to be added in the [requirements file](https://github.com/huggingface/transformers/blob/main/examples/pytorch/_tests_requirements.txt) (this is the link for PyTorch, there is another one for the Flax examples).

If you're interested in contributing, please reply to this issue with the examples you plan to move.

Comments:
Hi, I would like to try to move Pytorch examples.

Please go ahead and make a PR when you're ready then @atturaioe. You can check our [contributing guide](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) to get started.

@sgugger  , I would like to work on the tensorflow examples. 




Please go ahead @VijayKalmath :-)

@sgugger , I wanted to know if there are tests to run the examples in the tensorflow folder.

No tests for TensorFlow yet, we'll add those in the near future.

@sgugger , I don't think anyone is working on the flax examples. Will open a new PR for migrating the flax examples?

By all means, please go ahead! The only change will be that those are actually tested.

@sgugger, can I work on `research_projects` examples?

@atturaioe We don't update research projects to the new APIs as they use pinned versions of libraries. So this one should stay as is unless the author want to do a general update of them. Thanks for offering though!
Migrate metric to Evaluate in Pytorch examples
Author: atturaioe
Pull request: yes

# What does this PR do?
As metrics are being deprecated in Datasets, they need to be moved to Evaluate. This PR migrates function calls of `load_metric` from Datasets to `load` in Evaluate in Pytorch examples.


Fixes #18306 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sgugger 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
`local_files_only` is not passed to `_from_pretrained` in `PreTrainedTokenizerBase.from_pretrained`
Author: YouJiacheng
Pull request: no

I run
```python
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32", local_files_only=True)
```
But request is still sent to get etag.
https://github.com/huggingface/transformers/blob/b2e4b091f08f1aaf21855d588c6c8d284baba9eb/src/transformers/tokenization_utils_base.py#L1653-L1813
`local_files_only` is aborted when calling `_from_pretrained`, whether it is explicitly passed or implicitly set by `is_offline_mode()` check.
I think this behavior is buggy.
Fortunately, `transformers` check `is_offline_mode()` in `utils/hub.py/cached_path`, so I can globally and permanently force `local_files_only` as a workaround in my usecase.

Only fixing `PreTrainedTokenizerBase.from_pretrained` is not enough, `_from_pretrained` doesn't pass `local_files_only` to `AutoConfig.from_pretrained` either.
I'm working on a fix for it.

Comments:
Fix DeiT doc tets
Author: amyeroberts
Pull request: yes

# What does this PR do?

Fixes failing doctest for DeiT cause by copy-pasta from the PT model. 

The predicted class is determinstic with the set seed. However, as RNG for pytorch and tensorflow are different, it's a different class than in the pytorch Deit doctest. 

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
_The documentation is not available anymore as the PR was closed or merged._

From the description & the change, it seems to me that Deit has some random OP(s) used in the inference. Is this the case (and if yes, could you share which line that random op is 🙏 ) Thank you.

Sure :) `DeiTForImageClassification` loaded from the official facebook hub weights will have its head randomly initialized. You can't see it directly in this diff, but there's a comment above on line L928 mentioning this.
Fix from_pretrained kwargs forward
Author: YouJiacheng
Pull request: yes

I don't know whether `use_auth_token`, `cache_dir` and `local_files_only` should be passed to `(cls.slow_tokenizer_class)._from_pretrained`, but I guess it should.
Please correct me if anything is wrong.

# What does this PR do?

Fixes #18385

@sgugger @LysandreJik 

BTW, I find #13523 and #14508 addressing similar problems, which shows current implementation is vulnerable to this kind of problems, a refactor might be needed. A context manager might be suitable as a long-range dependency injector.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Unrecognized configuration class GPT2Config for AutoModelForSeq2SeqLM | Microsoft DialoGPT no longer working
Author: RuolinZheng08
Pull request: no

## Information

Model I am using: Microsoft's DialoGPT

The problem arises when using:
* [x] the official example scripts: 

Since the morning of July 14th, the inference API has been outputting errors on [Microsoft's DialoGPT](https://huggingface.co/microsoft/DialoGPT-medium). It was working fine before July 14th.

Error

```
{'error': "Unrecognized configuration class <class 'transformers.models.gpt2.configuration_gpt2.GPT2Config'> 

for this kind of AutoModel: AutoModelForSeq2SeqLM.\nModel type should be one of 

BigBirdPegasusConfig, M2M100Config, LEDConfig, BlenderbotSmallConfig, MT5Config, T5Config, PegasusConfig, MarianConfig, MBartConfig, BlenderbotConfig, BartConfig, FSMTConfig, EncoderDecoderConfig, XLMProphetNetConfig, ProphetNetConfig."}
```

Query script as given on Hugging Face's site:

```python
import requests

API_URL = "https://api-inference.huggingface.co/models/microsoft/DialoGPT-medium"
headers = {"Authorization": "Bearer API_TOKEN"}

def query(payload):
	response = requests.post(API_URL, headers=headers, json=payload)
	return response.json()

output = query({
    "inputs": {
		"past_user_inputs": ["Which movie is the best ?"],
		"generated_responses": ["It's Die Hard for sure."],
		"text": "Can you explain why ?",
	},
})
```

@patrickvonplaten, @LysandreJik
I'm mentioning these two people as the guide says they are working on gpt2. Sorry if I pinged the wrong people!


Comments:
Getting this error also. all fine-tuned models using Dialo giving this exact error.

Hi everyone.

Not really sure what happened here (the error is pretty confusing). It is fixed now anyway. 

If someone pinned a model while it was having issues, please let me know, we might have to update them to fix them too !

Hi @Narsil 

These 2 models need update:
- dbmdz/german-gpt2
- benjamin/gerpt2

dbmdz/german-gpt2 seems to be working  https://huggingface.co/dbmdz/german-gpt2?text=Heute+ist+sehr+sch%C3%B6nes+Wetter+in

It doesn't seem to be defined as `conversation`, is that what you're referring to ?
I am not sure how this model was defined and so if it actually works with conversation, but it doesn't seem to be the case.

The API works with text-generation for this model and it works fine.

`benjamin/gerpt2` seems to be exactly the same.

If you want to mark them as conversational you need to update the `pipeline_tag` https://huggingface.co/docs/hub/models-widgets#enabling-a-widget

Otherwise do you mind creating a new issue with the new error you're receiving to be able to reproduce (you can ping me) ?

Hope this helps.
Add balanced strategies for device_map in from_pretrained
Author: sgugger
Pull request: yes

# What does this PR do?

This PR brings to Transformers the functionality introduced in https://github.com/huggingface/accelerate/pull/534 .
Basically `device_map` can now take several options:
- `"sequential"` which corresponds to the current auto: fill each GPU sequentially (and if the user has lots of GPU spaces, some are not used at all)
- `"balanced"` which will split the model evenly across GPUs
- `"balanced_low_0"` which will split the model evenly across GPUs while leaving the most available memory on GPU 0, since that GPU might have more tensors on it when the outputs are used for some form of post-processing (generate and use_cache for instance)
- `"auto"` which now defaults to `"balanced"`.

When the user does not have enough GPU memory to accommodate the model, all the options are equivalent.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add `trace_device` argument to `smp.DistributedModel` call in `Trainer()`
Author: joehoover
Pull request: no

### Feature request

Allow `trace_device` to be passed to `smp.DistributedModel` so that `Trainer` jobs that use SageMaker Model Parallel can support models that exceed a single GPU's memory. 

Tagging @philschmid and @patil-suraj, because I think you both are involved in the SageMaker work?

### Motivation

The `Trainer` class provides native support for the SageMaker Model Parallel (smp) library; however, it does not support specifying the device where model tracing is conducted at the beginning of a smp training job. Because the default trace device is GPU, it is not possible to train a model that cannot fit in (a single) GPU memory, which, of course exactly when you would want to use model parallelism. 

This can be fixed by passing [trace_device](https://sagemaker.readthedocs.io/en/v2.20.0/api/training/smd_model_parallel_pytorch.html) to the `smp.DistributedModel` [call](https://github.com/huggingface/transformers/blob/d0acc9537829e7d067edbb791473bbceb2ecf056/src/transformers/trainer.py#L1236) in `Trainer()`. 

The value of `trace_device` could be specified in the [same way other smp parameters are specified](https://github.com/huggingface/transformers/blob/acb709d55150501698b5b500ca49683b913d4b3d/src/transformers/training_args.py#L904), which is via a json string of smp parameters, `mp_parameters`. 

As noted [elsewhere](https://github.com/huggingface/transformers/issues/14851#issuecomment-1013422175_), `trace_device` is not currently supported in the HF DLC, but it is apparently roadmapped. 

Accordingly, my current workaround is to use a SageMaker pytorch estimator and a custom Trainer class with the necessary overrides to ensure that `trace_device` is passed. 

It would be nice to be able to use the base trainer class rather than this workaround. 

### Your contribution

If this seems reasonable, I'd be happy to open a PR. 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.