noamaanMulla-03/transformers-issues huggingface dataset:
Incorrect Tokenization behavior when working with Hindi using RobertaTokenizer
Author: amankhandelia
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: https://github.com/huggingface/transformers@534f6eb9f1f8910a4912ccccd79f1f974c625168
- Platform: Linux
- Python version:  3.8.10
- PyTorch version (GPU?): 1.9.0 (Not using GPU)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


## Information

### Problem
[Our model](https://huggingface.co/flax-community/roberta-hindi) is tokenizing text in way longer sequences, more like a character level lm, than as sub-word level lm.

### Diagnosis
Tokenizer is at fault, when it tokenizes on a word with matra, it always breaks matra, i.e. diacritics into a seperate token, as in `'संसद' => [' स', 'ं', 'सद']`, where as a more plausible tokenization should be `'संसद' => ['सं', 'सद']`
For more details one can look at the analysis csv [here](https://gist.github.com/amankhandelia/f8192a6782714fab6537dc29aef8fbfc) where I have done the analysis for five different models. Given the pattern observed in all RoBERTa models I believe that the [BBPE tokenizer](https://arxiv.org/pdf/1909.03341.pdf) used in case of [RoBERTa-hindi-guj-san](https://huggingface.co/surajp/RoBERTa-hindi-guj-san), [HindiBERTa](https://huggingface.co/mrm8488/HindiBERTa) and [our model](https://huggingface.co/flax-community/roberta-hindi) is not upto the task, or need some modifications to work on Hindi. 

The tasks I was working on was Roberta Hindi LM as part of the recent flax community sprint. Although the sprint has ended, I was analyzing the tokenizers in order to learn how these model works, when I stumbled upon this observation, so this bug report is more of question than an explicit bug.
Is this the expected behavior of BBPE, if so, what is a possible remedy, can I use different tokenizer with Roberta instead using the RobertaTokenizer?

cc: @LysandreJik, @patil-suraj

Keeping Suraj in cc, as he might be able to better communicate the problem I am facing, with the HF team.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Allow the use of tensorflow datasets having UNKNOWN_CARDINALTY for TFTrainer
Author: prikmm
Pull request: no

# 🚀 Feature request
<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

I am currently working on a project which requires huggingface and tensorflow. I created a huggingface dataset and converted it into tensorflow dataset using `from_generator()`, instead of `from_tensor_slices()` like the example in the docs. The dataset that I am using is very big, hence, it takes a lot of time for it to get converted into a tensor using dictionary comprehension. `from_generator()` handles this well by processing the data on-the-fly. Tensorflow allows training via datasets having UNKNOWN_CARDINALITY. I would like to suggest the same to be implemented in TFTrainer.


Comments:
Hi! We're actually trying to deprecate the use of TFTrainer at the moment in favour of Keras - there will be a deprecation warning about this in the next release of Transformers. Using Keras you can train with a generator or tf.data.Dataset without problems.

In addition, we're looking to improve our Tensorflow integration in other ways, such as by making it easier to load huggingface datasets as Tensorflow datasets, which might allow us to retain the cardinality information regardless. This second part is still a work in progress, though!

> Hi! We're actually trying to deprecate the use of TFTrainer at the moment in favour of Keras - there will be a deprecation warning about this in the next release of Transformers. Using Keras you can train with a generator or tf.data.Dataset without problems.
> 
> In addition, we're looking to improve our Tensorflow integration in other ways, such as by making it easier to load huggingface datasets as Tensorflow datasets, which might allow us to retain the cardinality information regardless. This second part is still a work in progress, though!

That's great! Looking forward to the next release!:smile:

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Converting a tensor to a python boolean might cause the trace to be incorrect. We can't record the data flow of python values
Author: junleiz
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform: CENTOS 8.
- Python version: 3.7
- PyTorch version (GPU?): 1.9.0 + cuda
- Tensorflow version (GPU?): No
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 @patrickvonplaten, @patil-suraj  @LysandreJik
Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @Rocketknight1

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information
![image](https://user-images.githubusercontent.com/26025961/126442774-d5833dbb-3ebb-4eeb-a12b-4a47141c9a74.png)
I convert the hugging face bart model to onnx model using transformer.onnx (thanks for the excellent tools). But these warnings occured and they may have a influencec on the performance of model. 



Comments:
Hi @leoozy, you should safely disregard these warnings. The warnings should only apply to assertions we make to ensure that the input shape passed is correct.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
minor mistake in the documentation of XLMTokenizer
Author: ritwikmishra
Pull request: no

On [this page](https://huggingface.co/transformers/model_doc/xlm.html#xlmtokenizer), the default value of `cls_token` is given as `
` whereas it should be `
`.

Please fix it.


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Upgrade `os.path` to use `pathlib.Path` API for `from_pretrained` internals
Author: changjonathanc
Pull request: no

# 🚀 Feature request

Use `pathlib.Path` instead of `os.path` in places like `src/transformers/file_utils.py` and `src/transformers/configuration_utils.py`

## Motivation

I am using [cloudpathlib](https://github.com/drivendataorg/cloudpathlib), a library that wraps remote path as `pathlib.Path` like object. But I cannot use a cloudpathlib's remote directory for `from_pretrained` because the existing code uses `os.path`. Using `pathlib`'s API would solve this.

## Your contribution

I can submit a PR for this. I see pathlib used elsewhere inside `src/transformers`, so I think it's ok.

Comments:
This requires more code than I anticipated, and the caching isn't as fast as local. I think I will just download to local and call `from_pretrained` with local path.

Thanks for exploring

> I think I will just download to local and call from_pretrained with local path.

Yes it's the way we've been recommending 👍
[Slow tests] Disable Wav2Vec2 pretraining test for now
Author: patrickvonplaten
Pull request: yes

Wav2Vec2 pretraining seems not to be working curretnly. This PR disables the test: tests/test_modeling_wav2vec2.py::Wav2Vec2ModelIntegrationTest::test_inference_integration
Test will be re-enabled once succesfull wav2vec2 pretraining has been done

Comments:
Add Model Details for Pipeline API
Author: dhruvjimulia-sys
Pull request: no

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->
I think it will be great if we could add further details about what particular model the Pipeline API is using. In particular, we can add the following details relating to the models in a readable markdown file:
* Dataset used for training
* Type of transformer used
* Hyperparameters and neural net architecture
* Evaluation metrics: accuracy + precision + recall on test portion of dataset
## Motivation
I was working on different sentiment analysis datasets with different transformers when I stumbled across the HuggingFace pipeline API. I was impressed by the performance of the models, but I wasn't quantitively sure about the accuracy of the model, neither was I sure about how it was trained.

I believe that many other users of the Pipeline API will also benefit from such a change.

P.S. If there is some source of such information and I've missed it, please let me know


Comments:
This can be interesting. Are you interested in opening a PR? You could add a section with model details for every pipeline by editing the documentation pages (i.e. the .rst files).

Yes I would like to try opening a PR for this. However, it will take me some time to understand the code behind how the pipeline works.

Hello! The first step for this would be to expose the pretrained models used by the pipelines - it's currently hidden in the code here: https://github.com/huggingface/transformers/blob/master/src/transformers/pipelines/__init__.py#L110

Exposing which model is the default with a link to the model card would be nice indeed.

cc @Narsil, I think it would be very beneficial to document which model is used by default for each pipeline.

Should we do that at runtime with a log that we chose a default model, or you had something more static (definitely doable, but probably tedious to maintain.)


Yes I've been busy lately - although I thought I could examine the code, it has not been possible given my schedule. Given that this is an important problem others are facing too, feel free to open a PR.

@Narsil I personally feel that having a runtime log will be more than enough given the ease at which one can get started with using the pipeline API - one can simply implement it to find out what model is working behind the scenes

Yeah, a runtime log would be fine.

https://github.com/huggingface/transformers/pull/13276
LayoutLMv2 Model
Author: monuminu
Pull request: no
Labels: name: New model

# 🌟 New model addition

## Model description

LayoutLMv2 by pre-training text, layout and image in a multi-modal framework, where new model architectures and pre-training tasks are leveraged. Specifically, LayoutLMv2 not only uses the existing masked visual-language modeling task but also the new text-image alignment and text-image matching tasks in the pre-training stage, where cross-modality interaction is better learned.

<!-- Important information -->

## Open source status

* [ ] the model implementation is available: (give details) https://github.com/microsoft/unilm/tree/master/layoutlmv2
* [ ] the model weights are available: (give details) https://huggingface.co/microsoft/layoutlmv2-base-uncased/
* [ ] who are the authors: (mention them, if possible by @gh-username) 
@inproceedings{Xu2020LayoutLMv2MP,
  title     = {LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding},
  author    = {Yang Xu and Yiheng Xu and Tengchao Lv and Lei Cui and Furu Wei and Guoxin Wang and Yijuan Lu and Dinei Florencio and Cha Zhang and Wanxiang Che and Min Zhang and Lidong Zhou},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL) 2021},
  year      = {2021},
  month     = {August},
}


Comments:
What's the current status of this?
I see Microsoft has already pushed the code as mentioned above for this but i can't import it from huggingface.

I have completed the work but there is a problem with installing detectron2 using pip . Anyone can help on this ?

Can you tell me the issue with installing detectron2? I can prolly help

Its saying pip install detectron2 is giving issue . 

You have to install detectron from `git+https://github.com/facebookresearch/detectron2.git`, not pypi

See https://github.com/facebookresearch/detectron2/blob/master/INSTALL.md for instructions. 

Use
`python -m pip install 'git+https://github.com/facebookresearch/detectron2.git'`
[trainer] `--load_best_model_at_end` silently turns of `--save_steps` settings
Author: stas00
Pull request: no

Splitting off from https://github.com/huggingface/transformers/pull/12477#discussion_r668326212

Currently `--load_best_model_at_end` silently turns off `--save_steps` settings when `--do_eval` is off (or `--evaluation_strategy` is  set to other than `"no"`, which otherwise automatically turns on `--do_eval`)

The proposal is to assert if: 

`--load_best_model_at_end` is set  and `--evaluation_strategy` is `"no"`

Reproducible test:

```
export BS=16; rm -r output_dir; PYTHONPATH=src USE_TF=0 CUDA_VISIBLE_DEVICES=0,1 python -m torch.distributed.launch --nproc_per_node=2 examples/pytorch/translation/run_translation.py --model_name_or_path t5-small --output_dir output_dir --adam_eps 1e-06  --do_train --label_smoothing 0.1 --learning_rate 3e-5 --logging_first_step --logging_steps 500 --max_source_length 128 --max_target_length 128 --val_max_target_length 128 --num_train_epochs 1 --overwrite_output_dir --per_device_train_batch_size $BS --predict_with_generate --sortish_sampler --source_lang en --target_lang ro --dataset_name wmt16 --dataset_config "ro-en" --source_prefix "translate English to Romanian: " --warmup_steps 50 --max_train_samples 50 --save_steps 1 
```
which saves checkpoints.

then adding `--load_best_model_at_end` stops saving those.

@sgugger.

Comments:
Yes, as said in that comment, I think it's reasonable if we raise an error if `--load_best_model_at_end` is set and `--evaluation_strategy` is "no" since there is no "best model" to pick from in that case. I can do it later today if you want.

I'm still not 100% clear on how this feature's reliance on eval affects saving checkpoints, but if it solves the problem that's good enough for me.

Absolutely no rush on this one.

Thank you!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Hasn't this one been resolved already?

Yes, this was fixed by #12786 in the end.
[AutoFeatureExtractor] Fix loading of local folders if config.json exists
Author: patrickvonplaten
Pull request: yes

# What does this PR do?

Currently there is a problem when loading the feature extractor locally via `AutoFeatureExtractor` as spotted by @philschmid:

```bash
!git lfs install
!git clone https://huggingface.co/facebook/wav2vec2-base-960h
```

and then:

```python
from transformers import AutoFeatureExtractor

extractor = AutoFeatureExtractor.from_pretrained("wav2vec2-base-960h")
```

This leads to an error. This PR fixes it and also improves the error message.

Comments:
@sgugger - merging for now. Let me know if something is not well and we can change afterwards. Tests have been added so this PR should be more or less safe.
SageMaker: Fix sagemaker DDP & metric logs
Author: philschmid
Pull request: yes

# What does this PR do?

This PR fixes the fix introduced in #12853. Since `sm_dist.Barrier()` is not available in `smd 1.0.0 2021-01-26` release and smd `1.2.0`, which are used for the DLC with PyTorch 1.7.1 & 1.8.1 (they maintained ones). Further is the fix `sm_dist.barrier()` also working with `smd 1.0.0 2020-12-06`.  
cc @sgugger 

Additionally, does this PR update:
* the SageMaker test image_uris
* instances type for distributed training -> there are capacity issues with the 24xlarge
* and moves the adding of the `StreamHandler(sys.stdout)` for logs to the `trainer_pt_utils.py` to also cover the `log_metrics function` more to this below.

---
When running a training job on SageMaker all stdout or stderr are sent to Amazon CloudWatch Logs. With the introduction of the new `log_metrics` function, SageMaker lost its output. Therefore I moved the `StreamHandler(sys.stdout)` to the `trainer_pt_utils.py` and removed it in the `trainer`. More information can be found here #10633
 


Comments:
Thanks a lot @philschmid !
Improve documentation of pooler_output in ModelOutput
Author: navjotts
Pull request: yes

# What does this PR do?
Improves the doc string for pooler_output in modeling_outputs.py – making it more clear, and opening its availability to a more generic use-case than just BERT-family of models.

**Motivation**:
I was writing a `cls_pooler` for a sentence embeddings usage, and initially thought this is the CLS token output from the last layer – which is not the case, that would just be `last_hidden_state[0]`

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?
@sgugger 

Comments:
@sgugger noting: this PR is ready (tests have passed)

Thanks a lot!
Check None before going through iteration
Author: qqaatw
Pull request: yes

# What does this PR do?

This PR fixes the error mentioned in #13234. 

This is a quick solution. For long-term development, should we change the default value of `_keys_to_ignore_on_xxx` from `None` to `list`, so we can dismiss the check of whether it is `None` before running any iteration.
 
https://github.com/huggingface/transformers/blob/b1198a8440cc05f569b0bc22038993a1e5e707ab/src/transformers/modeling_utils.py#L444-L450

https://github.com/huggingface/transformers/blob/b1198a8440cc05f569b0bc22038993a1e5e707ab/src/transformers/modeling_tf_utils.py#L626-L629

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@LysandreJik @sgugger

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
torch longformer to tf longformer
Author: aixuedegege
Pull request: no

# 🚀 Feature request
1、how to use this model in triton
2、how to turn it into a tf model

## Motivation

I downloaded the model from "https://huggingface.co/xcjthu/Lawformer/tree/main" which is a torch model. It can not be transformed to a pt file using "torch.jit.trace". I want to use triton to server it. So I want to try to trans it to a tf model and try to transform it to savedmodel


## Your contribution
Sorry, I have no idea about how to Implement it I need your help haha. 
Thanks a lot.

Comments:
Can't help you with the first question, but for the second:

> how to turn it into a tf model

```
from transformers import TFLongformerModel

model = TFLongformerModel.from_pretrained("xcjthu/Lawformer", from_pt=True)
```

Hey @aixuedegege,

please use the forum: https://discuss.huggingface.co/ for questions like your first one as it's not really a bug of the library, but more a general question. Your chances of getting a good answer should be higher there :-)

@NielsRogge Thanks for you answer. But using your code I got some warning "Some weights or buffers of the TF 2.0 model TFLongformerModel were not initialized from the PyTorch model and are newly initialized". I will go to https://discuss.huggingface.co/ for discussing this.