noamaanMulla-03/transformers-issues huggingface dataset:
Rebase ESM PR and update all file formats
Author: Rocketknight1
Pull request: yes

This is a rebase and rework of the ESM PR at #13662. The old PR predates the `master -> main` rename and the conversion from `.rst` to `.mdx` documentation. As such, a straightforward rebase was very messy, so I copied the PR to a new branch and fixed the various file formats to be compatible with modern `transformers`. We're hoping to move quite quickly from here to get this merged.

Progress checklist:

- [X] Rebase
- [X] Convert documentation format
- [X] Convert imports in `__init__.py` and `modeling_auto.py`
- [x] Address comments on the old PR
- [x] Complete any `TODOs` left in the code
- [x] Figure out what the `encoder_keep_prob` hack in the conversion script is for
- [x] Use correct ESM classes in the conversion script
- [x] Checkout Meta's ESM repo and double-check tests to make sure outputs are still equivalent
- [x] Test that model output is still equivalent when there are `<mask>` tokens in the input
- [x] Test that model output is still equivalent when inputs are padded
- [x] Remove the unused `token_type_ids` from the code
- [x] Fix the copies now we're not using `token_type_ids`
- [x] Add support for ESM-2 `RotaryEmbedding`
- [x] Find out why the original repo has a loss of precision in `RotaryEmbedding` and whether we need the hack
- [x] Get the last few tests to pass
- [x] Make sure slow tests pass locally
- [x] Confirm uploaded model names with the Meta team
- [x] Figure out if we need custom heads/losses/classes for ESM-1v, and/or if that should be pushed to another PR

Models to convert/check:
- [x] ESM-1b (esm1b_t33_650M_UR50S)
- [x] ESM-1v (esm1v_t33_650M_UR90S_[1-5])

Models to convert/check if we include ESM-2 as well:
- [x] esm2_t6_8M_UR50D
- [x] esm2_t12_35M_UR50D
- [x] esm2_t30_150M_UR50D
- [x] esm2_t33_650M_UR50D
- [x] esm2_t36_3B_UR50D
- [x] esm2_t48_15B_UR50D

What we're **not** converting: MSA models (because HF doesn't have the MSA retrieval code yet) and ESM-1 (because it's been superceded by ESM-1b and we don't expect much usage).

Various people have expressed interest in this, so I'm going to ping them here so they're aware of this!
cc: @sgugger @patrickvonplaten @liujas000 @gianhiltbrunner @franzigeiger

Comments:
_The documentation is not available anymore as the PR was closed or merged._

cc @sgugger @LysandreJik this should now be ready for review! ESM-1b and ESM-2 models are both supported and the discrepancy between our output and the output from the original model is now 2e-5 or less.

I'm still chasing down one last question with the Meta team, but that only affects a very small part of the code.

(Note that tests will fail until I finish converting and uploading checkpoints)

@LysandreJik Everything renamed ESM -> Esm!

Thank you! Good to merge for me
chore: add expected output to the sample code.
Author: sayakpaul
Pull request: yes

Fixes the issue as described here: https://github.com/huggingface/transformers/pull/18815#issuecomment-1256338065

@ydshieh FYI.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@ydshieh here's what I have done so far:

* Update the weight conversion script, so we have the ImageNet-1k label mapping in the model config. 
* Update the `modeling_vit_msn.py` script with the expected output (which is basically an ImageNet-1k class).

I also pushed the model classes with the updated configs. I realized if I only updated the `config.json` in the respective Hub repositories, the member config associated with the model classes may not get appropriately updated. So, I pushed everything updated with the new config to my HF account. 

So, if you go to https://huggingface.co/models?filter=vit_msn now, you'd notice two versions of the same model checkpoint: one is hosted from `facebook` and another one is hosted from `sayakpaul`. I need someone from the HF team to transfer the updated model repositories to `facebook`. 

Let me know if anything here is unclear. 

> I realized if I only updated the config.json in the respective Hub repositories, the member config associated with the model classes may not get appropriately updated.

Could you explain a bit more about `the member config associated with the model classes may not get appropriately updated.
`?

The current model classes living under the Facebook org don't have the updated configs that have the ImageNet-1k labels. 

The model classes residing under `sayakpaul` do. They need to be transferred to the facebook org. 

If you search with https://huggingface.co/models?filter=vit_msn then you will get the complete list. 

Let me know if anything is unclear now. 

My question is about why uploading the new config.json is not enough - this is what I understand from your comments.

I think simply updating the config.json on the facebook model Hub repo. would be enough.

Okay, let me verify a few things and get back to you. 

@ydshieh @sgugger the PRs to the Hub repos are up:

* https://huggingface.co/facebook/vit-msn-small/discussions/4
* https://huggingface.co/facebook/vit-msn-base/discussions/2
* https://huggingface.co/facebook/vit-msn-large/discussions/2
* https://huggingface.co/facebook/vit-msn-base-4/discussions/2
* https://huggingface.co/facebook/vit-msn-large-7/discussions/2

Indeed, just updating the config.json made it work. I have verified it locally. 

Hub PRs all merged, thank you @sayakpaul (also for checking)
Can't load too big text file for dataset (RAM exhausts), HELP PLEASE !!!!!
Author: mv96
Pull request: no

Hello All,

I am trying to load a big text file for pertaining a BERT model from scratch the size of the txt file is about 11GB and trying to load the model for pertaining is exhausting all the RAM on the system.

Is it possible to load all the data in batches and then perform training.

I am a bit new to hugging face ecosystem so I would request you all to help me around if you have any clue about this.
I am using Google Colab for the purpose.

Please share code snippets, If possible.

Cheers !

`#construct dataset 
from transformers import LineByLineTextDataset

file_path="/content/drive/MyDrive/full_text_data.txt"
dataset = LineByLineTextDataset(
    tokenizer=tokenizer,
  file_path=file_path,
    block_size=32)`



Comments:
(duplicate of #19199)
Poc to use safetensors
Author: sgugger
Pull request: yes

# What does this PR do?

This PR introduces the necessary code changes to use safetensors weight file as a primary source.

Comments:
Thank you for being verified!

_The documentation is not available anymore as the PR was closed or merged._

This looks so clean !

Opened Hub Pull requests to convert the following weights:
- https://huggingface.co/roberta-base/discussions/3
- https://huggingface.co/roberta-large/discussions/1
- https://huggingface.co/gpt2/discussions/6
- https://huggingface.co/Jean-Baptiste/camembert-ner/discussions/1
- https://huggingface.co/openai/clip-vit-large-patch14/discussions/5

^will merge the 3 canonical ones to be able to test easily

note that you can also test from un-merged Hub PRs, you just have to pass the `refs/pr/:id` as a `revision`, for instance for gpt2:

```python
model = AutoModelForCausalLM.from_pretrained("gpt2", revision="refs/pr/6")
```
Add `beautifulsoup4` to the dependency list
Author: ydshieh
Pull request: yes

# What does this PR do?

Add `beautifulsoup4` to the dependency list - to enable the tests `MarkupLMFeatureExtractionTest` and `MarkupLMProcessorIntegrationTests`

On CircleCI, we now can have the following tested (those have `@require_bs4`)
```bash
PASSED tests/models/markuplm/test_feature_extraction_markuplm.py::MarkupLMFeatureExtractionTest::test_call
PASSED tests/models/markuplm/test_feature_extraction_markuplm.py::MarkupLMFeatureExtractionTest::test_feat_extract_from_and_save_pretrained
PASSED tests/models/markuplm/test_feature_extraction_markuplm.py::MarkupLMFeatureExtractionTest::test_feat_extract_to_json_file
PASSED tests/models/markuplm/test_feature_extraction_markuplm.py::MarkupLMFeatureExtractionTest::test_feat_extract_to_json_string
PASSED tests/models/markuplm/test_feature_extraction_markuplm.py::MarkupLMFeatureExtractionTest::test_init_without_params
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hi @sgugger I think I misread your message, and end up putting the installation in the CircleCI `config.yml` and GitHub workflow files. Should I instead put `beautifulsoup4` in `extras["testing"]`? That way might be much easier.

Yes I meant to add it in `extras["testing"]`.

test failure irrelevant, merge now.
Fix Encoder-Decoder testing issue about repo. names
Author: ydshieh
Pull request: yes

# What does this PR do?

Some tests for encoder-decoder models use repo. names `../gpt2`. It works before `huggingface-hub==0.10`, but failed with this version.

This PR changes `../gpt2` to `gpt2`. In any case, it is less confusing even if `../gpt2` worked before.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Also flagging to @Wauplin that this is another breaking change in the last release that is not spelled out in the [release notes](https://github.com/huggingface/huggingface_hub/releases/tag/v0.10.0).

Same answer as [here](https://github.com/huggingface/transformers/pull/19252#issuecomment-1263651027).
Docs - Guide to add a new TensorFlow model
Author: gante
Pull request: yes

# What does this PR do?

Adds a guide on how to add a new TensorFlow model (architecture and weights), as well as a few tips to debug TF<>PT mismatches. This means that we can stop relying on ad hoc tips and instructions to add TF models, and point towards this guide instead 🙌 

I've also updated the test paths in the guide to add new models, which were outdated, and added a cheeky "from" in the "Converting from TensorFlow checkpoints" guide title to clearly distinguish it from the guide I'm adding in this PR.

Review suggestion: read the [live docs](https://moon-ci-docs.huggingface.co/docs/transformers/pr_19256/en/add_tensorflow_model) themselves :)

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Awesome! I will merge the PR soon and make some guide-related announcements next Monday 🙌
Tokenizer Batch decoding  of predictions obtained from model.generate in t5
Author: rohanshingade
Pull request: no

How to do batch decoding of sequences obtained from model.generate in t5? Is there a function available for batch decoding in tokenizer `tokenizer.batch_decode_plus` similar to batch enocding `tokenizer.batch_encode_plus`?

Comments:
There is `batch_decode`, yes, the docs are [here](https://huggingface.co/transformers/internal/tokenization_utils.html?highlight=batch_decode#transformers.tokenization_utils_base.PreTrainedTokenizerBase.batch_decode).

@sgugger I wonder if we shouldn't make the docs of this method more prominent? The "Utilities for tokenizer" page mentions: "Most of those are only useful if you are studying the code of the tokenizers in the library.", but `batch_decode` and `decode` are only found here, and are very important methods of the tokenization pipeline.

We should add them to the `PreTrainedTokenizer` and `PreTrainedTokenizerFast` documentation. Or did you want to add them to all models?


@LysandreJik `tokenizer.batch_decode` and `tokenizer.decode` in loop, both the functions take almost the same time. can you suggest something, how can I speed up the decoding in T5? why is batch_decode not as fast as batch_encode_plus? Is there a way to make decoding even faster?

Unfortunately we have no way to go faster than that.

@LysandreJik this function is used in compute_metrics. and it seems it is limited to the number of GPUs ( it uses the same number of `--nproc_per_node` when doing ddp training, how is it possible to extend that to the maximum number of cores) any guide on how to fix and maybe do a PR?
Docstring says target_sizes is optional (in DETR post_process) but the code disagrees
Author: amgreene
Pull request: no
Labels: name: Good First Issue

name: Good First Documentation Issue

https://github.com/huggingface/transformers/blob/db7d6a80e82d66127b2a44b6e3382969fdc8b207/src/transformers/models/detr/feature_extraction_detr.py#L677

The Docstring marks the `target_sizes` parameter as `*optional*` but the parameter has no default value and is used without any conditional checking for whether it is `None`, so it would seem to be required.

Comments:
Thanks for pointing that out. Could you open a PR to remove the "optional" string?

Is this resolved?

This issue should be marked closed. The problem seems to be resolved.

We're actually updating this at the moment (see #19205), it will become optional. Closing this issue for now.
time series forecasting model
Author: kashif
Pull request: yes

# What does this PR do?

This PR implements a vanilla encoder-decoder Transformer for time-series forecasting.


Comments:
_The documentation is not available anymore as the PR was closed or merged._

cc'ing @mishig25 here - there seems to be an issue with the docs being built. The model is added to the toctree, but it's saying:

```
Traceback (most recent call last):
  File "/usr/local/bin/doc-builder", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/doc_builder/commands/doc_builder_cli.py", line 47, in main
    args.func(args)
  File "/usr/local/lib/python3.8/site-packages/doc_builder/commands/build.py", line 96, in build_command
    build_doc(
  File "/usr/local/lib/python3.8/site-packages/doc_builder/build_doc.py", line 427, in build_doc
    sphinx_refs = check_toc_integrity(doc_folder, output_dir)
  File "/usr/local/lib/python3.8/site-packages/doc_builder/build_doc.py", line 482, in check_toc_integrity
    raise RuntimeError(
RuntimeError: The following files are not present in the table of contents:
- model_doc/time_series_transformer
Add them to ../transformers/docs/source/en/_toctree.yml.
```

Mishig the failure on the doc was due to a typo (comment is hidden now since the suggestion was accepted) nothing to do for you :-)
AttributeError: 'T5Config' object has no attribute 'exponential_decay_length_penalty'
Author: Meet090201vora
Pull request: no

File "c:\users\meet vora\appdata\local\programs\python\python38\lib\site-packages\streamlit\scriptrunner\script_runner.py", line 443, in _run_script
    exec(code, module.__dict__)
File "C:\Users\MEET VORA\Desktop\fiver\ui.py", line 140, in <module>
    pred_summary = summarizeText(input_text)
File "C:\Users\MEET VORA\Desktop\fiver\ui.py", line 120, in summarizeText
    generated_ids = model.generate(
File "c:\users\meet vora\appdata\local\programs\python\python38\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
File "c:\users\meet vora\appdata\local\programs\python\python38\lib\site-packages\transformers\generation_utils.py", line 1259, in generate
    logits_processor = self._get_logits_processor(
File "c:\users\meet vora\appdata\local\programs\python\python38\lib\site-packages\transformers\generation_utils.py", line 727, in _get_logits_processor
    else self.config.exponential_decay_length_penalty
File "c:\users\meet vora\appdata\local\programs\python\python38\lib\site-packages\transformers\configuration_utils.py", line 257, in __getattribute__
    return super().__getattribute__(key)

Comments:
Hi,

It's pretty hard for us to figure out what the error is in case there's no code provided that can reproduce your issue.

Can I send you the code?

On Fri, 26 Aug, 2022, 1:11 pm NielsRogge, ***@***.***> wrote:

> Hi,
>
> It's pretty hard for us to figure out what the error is in case there's no
> code provided that can reproduce your issue.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/18761#issuecomment-1228162236>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ARIATPLZPZGZQPJHYYXQTVTV3BYI3ANCNFSM57S2BFHA>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fixed the error so the default matches with the coment
Author: Soham-West
Pull request: yes

Made it so that the num_output_group matches with the default setting.

Comments:
cc @xvjiarui @NielsRogge 

LGTM

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_18561). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
New Model: Charformer: Fast Character Transformers via Gradient-based Subword Tokenization 
Author: neel04
Pull request: no
Labels: name: New model

# 🌟 New model addition

## Model description

arXiv = https://arxiv.org/pdf/2106.12672.pdf (pre-print; under review)

In this paper, they introduce a soft gradient-based subword tokenization module (GBST) that automatically learns latent subword representations from characters in a data-driven fashion. More importantly, is the introduction of Charformer, a deep Transformer model that integrates GBST and operates on the byte level. 

> Via extensive experiments on English GLUE, multilingual, and noisy text datasets, we show that Charformer outperforms a series of competitive byte-level baselines while generally performing on par and sometimes outperforming subword-based models. Additionally, Charformer is fast, improving the speed of both vanilla byte-level and subword-level Transformers by 28%-100% while maintaining competitive quality. We believe this work paves the way for highly performant token-free models that are trained completely end-to-end.

## Open source status

* [ ] the model implementation is available: ----] [Implementation and weights](https://github.com/google-research/google-research/charformer)
* [ ] the model weights are available: ------------] [to be released soon here](https://github.com/google-research/google-research/charformer)
* [x] who are the authors: Yi Tay, Vinh Q. Tran, Sebastian Ruder*, Jai Gupta, Hyung Won Chung, Dara BahriZhen Qin, Simon Baumgartner, Cong Yu, Donald Metzler (Google and DeepMind-->`*`)


Comments:
Code is out now:

https://github.com/google-research/google-research/tree/master/charformer (please note the different url - compared to paper url)

An unofficial PyTorch implementation for Charformer https://github.com/lucidrains/charformer-pytorch

Thanks for the great work!
Will charformer be supported in the near future?


Still not supported yet
Restructure DETR post-processing, return prediction scores
Author: alaradirik
Pull request: yes

# What does this PR do?
- Restructures `DetrFeatureExtractor.post_process_instance_segmentation` and `DetrFeatureExtractor.post_process_panoptic_segmentation` methods, adds `score` key to output dictionaries
- Updates feature extractor tests

## Before submitting
- [ X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [X ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
     This was discussed on [Notion](https://www.notion.so/Post-processing-of-vision-models-33a8165333144581862014d83e96191b?d=462e95b396fc4ea49ffa602146bd91f5#d152a0d70b4c4cf599f642e9b891dba1).
- [ X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?



Comments:
_The documentation is not available anymore as the PR was closed or merged._
Update no_trainer script for summarization
Author: divyanshugit
Pull request: yes

# What does this PR do?

Update `run_summarization_no_trainer.py` in the example to include `accelerator.gather_metrics`

Fixes a part of #18437

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@muellerzr , @sgugger 
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.


Comments:
_The documentation is not available anymore as the PR was closed or merged._

> Thanks a lot for your PR! You also need to remove the initialization of `samples_seen` since that variable is not used anymore. This will fix the quality check.

Done; thanks for pointing it out.
'UserWarning: Module is put on CPU' when use FSDP by Accelerate
Author: scuyjzh
Pull request: no

script: https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_mlm_no_trainer.py

accelerate launch run_mlm_no_trainer.py \
    --model_name_or_path bert-base-chinese \
    --train_file $DATA_DIR/train.txt \
    --validation_file $DATA_DIR/val.txt \
    --line_by_line True \
    --per_device_train_batch_size 16 \
    --per_device_eval_batch_size 16 \
    --output_dir $MODEL_DIR/test-mlm

![image](https://user-images.githubusercontent.com/12816241/186734597-0c70137b-bfdb-44a4-add0-3a44ef6bc71e.png)

![image](https://user-images.githubusercontent.com/12816241/186734845-c99752c7-050c-49a5-abcf-2a266c32d238.png)

UserWarning: Module is put on CPU and will thus have flattening and sharding run on CPU, which is less efficient than on GPU. We recommend passing in `device_id` argument which will enable FSDP to put module on GPU device, module must also be on GPU device to work with `sync_module_states=True` flag which requires GPU communication.

**How can I solve this problem?** 



Comments:
Maybe cc @pacman100 ? :)

Hello @scuyjzh ,  as mentioned in https://github.com/huggingface/accelerate/issues/659, you can safely ignore that warning as it is only during model initialization under FSDP. I will look into this later as and when time permits because this doesn't impact model correctness or performance gains through fsdp.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.