noamaanMulla-03/transformers-issues huggingface dataset:
CoNLL2003 ner_tags order mismatch between the dataset from HF and the pretrained model
Author: vshampor
Pull request: no

@patrickvonplaten @dslim23 

@dslim23 's pretrained models such as:
https://huggingface.co/dslim/bert-base-NER
have the following NER tag order baked in:
`"O", "B-MISC", "I-MISC", "B-PER", "I-PER", "B-ORG", "I-ORG", "B-LOC", "I-LOC"`

while the https://huggingface.co/datasets/conll2003 dataset has:
`O (0), B-PER (1), I-PER (2), B-ORG (3), I-ORG (4) B-LOC (5), I-LOC (6) B-MISC (7), I-MISC (8)`

The mismatch leads to defunct accuracy measurements out of the box for the pretrained NER models; try, for instance:
`python examples/pytorch/token-classification/run_ner.py --model_name_or_path dslim/bert-base-NER --dataset_name conll2003 --output_dir /tmp/test-ner --do_eval`



Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

@julien-c - another good example for HF Hub issue support haha

@vshampor - let's see what @dslim23 thinks :-)

@patrickvonplaten thanks for the ping – though in the case it's the script that should be able to remap labels no? The model looks correctly defined with https://huggingface.co/dslim/bert-base-NER/blob/main/config.json

(a bit like the roberta case that's hardcoded in example scripts when it should be supported automatically?)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Ok sorry only took a closer look now. I think what happened here is that the model was trained with an older version of `run_ner.py` where the `id2label` was not adapted to the dataset yet which then leads to problems when one only wants to evaluate the model. @sgugger - should we maybe add some logic like:

"If --do_train is False and config has id2label list then use that" instead of forcing to overwrite it depending on the dataset?

That logic is in the text classification examples but not the token classification examples indeed. Will try to fix this during the week (or did you want to do it?)

> That logic is in the text classification examples but not the token classification examples indeed. Will try to fix this during the week (or did you want to do it?)

Happy to take it over - I have some free time this week

Go ahead then :-)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Avoid using get_list_of_files
Author: sgugger
Pull request: yes

# What does this PR do?

This PR removes all uses of `get_list_of_files` which uses a non-optimized call on the Hub, to only rely on the API to get files (which is better optimized). It should make the general `from_pretrained` APIs much more stable (cc @julien-c ).

To refactor some of the code we often get to just open a config, this PR adds a new utility in file_utils that will download and cache a given file from a repository and return the resolved path, but only fails if the repository or the revision does not exist. When the file itself does not exist, the function returns None, which make it easy for us to test if a given file (like a tokenizer config) is present or not, without relying on `get_list_of_files`.

The other case where this function was used is when we have a versioned config/tokenizer config in a given repo. In this instance, it will now be necessary to include the list of available configs/tokenizers in the config/tokenizer config so that we don't have to call `get_list_of_files`. There is only one repo in the wild (tested every one) that used this API (which is the last recourse to fix some backward compatibility issues), [microsoft/layoutxlm-base](https://huggingface.co/microsoft/layoutxlm-base), and I have manually fixed its config (the whole change is backward compatible in the sense that this repo will still work with older versions of Transformers).

Comments:
_The documentation is not available anymore as the PR was closed or merged._

We can definitely port it in a PR to `huggingface_hub`. I would leave it in this PR for now and wait for a release on `huggingface_hub` to replace it by the one there.
[Tests] Fix test
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR fixes the failing SwinModelIntegrationTest.test_inference_image_classification_head.

It also makes sure the appropriate device is used in one of ViLT's integration tests, although this test still fails due to the non-deterministic behaviour of the model. Setting torch.manual_seed(2) didn't help (it passes locally for me, but not on another machine).

Also removes a print statement in ViTMAE's test file.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Dependency parsing head for pretrained models
Author: saattrupdan
Pull request: no
Labels: name: Feature request

# 🚀 Feature request

Add a new classification head for pretrained models, for dependency parsing.

## Motivation

Current heads, such as `AutoModelForTokenClassification`, does not work well for finetuning a pretrained model for dependency parsing. [There are now such heads available](https://openreview.net/pdf?id=Hk95PK9le), so adding such would make it a lot easier to finetune models for such tasks. A `PyTorch` implementation of that paper (and others) can be found [in this repo](https://github.com/yzhangcs/parser).

## Your contribution

I could assist with a PR, but for now I'd like to start the discussion to see if this is something that the HF team and others would be interested in.

Comments:
Dan, did you start working on this? I would be glad to help with this feature.

Why me :| @huberemanuel 

Sorry, wrong mention

@huberemanuel I have a prototype implementation currently, but it probably needs a bunch of tweaking. You can find it at https://github.com/saattrupdan/ScandEval/blob/dependency-parsing/scandeval/dependency_parsing/parser.py#L17.

Right now it's basically a hack which tweaks the `forward` method of the `AutoModelForTokenClassification`. A proper implementation would use the underlying code instead of subclassing it, of course!

If you're interested in testing it and improving the implementation then let me know and we could work it out together 🙂 

Thanks! I will take a look into it and see how can I contribute to your initiative.

Hey Dan, just to send an update about this topic. I messed around with the `transformers` code and added an `AutoModelForDependencyParsing` class which for now is just a copy of `AutoModelForTokenClassification`. With that, I ran the NER training script, just to do a sanity test on token classification training. It all worked well and now I know which parts we will need to modify to add this feature. 
I will send more updates as I progress, and when I got to the point to merge your code with the `transformers` internals, we can discuss what design is better. If you advance more, please let me know :) 

@huberemanuel Amazing, great stuff! With the internals in place, it's hopefully doable to tweak it in (roughly) the way I did. Thanks for all the work you're putting into it, and would love to see your code when you're ready to share it! 😊 

@huberemanuel Any news on this?
GPT-NeoX checkpoint conversion?
Author: frankxu2004
Pull request: no

I found that there are conversion from gpt-neo and gpt-J to transformers. I wonder if the checkpoints trained using gpt-neox https://github.com/EleutherAI/gpt-neox could be loaded inside transformers? I think it's the same architecture as GPT-neo but I could be wrong. Thanks!

Comments:
This is something we're working on at eleuther in preparation for an upcoming release, and already have some working code for specific configurations, but for other configurations it breaks.

If it would be useful to HF, we can post up the code :) 

@sdtblck This sounds really useful and I'd like to take a look too! I wonder if you could share what configuration it works on? Right now I have a trained model that's using the 2.7B.yml and I'd like to do inference with HF (no training is fine). Thank you so much!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

I wonder if there are any updates on this @sdtblck ?
Any help would be greatly appreciated!
Fix table formatting in SegFormer docs
Author: deppen8
Pull request: yes

# What does this PR do?

This PR fixes the Markdown table on the main SegFormer doc page. It was missing the line below the header that makes it a valid Markdown table.

Fixes #15334 


## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

@sgugger, @NielsRogge

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix 'eval_split_name' described as defaulting to 'train'
Author: FremyCompany
Pull request: yes

# What does this PR do?

The default is correct (`test`) but the description is not.


## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

Cc: @sgugger, @patil-suraj

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Using HuggingFace Models for Text Translation of sensitive data
Author: prateekralhan
Pull request: no

Hello folks,

Wish you all a belated happy new year! 🎉

First of all, thanks a ton for creating such transformers which have "transformed" the entire NLP landscape. 🤗

### Objective 🎯
I am working on developing a web application for performing multilingual translation using mBart and transformers of textual information found in documents - PDFs, word files etc. A basic prototype has been implemented [here.](https://github.com/prateekralhan/Multilingual-Translator)

### Question/Doubt ⁉
I am planning to scale this up and use it for handling/processing extremely sensitive data at enterprise level. My question is to get an understanding how do the APIs are storing the responses returned by the models which are present in the HF servers.

1. Is the data copied to the backend servers via the browser, and contained in a BytesIO buffer in Python memory (i.e. RAM, not disk) and the data will persist in RAM until my app re-runs from top-to-bottom??
2. Is the data erased from memory when the user uploads another file, replacing the original one or when the user closes the browser tab/exits the web app?
3. Or is there any time period for which the data gets stored in the backend servers and then gets erased post that duration?

Would love to get complete idea on this. 

### Who can help
@patrickvonplaten

Sincere apologies from my end should I unknowingly followed wrong protocol for raising such a doubt. Please let me know if you need any further information from my end.

Cheers,
Prateek

Comments:
You're using the inference API, right? cc @Narsil 

Hi @prateekralhan ,

Your demo code doesn't use the API but seems to be using Streamlit, so I guess you envision using Spaces ?

If that's the case, I suggest you look a bit more into the API https://api-inference.huggingface.co/docs/python/html/index.html since Spaces currently has no commercial offering.

For the API.
Currently, all requests are logged in our secure servers.
Access is only granted to API maintainers, to help debug actual bugs (somes bugs are caused by specific requests, so we need to know what they are to trigger them again and fix them). It's also used as a backup for accounting so we can always recount the API use of a given user should the need arise (didn't so far to my knowledge).

The logs are never used for any other reason than that, not sold, not examinated, not used as a dataset source, and access is limited to a handful of employees.

That being said, if you plan to use the API extensively, being on a custom plan, you can also asks for logs to not be written on your account.  You can reach out to api-enterprise@huggingface.co (This isn't something you can do self served).
As for the sensitivity of your data, you seem to want extra security, this is something that should better discussed during a call with our teams I think.

@Narsil  and @LysandreJik , thank you for the clear explanation. This answers my doubt! 😄
Add YOSO
Author: novice03
Pull request: yes

# What does this PR do?

This PR adds the YOSO Transformer model to the repository.

Paper: [You Only Sample (Almost) Once: Linear Cost Self-Attention Via Bernoulli Sampling](http://proceedings.mlr.press/v139/zeng21a/zeng21a.pdf)
Code: [Official code](https://github.com/mlpen/YOSO) 
Checkpoints: [Uploaded to hub](https://huggingface.co/uw-madison/yoso-4096)

In this paper, the authors introduce a an efficient self-attention mechanism based on Locality Sensitive Hashing. They also implemented custom CUDA kernels in their [code](https://github.com/mlpen/YOSO/blob/6000487d9cd8e34519aa47650011f458ca0db64c/encoders/backbones/efficient_attentions/yoso/yoso_v1/kernel.py#L15). I've incorporated this in the modeling file, but some code qaulity checks fail.

## Who can review?

@NielsRogge @patrickvonplaten 

Comments:
Hey @novice03,

Thanks a lot for adding this complicated model! Looks super cool :-) Also great to see a revival of the LSH mechanism for long-range sequence modeling after Reformer.

Left a couple of suggestions above. It would be great if:

- We could remove the whole `use_cache` logic as the model is not able to use it
- Add one integration test with long inputs so that we can be sure that the model can process long inputs on a single GPU

Thank you for your feedback! I've pushed a few commits with the appropriate changes.


Oh, I ran `doc-builder convert` on the file and assumed that it fixed everything. Turns out there were still some parts of the code that weren't converted. Should be fixed now.

Looks good for merge to me! 

Thanks for all your work on this! Merging.
Fix deepspeed docs
Author: ngoquanghuy99
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes typos and misspellings in deepspeed docs.





## Who can review?

@sgugger @stas00 @LysandreJik 


<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
_The documentation is not available anymore as the PR was closed or merged._

I'm sorry I overlooked mistakes in those two sentences. It's true those are not fixes, though the first one is not gramatically correct anyway.

Do as you wish to fix those, I'll make sure to stay very far away form the deepspeed file in the future.

I tried revert, but I thought it'd magically resume this PR and it didn't, so it'd be a bad workflow. So if I have to make a new PR, it's much simpler to just do a new PR with just the fixes: https://github.com/huggingface/transformers/pull/15355
Push to hub save
Author: sgugger
Pull request: yes

# What does this PR do?

This PR changes the documentation of the `push_to_hub` training argument to be in sync with the actual behavior, and also makes sure that when `push_to_hub=True`, a push is done every time the model is saved.

Fixes #15313

Comments:
_The documentation is not available anymore as the PR was closed or merged._

I think you're missing the fact that, apart from a push asked explicitly with `trainer.push_to_hub()` (or now `trainer.save_model()`) all other pushes are only done if the previous one is completely finished (see [here](https://github.com/huggingface/transformers/blob/637e81752aad01738d36ef816fadee21fa392317/src/transformers/trainer.py#L2712)). So, there are no frequent pushes (except if the user asks for it by putting `trainer.save_model()` in a loop but I don't think that will happen).

Then that's perfect :) 

Ugh, sorry @osanseviero I was convinced I had accepted your suggestions before merging (must have skipped one click), will make a new commit with you as Co-author...
Fix KerasMetricCallback prediction with generate() and inference of column names
Author: Rocketknight1
Pull request: yes

A number of fixes to the Keras metric callback based on testing with the notebooks. I think I've caught all the implementation differences, and in most cases we can now pass exactly the same `compute_metrics` function to this callback as to `Trainer`.

I'm still in the process of overhauling the notebooks, so I might need to make a few more tweaks to this PR if I encounter any other problems!

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix encoder-decoder models when labels is passed
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix encoder-decoder models when labels is passed and `return_dict` is `False`.

## Details

This line 

https://github.com/huggingface/transformers/blob/669e3c50c98ad5b506555a551d2ecbf72ceb3c99/src/transformers/models/encoder_decoder/modeling_encoder_decoder.py#L526

fails when `labels` is passed to `EncoderDecoderModel` + `return_dict = False`
Since we don't pass `labels` to decoder, `decoder_outputs` won't return `loss`. When `return_dict` is `False`, `logits` will be the first element in the returned tuple `decoder_outputs`.

## The issue

```
import os, tempfile
from transformers import EncoderDecoderModel, BertConfig, AutoTokenizer, BertModel, BertLMHeadModel


config = BertConfig(num_hidden_layers=2, hidden_size=4, num_attention_heads=2, intermediate_size=4)
enc = BertModel(config)
dec = BertLMHeadModel(config)

with tempfile.TemporaryDirectory() as tmpdir:
    enc.save_pretrained(os.path.join(tmpdir, "enc"))
    dec.save_pretrained(os.path.join(tmpdir, "dec"))
    enc_dec = EncoderDecoderModel.from_encoder_decoder_pretrained(
        os.path.join(tmpdir, "enc"), os.path.join(tmpdir, "dec")
    )

enc_dec.config.pad_token_id = 0
enc_dec.config.decoder_start_token_id = 1

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

enc_text = "i love cats"
dec_text = "my cat is cute"

enc_inputs = tokenizer(enc_text, return_tensors="pt")
dec_inputs = tokenizer(dec_text, return_tensors="pt")

# This fails
inputs = {
    "input_ids": enc_inputs["input_ids"],
    "labels": dec_inputs["input_ids"]
}
outputs = enc_dec(**inputs, return_dict=False)
```
This gives the error
```
    loss = loss_fct(logits.reshape(-1, self.decoder.config.vocab_size), labels.view(-1))
AttributeError: 'tuple' object has no attribute 'reshape'
```

## Who can review?

@NielsRogge 

Comments:
@NielsRogge Could you take a look whenever you have sometime? Or tag another person if you think more properly. Thanks!

@NielsRogge - would be amazing if you could wait a day or so here so that I can quickly take a look as well before merging :-)
Adafactor lacking min_dim_size_to_factor
Author: ZhaofengWu
Pull request: no

The Adafactor implementation in mesh_tensorflow has a `min_dim_size_to_factor` flag where tensors with size smaller than this will not be factored.
https://github.com/tensorflow/mesh/blob/57ed4018e6a173952501b074daabad32b6449f3d/mesh_tensorflow/optimize.py#L220

However, this flag is lacking in HF's version. And there are indeed parameters that are smaller than this, for example, T5's `relative_attention_bias`.

First of all, this causes an implementation mismatch. Secondly, this prevents the loading of pretrained optimizer states into HF.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.