noamaanMulla-03/transformers-issues huggingface dataset:
BERT with sequence pairs & padding
Author: domaala
Pull request: no

## ❓ Questions & Help

I am having trouble understanding how to setup BERT when doing a classification task like STS, for example, inputting two sentences and getting a classification of some sorts. I am using `BertForSequenceClassification` for this purpose. However, what boggles me is how to set up `attention_mask` and `token_type_ids` when using padding.

Let's assume two sentences: `I made a discovery.` and `I discovered something.`

Currently, I'll prepare the input as follows (assume padding).

1. Input IDs (encoded): `[CLS] I made a discovery. [SEP] I discovered something. [SEP] [PAD] [PAD] [PAD]`
2. `token_type_ids`: everything `0` by the first `[SEP]` (also included), **after** which everything will be marked as `1` (padding included).
3. `attention_mask`: `1` for everything but the padding.

And, of course, labels are trivial as they are not affected by padding. Anything wrong with my setup? Am I missing anything?

Comments:
Hi! Yes, I think your understanding is correct. Your setup seems fine to me!

regarding the token_type_ids, shall we mark [PAD] token as 0 or 1? [PAD] by default does not belong to any of the two input sequences. Therefore it is ambiguous to determine whether it should be 0 or 1.

The default padding value for `token_type_ids` is 0 which is defined by `tokenizer._pad_token_type_id`. You can specify it to 1 by `tokenizer._pad_token_type_id=1`.
[urls to hub] Replace outdated model tags with their now-canonical pipeline types
Author: julien-c
Pull request: yes

> ### [Internal tracker 🌔 ](https://github.com/huggingface/moon-landing/pull/1558)

#### Context:

On the hub side, we are using the metadata generated by transformers (hat/tip @sgugger) to replace the pattern matching, BUT I'm also proposing to take this opportunity to remove the `lm-head`, `seq2seq`, `causal-lm`, `masked-lm` auto-generated tags (it's still possible to add them manually to a model card, of course)

(they are superseded by the pipeline tag aka. what we call the Task on hf.co/models)

This might break some stuff though, which is why your feedback will be important. 

Comments:
also cc'ing @osanseviero for info before merging!
Add mLUKE
Author: ryokan0123
Pull request: yes

# What does this PR do?
This PR adds mLUKE, [mLUKE: The Power of Entity Representations in Multilingual Pretrained Language Models](https://arxiv.org/abs/2110.08151), by Studio Ousia.

The model is a multilingual extension of [LUKE](https://huggingface.co/transformers/model_doc/luke.html), which is trained on the Wikipedia articles in 24 languages.
We have uploaded the model weights on [the hugging face hub](https://huggingface.co/studio-ousia/mluke-base).

The only difference from LUKE is the tokenizer, so I’ve just added `MLukeTokenizer`, which is a mix of `LukeTokenizer` and `XLMRobertaTokenizer`.

I’ve also added `LukeForMaskedLM` to perform the cloze prompt task experimented in [the original paper](https://arxiv.org/abs/2110.08151).
The example can be found in [this notebook](https://colab.research.google.com/drive/1FAJimyuzUefZyWS1ckSEitglLfFRofi4?hl=ja#scrollTo=jIXV3onW2ITm).

## Who can review?
@NielsRogge, could you help us review the code as you have worked on LUKE?🙂

Comments:
Thank you for reviewing, @NielsRogge!
I’ve reflected the reviews to the code and also added mLUKE to the Multilingual-models section in the doc.

Thanks for taking into account the comments. Can you rebase with master? The index.rst file is now replaced by an [index.mdx](https://github.com/huggingface/transformers/blob/master/docs/source/index.mdx) file because of the new front-end website.

Also, in the main README, make sure the links are correct (they should not include the .html and have a docs in the url as we will move to https://huggingface.co/docs/transformers). You can then run make fix-copies to fix the other READMEs.

I think there was something wrong in your rebase, and the diff now shows much more than what you are actually changing. Could you close your PR and reopen a fresh one from your branch?

Sorry, seems like I messed it up.
I will reopen a new pull request.
[Hub] 403 when trying to download models
Author: etiennedi
Pull request: no

Cross-posting from https://github.com/UKPLab/sentence-transformers/issues/1297 as I'm not sure if this is an issue with the `SentenceTransformers` library or with the model hub directly.

Similar to the author of the linked post, we are also running into this both locally and from CI, so it doesn't seem to be one particular blocked or rate-limited IP. Interestingly enough, I am not running into this issue when using `huggingface/transformers` directly, but only when it's wrapped through `SentenceTransformers`. 

Comments:
Update: The original author found that the [same error also occurs on non SBERT-related activity](https://github.com/UKPLab/sentence-transformers/issues/1297#issuecomment-986724370), suggesting that it is indeed an issue on the model hub directly.

Thanks in advance for timely investigation and fix, as I assume this is blocking a lot of users at the moment :) 

We're also affected, our `all-MiniLM-L6-v2` model is not working in production. Are there any workarounds? How can we download them manually and load them locally instead?

In case this helps with the investigation, I can also reproduce this in the browser:

- This page works fine: https://huggingface.co/sentence-transformers/clip-ViT-B-32/tree/afe976e3f4ea04b633edb0089a7e5088ddbe9212
- When I now click on `.gitattributes`, I am directed to a page that returns `403`: https://huggingface.co/sentence-transformers/clip-ViT-B-32/blob/afe976e3f4ea04b633edb0089a7e5088ddbe9212/.gitattributes (Update: It's not really specific to `.gitattributes`, it's any files, the file with the leading dot just happens to be the first that is checked)

@hfjallemark 
You can download all models from here:
https://public.ukp.informatik.tu-darmstadt.de/reimers/sentence-transformers/v0.2/

Unzip them to a path and load from disc.

We currently look into this

Thanks @nreimers -- managed to do that and load locally.

> We currently look into this

Thanks @nreimers for the quick response. If there is any way we can help, do let us know!

Same here with `stsb-distilbert-base`
Digging in code, found that server says model should contain files [".gitattributes", "README.md"] which are 403
Ugly workaround - just skip them by adding at `site-packages/sentence_transformers/util.py`@434 (`snapshot_download` func, loop over `.sublings`) smth like
```
if model_file.rfilename in [".gitattributes", "README.md"]:
      print("[adhoc hack] Skip ", model_file.rfilename)
```

Hope you will fix that 

The download from the hub should be working again.

Let me know if you still have issues.

I can confirm that the download works again. Thanks to you and everyone on the team for the quick action! 

Post-mortem: AWS WAF (Web Application Firewall) had a new ruleset merged today (https://docs.aws.amazon.com/waf/latest/developerguide/aws-managed-rule-groups-list.html) that didn't like our URL structure
Couldn't reproduce DistilBERT downstream tasks performance on SQuAD dataset.
Author: liz-badada
Pull request: no

Hi, 

I tried the DIstilBERT finetuning (script path: transformers/examples/tensorflow/question-answering) for QuestionAnswering downstream task without changing any code, and this is my command:

`
python3 run_qa.py --model_name_or_path distilbert-base-uncased --output_dir output --dataset_name squad --do_train --do_eval
`

And finally, I got the following performance: SQuAD(EM/F1): 72.441/81.188

But the performance was shown on the paper: SQuAD(EM/F1): 79.1/86.9

Is there anything wrong with my command?

transformers version: 4.11.3

Comments:
The `run_qa.py` script is very general, to be used by any `xxxForQuestionAnswering` model. To reproduce the results of a specific paper, you may need to tweak the hyperparameters, such as number of epochs, learning rate, batch size.

I see, thanks! @NielsRogge 

May I get the recommended hyperparameters to reproduce the performance of SQuAD(EM/F1): 79.1/86.9.

The DistilBERT author (@VictorSanh) used the (now legacy) `run_squad.py` script which can be found [here](https://github.com/huggingface/transformers/tree/master/examples/legacy/question-answering). So you might take a look at the hyperparameters used there. The following command is still included in the README of the question-answering examples:

```
python run_squad.py \
  --model_type distilbert \
  --model_name_or_path distilbert-base-uncased \
  --do_train \
  --do_eval \
  --do_lower_case \
  --train_file $SQUAD_DIR/train-v1.1.json \
  --predict_file $SQUAD_DIR/dev-v1.1.json \
  --per_gpu_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir /tmp/debug_squad/
```
So might try this out. The checkpoint is also available on the hub: https://huggingface.co/distilbert-base-cased-distilled-squad

@NielsRogge Thanks so much!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Update the example of exporting Bart + BeamSearch to ONNX module to resolve comments.
Author: fatcat-z
Pull request: yes

This is a PR to resolve all of comments left in previous PR:

https://github.com/huggingface/transformers/pull/13765

Also provides a README.md for this example.

Comments:
@NielsRogge  @garymm  

Please help to take a review. Thanks.

@michaelbenayoun 
 Could you please help to take a review? Thanks!

@garymm Please help to take a review.

Thanks for integrating all the feedback @fatcat-z ! Is it OK if I merge this?
Add GPTJForQuestionAnswering
Author: tucan9389
Pull request: yes

# What does this PR do?

- Add `GPTJForQuestionAnswering` class for GPTJ upstream and QA downstream task

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@patrickvonplaten, @LysandreJik
@sgugger, @patil-suraj


Comments:
Hi, @sgugger and @patrickvonplaten again.
Like the https://github.com/huggingface/transformers/pull/13290 PR that I made before, I made QA for GPT-J class and made this PR for that.
If you have time, I'd appreciate your review and feedback on it.

This seems good to me - @patil-suraj, do you mind taking a look at this?

Looks good to me as well! @patil-suraj - could you also take a quick look? :-)

Thanks again for your PR!
doc: mismatch between pooler/d_output
Author: guhur
Pull request: yes

The model outputs a pooler_output whereas the doctype examples were using a pooled_output.

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

@sgugger



Comments:
fix flax example tests
Author: patil-suraj
Pull request: yes

# What does this PR do?

The `conftest.py` was missing from the flax examples directory. so the flax example tests are failing on master.
(cf [here](https://app.circleci.com/pipelines/github/huggingface/transformers/30924/workflows/7c0b3b8b-d664-4dc2-9c7e-69520347f0d5/jobs/317301))

This PR adds the `conftest.py` file in the flax example directory.

Comments:
Add a job to test the documentation build
Author: sgugger
Pull request: yes

# What does this PR do?

This PR adds a GitHub action to test the documentation build. While we will in the future have something to check the built documentation, it's important to test right now that there is no failure in the build at least, even if we can't see the result.

Comments:
The caching only gets us one minute back in the setup :-(
Once the dummies have the docstrings of their real counterpart, we can stop installing all dependencies and have better timing though!
Add mLUKE
Author: ryokan0123
Pull request: yes

# What does this PR do?

Reopen the following PR:
https://github.com/huggingface/transformers/pull/14570

I have reflected the comments in the previous PR (add `# Copied from ...` comments, replace `assert` with `ValueError`).
I have also added few tests to make sure `LukeTokenizer` and `MLukeTokenizer` correctly handle padding with entity inputs.

## Who can review?
@NielsRogge  @LysandreJik  @sgugger 

Comments:
It looks like there a quite a few failures in the Luke tests. You can see them locally with:
```
pytest tests/test_modeling_luke.py
```
from the root repo.

Thanks, I’ve fixed that.
I still see some tests failed in CI (`run_tests_tf`), but I guess it is not from the changes in this PR?

Yes, they're timing out for some reason, it's not a failure due to your PR.

Thanks a lot for adding this new model!
[deepspeed] fix --load_best_model_at_end
Author: stas00
Pull request: yes

This PR is making `--load_best_model_at_end` reported in https://github.com/huggingface/transformers/issues/14628 work.

There is a problem with resuming from a checkpoint after deepspeed engine was used https://github.com/microsoft/DeepSpeed/issues/1394 - I made a new Issue there to stress out that this is a problem. https://github.com/microsoft/DeepSpeed/issues/1612

This PR:
- adds a workaround that re-creates the deepspeed engine when resuming from `--load_best_model_at_end`
- fixes the timing when deepspeed checkpoint loading kicks in
- adds a test
- adds a commented out test that requires fixing on the deepspeed side

Additionally:
- I added a saving of the DS checkpoint on model_save - I think that would be useful to some users - it's a very quick operation.

Fixes: https://github.com/huggingface/transformers/issues/14628

@sgugger 

Comments:
Encoder Decoder Model Generation Issue
Author: dobby-seo
Pull request: no

Thank you for your great works!

I'm customizing Enc-Dec model, but it can't get additional parameters. 

How about modifying from below, 
https://github.com/huggingface/transformers/blob/0ddadbf0a82c3d9f3df864e1302e3f04e25c2967/src/transformers/models/encoder_decoder/modeling_encoder_decoder.py#L471

to
```
decoder_inputs = self.decoder.prepare_inputs_for_generation(input_ids, past=past, **kwargs)
```

Comments:
Hi! It should be possible to do that, but could please explain the issue in more detail so we could better understand it? Thanks!

@patil-suraj @patrickvonplaten 
I did pull request https://github.com/huggingface/transformers/pull/13889, it's described below.
When implementing [Pointer Network], it needs to get tokens of the source text. But that returns only default keys(e. g. decoder_input_ids). And it's bound "prepare_inputs_for_generation" by "generation_utils", so it can't add other parameters.
So i add parameter "use_default_keys". Then variable "input_ids" can be extended from each language model head's  "prepare_inputs_for_generation" modefied by users.

Let's say, if using Bert2Bert model implementation of below, it can be getting "decoder_src_input_ids" on decoding when use **kwargs in parent function of "prepare_inputs_for_generation".
``` 
def prepare_inputs_for_generation(self, input_ids, past=None, attention_mask=None, **kwargs):
        input_shape = input_ids.shape
        # if model is used as a decoder in encoder-decoder model, the decoder attention mask is created on the fly
        if attention_mask is None:
            attention_mask = input_ids.new_ones(input_shape)

        # cut decoder_input_ids if past is used
        if past is not None:
            input_ids = input_ids[:, -1:]
        src_input_ids = kwargs.get('decoder_src_input_ids', None)
        return {"input_ids": input_ids, "attention_mask": attention_mask, "past_key_values": past, "decoder_src_input_ids": src_input_ids}
```


[Pointer Network]: https://arxiv.org/abs/1704.04368

Thank you for reply!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Closing due to inactivity on the pull request
fx: don't copy by reference
Author: dar-tau
Pull request: no

Here:
https://github.com/huggingface/transformers/blob/9fc1951711e5377ffa1f06614ca37d4d5ad281a8/src/transformers/utils/fx.py#L307

there's a copy by reference which is problematic because you append values to `shape`. 
You should make it a copy by value, e.g. by adding the following line right after:
``
shape = shape[:]
``

Cheers, Guy :)

Comments:
cc @michaelbenayoun 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
id2label is list but we need dict to load models properly from tf.saved_model
Author: Abhishek-krg
Pull request: yes

id2label is list and is consistent throughout pytorch and transformers the config.json file created through TFmodel.save_pretrained keeps it as list as well, the code above expected the id2label to be a dict obj but was list instead causing errors in loading saved models in tensorflow.

# What does this PR do?

<!--
prevented properly loading models from tensorflow
-->

<!-- Remove if not applicable -->

Fixes # (issue)

# id2label AttributeError : 'list' object has no attribute 'items' when loading saved model in TF
<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 @LysandreJik

 -->


Comments:
Hi @Abhishek-krg, I'm sorry for the slow reply. There are a couple of issues here:

1. The PR has a trailing ), so the code is invalid. Can you fix that issue so we can run tests?
2. The attached code is missing. Can you show some sample code that's failing without this PR?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
[WIP] Add MarianMT to models exportable with ONNX
Author: Maxinho96
Pull request: yes

Resolves #13823

@patil-suraj @michaelbenayoun 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
quick fix SummarizationPipeline error messages
Author: nmntz
Pull request: yes

Fix error messages to avoid spam errors, and errors of type:
`Your max_length is set to 50, but you input_length is only 46. You might consider decreasing max_length manually, e.g. summarizer('...', max_length=50)`

Comments:
Pinging @Narsil and @patrickvonplaten 

Thanks for your contribution @NouamaneTazi :)

You have an error in your code quality; do you mind running the following commands at the root of your clone?
```
pip install -e ".[quality"]
make fixup
```

This should fix most quality issues and let you know if there are some the scripts cannot resolve.

@NouamaneTazi .

I am not sure you PR achieves your desired goal:

You are changing another error than the error you're posting in the description.
You report `max_length is set to ....`    and are modifying another warning. Actually the error you're modifying is different and linked to using `min_length` and having a length very well smaller than `min_length`.


FWIW. Summarization usually ingests large texts, and tries to generate something between `min_length` and `max_length` of size as a summary.

If `input_length` < `min_length` or `input_length` < `max_length` then the summary is likely to be not very summarized (since the original string already fits the requirements, or worse, the summary is supposed to expand on it. (Since original text is smaller than the supposedly `min_length`.

Do that clarify why the warnings are originally here ? If yes, then maybe we could clarify the warning instead to make them self sufficient ?

The warnings originated here 9c683ef01e19c4dc1216dcd1ae3c8e7c44d7b2b9 (tried to get a better rationale for the `// 2` but doesn't seem to be explained)



Hello @Narsil , thanks for your clarification.
Indeed, since I tried to make the quick fix using Github web IDE, it didn't save all my changes. Here are my intended changes:
* assert `max_length > min_length`
* assert `input_length > max_length` (if not suggest `max_length = input_length // 2`)
Do you think there is more checks to do?

I think it should be fine to check both, but would that avoid the spam you were referring to ?

Yes, because before the solution proposed wasn't clear:
`Your max_length is set to 50, but you input_length is only 46. You might consider decreasing max_length manually, e.g. summarizer('...', max_length=50)`
Whereas now, we do propose a `max_length = input_length // 2` which should stop showing the error if fixed.

Makes sense.
[WIP] Add Nystromformer
Author: novice03
Pull request: yes

# What does this PR do?

This PR adds the Nystromformer transformer model to the repository.

Paper: [https://arxiv.org/abs/2102.03902](https://arxiv.org/abs/2102.03902)
Code: [https://github.com/mlpen/Nystromformer](https://github.com/mlpen/Nystromformer)
Checkpoint: [Nystromformer sequence length 512](https://www.dropbox.com/s/8uv4f6q52oaqwkh/Nystromformer.model?dl=0)


## Who can review?

@NielsRogge 


Comments:
Getting ambiguous messages for SummarizationPipeline
Author: nmntz
Pull request: no

When working with the SummarizationPipeline I get errors every input of type
`Your max_length is set to 50, but you input_length is only 46. You might consider decreasing max_length manually, e.g. summarizer('...', max_length=50)`

Comments:
I tried to fix this in this PR #14618

Fixed in #14618