noamaanMulla-03/transformers-issues huggingface dataset:
Bump notebook from 6.4.1 to 6.4.10 in /examples/research_projects/lxmert
Author: dependabot[bot]
Pull request: yes
Labels: name: dependencies

Bumps [notebook](http://jupyter.org) from 6.4.1 to 6.4.10.


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=notebook&package-manager=pip&previous-version=6.4.1&new-version=6.4.10)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
- `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
- `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
- `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
- `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/huggingface/transformers/network/alerts).

</details>

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Update the _test_requirements to use the dev version of Accelerate
Author: muellerzr
Pull request: yes
Labels: name: Tests

# What does this PR do?

Since there is rapid development of the Accelerate package, this includes the git version as a requirement for the examples in `_test_requirements.txt` for the examples, and also updates the CI to use that for installation only.

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sgugger @ydshieh 


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Herbert polish model
Author: rmroczkowski
Pull request: yes
Labels: name: model card

The HerBERT model is a transformer model pretrained using masked language modeling (MLM) and Sentence Structural (SSO) objectives for the Polish language. 

It was added to the library in PyTorch with the following checkpoints:

- `allegro/herbert-base-cased`
- `allegro/herbert-large-cased`


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dimiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests), 
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests? 


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.
@patrickvonplaten
@julien-c
@LysandreJik 


Comments:
Great point! I've already simplified the code and left only the tokenizer. Ideally, `XLMTokenizer` should have a converter to the appropriate `Fast` class. On second thought, I can see the problem would be to program the Moses pretokenization in the `tokenizers` library.

Hi @rmroczkowski 

As far as I know, you employed fastBPE to train a tokenizer: https://github.com/huggingface/transformers/tree/b23d3a5ad4aa08decd10671f85be5950767dd052/model_cards/allegro/herbert-klej-cased-tokenizer-v1

I also employed fastBPE for the Vietnamese BERT-based tokenizer (i.e. PhoBERTTokenizer https://github.com/huggingface/transformers/pull/6129 ), but I am still struggling to implement a fast tokenizer based on fastBPE, e.g. handling the suffix "@@" of subword tokens. In particular, given https://huggingface.co/vinai/phobert-base/tree/main I can convert "bpe.codes" into a "merge.txt"-style file, but I am not sure about how to convert our "vocab.txt" into your "vocab.json"-style file.

How can you convert your fastBPE's code and vocab outputs into HuggingFace's tokenizers? So that you can call the tokenizer with the use_fast=True option.  

cc: @LysandreJik  is there any idea for implementing a fast version of a fastBPE-based slow one?

Thank you both.
`DebertaTokenizer` always assigns token type ID 0
Author: daniel-ziegler
Pull request: no
Labels: name: Good First Issue

## Environment info

- `transformers` version: 4.16.2
- Platform: Linux-5.15.13-051513-generic-x86_64-with-glibc2.34
- Python version: 3.9.7
- PyTorch version (GPU?): 1.9.0+cu111 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


### Who can help
@LysandreJik 

## Information

Model I am using (Bert, XLNet ...): `microsoft/deberta-large`

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

Run this code:
```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("microsoft/deberta-large")
print(tokenizer("Hello", "World"))
```

It outputs:
```
{'input_ids': [1, 31414, 2, 10988, 2], 'token_type_ids': [0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1]}
```

Even though I put in two sequences, all `token_type_ids` are 0.

## Expected behavior

The tokens from the second sequence should get type ID 1. `token_type_ids` should be `[0, 0, 0, 1, 1]`.

Comments:
Looks like this is the change that introduced this behavior.

https://github.com/huggingface/transformers/commit/57c1749efabf5c86bcfd4e4e078567a63a7c8a81#diff-7ff4f35b72b8541520ea52c851b55bc2682da83e01e6e0ceeb5289f7dd2f0620R217


Good catch! Would you like to open a PR to fix this?

I'll give this a try!
Fix DeBERTa token_type_ids
Author: deutschmn
Pull request: yes

# What does this PR do?

This PR fixes #15735. It changes the behavior of `DebertaTokenizer` and `DebertaTokenizerFast` when passing pair inputs. Before, the token type IDs were all `0`. This PR changes this so that the `token_type_id`s for the tokens of the second sentence are `1`. 

It also adds a test case to test this behavior (`DebertaTokenizationTest.test_token_type_ids`). Failed before, passes now.

## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. https://github.com/huggingface/transformers/issues/15735
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@LysandreJik?

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Great, thanks for looking into this! I also checked Microsoft's implementation and [it looks like](https://github.com/microsoft/DeBERTa/blob/c558ad99373dac695128c9ec45f39869aafd374e/DeBERTa/deberta/deberta.py#L77) they use `1` for sentence B as well 😊
Fix pipeline doctests
Author: stevhliu
Pull request: yes
Labels: name: Documentation

Clean PR of #17013 to fix doctests for the `pipeline` tutorial:

- Updated output values in vision example as suggested by @ydshieh.
- Fixed output format issues. Showing each output result on a new line fails :(

Comments:
_The documentation is not available anymore as the PR was closed or merged._

(I don't really follow the `... ) # doctest: +SKIP`) thing. If you have a bit time, it would be great if you can explain a bit to me what issue we have previously.

Generation is random @ydshieh so the results will be different for each run. `# doctest: +SKIP` tells `doctest` to not test this particular example.
Minor change about ARCHIVE_LIST for TF Data2Vec
Author: ydshieh
Pull request: yes

# What does this PR do?

Just not to declare `DATA2VEC_VISION_PRETRAINED_MODEL_ARCHIVE_LIST` in TF test file.
But I don't mean to introduce it in the model (it was there already in that PR).

As we discussed before, it's better to just drop this variable, and use the checkpoint name directly in `test_model_from_pretrained`. I can change to that, just let me know :-)


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Added spanish translation of autoclass_tutorial.
Author: duedme
Pull request: yes

# Translation of autoclass_tutorial.mdx into spanish

I made the translation of autoclass_turorial.mdx into Spanish (fixes #15947). The document is located in the docs/source/es folder.

This PR also includes the translation of _toctree.yml to include autoclass_tutorial.

FYI @omarespejel 


## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section? 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

LGTM! Just that little change in the `_toctree.yml`, @Duedme. Thanks!

Is it ok if I merge when the change is made @sgugger?
[Fix doc example] Fix 2 PyTorch Vilt docstring examples
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix 2 PyTorch Vilt docstring examples

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks for fixing, can we add ViLT to the doctests? Namely this file: https://github.com/huggingface/transformers/blob/master/utils/documentation_tests.txt

> Thanks for fixing, can we add ViLT to the doctests? Namely this file: https://github.com/huggingface/transformers/blob/master/utils/documentation_tests.txt

Added (after discussed with Patrick)

> Thanks for fixing, can we add ViLT to the doctests? Namely this file: https://github.com/huggingface/transformers/blob/master/utils/documentation_tests.txt

@NielsRogge I added `ViLT` to `documentation_tests.txt`. Do you think this PR is ready for merge? Thanks.

Will merge later today (unless there is further comment 🙂 )

Can you confirm the doc tests are passing?

> Can you confirm the doc tests are passing?

Hi, Yes, I confirmed! Here is the doctest run

https://github.com/huggingface/transformers/runs/5543735270?check_suite_focus=true
fix last element in hidden_states for XGLM
Author: ydshieh
Pull request: yes

# What does this PR do?

Same fix as in #16167 (where the last element in `hidden_states` is different between PT/Flax version.)

This was missed in the previous PR - because XGLM model test overwrites the common (more thorough) PT/Flax equivalence test.

This fix is tested with #16280 and passed !

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[Doctests] Make TFRoberta-like meaningfull 
Author: ydshieh
Pull request: yes

# What does this PR do?

Similar to #16363, but for TF

I made sure the doctests run without failure, but 2 models currently use `from_pt=True` --> need to convert and upload the TF ckpts.

@patrickvonplaten 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@patrickvonplaten I uploaded the TF checkpoints and removed `from_pt`.

~~I will need to check and apply Sylvain's suggestion in your PR to finalize this PR.~~

Ready for review :-)
Fix FlaxRoFormerClassificationHead activation
Author: ydshieh
Pull request: yes

# What does this PR do?

Current `FlaxRoFormerClassificationHead` use hard-coded activation `nn.tanh`, but the PyTorch version `RoFormerClassificationHead` use `ACT2FN[self.config.hidden_act]` which is by default `gelu` (and this is the one used by the default checkpoint's config).

This PR fixes this activation issue in `FlaxRoFormerClassificationHead`.

Comments:
To get more familiar with the library management/maintenance, I would like hear from you regarding: does this count as a breaking change?


_The documentation is not available anymore as the PR was closed or merged._

Merge now since this is clearly a bug, as well as a tiny change
use mean instead of elementwise_mean in XLMPredLayer
Author: ydshieh
Pull request: yes

# What does this PR do?

Super tiny change:

`XLMPredLayer` currently uses

```
                loss = nn.functional.cross_entropy(
                    scores.view(-1, self.n_words), y.view(-1), reduction="elementwise_mean"
                )
```
I believe `elementwise_mean` is deprecated:

https://github.com/pytorch/pytorch/pull/13419

https://pytorch.org/docs/stable/generated/torch.nn.functional.cross_entropy.html


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix TFLEDModel
Author: ydshieh
Pull request: yes

# What does this PR do?

The computation of `diagonal_mask` in `TFLEDEncoderSelfAttention`, and `inputs["attention_mask"]` in `TFLEDEncoder` differ from `LEDEncoderSelfAttention` and `LEDEncoder`.

This cause the `logits` and `decoder hidden states` differs in the range `1e-5 ~ 5e-5`, and `encoder hidden states` differs in the range `1e-2 ~ 1e-1`.

This PR fixes these inconsistency, which reduces the difference to the range `1e-7 ~ 1e-6`. I can provide an example if you prefer to see one.

(#15256) 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@patrickvonplaten was very involved in those two models, but @Rocketknight1 and @gante may also take a look

> It looks good to me, as it recovers a pattern lost in the transition and lowers the error. I'm assuming this PR is also a dependency of #15256, correct?
> 

Yes!



Looks clean - thanks a mille for diving into this complicated code @ydshieh! Could we maybe add a test between PT and TF for LED that would have failed previously without this change? Also Longformer should be affected by this problem as well no? 

> Looks clean - thanks a mille for diving into this complicated code @ydshieh! Could we maybe add a test between PT and TF for LED that would have failed previously without this change? Also Longformer should be affected by this problem as well no?

For Longformer, 

 - about `merge "global_attention_mask" and "attention_mask"`: TF version works correctly in `_merge_to_attention_mask`.
 - about the `diagonal_mask`, it turns out `TFLongformerMainLayer` use something different `extended_attention_mask = tf.cast(tf.math.abs(1 - extended_attention_mask), tf.dtypes.float32) * -10000.0` (note the `tf.math.abs` here), which makes all of masked places negative (including global attn), while in `TFLED`, it uses `_expand_mask`, which make global attn `positive`.
This difference makes `TFLongformer` gives the same results as PT models in the test. (I feel there is something strange of using abs here though!)






For the test, I have a more general `test_pt_tf_model_equivalence` in mind (already in another PR), which will make the previous version fail. However, it will also fail (quite a lot) other models/tests. Maybe I can add it specifically to LED model in this PR, and remove it once the more general test could be put into `test_modeling_tf_common.py` (after fixing other tf/pt inconsistency)
Add Flax image captioning example
Author: ydshieh
Pull request: yes

# What does this PR do?

Add `run_image_captioning_flax.py` (modified from `run_summarization_flax.py`).

## Who can review

Examples:  @patil-suraj + cc @patrickvonplaten @NielsRogge @sgugger for info


Comments:
@patil-suraj , @sgugger 

Thank you for the comments. I will make the example much simpler by only supporting loading pre-trained vision encoder and text models.

About casting jax array to np arrary, there is a significant slow down (at least, in image examples) when using jax array as indices for accessing `datasets.Dataset`. I will re-produce the timing comparison.




Hi, @patil-suraj @sgugger 

I simplified the config/model initialization parts (only support loading pretrained encoder & decoder).

--------------------

For @patil-suraj 

About using `numpy` array instead of `jnp.array` when it comes to `datasets`,

For this line

https://github.com/huggingface/transformers/blob/650fb4aa2adfe746f8c3e1aec1d601c4a0e9f40c/examples/flax/image-captioning/run_image_captioning_flax.py#L854

takes 30 seconds (for selecting `16384` elements) using `jax.numpy`,  while using `numpy` only takes `0.005` second.

For this line (take 256 elements - with image data)

https://github.com/huggingface/transformers/blob/650fb4aa2adfe746f8c3e1aec1d601c4a0e9f40c/examples/flax/image-captioning/run_image_captioning_flax.py#L348

jax.numpy: 0.45 second / numpy: 0.10 - 0.15 second

A singe training step (global batch size: 256 images) takes < 0.5 seconds on TPU. 

Due to this significant differences in processing speed, I think it is worth keeping using `numpy` when dealing with `datasets`.
Let me know if you have different opinions about this :-)


Hi

- support `from_pretrained` (the model creation from encoder/decoder is done in another script)
- README updated
- rename to `coco_dataset_script` to avoid confusion
- other nits applied

Thanks for the reviews :-)
[Fix doc examples] missing from_pretrained
Author: ydshieh
Pull request: yes

# What does this PR do?

In a docstring,

```
model = SegformerModel("nvidia/segformer-b0-finetuned-ade-512-512")
```
fails -> should use `from_pretrained`.

This PR fixes it. 

## Who can review?

@NielsRogge 

Comments:
update the file list for doc testing
Author: ydshieh
Pull request: yes

# What does this PR do?

As discussed in a previous email, I updated the file `utils/documentation_tests.txt`:
  - change the current 2 `.rst` files to the new `.mdx` files
  - add the pytorch modeling files that have no example issue (I tested them locally)

There are still some modeling files containing example issues, but we can fix them progressively.
I think it is a good start for 2022 :-)

## Who can review?

@LysandreJik @sgugger 



Comments:
Oh wow that's quite a big change! Also cc @patrickvonplaten 

Thanks for the change @ydshieh - I'm quite sure that most of the docstring tests in the mentioned files fail at the moment. Could you maybe do those changes: https://github.com/huggingface/transformers/pull/15031/files#r778215725 in your PR as well so that we can see which models have failing doc tests? :-)

> Thanks for the change @ydshieh - I'm quite sure that most of the docstring tests in the mentioned files fail at the moment. Could you maybe do those changes: https://github.com/huggingface/transformers/pull/15031/files#r778215725 in your PR as well so that we can see which models have failing doc tests? :-)

Hi, @patrickvonplaten , do you mean merge that PR into this one? Or just apply a specific change in that PR (if so, which one)?

I am not familiar with the test. The way I checked the doc examples locally is to extract those examples and run them (in some automatic way), and I only added the files without having doc example issue.

Thanks! 

> Thanks for the change @ydshieh - I'm quite sure that most of the docstring tests in the mentioned files fail at the moment. Could you maybe do those changes: https://github.com/huggingface/transformers/pull/15031/files#r778215725 in your PR as well so that we can see which models have failing doc tests? :-)

Hi, I rebased my PR on your PR #15031. Let's see what it gives.



@patrickvonplaten , in your PR, you added new lines in some doc examples, like

```
    >>> list(last_hidden_states.shape)
    {expected_output}
```

Is this necessary to make doctest work? Should I do the same for the doc examples in the files I added here?
(Could you point me a guide to deal with this doctest thing, please? Thanks)

Hey @ydshieh,

sorry for answering so late here. Let me discuss the doctests quickly internally and come back to you :-)

Hey @ydshieh,

We are currently working on adding a section that explains in detail how to test/check the doc tests. Would it be ok to close this PR for now and open a new one, once the documentation is up? 

> Hey @ydshieh,
> 
> We are currently working on adding a section that explains in detail how to test/check the doc tests. Would it be ok to close this PR for now and open a new one, once the documentation is up?

Sure! Thanks for this update :-)
[VisionTextDualEncoder] Add token_type_ids param
Author: ydshieh
Pull request: yes

# What does this PR do?

This line will fail since there is `token_type_ids` in `inputs` but `get_text_features` has no such param:
https://github.com/huggingface/transformers/blob/ac224bb0797c1ee6522d814139f3eb0a8947267b/src/transformers/models/vision_text_dual_encoder/modeling_vision_text_dual_encoder.py#L235

Since `token_type_ids` appears in other places in `modeling_vision_text_dual_encoder.py`, I think it's reasonable to add it to `get_text_features`.

## Who can review?
@patil-suraj



Comments: