noamaanMulla-03/transformers-issues huggingface dataset:
bert（+lstm）+crf
Author: michael-wzhu
Pull request: yes
Labels: name: wontfix

add crf layer for better performance in NER tasks

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2249?src=pr&el=h1) Report
> Merging [#2249](https://codecov.io/gh/huggingface/transformers/pull/2249?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/ac1b449cc938bb34bc9021feff599cfd3b2376ae?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2249/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2249?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2249   +/-   ##
=======================================
  Coverage   79.82%   79.82%           
=======================================
  Files         131      131           
  Lines       19496    19496           
=======================================
  Hits        15562    15562           
  Misses       3934     3934
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2249?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2249?src=pr&el=footer). Last update [ac1b449...ea25498](https://codecov.io/gh/huggingface/transformers/pull/2249?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


@michael-wzhu did this increase f1 on CoNLL?

> add crf layer for better performance in NER tasks

Hi. How do you solve the tokens with "##..." when they are fed into the crf layer?
e.g. De ##duct ##ive reasoning
Do "##duct" and ""##ive" are fed into the crf layer? If they are, do they have chance to be transfered in the transition matrix?

> e" are fed into the crf layer? If they are, do they have chance to be transfered in the transition matrix?

I have used this to develop my own version. To answer your quesiton, ## sub word tokens are treated as padding, as suggested by the original BERT authors. This code relies on The padding label token being "X", at the first position (to get0th index) from the output of get_labels function in crf_utils_ner.py 

The pad token label id might need to be 0 for calculations in CRF, but you should be careful in declaring your mask so your model does not confuse padding with one of the tokens. 

If ## sub word tokens are treated as padding, it will break the tag-tag dependencies, so definitely not ideal.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@srush @mezig351  What did you find after your work on this issue? Why didn't you merge it?

I think you should add it, it's not trivial, and I myself spent 2 days to make it work...
and then I just found this thread...
Fix mask token handling
Author: qqaatw
Pull request: yes

# What does this PR do?

This PR fixes the problem that the mask token is trying to incorrectly match normalized input texts.

This is a related PR of #13594 and #13930.

@SaulLu @LysandreJik 

Comments:
Hey @qqaatw, are you sure this is an issue with all tokenizers you refactored here? If so, then ideally there would be a test for all of them. The test would fail on current `master`, and would be solved by your PR.

If the problem is as widespread as you show it here, then it might even make sense to add it to the common tests.



Hey @LysandreJik, thank you for your response.

I think this change will be covered in this test after we extend the test for both python and rust tokenizers. (discussed on this [thread](https://github.com/huggingface/transformers/pull/13594#discussion_r714795220)) 

https://github.com/huggingface/transformers/blob/1cc453d33c5d0be01eaf3050082c125ce87491aa/tests/test_tokenization_common.py#L651-L656

As a matter of fact, not all tokenizers would fail on current master as it depends on different kinds of special tokens.
For example, if the mask token is `[MASK]`, then on current master the tokenizer will incorrectly normalize input texts first, and then try to match the mask token, resulting in:

`Today is a [MASK] day` normalized-> `today is a [mask] day` -> cannot match `[MASK]`

However, if the mask token is `<mask>`, whether we're on current master or this PR, the test will always pass as there is no difference on the mask token before and after normalization.

`Today is a <mask> day` normalized-> `today is a <mask> day` -> can match `<mask>`

Therefore, changing all tokenizers with special mask token handling just wants to make sure they have a consistent behavior throughout the codebase.

Thank you very much for the additional information @qqaatw . 

I agree with you that it is more "intuitive" that the default behavior for a mask token is `Normalized=False`. 

However, since this doesn't necessarily solve a problem and potentially introduce changes for our users, maybe it's worth leaving the settings as they were before. What do you think about it?

@SaulLu I agree with your point. Except for `Albert` and `FNet` tokenizers, other tokenizers having specifal mask token handling don't have the `do_lower_case` option; therefore, they would not fail `test_added_tokens_do_lower_case` test.

So we only need to modify `FNet`  because `Albert` was already addressed by another PR.
[doc] bf16/tf32 guide
Author: stas00
Pull request: yes

This PR adds performance-specific docs as a follow up to https://github.com/huggingface/transformers/pull/13207

- adds the missing `--fp16_full_eval`
- adds `--bf16` and `--bf16_full_eval`
- adds discussion of `tf32`

This is the first pass, surely will add more down the road.

@sgugger 

Comments:
Thanks a lot for the proofreading and the corrections, @manuelciosici 

You couldn't make suggestions since the PR was merged already.

I applied your corrections to this ongoing PR https://github.com/huggingface/transformers/pull/14580 that collects small fixes.

I even figured out how to make the right attribution to your suggestions ;)

```
git commit --author "Manuel R. Ciosici <manuelrciosici@gmail.com>" -ma "corrections"
```
[Flax] Add FlaxBlenderbotSmall
Author: stancld
Pull request: yes

# What does this PR do?

This PR adds `FlaxBlenderbotSmall` models.

**TODO:** Add model checkpoints for flax on the Hub.

Fixes #14188 


## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?
@patrickvonplaten @patil-suraj 

Comments:
Good to merge for me! Sorry for being so late and causing the merge conflict now @stancld :-/ Could you fix it quickly? Then I think we're good to merge this PR

@patrickvonplaten No worries, it's okay :] Merge conflict should be resolved.

Thanks, @stancld! Will push the checkpoints shortly and merge :)
[Flax] Add Flax implementation of `BlenderbotSmallModel`
Author: stancld
Pull request: no

# 🚀 Feature request

Add Flax implementation of `BlenderbotSmall` models.

## Motivation

Improve the level of Flax/Jax support.

## Your contribution

I'll be happy to do it once the problems in #13633 will be resolved and the PR will be merged.

@patrickvonplaten 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

waiting for #13633
Add CodeParrot 🦜 codebase
Author: lvwerra
Pull request: yes

# Add CodeParrot 🦜 codebase

This PR adds the CodeParrot 🦜 codebase to `examples/research_projects/`. The folder `scripts/` includes files for the following steps:

- data preprocessing
- model initialization
- training with accelerate
- validation loss
- evaluate on HumanEval benchmark

In addition the README gives an overview and highlights the results. The requirements file fixes the dependencies.

cc @LysandreJik @thomwolf

Comments:
> * For each step (preprocessing, training, evaluation etc), I think it would be useful to have a CLI with some default values. Currently, there are some hard-coded variables / configs that require the end-user to read the source code to understand what's going on.

The reason I went without a CLI is that most scripts are quite lean and a CLI would decrease readability. E.g. for initialisation would significantly longer with a CLI. Due to the required compute scale I also expect this to be run by "advanced" users. This reminded me that the training script has to be executed from the accelerate CLI so I'll add a remark about this in any case.

What do you think @LysandreJik?
Adds a git pull instruction to the documentation builder
Author: LysandreJik
Pull request: yes

If two consecutive commits happen in rapid succession, then the clone of the doc-builder repository might not be updated by the time the pip installations are over. This greatly reduces the risk this happens by putting a `git pull` instruction after the environment setup.

Comments:
you could do it even later, i.e. 1/ building the doc to a temp folder, 2/ pulling, 3/ mv'ing the doc (instant), 4/ pushing
Computation of mask indices in Wav2vec2Model fails with low probabilities
Author: nikvaessen
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.12.2
- Platform: Linux
- Python version: 3.8
- PyTorch version (GPU?): 1.10

### Who can help

@patrickvonplaten 

## Information

I'm trying to reproduce fine-tuning with Wav2vec2 on Librispeech, however using feature mask probability 0.0012 as in the paper makes the code crash at some point (after ~3_000 steps). 

## To reproduce
```
from transformers.models.wav2vec2.modeling_wav2vec2 import _compute_mask_indices

mask = _compute_mask_indices(
    shape=(10, 500),
    mask_prob=0.0012, # or even lower
    mask_length=10,
)

print(mask)
```
raises
```
Traceback (most recent call last):
  File "/home/nik/workspace/phd/repo/w2v2-mt-learning/playground/buggy_mask.py", line 3, in <module>
    mask = _compute_mask_indices(
  File "/home/nik/workspace/phd/repo/w2v2-mt-learning/.venv/lib/python3.8/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 201, in _compute_mask_indices
    dummy_mask_idx = spec_aug_mask_idx[0]
IndexError: index 0 is out of bounds for axis 0 with size 0
```

Note that using `min_mask=1` prevents this issue as well.

## Expected behavior

If the probability is so low that no features are masked, the method shouldn't raise an `IndexError`.


Comments:
Improve tokenizer tests
Author: qqaatw
Pull request: yes

# What does this PR do?

Improve tokenizer common tests in `tests/test_tokenization_common.py`.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@LysandreJik 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
Pinging @SaulLu who has worked on this in the past

I'll take the liberty of pinging @Narsil if he can give any leads on how to unblock failed tests on pipelines.

I believe the pipeline tests are currently passing?

Hi @SaulLu could you review all the changes are good to you when you're back?

So that we don't get lost, this PR is awaiting the outcome of the PR #13930. :slightly_smiling_face: 

@SaulLu All tests passed. Thanks.
Fixing mishandling of `ignore_labels`.
Author: Narsil
Pull request: yes

Fixes #14272


# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@LysandreJik

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

Comments:
Support for new LayoutLMv2 in Auto Classes (specifically AutoModelForMaskedLM)
Author: nataliebarcickikas
Pull request: no

# 🚀 Feature request

I am trying to finetune a LayoutLMv2 language model on my dataset using the Auto Class functionality, namely AutoModelForMaskedLM. However, it does not support this model. (https://huggingface.co/transformers/model_doc/auto.html#automodelformaskedlm) 

## Motivation

It is hard to develop a streamlined and consistent pipeline if I have to jump back and forth between ways to finetune depending on the model I use. 

Comments:
As the authors did not release any pretraining code, we did not implement a `LayoutLMv2ForMaskedLM`, as LayoutLMv2 was actually pre-trained on 3 tasks:

* Masked Visual-Language Modeling
* Text-Image Alignment
* Text-Image Matching

Of course, we could add a `LayoutLMv2ForMaskedLM`, which adds a language modeling head on top of the base model. This could be used to replicate the first pre-training objective. Feel free to open a PR if you would like this to be added.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
how to reproduce distilbert pretrain on TF2.x?
Author: ZhaiFeiyue
Pull request: no

hi ALL,
I notice that there is a script for train distilbert on PyTorch([Link](https://github.com/huggingface/transformers/blob/master/examples/research_projects/distillation/train.py)), is there same code for TF2.x for reference?

Thanks a lot.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
ImportError: cannot import name 'TrOCRProcessor'
Author: saqibns
Pull request: no

## Environment info

- `transformers` version: 4.11.3
- Platform: Linux-5.3.0-51-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7.0+cu101 (True)
- Tensorflow version (GPU?): 2.3.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


### Who can help
@LysandreJik please help me out reaching the correct contributor. Tagging @NielsRogge since it is related to vision.
## Information

Model I am using: TrOCR

The problem arises when using:
* [x] the official example scripts: (give details below)
I am trying to run the code sample given on the HuggingFace website [here](https://huggingface.co/microsoft/trocr-base-printed#how-to-use).

The tasks I am working on is:
* [x] my own task or dataset: (give details below)
OCR on black and white images with blurry text. 

## To reproduce

Steps to reproduce the behavior:

1. Install the latest version of `transformers`
2. Run the code snippet provided [here](https://huggingface.co/microsoft/trocr-base-printed#how-to-use).

You should get the following error:
```
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-13a8ac45c2a8> in <module>
----> 1 from transformers import TrOCRProcessor, VisionEncoderDecoderModel
      2 from PIL import Image
      3 import requests

ImportError: cannot import name 'TrOCRProcessor'
```

## Expected behavior

I should be able to run the code example [here](https://huggingface.co/microsoft/trocr-base-printed#how-to-use) without getting any import errors.


Comments:
Thanks for your interest in TrOCR! For now, you should install Transformers from master: 

`pip install git+https://github.com/huggingface/transformers.git`

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Which is better ?
Author: MohamedAliRashad
Pull request: no

I want to train a GPT-J-6B model on Arabic dataset but i am facing an important question.
Should i train the model from scratch or fine tune it on the new language … which is better ?

Comments:
You should use the [forums](https://discuss.huggingface.co/) to discuss this, as we keep issues for bugs and feature requests only.

@MohamedAliRashad did you ask it on the forum?
If yes, can you provide the link?
thanks.

@srulikbd 
Found similar questions not answered or recommended the training from scratch part but knew it is impossible because it requires huge amount of computation. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.