noamaanMulla-03/transformers-issues huggingface dataset:
use transformers on apple mac m1 (TF backend)
Author: srulik-ben-david-hs
Pull request: no

is it possible?
I saw that there is an option to use tensorflow-metal..but does it gonna work seamlessly with transformers?
https://developer.apple.com/metal/tensorflow-plugin/

for pytorch it seems not possible yet. 



Comments:
I believe it should work with both on M1, @NielsRogge you have experience using `transformers` on M1 macs right?

I'm using PyTorch only on my Mac, but if you use TF for Transformers you get a nice speedup, see here for a nice thread: https://twitter.com/lvwerra/status/1470818833619468290?t=1bDbo3Kl-MHBF2MbDh5GiA&s=19

thanks a lot!
I'll try it.
so do you mean that you use pytorch without the m1 gpu?

> so do you mean that you use pytorch without the m1 gpu?

Yes, for now. However the PyTorch team is planning to add support for it.
NER at the Inference Time
Author: ndenStanford
Pull request: no

Apology in advance if this question might have already been asked. However, I have not been able to find a convincing answer or the optimum way to deal with this issue. 

To my understanding, NER makes a prediction at a token level. Since BERT is using a sub-word tokenizer, it is entirely possible that some part of the word won't get labeled or we have a different label within the same word. Both of these are undesirables because in the end we want the final result to be NER on a word, not token, level. 

For example, see [link](https://huggingface.co/dslim/bert-base-NER?text=Seven+of+the+men+are+from+so-called+%22red-flagged%22+countries%2C+including+Egypt%2C+Turkey%2C+Georgia%2C+Pakistan+and+Mali.Her+eighth+husband%2C+Rashid+Rajput%2C+was+deported+in+2006+to+his+native+Pakistan+after+an+investigation+by+the+Joint+Terrorism+Task+Force.If+convicted%2C+Barrientos+faces+up+to+four+years+in+prison.++Her+next+court+appearance+is+scheduled+for+May+18.). 

Barr(PER) ien(O) tos(PER)

This should have been Barr(PER) ien(PER) tos(PER). 

Another more confusing prediction [here](https://huggingface.co/dslim/bert-base-NER?text=27+Colors+for+Main+Fabric+available+in+singles+%26+together+in+Fatpack+Sweater)

F(MISC) abric (O) .... Fat (LOC) pack Sweater (ORG) 

We have inconsistent token production within the same words. 

So my questions are the following 

1) How can I best convert token-level NER labels to word-level labels? What is the best policy to deal with inconsistent token-level prediction within the same word? Is there a standard way to do this? Has this already been implemented in the huggingface library. 

2) Should not there be a way to force the model to recognize that those three tokens came from the same word so they need to have the same token-level label in the first place? 

Any suggestions are greatly appreciated. Thank you!


Comments:
Hey @ndenStanford !
Thanks for the issue but I think the [forum](https://discuss.huggingface.co/) would be the best place to ask such general questions. We use issues to report bugs and for feature request. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
add type hints for gpt2 model classes (PyTorch)
Author: KristijanArmeni
Pull request: yes

# What does this PR do?

1. This PR adds type hints for GPT2 as mentioned in #16059 (following docs and other PRs for GPT-like model classes).

Notes:
- I've specified tensor subtypes where I could, e.g. `Optional[torch.LongTensor]`, `Optional[torch.FloatTensor]` as opposed to `Optional[torch.Tensor]` (assuming that's desired)
- I was not entirely sure about `encoder_hidden_states`, as those don't seem to be documented yet (Tuple?, assumed it's `torch.FloatTensor`)
- I was not able to run `pytest` yet due to local (Flask version) conflicts
- I was not able to run `make fixup` locally

(first time contributing in Python, so happy to work on and learn setting up tests)

## Who can review?

@Rocketknight1 




Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_16240). All of your documentation changes will be reflected on that endpoint.

@KristijanArmeni thanks for this PR! I think the types you chose look good. Running `make fixup` can be tricky, though - it depends on having a working `transformers` dev environment. You need to follow the instructions here, but if you're running on Windows you might find that this is impossible: https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md

If you can't get `make fixup` working, feel free to ping me and I'll do it! Make sure you've enabled contributions from maintainers on your PR.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Question about `query_length` in modeling_t5.py 
Author: GeneZC
Pull request: no

https://github.com/huggingface/transformers/blob/0a057201a96565df29984d716f660fd8d634329a/src/transformers/models/t5/modeling_t5.py#L464

If the `past_key_value` is not None, the `query_length` induced from `SelfAttention` can be the length of `past_key_value` + `hidden_states`, which is actually wrong.

Luckily, this problematic behavior would not affect the `EncDecAttetion` since the `position_bias` are always zeros.

Correct me if I am wrong.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
ValueError: HFArgumentParser on running run_glue.py
Author: vr25
Pull request: no

Hi,

I get the following error with HFArgumentParser on running run_glue.py. Any help would be appreciated. Thanks!

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.16.2
- Platform: CentOS
- Python version: 3.9
- PyTorch version (GPU?): 1.10
- Tensorflow version (GPU?): 2.4.1
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- BERT

 -->

## Information

Model I am using (Bert):

The problem arises when using:

```
export GLUE_DIR=/work/vrawte/glue_data
export TASK_NAME=STSB

python run_glue.py \
  --model_type=bert \
  --model_name_or_path=bert-base-cased \
  --task_name=$TASK_NAME \
  --do_train \
  --do_eval \
  --data_dir=$GLUE_DIR/$TASK_NAME \
  --max_seq_length=128 \
  --train_batch_size=32 \
  --learning_rate=3e-5 \
  --num_train_epochs=3.0 \
  --output_dir=/tmp/$TASK_NAME \
  --overwrite_output_dir \
  --logging_steps=50 \
  --save_steps=200 \
  --num_cores=8 \
  --only_log_master
```

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)

![image](https://user-images.githubusercontent.com/22553367/158710662-9ac2142a-96c3-4f63-9d71-847bf78bc019.png)




Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Use the target tokenizer in text generation pipeline
Author: AmitMY
Pull request: yes

# What does this PR do?
A subset of https://github.com/huggingface/transformers/pull/15946
This PR addresses the issue with pipelines in decoding text of different target side tokenizer

This is split from the main PR because this affects the huggingface model hub and API, while the other code is applicable to run in a local copy

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_16049). All of your documentation changes will be reflected on that endpoint.

Hi @AmitMY ,

Thanks for this PR ! It would be nice to get it working with Marian correctly.

Unfortunately, this approach cannot work inside within pipelines.

`.as_target_tokenizer()` does not exist in most tokenizers, and therefore cannot be used (pipeline code is very generic), It's also mutating internal state which is likely going to cause issues in threaded/multiprocessing context.

Here I can think of a better approach if we do indeed need two very distinct tokenizers (it seems to be the case, but the solution for #15946 might change the direction).

The way the pipeline is set, we would need to define two tokenizer, a `preprocess_tokenizer` and a `postprocess_tokenizer` (or `source`/`target`). Each would be unique, and be used accordingly.

By default since most models use the same tokenizer, we could just set both variables to the same object and that should solve the issue at hand.

Before jumping to coding this, I think we should fix the first issue at hand #16050.

Btw, in that issue you mention Mbart, but in the code you modify Marian.


And a note for readers: there's also an open issue to support fast Marian tokenizers https://github.com/huggingface/transformers/issues/15982 which could end up being linked to this work.


Not reviewing but commenting for information.

> `.as_target_tokenizer() does not exist in most tokenizers, and therefore cannot be used `

@Narsil `as_target_tokenizer` defined in `PreTrainedTokenizerBase` so it should be available  for all tokenizers. By default it's a no op. cf 
https://github.com/huggingface/transformers/blob/e66743e6c9601a4b12ffc2335a9f60a41d1ca60c/src/transformers/tokenization_utils_base.py#L3405-L3411

> By default since most models use the same tokenizer, we could just set both variables to the same object and that should solve the issue at hand.

Right now I think there are only two models that support two vocabs: FSMT and Marian (#15831), but for these two tokenizers it's not necessary to call `as_target_tokenizer` since they use the target tokenizer in `.decode/batch_decode`

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[HELP NEEDED] Use target tokenizer for all functionality in target side
Author: AmitMY
Pull request: yes

# What does this PR do?

When using the MarianTokenizer with a different source and target SPMs, it is expected to get the length of the target vocabulary when performing:
```python
with tokenizer.as_target_tokenizer():
    print(len(tokenizer))
```
Along with the expectation to be able to get the target vocabulary, and in translation time to use the target tokenizer.


## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Did you write any new necessary tests?

## HELP NEEDED

Tests do not pass, as the token IDs returned change.
However, the IDs are taken directly from the SPM model and not the VOCAB file, and I do not understand why my new values are less correct compared to the ones in the test files.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15946). All of your documentation changes will be reflected on that endpoint.

Hi @AmitMY ,

I am not 100% sure not knowing all the internals here, but it's pretty frequent that researchers add code on top of SPM and modify the ids. A behavior which is mimicked in `transformers` so that the actual model works.

So the difference of ids could be just that, and we really the code to stay the same.

I don't have a lot of background on this code to give you a possible better approach, sorry :(

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[WIP]Resnet with variants
Author: FrancescoSaverioZuppichini
Pull request: yes

# What does this PR do?
This PR shows how including the required changes to make `resnet-d` variant look but also to quickly decide to use one embedding type over another.

Usage:

```python
# normal embeddings
config = ResNetConfig(embedding_type='basic')
# embeddings is replace by 3 3x3 convs
config = ResNetConfig(embedding_type='deep')
# normal shortcut (stride=2) in a conv layer to downsample
config = ResNetConfig(shortcut_type='basic')
# resnet-d shortcut, avg pool to downsample (no stride = 2) in the conv
config = ResNetConfig(shortcut_type='avg_down')
# resnet-d like config
config = ResNetConfig(embedding_type='deep', shortcut_type='avg_down')
# use it as usual
model = ResNetForImageClassification(config)
```

For example, [mmlab*** ](https://github.com/open-mmlab/mmdetection/blob/master/mmdet/models/backbones/resnet.py) allows creating different ResNet architectures similar to how this pr does by allowing the user to pass `embedding_type` and `shortcut_type`

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15912). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
update world_size and process_index for SMP 
Author: cavdard
Pull request: yes



This PR updates world_size and process_index for SMP when prescaled_batch is True/False. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15811). All of your documentation changes will be reflected on that endpoint.

Prescaled batch is a new construct in SMP. SMP does tensor parallelism within a DP group. What this means is that it shuffles around the samples read by all ranks so that the rank which owns the shard of a layer/parameter can see relevant portion of samples for all samples across the data parallel group. To reduce the communication before layer execution, and to support cases where we can't use a large batch size (i.e. all ranks can't get their own sample), we recommend using prescaled batch, where all ranks in same rdp group read the same batch of data. Hence this usage affects the number of examples seen. From data loading perspective, the number of ranks which read different data changes in prescaled and non-prescaled case. World size and process index in transformers seem to relate to number of times data parallelism happens in the job, hence this change.

More details here
https://sagemaker.readthedocs.io/en/stable/api/training/smd_model_parallel_general.html#prescaled-batch

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Update distributed sampler for smp
Author: cavdard
Pull request: yes

# What does this PR do?

This PR updates world_size and process_index for SMP when prescaled_batch is True/False. 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15809). All of your documentation changes will be reflected on that endpoint.

 Please close this as it's replaced by PR https://github.com/huggingface/transformers/pull/15811 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
update dp_rank to rdp_rank for opt_state_dict
Author: cavdard
Pull request: yes


This PR fixes hang for SMP when calling state_dict(). 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15804). All of your documentation changes will be reflected on that endpoint.

When only pipeline parallelism is used, dp_rank works. But with newer features of SageMaker model parallel library we support Tensor parallelism too. Tensor parallelism is done within a DP group. In such a case, we need to save the state on all ranks with rdp_rank=0. Essentially two ranks in an RDP group have copies of the same shard of parameter. More details on ranking mechanism here https://docs.aws.amazon.com/sagemaker/latest/dg/model-parallel-extended-features-pytorch-ranking-mechanism.html


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Create optimizer after model creation for SMP
Author: cavdard
Pull request: yes



This PR enables optimizer creation after model creation which is a requirement for the newer version of SMP.




Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15796). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
changed documentation for Trainer.predict()
Author: cbrew
Pull request: yes

# Fix a documentation error in Trainer.predict





Fixes # (issue)


## Before submitting
- [*] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [*] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15743). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add missing ckpt in config docs
Author: ydshieh
Pull request: yes

# What does this PR do?

As discussed on Slack, I worked on the `Config` files to add missing information about checkpoints, or correct them.

- I tried to check the mentioned checkpoints are actually on the Hub
- also tried to make sure the checkpoints are for the target architecture
- I didn't verify the statement `Instantiating a configuration with the defaults will yield a similar configuration to that of the Speech2Text2 [mentioned checkpoint]`
   - in particular, the hyperparameters like `hidden_dim`, `num_layers` might be different
   - it says `similar`, so I think it is fine (..?)

@patrickvonplaten Could you take a look on the speech models?
@NielsRogge Could you take a look on the vision models?

Comments:
_The documentation is not available anymore as the PR was closed or merged._

> Thanks for fixing all of those! It would be awesome to have some kind of quality script to check we don't introduce new faulty checkpoints.

Yes, I do have some (draft) check locally. I plan to add it in another PR (unless it's necessary to do so in this PR).

Thank you @NielsRogge I should try to use the correct names, as defined in `MODEL_NAMES_MAPPING`.

> Thanks a lot for this PR, awesome that this gets improved.
> 
> Left some comments, just for consistency, I would always use the template:
> 
> > "will yield a similar configuration of that of the - snake-cased model name -  [checkpoint name](link) architecture".

I will add this to the check I currently have (locally, but will push to another PR), thanks!

Merge now. Thanks for the review.

With this PR, all configs are good except the following (which are expected, since those composite models don't have full default config arguments - they rely on the encoder and decoder configs.)

- DecisionTransformerConfig
- VisionEncoderDecoderConfig
- VisionTextDualEncoderConfig
- CLIPConfig
- SpeechEncoderDecoderConfig
- EncoderDecoderConfig
- RagConfig