noamaanMulla-03/transformers-issues huggingface dataset:
Custom pipeline
Author: sgugger
Pull request: yes

# What does this PR do?

This PR adds the ability to support custom pipelines on the Hub and share it with everyone else. Like the code in the Hub feature for models, tokenizers etc., the user has to add `trust_remote_code=True` when they want to use it. Apart from this, the best way to get familiar with the feature is to look at the [added documentation](https://moon-ci-docs.huggingface.co/docs/transformers/pr_18079/en/add_new_pipeline#adding-it-to-the-list-of-supported-tasks).

Note: this PR changes the newly added `PIPELINE_REGISTRY.register_pipeline` API to accept all the arguments one by one instead of inside a big dictionary. This makes the API easier to use in my opinion.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@sgugger what is your opinion on using attrs for validation with the representation of the pipeline task? I think it would be nice if it has a validation step here.  (this means introducing `attrs` as a transformers' dependency).

Yes, the doc clearly states that you need to save your custom pipeline in a module and import it from there. I could add support from writing something from `__main__.py` but maybe in a followup PR that also deals with custom models/tokenziers/configs etc?

[Line 655](https://github.com/huggingface/transformers/blob/f4e172716b91b477ce3cddc9a253094b7121a4b8/src/transformers/pipelines/__init__.py#L655) in
https://github.com/huggingface/transformers/blob/f4e172716b91b477ce3cddc9a253094b7121a4b8/src/transformers/pipelines/__init__.py#L649-L657
will call
https://github.com/huggingface/transformers/blob/f4e172716b91b477ce3cddc9a253094b7121a4b8/src/transformers/pipelines/base.py#L257

In the current version, we have `model_class` not being an `Auto` class for `(TF) ImageClassificationPipelineTests`, and we get test failure `TypeError: ('Keyword argument not understood:', 'trust_remote_code')`

https://github.com/huggingface/transformers/runs/7421505300?check_suite_focus=true

Adding `TFAutoModelForImageClassification` in `src/transformers/pipelines/__init__.py` will fix the issue.
NER training crash
Author: yanan1116
Pull request: no

transformers version: 4.17.0

Hi, I run the script for NER task for [few-nerd](https://huggingface.co/datasets/dfki-nlp/few-nerd) dataset
https://github.com/huggingface/transformers/blob/main/examples/pytorch/token-classification/run_ner_no_trainer.py


```
 CUDA_VISIBLE_DEVICES=3  python -u run_ner_no_trainer.py \
  --model_name_or_path roberta-large \
  --dataset_name dfki-nlp/few-nerd \
  --dataset_config_name "supervised" \
  --output_dir /scratch/w/wluyliu/yananc/finetunes/roberta_nerd_fine \
  --text_column_name "tokens" \
  --label_column_name "fine_ner_tags" \
  --num_train_epochs 7  --local_files_only --debug  --debug_cnt 50000
```
When I use small samples, for example below 30000, things go smoothly and the precision, recall and F1 are in good alignment with the original paper.

However, when I increase the samples used for training, for example, 50000, or full set, the metrics become zero, where the predictions from model are all "O".  It is quite weird. 

I also try the conll2003 dataset, it is the same. 

Am I miss something ?

Thanks.



Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Was that issue resolved? I am facing a similar problem with the HF implementation of LUKE: https://github.com/huggingface/transformers/tree/main/examples/research_projects/luke
My model's Hosted Inference API is returning "Internal Server Error" and when fetching the API it eternally loads.
Author: CakeCrusher
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.6.1
- Platform: Windows-10-10.0.19043-SP0
- Python version: 3.8.10
- PyTorch version (GPU?): 1.8.1 (True)
- Tensorflow version (GPU?): 2.6.0 (False)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No        

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Visit https://huggingface.co/SebastianS/DEL2
2. Try and use the Hosted Interface API.
3. It will return "Internal Server Error"

![image](https://user-images.githubusercontent.com/37946988/180653047-29db495d-30d6-4b36-8240-7e9ac9eb7933.png)
![image](https://user-images.githubusercontent.com/37946988/180653060-3d150330-9f72-4628-8ffd-cef58d672935.png)


### Expected behavior

The model should be loaded and then generate text.

Comments:
Its working again, seems like something did happen with the servers.
Model parallelism for m2m100
Author: elricwan
Pull request: no
Labels: name: New model

### Model description

The translation model m2m100 proposed by Facebook is too huge to train using DDP, is there any open solution for model parallelism of m2m100, just like GPT2? Thank you.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

_No response_

Comments:
We recommend using accelerate to achieve parallelisation now: https://github.com/huggingface/accelerate
Owlvit docs test
Author: alaradirik
Pull request: yes

- Fixes a typo in OwlViTForObjectDetection forward function docs: transformers/models/owlvit/modeling_owlvit.py
- Adds docs test for OWL-ViT
- Makes `OwlViTFeatureExtractor.post_process` callable from `OwlViTProcessor`
- Improves code examples to demonstrate how to use the post_process method

Comments:
_The documentation is not available anymore as the PR was closed or merged._

> LGTM! Just wondering why there are 90+ commits

That was my mistake, I merged the owlvit branch of my forked transformers repo with the main and created this branch. I squashed the commits on the main but don't know how to fix this one.

Let me know if you'd like some help to squash the commits of this PR @alaradirik!
Add ViltForTokenClassification e.g. for Named-Entity-Recognition (NER)
Author: gilad19
Pull request: yes

# What does this PR do?

Adding ViltForTokenClassification in order to be able to fine-tune ViLT for a token classification task (e.g. as Named-Entity-Recognition). This allows leveraging both image and text for token classification tasks using ViLT. 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [X] Did you write any new necessary tests?


## Who can review?

@xhluca , @LysandreJik, @NielsRogge, @ydshieh 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hey @gilad19, the implementation and internals look good to me! 

I'll defer to @NielsRogge regarding the forward call implementation and documentation.

Regarding the use case - exactly, apply NER on a piece of text for which you also have a visual information. 

Hi @NielsRogge - a gentle reminder :) 

Feel free to merge when satisfied @NielsRogge
Replace false parameter by a buffer
Author: sgugger
Pull request: yes

# What does this PR do?

The weights of the sinusoidal embedding are defined as a parameter with no grad in M2M100 (and thus XGLM), and never saved in the state dict. The problem is that when loading this with `low_cpu_mem_usage=True`, this false parameter will be replaced by an empty weight on the meta device, which is not re-initialized afterward (since it's not in the state dict). As a result, the model is not usable if `low_cpu_mem_usage=True` is used.

By replacing it with a buffer, the weight is ignore by init_empty_weights and thus preserved.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add TFAutoModelForImageClassification to pipelines.py
Author: ydshieh
Pull request: yes

# What does this PR do?

Add `TFAutoModelForImageClassification` to `pipelines.py`.

Fix the test failure mentioned [here](https://github.com/huggingface/transformers/pull/18079#issuecomment-1194352739).

Here is the [failed job run](https://github.com/huggingface/transformers/runs/7492640649?check_suite_focus=true)

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[ create_a_model.mdx ] translate to pt
Author: Fellip15
Pull request: yes

# What does this PR do?

Creates a new file called create_a_model.mdx in docs/source/pt
Translates all the content of the base create_a_model to pt-br

Fixes issue #16824

## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [X] Did you write any new necessary tests?

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Obrigado @Fellip15! Thank you for your translation, and sorry for my late review. 

The text looks good to me, but the tests seem to have a weird loop. I can not find the reason, WDYT @sgugger?
Add PyTorch 1.11 to past CI
Author: ydshieh
Pull request: yes

# What does this PR do?

Add PyTorch 1.11 to past CI (as PyTorch 1.12 is the current latest version).

I will re-run past CI once this PR is merged (for [this comment](https://github.com/huggingface/transformers/issues/18181#issuecomment-1189047657)).

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Snacky Brain Bites for HF Transformers
Author: zolekode
Pull request: no

### Feature request

Nugget Republic has these very cool nuggets based on the HF course.
I would like to submit a PR to integrate these nuggets into either the Readme or into the official documentation if allowed.
Can anyone point me to the right place or person?

Here are a few examples of said nuggets (4 / 11 nuggets):

https://app.flexudy.com/story?mId=W7YKXTS0&dId=FMCSLFZH
https://app.flexudy.com/story?mId=W7YKXTS0&dId=X1PL9FZG
https://app.flexudy.com/story?mId=W7YKXTS0&dId=AFG8XX5S
https://app.flexudy.com/story?mId=W7YKXTS0&dId=UAYHMCSL


### Motivation

Simple. To make HF Transformers even more accessible to the general public. 

### Your contribution

Sure thing!

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Ignore small batches of examples for CLM training
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix #17875.

In https://github.com/huggingface/transformers/blob/a5d504834d01dd1c7edf7c46d93c080cb5274eec/examples/pytorch/language-modeling/run_clm.py#L440

we drop remainder when `if total_length >= block_size`. However, If a batch of examples is too small to have long enough length, that batch will be used. If the whole dataset has only that batch, it is fine. Otherwise, we might get examples of different lengths, and get an error `expected sequence of length X at dim 1 (got Y) during training`.

The same question is also asked on [StackOverflow](https://stackoverflow.com/questions/71166789/huggingface-valueerror-expected-sequence-of-length-165-at-dim-1-got-128).

With this fix, an edge case is that the whole dataset is dropped (if it contains only small batches that are all dropped) - but this is very unlikely. **We can throw an error in this case however**.

**Once the fix is approved, I can apply the same change to TF/Flax example scripts too (if relevant there)**

Comments:
_The documentation is not available anymore as the PR was closed or merged._

I was thinking to throw an error with some information. But since this has been discussed several times and the decision is made, I am going to close this PR.

If you throw an error later on if the dataset is empty, that fix then works for me (but there is a simpler way which is just to remove the test).
Add PYTEST_TIMEOUT for CircleCI test jobs
Author: ydshieh
Pull request: yes

⚠️  Before merging, we need to **run the full tests on CircleCI** to see if there are slower tests that will fail and decide what to do with them.

# What does this PR do?
Add `PYTEST_TIMEOUT: 30` for CircleCI jobs:
```
environment:
    ...
    PYTEST_TIMEOUT: 30
```
The main goal is to avoid CircleCI's default 10 minute timeout that cancels the jobs. Also with this PR, we can see clearly which test (s) timeout.



Comments:
_The documentation is not available anymore as the PR was closed or merged._

I like the idea of getting a hard error instead of silently getting new tests that slow down the CI by quite a lot! Now we just ahve to get to all tests passing below that threshold 😅 
For the examples, you can authorize 60s before timing out as those are end-to-end small training so take more time.

Currently set to `PYTEST_TIMEOUT: 120`. As mentioned on Slack, tests sometimes get much longer to run. For example,

```
test_modeling_data2vec_audio.py::Data2VecAudioModelTest::test_mask_time_prob_ctc

44.16s call 
37.64s call     
12.46s call     
12.60s call 
```

and 

```
test_modeling_plbart.py::PLBartBaseIntegrationTest::test_base_generate

48.18  call
11.20s call 
11.53s call
```

This makes it difficult to determine a good threshold that won't be flaky.

### Current longest 2 tests 
(observed on a CircleCI workflow run):

```
73.08s call     
test_pipelines_image_segmentation.py::ImageSegmentationPipelineTests::test_pt_DetrConfig_DetrForSegmentation_notokenizer_DetrFeatureExtractor

65.33s call
longt5/test_modeling_flax_longt5.py::FlaxLongT5ModelTest::test_jit_compilation
```

Reverted the change in `setup.py` (was for running the full tests). The final timeout limit is 2 minutes (to avoid flaky failures).
I will merge this PR today unless @sgugger has a different opinion.
T5 generate with do_sample doesn't work on DeepSpeed Stage 3
Author: lkm2835
Pull request: no
Labels: name: bug

### System Info

transformers == 4.20.1
python == 3.8.13
OS == ubuntu 20.4
DeepSpeed == 0.6.7

### Who can help?

@patrickvonplaten

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

https://github.com/microsoft/DeepSpeed/issues/2022#issuecomment-1158389764

### Expected behavior

All processes run and finish.

Comments:
cc @stas00 

@lkm2835, looking at your code you linked to, you must use `generate(..., synced_gpus=True)` when using ZeRO stage-3

@stas00, Thank you for your advice!

It works using `synced_gpus`!

https://huggingface.co/docs/transformers/v4.20.1/en/main_classes/text_generation#transformers.generation_utils.GenerationMixin.generate.synced_gpus

If possible, could you explain to me why it is not possible in T5 since it works in gpt2 and gptj without `synced_gpus`?

oops, apologies for the typo - fixed! glad you figured it out, @lkm2835 

This has nothing to do with t5 specifically, but just how ZeRO stage3 works. It needs to have all gpus work in sync. So if one gpu finished generating, it has to continue running `forward` because ZeRO distributes all the weight shards to all gpus and if one stops the other gpus can't get the shards they are missing.

So it really depends on the situations - sometimes all gpus generate the same output length and then it works w/o syncing, but that's just an accident and can easily break down the road.

For more details please see:
https://huggingface.co/docs/transformers/main/perf_train_gpu_many#zero-data-parallelism
Freeze the token embeddings for finetuning
Author: santhoshkolloju
Pull request: no
Labels: name: wontfix

Hi ,

I have finetuned the T5 model using the community notebooks given . But when i looked into the code under examples/seq2seq finetuning code token embedding weights are frozen.

Can some one throw some light on how does this effect the finetuning process?


Thanks

Comments:
freezing embedding weights will result in faster training and will also consume less GPU memory. It might result in slightly less performance but I guess it ultimately depends on the task, the dataset etc.

A tangent question: is it possible to freeze only a subset of the token embedding weights? Background: I am adding some new tokens and I want the model to only fine tune on those.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


It would be a very interesting feature proposed by @AzharSultan like for ***prompt learning*** purpose. 
PyTorch doesn't allow to only update a part of the embeddings lookup table. A solution would be either masking gradients or creating a new lookup table just for the added tokens. Any plan to add that ? 

@montellasebastien did you experiment with these workarounds?

I am also interested in @AzharSultan's question. What is the recommended for doing this right now?
patch for smddp import
Author: carolynwang
Pull request: yes

# What does this PR do?

Fixes an `invalid backend` error when starting a HF job with smddp in a method that goes through src/transformers/training_args.py by adding the import statement for smddp, which registers smddp as a torch.distributed backend. 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
_The documentation is not available anymore as the PR was closed or merged._
[EncoderDecoder] Improve docs
Author: NielsRogge
Pull request: yes

# What does this PR do?

As a follow-up of #17815, this PR improves the docs of `VisionEncoderDecoderModel` and `SpeechEncoderDecoderModel`.

It also fixes some typos in the docs of `EncoderDecoderModel`.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Apply type correction to `TFSwinModelOutput`
Author: harrydrippin
Pull request: yes

# What does this PR do?

## Problem & Fix

This PR fixes small type error on `TFSwinModelOutput`.

`TFSwinModelOutput`'s `pooler_output` is actually optional, which can be disabled by using `add_pooling_layer=False` option on `TFSwinModel` or `TFSwinMainLayer`. This fix applies `Optional` type on `pooler_output` to fix this error.

## Review
This PR is related with Swin Transformer.
R: @amyeroberts, @NielsRogge


Comments:
_The documentation is not available anymore as the PR was closed or merged._
[XLA] Improve t5 model performance
Author: ymwangg
Pull request: yes

This PR improves the performance of t5 model on XLA device. I tested T5ForConditionalGeneration with t5-small config on both GPU and colab TPU and the performance on GPU was improved by >10% while the speedup on TPU is not significant (<1%).

I'm not sure if it's ok to skip the `shifted_input_ids` checking on XLA device since this can trigger device host sync. The `torch.ones().to(device)` operation will trigger unnecessary data uploading to XLA device.
cc @sgugger

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@sgugger I've removed this check completely. I think maybe it's fine to skip it since `nn.Embed` will throw an error for negative indices.

Makes sense, LGTM! Wdyt @patrickvonplaten ?

Ok to skip this check for me!
Update CodeParrot readme to include training in Megatron
Author: loubnabnl
Pull request: yes

This PR updates the README to explain how to train CodeParrot with [Megatron](https://github.com/NVIDIA/Megatron-LM), and redirects model and dataset imports to [CodeParrot organization](https://huggingface.co/codeparrot).

Comments:
_The documentation is not available anymore as the PR was closed or merged._

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.