noamaanMulla-03/transformers-issues huggingface dataset:
Update TAPAS script to convert TAPAS-NQ weights (re-attempt #11907)
Author: xhluca
Pull request: yes

Closes https://github.com/huggingface/transformers/issues/14494

This is a reattempt of #11907, as it became stall and was closed. I had to use the latest version and copy the code over.

Instructions:
* Clone fork
* go to `transformers/`
* `pip install torch pandas tensorflow`
* `pip install torch-scatter -f https://data.pyg.org/whl/torch-1.10.0+102.html` (or whatever your cu version is, just do `pip show torch` to find out)
* Download the retriever medium or large weights with HN from the official source: https://github.com/google-research/tapas/blob/master/DENSE_TABLE_RETRIEVER.md
* Run this:
```
# Medium model
python src/transformers/models/tapas/convert_tapas_original_tf_checkpoint_to_pytorch.py --task RETRIEVAL --tf_checkpoint_path tapas_nq_hn_retriever_medium/model.ckpt --tapas_config_file tapas_nq_hn_retriever_medium/bert_config.json --pytorch_dump_path tapas_od/ --vocab_file tapas_nq_hn_retriever_medium/vocab.txt 

# lareg model
python src/transformers/models/tapas/convert_tapas_original_tf_checkpoint_to_pytorch.py --task RETRIEVAL --tf_checkpoint_path tapas_nq_hn_retriever_large/model.ckpt --tapas_config_file tapas_nq_hn_retriever_large/bert_config.json --pytorch_dump_path tapas_nq_large/ --vocab_file tapas_nq_hn_retriever_large/vocab.txt 
```

## Results

https://huggingface.co/xhlu/tapas-nq-hn-retriever-medium-1
https://huggingface.co/xhlu/tapas-nq-hn-retriever-medium-0


https://huggingface.co/xhlu/tapas-nq-hn-retriever-large-1
https://huggingface.co/xhlu/tapas-nq-hn-retriever-large-0

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15592). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@xhluca Thanks for this contribution! Can you elaborate how to use this for inference?

This script isn't meant for inference. You should use load the regular TAPAS model. I have uploaded them to Huggingface already so you can find my name and use them if you don't want to run this script.
Silero Models License Infringement
Author: snakers4
Pull request: no
Labels: name: New model

### Model description

Hi,

My name is Alexander, I am writing to you on behalf of Silero, a company maintaining our project [silero-models](https://github.com/snakers4/silero-models).
We noticed that our models are rehosted here - https://huggingface.co/spaces/pytorch/silero_tts or here https://huggingface.co/spaces?search=silero.
We did not explicitly grant Hugging Face, Inc. any sort of permission to rehost, relicense and profit from our work and models. 
Moreover it openly disregards our CC BY-NC-SA license.
Please immediately remove any of our models from your website and / or any of your resources.



Best,
Alexander

### Open source status

- [ ] The model implementation is available
- [ ] The model weights are available

### Provide useful links for the implementation

_No response_

Comments:
closing as duplicate of https://github.com/huggingface/hub-docs/issues/216 let's address the issue there
Update expected values in constrained beam search tests
Author: ydshieh
Pull request: yes

# What does this PR do?

Update expected values in constrained beam search tests.

#17814 changed `generation_utils.py` which gives new expected values in the test.

(otherwise test failed - as in the current CI report)

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks the corrections sound better as well :-)
Fix `test_number_of_steps_in_training_with_ipex`
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix `test_number_of_steps_in_training_with_ipex`.

## Details
This test uses `no_cuda=True`, which will change `n_gpu` to `0` (see `_setup_devices`), and `train_batch_size` will be `8` (with the default training args). However this line
https://github.com/huggingface/transformers/blob/93f48da2740ab69fd14e6bbb38d53c87b4809eda/tests/trainer/test_trainer.py#L590
is computed (earlier) with GPUs, and therefore the (total) batch size is `16` when 2 GPUs is available.

This cause the following error.

#### Currently test error
```bash
tests/trainer/test_trainer.py::TrainerIntegrationTest::test_number_of_steps_in_training_with_ipex
(line 652)  AssertionError: 24 != 12.0
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Properly check for a TPU device in is_torch_tpu_available
Author: muellerzr
Pull request: yes
Labels: name: External

name: bug

# What does this PR do?

This PR adds a better check to see if a TPU is available in the environment when checking `is_torch_tpu_available` by adding a quick import check for `xm.tpu_device()`. This raises a `RuntimeError` if a TPU device is not found. 

Mimics solution in https://github.com/huggingface/accelerate/pull/456

Fixes # (issue)
https://github.com/huggingface/transformers/issues/17752

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sgugger 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

I have the same problem in vertex AI. it is not solver even with `pip install git+https://github.com/huggingface/transformers` 
The weird thing is that I dont even use TPU!

I made it work by `!pip uninstall torch-xla` (at least on vertex, I dont know about other env.)

thanks :)
May i just train a translation task with T5 from scratch without pretrain a language model?
Author: 520jefferson
Pull request: no

### Feature request

May i use bpe in preprocessing  and train a translation model from scatch without pretrain a language model?
@patrickvonplaten

### Motivation

I want to distill a big model to T5 model  and the T5 vocab should be the same as big model. 

### Your contribution

I can verify the process.

Comments:
Hey @520jefferson,

I understand your question as whether it's possible to fine-tune T5 from scratch and to **not** use a pretrained checkpoint.

Yes, this should definitely be possible, but I wouldn't really recommend it given the power of transfer learning. 

Here also some very nice explanation by @sgugger on how powerful transfer learning is that might be interesting: https://huggingface.co/course/chapter1/4?fw=pt#transfer-learning

Why not fine-tune a pretrained T5 model on translation?

Hey @patrickvonplaten 

I want to distill a big model (pytorch version) to t5 model (considering  the  FasterTransformer Backend
https://github.com/triton-inference-server/fastertransformer_backend has provide  origin T5 (not t5.1) triton backend optimization , this reasoning optimizaiton will be conducive to carrying more online traffic) ,  why i don't use transformer as the student model because i haven't find the pytorch version transformer with reasoning optimization and combining with triton. 
 
And the big model use bpe not sentencepiece, So the tokenizer should be load the bpe codes and the vocabs is differenct from the origin t5 model. Therefore i want to distill the big model to t5 model and use the vocab in the same time. 

So I need to figure out two things:
1, whether the t5 can be train in dialogue without pretrain which treat the t5 like transformer without pretrain and i haven't find a relate case finetune from scratch.
2, how to set the tokenizer to just use bpe codes?



Sorry I'm a bit lost here @520jefferson, 

I don't fully understand what you want to do here, but I guess the target task is distillation? Should we maybe try to get help on the forum: https://discuss.huggingface.co/ for distillation? 

@patrickvonplaten 
I just need to finetune t5 from scratch without pretrain, and the tokenizer can just load vocab.txt (not json) or merges.txt (bpe codes).

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

the tokenizer can be built by hand.
Fix loss computation in TFBertForPreTraining
Author: Rocketknight1
Pull request: yes

With thanks to @sreyan88 for writing up a clean bug report and reproducer, and to @ydshieh for locating the problematic code!

Our `hf_compute_loss()` function for `TFBertForPreTraining` was incorrect. However, it still appeared to work when the number of masked positions was evenly divisible by the batch size. Other, more commonly-used models like `TFBertForMaskedLM` do not have this issue.

The problem was incorrect handling of the reduction for the masked loss, so I took the opportunity to rewrite the function in modern TF. All shapes are now static in the rewritten function as well, which means it should now compile with XLA.

Fixes #17883 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@sgugger I believe it's unique to BERT, because I tried searching the codebase for any similar lines and it couldn't find any. I suspect this is how it stayed undetected for so long - it uses the NSP loss and people generally don't train with that anymore.
Pin black to 22.3.0 to benefit from a stable --preview flag
Author: LysandreJik
Pull request: yes

Pins black to 22.3.0 in order to benefit from the `--preview` flag continuously. This flag adds reformats for strings, exceptions, logs, and others.

The recent black 22.6.0 version's `--preview` flag isn't compatible with the 22.3.0 and results in line changes. 

This PR pins 22.3.0 as it was deemed the path with the least friction.

Comments:
Merging now as the code quality passes so that as few PRs are impacted as possible.

_The documentation is not available anymore as the PR was closed or merged._

Thanks a lot!
Fix bug in gpt2's (from-scratch) special scaled weight initialization 
Author: karpathy
Pull request: yes


I randomly noticed a minor bug in the (from-scratch) weight initialization of gpt2, where the same tensor gets re-initialized over and over many times. I don't believe the significantly more common `from_pretrained` is impacted. The original discussion and explanation is here https://github.com/huggingface/transformers/pull/13573#discussion_r906288955 . 

The simplest reproduction is 

```python
from transformers import GPT2Model, GPT2Config
configuration = GPT2Config()
model = GPT2Model(configuration)
```

Then if you insert `print(id(p), name)` inside the if statement you'll see 4 inits of the same tensor, at each layer of the onion, e.g.:

```
139851709684832 c_proj.weight
139851709684832 attn.c_proj.weight
139851709684832 0.attn.c_proj.weight
139851709684832 h.0.attn.c_proj.weight
```

I verified that original code triggers the `if` statement 96 times, while this version triggers it 24 times, which is correct for a 12-layer model. I also ran `pytest tests/models/gpt2/test_modeling_gpt2.py`, without issues. The code is still not super satisfying (e.g. there is still one layer of overwriting present due the init in code block above, it only happens in the right order because of the way `self.apply` iterates depth-first over children, and we're "hard-coding" variable names present in different modules all the way up, but fixing this would be a bit bigger refactor.

cc potential gpt2 reviewers @patrickvonplaten, @LysandreJik and @sgugger, @siddk  from original thread

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Coming to this PR a bit late (sorry!) and this is a nice fix! I think @karpathy raises a good point about maybe factoring out a special post-init (probably not for GPT-2 since folks are used to this API, but for future models others may want to hack on) that prevent the extra initialization calls.

FWIW this code does only execute once at the beginning of a from-scratch training run; I could see this becoming a problem if we tried to naively scale to much larger models. I'll see if we can come up with a better fix for other models.

Thanks for the PR @karpathy - super excited to see you contributing to `transformers`!
Adding GroupViT Models
Author: xvjiarui
Pull request: yes

# What does this PR do?

This PR implements paper [GroupViT: Semantic Segmentation Emerges from Text Supervision](https://arxiv.org/abs/2202.11094), the model is converted from [official implementation](https://github.com/NVlabs/GroupViT).

- [x] Inference accuracy matched
- [x] Complete docstring and model cards

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@NielsRogge 



Comments:
_The documentation is not available anymore as the PR was closed or merged._

CI somehow failed due to connection issue. Could you please restart it?

Hi @NielsRogge 
I have resolved all the comments. 

Regarding your concern, I think it's a little bit hard to combine them since `GroupViTVisionLayer` follows `ViT` and `GroupViTTextEncoderLayer` follows `CLIP`. 

Btw, I still don't know why PR doc build failed. Any idea?

@patil-suraj are you fine with the fact that the vision and text encoder use separate classes (`GroupViTVisionLayer` and `GroupViTTextEncoderLayer` respectively), even though they do the same thing? The only reason we keep `GroupViTVisionLayer` is because the vision encoder implementation is copied from ViT.

So either we keep the two separate classes with `# Copied from` statements, or we remove the copied from and just create a single `GroupViTEncoderLayer` class (as is done in CLIP).

Pinging @mishig25 regarding the build_documentation CI issue


Sorry to only reply now.

> are you fine with the fact that the vision and text encoder use separate classes (GroupViTVisionLayer and GroupViTTextEncoderLayer respectively), even though they do the same thing? The only reason we keep GroupViTVisionLayer is because the vision encoder implementation is copied from ViT.

If  `GroupViTVisionLayer` and `GroupViTTextEncoderLayer` are exactly similar then IMO it's better to just have one `GroupViTEncoderLayer`, this will make the code much more readable.  I am not in favor of adding extra modules just for `#copied from..` statememts.

Yes same, so @xvjiarui you can remove the copied from statements from the vision encoder in favor of simpler code. 

Awesome work, thanks a lot!! Merging :)
fix:  eval logs is not saved
Author: iamtatsuki05
Pull request: yes

# What does this PR do?

<!--

This PR fix the If eval tries to log eval logs with prediction_loss_only and logging_dir, it will not be logged, so I changed it so that logs will be saved.

-->

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@sgugger  @patil-suraj

Comments:
Sorry, I forgot to ping reviewers @patil-suraj @sgugger

_The documentation is not available anymore as the PR was closed or merged._
Compute min_resolution in prepare_image_inputs
Author: ydshieh
Pull request: yes

# What does this PR do?

If `feature_extract_tester.min_resolution` is specified, the images have to be at least that large, otherwise we will get image width and/or height `0` and it gives error.

An error is [here](https://github.com/huggingface/transformers/runs/7071766841?check_suite_focus=true):
```
>       return self._new(self.im.resize(size, resample, box))
E       ValueError: height and width must be > 0
```


So far, we have the following in `GLPNFeatureExtractionTester` and other testers
```
        min_resolution=30,
        ...
        size_divisor=32,
```

issue spotted by @Rocketknight1 , thanks!

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add missing comment quotes
Author: leondz
Pull request: yes

# What does this PR do?

This minor fix adds missing quote marks round some explanatory comments in the "new model" tokenizer cookiecutter template.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
`blame` suggests @LysandreJik 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Sorry for letting this fall through the cracks, just merged it!
Fix job links in Slack report
Author: ydshieh
Pull request: yes

# What does this PR do?

The current `notification_service.py` doesn't take `artifact_path['gpu']` (`single` or `multi`) into account when storing the `job_link` information, which leads to wrong pages (sometimes) when we click the `GitHub Action Job` button on Slack.

This PR fixes this issue.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
new Transformer update causes an error with TPU XLA implementation
Author: salrowili
Pull request: no
Labels: name: bug

Hi,
I notice the new release of the Transformer model (4.20) causes an issue with PyTorch XLA implementation and the new error message says "Cannot replicate if number of devices (1) is different from 8" appears.



### Who can help?

_No response_

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

`!pip install cloud-tpu-client==0.10 torch==1.11.0 https://storage.googleapis.com/tpu-pytorch/wheels/colab/torch_xla-1.11-cp37-cp37m-linux_x86_64.whl`

```
!pip3 install git+https://github.com/huggingface/transformers
!git clone https://github.com/huggingface/transformers
```


`!pip3 install -r /content/transformers/examples/pytorch/question-answering/requirements.txt`

```
!python /content/transformers/examples/pytorch/xla_spawn.py --num_cores=8 /content/transformers/examples/pytorch/question-answering/run_qa.py --model_name_or_path sultan/BioM-ELECTRA-Large-Discriminator \
--dataset_name squad_v2 \
--do_train \
--do_eval \
--dataloader_num_workers 4 \
--preprocessing_num_workers 4 \
--version_2_with_negative \
--num_train_epochs 2 \
--learning_rate 5e-5 \
--max_seq_length 384 \
--doc_stride 128 \
--per_device_train_batch_size 4 \
--per_device_eval_batch_size 4 \
--logging_steps 500 \
--save_steps 1000 \
--overwrite_output_dir \
--output_dir out
```


To fix the issue we git clone and git+install with 4.19 release:

```
!pip3 install git+https://github.com/huggingface/transformers.git@v4.19.4
!git clone --depth 1 --branch v4.19.4 https://github.com/huggingface/transformers
```


### Expected behavior

```shell
A new error message says "Cannot replicate if number of devices (1) is different from 8" appears.
```


Comments:
Maybe of interest to @sgugger 

cc @muellerzr It's probably linked to your recent changes for selecting the TPU device.