noamaanMulla-03/transformers-issues huggingface dataset:
[Tests] Fix slow opt tests
Author: patrickvonplaten
Pull request: yes

Fixes failing circle ci tests: 

```
tests/models/opt/test_modeling_opt.py::OPTEmbeddingsTest::test_logits
tests/models/opt/test_modeling_opt.py::OPTModelIntegrationTests::test_inference_no_head
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hey, will review that ASAP (~1-2h)
Fix test_model_parallelization
Author: lkm2835
Pull request: yes

# What does this PR do?

When number of gpus is greater than ```len(model.device_map.keys())```, exceptional case happens.

Fixes #17248


## Who can review?

@patrickvonplaten


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Change is fine by me! @sgugger @stas00 what do you think?
Fix missing job action button in CI report 
Author: ydshieh
Pull request: yes

# What does this PR do?

Current CI reports lack the `GitHub Action Job` button, due to the recent changes in workflow files:
 - `models/bert` -> `models_bert` (was done in artifact names, but not in the matrix)
 - `[single|multi]-gpu-docker` -> `[single|multi]-gpu` (was done in `notification_service.py`, but not in scheduled CI workflow)

This PR fixes the issues by:
- Let the workflow files use `single-gpu` and `multi-gpu` as matrix and artifact names. Only adds `-docker` in `runs-on:` for scheduled CI.
- Add `model.replace('models_', 'models/'`) at a proper place in `notification_service.py`

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fix wrong PT/TF categories in CI report
Author: ydshieh
Pull request: yes

# What does this PR do?

Current `notification_service.py` has
```
if re.search("_tf_", line):
    model_results[model]["failed"]["TensorFlow"][artifact_path["gpu"]] += 1
```
which will put all `test_pt_tf_model_equivalence` under `TensorFlow` even it is from the PT (cross) tests, like
```
tests/models/albert/test_modeling_albert.py::AlbertModelTest::test_pt_tf_model_equivalence
```
This PR fixes this issue.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
why set masked_bias as -10000 in GPT2
Author: xu-song
Pull request: no
Labels: name: wontfix

## Information

`masked_bias` is set as `-10000` in GPT2, why not `-inf`?

https://github.com/huggingface/transformers/blob/e43f3b6190cfd98a38912411b8bc8ecbb6629280/src/transformers/models/gpt2/modeling_gpt2.py#L133

## openai/gpt-2

In [openai/gpt2](https://github.com/openai/gpt-2/blob/a74da5d99abaaba920de8131d64da2862a8f213b/src/model.py#L88), the bias is set as `-1e10` 
```py
w = w*b - tf.cast(1e10, w.dtype)*(1-b)
```
## Other implementation, such as bert, transformer

https://github.com/huggingface/transformers/blob/82498cbc37d5c15520c7bddde5d804c804eee498/src/transformers/models/bart/modeling_bart.py#L81

<!-- A clear and concise description of what you would expect to happen. -->


Comments:
This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.
[ConvNeXT] Fix drop_path_rate
Author: NielsRogge
Pull request: yes

# What does this PR do?

As pointed out by #16699, the drop path rate attribute of `ConvNextConfig` wasn't implemented correctly. 

This PR fixes that, for both the PyTorch and Tensorflow implementations.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@sgugger ok for you?
How to input word2vec embeddings to gpt2 model?
Author: tejaravi675
Pull request: no

Hi,

I am working on the huggingface gpt2 model. I have a word2vec model trained on a dataset (dimensions similar to gpt2, 768). Now I want to input these embeddings to gpt2. I understand I must use inputs_embeds to input the embeddings but I am little unclear about how exactly to do it. Any source or help would be appreciated.

Comments:
Hi @tejaravi675 👋  Yes, you can pass embeddings to the model as you described (through `inputs_embeds`), but the embeddings will be unknown to the model, as it was not trained with them. In essence, you will have to build a script to finetune the model using your embeddings (for GPT-2, with the causal language modeling task). You can see a few examples [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling) (pytorch) and [here](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/language-modeling) (tensorflow).

As per our [issues guidelines](https://github.com/huggingface/transformers/blob/main/ISSUES.md), we reserve GitHub issues for bugs in the repository and/or feature requests. For any other requests, we'd like to invite you to use our [forum](https://discuss.huggingface.co/) 🤗  I'm closing this issue, but feel free to reopen with queries that fit the criteria I described.

Hi, I understand that I have to finetune the model with my embeddings. My problem is I am unable to understand how exactly to code (Not very handy with complex coding). So I wanted to know if there were any examples which I can refer to to understand the finetuning part and understand how to add my word embeddings to gpt2 model.

The examples I linked above are the closest we have to the task you are describing, but they require some modification to run your use case :) Sadly, we don't have the capacity to further help you with your task -- try in the forums, maybe some other user tried to do a similar thing.
Rewrite TensorFlow train_step and test_step
Author: Rocketknight1
Pull request: yes

Draft PR for a full rewrite of the TF train/test steps. I swear this will fix like 50% of our TF issues in one PR.

Current status:
- Correctly handles output mapping across most model classes for losses + metrics
- Keras metrics are back, even with the dummy loss. (!!!!)
- Keras metrics work correctly even for multi-output models (like QA)
- In most cases, users can pass tensors in either the input dict or the labels and the model will handle them correctly.
- No more errors when calling `fit()` when the model has nested output structure (e.g. the model outputting a `past` tuple)

What's left to do:

- [X] Models with multiple unusual outputs that do not match label names may still have issues with metrics. This is relatively uncommon. We support adding a property to those classes to tell Keras what to do with the labels, but we haven't added it to any models yet. (None are failing in tests, so hopefully we won't need to worry too much about this!)
- [x] Testing testing testing! I want to rerun all notebooks/examples and make sure the user experience is good.
- [X] CI testing - We need to make sure we don't regress on any of this
- [ ] Discoverability: After this is merged we should update notebooks/examples to show off the cool new features, and document our TF workflow/philosophy somewhere that new users will find.


Comments:
_The documentation is not available anymore as the PR was closed or merged._

(Requesting reviews now that @gante is back)
Fix tests of mixed precision now that experimental is deprecated
Author: Rocketknight1
Pull request: yes


Comments:
Thanks, @Rocketknight1 

I saw we have a few

```
tf.keras.mixed_precision.experimental.Policy
```

in `src/transformers/training_args_tf.py`. Maybe we also need to update these places? (I am OK if you prefer to do it in another PR, if the change is indeed necessary)

@ydshieh I'll fix them here too!

_The documentation is not available anymore as the PR was closed or merged._

![image](https://user-images.githubusercontent.com/12240844/168825240-bbf3d9bf-f113-4621-9ccd-cdeb183c9695.png)

Doesn't this mean TF<=2.8 will fail in these lines?

(@Rocketknight1 )

For push/scheduled CI, I think it is fine, as the docker image is built with

```
RUN python3 -m pip install --no-cache-dir -U torch tensorflow
```
(right ..?)

I am going to print TF version in CI jobs.

Update: The latest docker image is built with 2.9

```
2022-05-17T01:34:05.0014817Z #10 15.23 Collecting tensorflow>=2.3
2022-05-17T01:34:05.0016257Z #10 15.24   Downloading tensorflow-2.9.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (511.7 MB)
```

Yes, CI should be fine :D but users with older versions (which we support) will experience errors, right? Or is it backwards compatible?

You are right! At least for `src/transformers/training_args_tf.py`.

I am not very sure what's our policy regarding the test backward compatible (regarding the change in `tests/utils/test_modeling_tf_core.py`)
Doctest longformer
Author: KMFODA
Pull request: yes

# What does this PR do?

adds LONGFORMER PT to doctests

@patrickvonplaten - `LongformerForMaskedLM` and `LongformerForQuestionAnswering` have `@replace_return_docstrings `instead of `@add_code_sample_docstrings`. Should I be replacing them with that instead so that I can test for `expected_output` & `expected_loss`?


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hey @KMFODA, thanks for your PR! Could you rebase/merge on `main` so that the CI passes? Thank you!

apologies @LysandreJik, had merged with master not main. Fixed now :).

@KMFODA would it be ok if we go into the PR to finish it in case you don't have enough time? :-)

hey @patrickvonplaten, apologies for the delay. I've managed to work on the proposed changes. Let me know if anything else is missing?

Great! Looks good to me actually :-) @ydshieh could you take a look maybe?

@patrickvonplaten  could you also check if my comment about `randomly initialized` is valid, please? I haven't run the examples myself though. I can double check tomorrow.

**Update**: I ran it, and the 2 tests failed as I think. Depending on @KMFODA, maybe we can merge this  PR with only the PT file (and we work on TF part internally)? To address this situation, either a conversion of PT -> TF checkpoint, or using a tiny model checkpoint is required.

@KMFODA 

If you are willing to deal with the TensorFlow version , could you try to use the TF checkpoint [hf-internal-testing/tiny-random-longformer](hf-internal-testing/tiny-random-longformer) for `TFLongformerForSequenceClassification` and `TFLongformerForTokenClassification`.

Otherwise, it's totally OK that we revert the changes in `src/transformers/models/longformer/modeling_tf_longformer.py` + remove `src/transformers/models/longformer/modeling_tf_longformer.py` from `utils/documentation_tests.txt`, and we can merge this PR :-). Thank you!

hey @ydshieh thanks for the comments. Yes sure I just switched those 3 checkpoints now. I'm having issues with TF and my new M1 Mac so I can't test these now unfortunately. Thought I'd just commit these changes now in case you're in a rush but will try and test these changes soon as I get my TF library working.

Hi, @KMFODA , no rush on our side :-). But if you have trouble with TF on Mac, we can run it and update the expect values, just let us know. Thank you!

No problem at all. Thanks for all the helpful comments. Changes made :).

Thanks a mille for all your work here @KMFODA !
Add PR author in CI report + merged by info
Author: ydshieh
Pull request: yes

# What does this PR do?

As title. The result looks like


<img width="502" alt="Screenshot 2022-05-17 093737" src="https://user-images.githubusercontent.com/2521628/168756091-6969528d-a7c7-492e-89f4-ae3649fb10bd.png">



Comments:
_The documentation is not available anymore as the PR was closed or merged._
Train model from scratch with Tensorflow
Author: tqdo
Pull request: no
Labels: name: wontfix

First of all thank you for a great library. I am quite new to the library so sorry if my questions sound dumb. I see a tutorial to train a model from scratch (https://huggingface.co/blog/how-to-train) which is very useful. My questions are:

1/ It seems like that tutorial is for training using PyTorch. Is there any tutorial for Tensorflow?

2/ I am thinking about training an ALBERT model with just 6 layers. If possible I want to use the weights of the pre-trained ALBERT (since all layers share parameters) and also the structure but just change the number of layers from 12 to 6. Is it possible or I need to train entirely from scratch?

Comments:
> 1/ It seems like that tutorial is for training using PyTorch. Is there any tutorial for Tensorflow?

[Here](https://github.com/stefan-it/turkish-bert/blob/master/CHEATSHEET.md) is a cheatsheet for pretraining using TF code. It is written for BERT but the training for ALBERT is similar. 

Regarding pretraining code for ALBERT in TensorFlow, you will find relevant resources in [the official ALBERT repository](https://github.com/google-research/ALBERT). You will need a corpus in txt format with one sentence per line. Ideally after splitting the corpus in smaller shards, you can use the `create_pretraining_data.py` file to transform the txt files in tfrecords. After that, you can run `run_pretraining.py` to pretrain your model. 

> 2/ I am thinking about training an ALBERT model with just 6 layers. If possible I want to use the weights of the pre-trained ALBERT (since all layers share parameters) and also the structure but just change the number of layers from 12 to 6. Is it possible or I need to train entirely from scratch?

This makes me think of what @VictorSanh et al. did in [DistilBERT](https://arxiv.org/abs/1910.01108), namely initializing with every other layer from a vanilla BERT base. The difference is they added a distillation loss for the student to learn from the teacher. I'm not sure whether you can modify the number of layers in the configuration (that could be) but pretraining with less layers might severely impact your model performance. One thing you could do to maximize performance is to pretrain a normal 12-layer ALBERT base and then distil it (more on how to do this [here](https://github.com/huggingface/transformers/tree/master/examples/distillation)). 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Hello, Has there been anything done on this issue, if yes can you please share?
Change `config.encoder_ffn_dim` to `config.decoder_ffn_dim` for decoder impl
Author: cloudhan
Pull request: yes

Change `config.encoder_ffn_dim` to `config.decoder_ffn_dim` for decoder. 

These typos are detected during loading flax from pytorch checkpoint where `encoder_ffn_dim` != `decoder_ffn_dim`

# What does this PR do?

This PR fix typos, these typos is critical!!!

@patrickvonplaten @patil-suraj


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add support for pretraining recurring span selection to Splinter
Author: jvcop
Pull request: yes

This pull request aims to add support for recurring span selection pretraining as proposed by the authors of [Splinter](https://arxiv.org/abs/2101.00438). The pretraining objective differs from the question answering task in a couple of ways:

- There is not a question, but a number of question tokens replacing recurring spans.
- The shape of `start_positions` and `end_positions` is `(batch_size, num_questions)` instead of `(batch_size, )`.
- The shape of `start_logits` and `end_logits` is `(batch_size, num_questions, sequence_length)` instead of `(batch_size, sequence_length)`.
- The loss should ignore zero positions, i.e. `ignore_index=0`. Zeros are used in the original code to denote padded question tokens and their start and end positions.

To this end, we added `SplinterForPreTraining`.

Minimal training example:

```python
import torch
from torch.utils.data import IterableDataset
from transformers import SplinterConfig
from transformers import SplinterForPreTraining
from transformers import Trainer
from transformers import TrainingArguments


class QuestionAnsweringDataset(IterableDataset):
    def __iter__(self):
        yield {
            "input_ids": torch.tensor([101, 104, 123, 456, 104, 234, 567, 102]),
            "attention_mask": torch.tensor([1, 1, 1, 1, 1, 1, 1, 1]),
            "token_type_ids": torch.tensor([0, 0, 0, 0, 0, 0, 0, 0]),
            "question_positions": torch.tensor([1, 4]),
            "start_positions": torch.tensor([2, 5]),
            "end_positions": torch.tensor([3, 6]),
        }


config = SplinterConfig()
model = SplinterForPreTraining(config)
dataset = QuestionAnsweringDataset()

trainer = Trainer(
    model=model,
    args=TrainingArguments(max_steps=3, output_dir="/tmp"),
    train_dataset=dataset,
)
trainer.train()
```

CC @tobigue

@patil-suraj @LysandreJik @patrickvonplaten @oriram 


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks for reviewing the PR! :) I added the suggested changes.

@jvcop will add an answer to this https://github.com/huggingface/transformers/pull/17247#discussion_r873691809

As far as I can see all comments have been addressed - merging! Thanks a lot for your work here @jvcop !

Thanks a lot for the fast review! And to @tobigue who was an integral part of this :tada:
Remove Bert's position_embeddings init in modeling_roberta.py
Author: AlexandreNap
Pull request: yes

# What does this PR do?

Remove useless position_embeddings initialisation in modeling_roberta. 
First one was from Bert's code and second one is the tweaked Roberta's one.
Removed first one.

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@LysandreJik

Comments:
Add type hints for ProphetNet (Pytorch)
Author: jQuinRivero
Pull request: yes

Adding type hints for forward methods in user-facing class for Pegasus model (PyTorch) as mentioned in #16059

@Rocketknight1


Comments:
_The documentation is not available anymore as the PR was closed or merged._

> This looks good to me now, thank you! Let me know when you're ready and I'll merge it.

Go ahead! Thanks!
Accepting torch device objects in the Pipeline init 
Author: courtneysprouse
Pull request: no

### Feature request

Currently Pipeline init only takes an integer as a device argument in the constructor. It would make it a little easier to interface with and integrate with existing code if it also took a pytorch device object. 

### Motivation

It's frustrating not be to be able to use the model.device/self.device object when instantiating the pipeline object 

### Your contribution

I feel like I could do this and I'm happy to take a whack, but I haven't contributed to Transformers yet 

Comments:
cc @Narsil 

@courtneysprouse you're entirely correct, there's no real reason why we don't accept those.

The reason why we want to be able to accept `int` is to support seamless `TF` and `PyTorch` (which don't use the same conventions for devices) but the pipeline abstracts that away. But if you want to use native objects, you should always be able to for sure.

Awesome! Thank you so much!
The accuracy of test set is different in training and evaluating Bert
Author: klyuhang9
Pull request: no

Hi,friends, I meet a problem which puzzled me for couple of days.

I want to training a Bert for two classification task on my custom data.
I choose BertForSequenceClassification model.
And I use the 'Trainer' from 'transformers', the tokenizer, model, metrics and Trainer code as follows:
`


tokenizer=BertTokenizerFast.from_pretrained("bert_base_chinese")

model = BertForSequenceClassification.from_pretrained("bert-base-chinese", return_dict=True)


def computed_metric(eval_pred):

    predictions, labels = eval_pred
    predictions = np.argmax(predictions, axis=1)

    precision, recall, f1, _ =precision_recall_fscore_support(labels,predictions, average="macro")

    acc = accuracy_score(labels, predictions)

    return {"accuracy": acc, 'precision: ':precision, "f1" : f1, "recall" : recall }

trainer = Trainer(
    model,
    args,
    train_dataset= dataset_train,
    eval_dataset= dataset_test,
    tokenizer= tokenizer,
    compute_metrics= computed_metric,

)
`

The problem is: in the training process "eval_accuracy: 0.82", however,  after t he training, I tested the model separately, and the statistical accuracy was only 0.74.
The code for the test as follows：
`
def tokenize(x):

    return tokenizer([x[0]], [x[1]], truncation=True, padding='max_length',max_length=512)


answer_model = []  #record the answer from model

model.eval()

for i in range(len(data_text)):

    encoded_input = tokenize(data_text[i])

    encoded_input["input_ids"] = torch.tensor(encoded_input["input_ids"])
    encoded_input["token_type_ids"] = torch.tensor(encoded_input["token_type_ids"])
    encoded_input["attention_mask"] = torch.tensor(encoded_input["attention_mask"])

    output = model(**encoded_input)
    
    print(i)
    print(output)  # [-2.9663,  2.3750]

    if output["logits"][0][0] < output["logits"][0][1]:
        answer_model.append(1)
    else:
        answer_model.append(0)


acc = accuracy_score(data_label, answer_model)
print("accuracy ",acc)   #only 0.74
`

 I have carefully checked the code for two days and retrained it, but the result is still different. I don't know where the problem might be. I hope to get your help！


Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co) instead?

Thanks!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
DebertaV2Tokenizer wasn't added to the __init__ file  
Author: AhmedIdr
Pull request: no
Labels: name: bug

### System Info

```shell
- `transformers` version: 4.19.0.dev0
- Platform: Linux-5.11.0-46-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.5.1
- PyTorch version (GPU?): 1.10.1+cu113
- Tensorflow version (GPU?): not installed
- Flax version (CPU?/GPU?/TPU?): not installed 
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No
```


### Who can help?

Hey, I just saw that the pull request for DebertaV2Tokenizer and DebertaV2TokenizerFast #15529 was merged around two weeks ago. Unfortunately, importing DebertaV2Tokenizer or DebertaV2TokenizerFast did not work. This is seems to happen because "DebertaV2Tokenizer" wasn't added to the `__init__` file https://github.com/huggingface/transformers/blob/main/src/transformers/__init__.py#L181
`"models.deberta_v2": ["DEBERTA_V2_PRETRAINED_CONFIG_ARCHIVE_MAP", "DebertaV2Config"],`

Adding it to the file fixes the problem

@SaulLu


### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

`from transformers import DebertaV2TokenizerFast, DebertaV2Tokenizer`

### Expected behavior

```shell
To import the tokenizer and not throw an error.
```


Comments:
Hello @AhmedIdr! Now that a new version has been released, you should be able to import it by upgrading to the latest version.

Yeah it seems to be working, thank you.
Add CvT
Author: NielsRogge
Pull request: yes

# What does this PR do?

Co-authored-by: AnugunjNaman <anugunjjha@gmail.com>

This PR adds CvT (Convolutional Vision Transformer) by Microsoft Research.

I just cleaned up the branch of @AnugunjNaman. 

To do:

- [x] make @AnugunjNaman co-author of this PR

Comments:
_The documentation is not available anymore as the PR was closed or merged._