noamaanMulla-03/transformers-issues huggingface dataset:
Deepseep configs keys probelm
Author: 7AM7
Pull request: no
Labels: name: DeepSpeed

https://github.com/huggingface/transformers/blob/24881008a6743e958cc619133b8ee6994ed1cb8c/src/transformers/integrations.py#L288

I guess it should be `if len([x for x in bs_keys if x in config.keys()]) <= 0: ` or `if not len([x for x in bs_keys if x in config.keys()]) <= 0: `

Comments:
cc @stas00 

Could you please explain what is the problem that you're encountering?

These keys **shouldn't be  in the config**, so len() will be > 0 if they are and then the assert happens, so I'm not sure why you're trying to reverse the logic.

```
config = {
    'train_batch_size': 1,
    "train_micro_batch_size_per_gpu": 1,
}
bs_keys = ["train_batch_size", "train_micro_batch_size_per_gpu"]
if len([x for x in bs_keys if x in config.keys()]):
    raise ValueError(
        f"Do not include {bs_keys} entries in the ds config file, as they will be set via --per_device_train_batch_size or its default"
    )
```

Please see: https://huggingface.co/transformers/master/main_classes/trainer.html#shared-configuration
In which directory the downloaded roberta-base models will be stored on linux server conda environment
Author: jonanem
Pull request: no
Labels: name: wontfix

## In which directory the downloaded roberta-base models will be stored on linux server conda environment



Comments:
Models downloaded with the `XXXModel.from_pretrained` method are usually in the torch home folder, which is `~/.cache/torch/transformers`

Thanks for your response. I could see there are some files with below names
b35e7cd126cd4229a746b5d5c29a749e8e84438b14bcdb575950584fe33207e8.70bec105b4158ed9a1747fea67a43f5dee97855c64d62b6ec3742f4cfdb5feda.json

Can we rename those files something like below, the naming convention mentioned in https://s3.amazonaws.com/models.huggingface.co/

roberta-base-config.json

and load them as RobertaTokenizer.from_pretrained('roberta-base')



#2157 is very similar; perhaps it'll answer your question.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


If we have the model loaded, can we then find from where on disk it was loaded?
Add head_mask and decoder_head_mask to PyTorch LED
Author: stancld
Pull request: yes

This PR implements `head_mask` and `decoder_head_mask` for PyTorch LED (and Longformer as there's a copy dependency) and it is the follow-up to the open issue #9814. 

**Motivation:** This PR is a part of an endeavour to enable the usage of `head_mask` and `decoder_head_mask` for all encoder-decoder transformers following the recent work on BART-like models (#9569).

<hr>

Fixes: https://github.com/huggingface/transformers/issues/9814

Reviewers: @patrickvonplaten @LysandreJik @stas00 

Comments:
Add head_mask and decoder_head_mask to FSMT
Author: stancld
Pull request: yes

This PR implements `head_mask` and `decoder_head_mask` for FSMT and it is the follow-up to the open issue #9814. 

**Motivation:** This PR is a part of an endeavour to enable the usage of `head_mask` and `decoder_head_mask` for all encoder-decoder transformers following the recent work on BART-like models (#9569).

<hr>

Fixes: https://github.com/huggingface/transformers/issues/9814

Reviewer: @stas00 

Comments:
I know than one can add, for example, a line like this
```
# Copied from transformers.models.bart.modeling_bart.BartAttention with Bart->FSMT
```
before `Attention` module in FSMT. However, this does not copy only additions, but the whole module from BART, which is, in this case, undesirable, I guess, as these modules are a little bit different. But maybe there is another way I am not aware of.

@LysandreJik, @patrickvonplaten - how can we make sure fsmt gets tracked and synced with all the bart-family changes? while the tokenizer is different, the model is ~95% identical.

as @stancld said, we can do that with some statements of the following kind:

```
# Copied from transformers.models.bart.modeling_bart.BartAttention with Bart->FSMT
```

The difference between the BART and FSMT implementation of the targeted object must only be the "BART" occurrences that change to "FSMT". @sgugger can tell you more about it.

Thank you, @LysandreJik

I think this is really a question to @patrickvonplaten  - who I remember was planning to refactor FSMT to match what he did for Bart. So if this is still planned, Patrick perhaps you could add this item to the agenda - keeping FSMT in sync with the Bart-family (modeling only - tokenizer is similar to xlm).

So the currently proposed solution can't be used, since Bart diverged since FSMT forked it.

It might help to treat FSMT as Bart with the main difference of it having a dual vocab and no tied weights - and a few layers that are different - but identical otherwise. (again for the model only).

> I think this is really a question to @patrickvonplaten - who I remember was planning to refactor FSMT to match what he did for Bart. So if this is still planned, Patrick perhaps you could add this item to the agenda - keeping FSMT in sync with the Bart-family (modeling only - tokenizer is similar to xlm).

Yes, the FSTM / ProphetNet refactor is still on my ToDo List (think next week is reasonable). After the refactor I'll try to add as many # Copied from statements to keep the models in sync. Nevertheless, this PR can be merged as it is now!

Great work @stancld
Add a new model ConvBert
Author: RyanHuangNLP
Pull request: no
Labels: name: New model

# 🌟 New model addition

Pre-trained language models like BERT and its variants have recently achieved impressive performance in various natural language understanding tasks. However, BERT heavily relies on the global self-attention block and thus suffers large memory footprint and computation cost. Although all its attention heads query on the whole input sequence for generating the attention map from a global perspective, we observe some heads only need to learn local dependencies, which means the existence of computation redundancy. We therefore propose a novel span-based dynamic convolution to replace these self-attention heads to directly model local dependencies. The novel convolution heads, together with the rest self-attention heads, form a new mixed attention block that is more efficient at both global and local context learning. We equip BERT with this mixed attention design and build a ConvBERT model. Experiments have shown that ConvBERT significantly outperforms BERT and its variants in various downstream tasks, with lower training cost and fewer model parameters. Remarkably, ConvBERTbase model achieves 86.4 GLUE score, 0.7 higher than ELECTRAbase, while using less than 1/4 training cost.

<!-- Important information -->

## Open source status

* [x] the model implementation is available: (https://github.com/yitu-opensource/ConvBert)
* [x] the model weights are available: (https://drive.google.com/drive/folders/1pSsPcQrGXyt1FB45clALUQf-WTNAbUQa)
* [x] who are the authors: (@zihangJiang @zhoudaquan)


Comments:
there is one implement:https://github.com/JunnYu/ConvBert_huggingface

I have implemented this model in [https://github.com/gitabtion/ConvBert-PyTorch](https://github.com/gitabtion/ConvBert-PyTorch), and  it pass the unittest right now. :heavy_check_mark:

> I have implemented this model in https://github.com/gitabtion/ConvBert-PyTorch, and it pass the unittest right now. heavy_check_mark

how about a conversion script?
[research proj] [lxmert] remove bleach dependency
Author: stas00
Pull request: yes

github reports `bleach==3.1.5` to have a vulnerability and it's not really used anywhere in the code, and because it has a fixed version set that is vulnerable, so just as well remove it completely from deps.
https://github.com/huggingface/transformers/security/dependabot/examples/research_projects/lxmert/requirements.txt/bleach/open

@LysandreJik, @sgugger, @patrickvonplaten 

Comments:
Missing None verification in the CLM language modeling example
Author: Aunsiels
Pull request: no

Here: https://github.com/huggingface/transformers/blob/1682804ebd504d3381523116773583a52f35afd1/examples/language-modeling/run_clm.py#L230, data_args.train_file can be None (as it is checked some lines above). Therefore, there should be a check to see if it is the case or not.


Comments:
HI @Aunsiels you are right, thanks! Feel free to open PR to fix it :)

If `train_file` is `None` it should use `validation_file` to get the extension
Added integration tests for TensorFlow implementation of the ALBERT model
Author: spatil6
Pull request: yes

Added integration tests for TensorFlow implementation of the ALBERT model


Fixes #9946 


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

@LysandreJik 

Comments:
[Good first issue] ALBERT TensorFlow Integration tests
Author: LysandreJik
Pull request: no
Labels: name: TensorFlow

name: Tests

name: Good First Issue

The TensorFlow implementation of the ALBERT model currently has no integration tests. This is problematic as the behavior can diverge without being noticed.

The [test_modeling_tf_albert.py](https://github.com/huggingface/transformers/blob/master/tests/test_modeling_tf_albert.py) file should be updated to include integration testing.

An example of a good modeling integration test is visible in the [test_modeling_tf_bert.py#L365-L387](https://github.com/huggingface/transformers/blob/1809de5165804666ba6c6a02a9d177f6683869cc/tests/test_modeling_tf_bert.py#L365-L387) file:

https://github.com/huggingface/transformers/blob/1809de5165804666ba6c6a02a9d177f6683869cc/tests/test_modeling_tf_bert.py#L365-L387

Some additional tips:
- The test must be marked as slow using the `@slow` decorator, so as to be run *daily*, and not on every commit of every branch/pull request of this repository.
- The test must be decorated with the `@require_tf` decorator so as to only be run in environments using PyTorch.
- A single test is necessary. If you feel like implementing multiple of these, then sharing the same checkpoint would be ideal so as to reduce download time.

Comments:
TF DistilBERT integration tests 
Author: spatil6
Pull request: yes

Added integration tests for TensorFlow implementation of the DistilBERT model


Fixes #9953 


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


Who can review?
@LysandreJik 


Comments:
[Good first issue] DistilBERT TensorFlow Integration tests
Author: LysandreJik
Pull request: no
Labels: name: TensorFlow

name: Tests

name: Good First Issue

The TensorFlow implementation of the DistilBERT model currently has no integration tests. This is problematic as the behavior can diverge without being noticed.

The [test_modeling_tf_distilbert.py](https://github.com/huggingface/transformers/blob/master/tests/test_modeling_tf_distilbert.py) file should be updated to include integration testing.

An example of a good modeling integration test is visible in the [test_modeling_tf_bert.py#L365-L387](https://github.com/huggingface/transformers/blob/1809de5165804666ba6c6a02a9d177f6683869cc/tests/test_modeling_tf_bert.py#L365-L387) file:

https://github.com/huggingface/transformers/blob/1809de5165804666ba6c6a02a9d177f6683869cc/tests/test_modeling_tf_bert.py#L365-L387

Some additional tips:
- The test must be marked as slow using the `@slow` decorator, so as to be run *daily*, and not on every commit of every branch/pull request of this repository.
- The test must be decorated with the `@require_tf` decorator so as to only be run in environments using PyTorch.
- A single test is necessary. If you feel like implementing multiple of these, then sharing the same checkpoint would be ideal so as to reduce download time.

Comments:
Added integration tests for TensorFlow implementation of the mobileBERT
Author: spatil6
Pull request: yes

Added integration tests for TensorFlow implementation of the ALBERT model

Fixes #9955

Before submitting

  This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  Did you read the contributor guideline,
Pull Request section?
  Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case.
  Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
  Did you write any new necessary tests?

@LysandreJik

Comments:
[Good first issue] MobileBERT TensorFlow Integration tests
Author: LysandreJik
Pull request: no
Labels: name: TensorFlow

name: Tests

name: Good First Issue

The TensorFlow implementation of the MobileBERT model currently has no integration tests. This is problematic as the behavior can diverge without being noticed.

The [test_modeling_tf_mobilebert.py](https://github.com/huggingface/transformers/blob/master/tests/test_modeling_tf_mobilebert.py) file should be updated to include integration testing.

An example of a good modeling integration test is visible in the [test_modeling_tf_bert.py#L365-L387](https://github.com/huggingface/transformers/blob/1809de5165804666ba6c6a02a9d177f6683869cc/tests/test_modeling_tf_bert.py#L365-L387) file:

https://github.com/huggingface/transformers/blob/1809de5165804666ba6c6a02a9d177f6683869cc/tests/test_modeling_tf_bert.py#L365-L387

Some additional tips:
- The test must be marked as slow using the `@slow` decorator, so as to be run *daily*, and not on every commit of every branch/pull request of this repository.
- The test must be decorated with the `@require_tf` decorator so as to only be run in environments using PyTorch.
- A single test is necessary. If you feel like implementing multiple of these, then sharing the same checkpoint would be ideal so as to reduce download time.

Comments:
Added integration tests for TensorFlow implementation of the MPNet model
Author: spatil6
Pull request: yes

Added integration tests for TensorFlow implementation of the ALBERT model


Fixes #9956


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

@LysandreJik 

Comments:
[Good first issue] MPNet TensorFlow Integration tests
Author: LysandreJik
Pull request: no
Labels: name: TensorFlow

name: Tests

name: Good First Issue

The TensorFlow implementation of the MPNet model currently has no integration tests. This is problematic as the behavior can diverge without being noticed.

The [test_modeling_tf_mpnet.py](https://github.com/huggingface/transformers/blob/master/tests/test_modeling_tf_mpnet.py) file should be updated to include integration testing.

An example of a good modeling integration test is visible in the [test_modeling_tf_bert.py#L365-L387](https://github.com/huggingface/transformers/blob/1809de5165804666ba6c6a02a9d177f6683869cc/tests/test_modeling_tf_bert.py#L365-L387) file:

https://github.com/huggingface/transformers/blob/1809de5165804666ba6c6a02a9d177f6683869cc/tests/test_modeling_tf_bert.py#L365-L387

Some additional tips:
- The test must be marked as slow using the `@slow` decorator, so as to be run *daily*, and not on every commit of every branch/pull request of this repository.
- The test must be decorated with the `@require_tf` decorator so as to only be run in environments using PyTorch.
- A single test is necessary. If you feel like implementing multiple of these, then sharing the same checkpoint would be ideal so as to reduce download time.

Comments:
Added integration tests for Pytorch implementation of the ALBERT model
Author: spatil6
Pull request: yes

Added integration tests for Pytorch implementation of the ALBERT model


Fixes #9945 


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

@LysandreJik 

Comments:
[Good first issue] ALBERT PyTorch Integration tests
Author: LysandreJik
Pull request: no
Labels: name: PyTorch

name: Tests

name: Good First Issue

The PyTorch implementation of the ALBERT model currently has no integration tests. This is problematic as the behavior can diverge without being noticed.

The [test_modeling_albert.py](https://github.com/huggingface/transformers/blob/master/tests/test_modeling_albert.py) file should be updated to include integration testing.

An example of a good modeling integration test is visible in the [test_modeling_bert.py#L552-L565](https://github.com/huggingface/transformers/blob/master/tests/test_modeling_bert.py#L552-L565) file:

https://github.com/huggingface/transformers/blob/1809de5165804666ba6c6a02a9d177f6683869cc/tests/test_modeling_bert.py#L552-L565

Some additional tips:
- The test must be marked as slow using the `@slow` decorator, so as to be run *daily*, and not on every commit of every branch/pull request of this repository.
- The test must be decorated with the `@require_torch` decorator so as to only be run in environments using PyTorch.
- A single test is necessary. If you feel like implementing multiple of these, then sharing the same checkpoint would be ideal so as to reduce download time.

Comments:
Hi Can I have a go at this issue?

Please do!
[Good first issue] MPNet PyTorch Integration tests
Author: LysandreJik
Pull request: no
Labels: name: PyTorch

name: Tests

name: Good First Issue

The PyTorch implementation of the MPNet model currently has no integration tests. This is problematic as the behavior can diverge without being noticed.

The [test_modeling_mpnet.py](https://github.com/huggingface/transformers/blob/master/tests/test_modeling_mpnet.py) file should be updated to include integration testing.

An example of a good modeling integration test is visible in the [test_modeling_bert.py#L552-L565](https://github.com/huggingface/transformers/blob/master/tests/test_modeling_bert.py#L552-L565) file:

https://github.com/huggingface/transformers/blob/1809de5165804666ba6c6a02a9d177f6683869cc/tests/test_modeling_bert.py#L552-L565

Some additional tips:
- The test must be marked as slow using the `@slow` decorator, so as to be run *daily*, and not on every commit of every branch/pull request of this repository.
- The test must be decorated with the `@require_torch` decorator so as to only be run in environments using PyTorch.
- A single test is necessary. If you feel like implementing multiple of these, then sharing the same checkpoint would be ideal so as to reduce download time.

Comments:
@LysandreJik  "test_modeling_mpnet.py"  it already have integration test.

You're correct! That's on me, thanks for letting me know.
distilbert: fix creation of sinusoidal embeddings
Author: stefan-it
Pull request: yes

Hi,

similar issue as reported by @stas00 with BART, see #8226.

The creation of sinusoidal embeddings is currently not working on PyTorch 1.8+.

It fails with:

```bash
  File "/mnt/europeana-bert/flair/flair/embeddings/token.py", line 820, in __init__
    self.model = AutoModel.from_pretrained(model, config=config, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/transformers/models/auto/modeling_auto.py", line 728, in from_pretrained
    return MODEL_MAPPING[type(config)].from_pretrained(
  File "/opt/conda/lib/python3.8/site-packages/transformers/modeling_utils.py", line 1034, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
  File "/opt/conda/lib/python3.8/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 419, in __init__
    self.embeddings = Embeddings(config)  # Embeddings
  File "/opt/conda/lib/python3.8/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 88, in __init__
    create_sinusoidal_embeddings(
  File "/opt/conda/lib/python3.8/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 76, in create_sinusoidal_embeddings
    out[:, 0::2] = torch.FloatTensor(np.sin(position_enc[:, 0::2]))
RuntimeError: a view of a leaf Variable that requires grad is being used in an in-place operation.
```

I've seen this problem when trying to train a model in Flair with DistilBERT as feature-based embeddings, as well as when training a DistilBERT model from scratch using the official example.

It can be reproduced in a `nvcr.io/nvidia/pytorch:20.12-py3` container, that comes with PyTorch 1.8.

Comments: