noamaanMulla-03/transformers-issues huggingface dataset:
pip install transformers by default install 2.5.1
Author: guoxuxu
Pull request: no
Labels: name: wontfix

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     

https://huggingface.co/transformers/installation.html

Running '''pip install transformers''' by default installs 2.5.1
Could you also specify TensorFlow CPU or GPU version should be installed

- `transformers` version:
- Platform:
- Python version: 3.7.4
- PyTorch version (GPU?): yes
- Tensorflow version (GPU?): 2.2.0
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: 

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 
 albert, bert, GPT2, XLM: @LysandreJik 
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao 
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao 
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->


Comments:
Running `pip install transformers` installs the latest version for me. Which pip version are you running on?

You can install both cpu or gpu for tensorflow, if you want to run on a cpu or a gpu.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
example for passage re-ranking using bert-multilingual-passage-reranking-msmarco
Author: lalitkishorevyas
Pull request: no
Labels: name: wontfix

can you give me a working example this

`from transformers import AutoTokenizer, AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained("amberoad/bert-multilingual-passage-reranking-msmarco")

model = AutoModelForSequenceClassification.from_pretrained("amberoad/bert-multilingual-passage-reranking-msmarco")
`


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
what's the values of start_positon and end_position while the answer is impossible in run_squad.py
Author: ppyu
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help
### Details
 I want to know what's the values of `start_positon` and `end_position` while the answer is **impossible** in `run_squad.py`.
`start_positions = end_postions = -1` OR `start_positions = end_postions = 0`

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
`add_prefix_space=True` option in the BPE tokenizer 
Author: h56cho
Pull request: no
Labels: name: wontfix

Hello,
I understand that when I add the `add_prefix_space=True` option in the BPE tokenizer statement, the tokenizer will add a space in the beginning of every sequence.

Is there some specific advantages of using the `add_prefix_space=True` option for BPE tokenizer (compared to when I don't use the option), given that all my sequences start without a space in the beginning.?

Thanks,

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
ModuleNotFoundError: No module named 'tokenizations.tokenizations'
Author: louisabraham
Pull request: no

```
    nlp = en_trf_bertbaseuncased_lg.load()
  File "/usr/lib/python3.9/site-packages/en_trf_bertbaseuncased_lg/__init__.py", line 12, in load
    return load_model_from_init_py(__file__, **overrides)
  File "/usr/lib/python3.9/site-packages/spacy/util.py", line 239, in load_model_from_init_py
    return load_model_from_path(data_path, meta, **overrides)
  File "/usr/lib/python3.9/site-packages/spacy/util.py", line 202, in load_model_from_path
    cls = get_lang_class(lang)
  File "/usr/lib/python3.9/site-packages/spacy/util.py", line 74, in get_lang_class
    if lang in registry.languages:
  File "/usr/lib/python3.9/site-packages/catalogue.py", line 56, in __contains__
    has_entry_point = self.entry_points and self.get_entry_point(name)
  File "/usr/lib/python3.9/site-packages/catalogue.py", line 140, in get_entry_point
    return entry_point.load()
  File "/usr/lib/python3.9/importlib/metadata.py", line 77, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/lib/python3.9/site-packages/spacy_transformers/__init__.py", line 2, in <module>
    from .pipeline.tok2vec import TransformersTok2Vec  # noqa
  File "/usr/lib/python3.9/site-packages/spacy_transformers/pipeline/__init__.py", line 3, in <module>
    from .wordpiecer import TransformersWordPiecer  # noqa
  File "/usr/lib/python3.9/site-packages/spacy_transformers/pipeline/wordpiecer.py", line 3, in <module>
    from tokenizations import get_alignments
  File "/usr/lib/python3.9/site-packages/tokenizations/__init__.py", line 2, in <module>
    from .tokenizations import (
ModuleNotFoundError: No module named 'tokenizations.tokenizations'
```


Comments:
Sorry, wrong repo
No module named 'transformers.modeling_albert'
Author: hjzhang1018
Pull request: no

- Platform: Colab
- Python version:
- PyTorch version (GPU?):GPU 
- Tensorflow version (GPU?):GPU
- Using GPU in script?:Yes

 examples/token-classification: @stefan-it
 -->

The problem arises when using:
* [ ] the official example scripts: I'm following the tutorial "23_Transfer_Learning_With_ChemBERTa_Transformers_Pt_2.ipynb" to reproduce the results. However I got this error message " No module named 'transformers.modeling_albert". I cannot figure out the reason. The following is the snippet I used:

```
!curl -Lo conda_installer.py https://raw.githubusercontent.com/deepchem/deepchem/master/scripts/colab_install.py
import conda_installer
conda_installer.install()
!/root/miniconda/bin/conda info -e
!pip install --pre deepchem 
import deepchem  
deepchem.__version__
from rdkit import Chem

!git clone https://github.com/NVIDIA/apex
!cd /content/apex
!pip install -v --no-cache-dir /content/apex
!pip install transformers
!pip install git+https://github.com/seyonechithrananda/simpletransformers.git@pip
!pip install wandb
!cd ..

!git clone https://github.com/seyonechithrananda/bert-loves-chemistry.git

%cd /content/bert-loves-chemistry

import os
import numpy as np
import pandas as pd
from typing import List
# import molnet loaders from deepchem
from deepchem.molnet import load_bbbp, load_clearance, load_clintox, load_delaney, load_hiv, load_qm7, load_tox21
from rdkit import Chem
# import MolNet dataloder from bert-loves-chemistry fork
from utils.molnet_dataloader import load_molnet_dataset, write_molnet_dataset_for_chemprop

tasks, (train_df, valid_df, test_df), transformers = load_molnet_dataset("clintox", tasks_wanted=None)

from simpletransformers.classification import ClassificationModel
import logging
logging.basicConfig(level=logging.INFO)
transformers_logger = logging.getLogger("transformers")
transformers_logger.setLevel(logging.WARNING)
```
Any suggestions and help are appreciated! Thank you!

Comments:
Hey @hjzhang1018,

This does not seem to be a bug in Transformers, but rather in `seyonechithrananda/simpletransformers.git` so I'm not sure here is the correct place to post the issue. I think one has to change the line `from transformers.modeling_albert import ....` to `from transformers.models.albert.modeling_albert import ...` in the respective repo.
Fail to reload tokenizer from save_pretrained method
Author: jc-hou
Pull request: no

Hi,
to reproduce:
```
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
tokenizer.save_pretrained(".")
tokenizer = AutoTokenizer.from_pretrained(".")
```
with error msg:
```
file ./config.json not found
Traceback (most recent call last):
  File "/data/stars/user/jhou/Test/pytorch_test/huggingface/jc-hou_fork/transformers/src/transformers/configuration_utils.py", line 389, in get_config_dict
    local_files_only=local_files_only,
  File "/data/stars/user/jhou/Test/pytorch_test/huggingface/jc-hou_fork/transformers/src/transformers/file_utils.py", line 1015, in cached_path
    raise EnvironmentError("file {} not found".format(url_or_filename))
OSError: file ./config.json not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/stars/user/jhou/Test/pytorch_test/huggingface/jc-hou_fork/transformers/src/transformers/models/auto/tokenization_auto.py", line 337, in from_pretrained
    config = AutoConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)
  File "/data/stars/user/jhou/Test/pytorch_test/huggingface/jc-hou_fork/transformers/src/transformers/models/auto/configuration_auto.py", line 341, in from_pretrained
    config_dict, _ = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/data/stars/user/jhou/Test/pytorch_test/huggingface/jc-hou_fork/transformers/src/transformers/configuration_utils.py", line 401, in get_config_dict
    raise EnvironmentError(msg)
OSError: Can't load config for '.'. Make sure that:

- '.' is a correct model identifier listed on 'https://huggingface.co/models'

- or '.' is the correct path to a directory containing a config.json file

```
Thanks.
transformers:4.1.0
 tokenizers: @mfuntowicz
 


Comments:
Hi @jc-hou,

The `Auto*` classes require the `config.json` (which is saved when you save the model) file to find the correct model/tokenizer class for loading the model/tokenizer. To directly load the tokenizer without the model use the specific tokenizer class, in this case, `BertTokenizer`.

Hi, thanks. I understand.
[Seq2Seq Templates] Correct some TF-serving errors and add gradient checkpointing to PT by default.
Author: patrickvonplaten
Pull request: yes

# What does this PR do?

This PR improves the Seq2Seq model templates. 
Notably:

- a too model-specific test is removed from PyTorch
- gradient checkpointing is added to PyTorch
- some tf-serving incompatible statements are removed


Comments:
Is there a way to evaluate models during training in Multi-gpu setting
Author: ereday
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

Hi all,
I always see comments in examples saying that "when single GPU otherwise metrics may not average well". So is this really something that shouldn't be done? I mean, is there a way to evaluate the model safely after each epoch in the multi-gpu training setting? Thanks.
<!-- A clear and concise description of the question. -->


Comments:
Yes. After each batch has been completed by a GPU, you can store its results and corresponding labels in a shared space (e.g. CPU/memory). Then, when all batches are done, you can evaluate the epoch by calculating your metric/avg loss over all gathered results. It has been suggested to only keep track of the batch averages, but I feel like that is just an approximation of an approximation.

My code is a bit too bombastic to share, but I use something like this:

- custom Trainer class with a train/evaluate/test loop. It also has a 'performer' property
- the performer is an instance of a custom Performer singleton class that keeps track of losses, labels, and/or predictions at the end of each processed batch. Note that this means that each separate process keeps track of its own progress. Results aren't shared between processes until the end of the epoch
- at the end of each epoch, all results are `gather`ed from the different processes to a single one, which then calculates the average over all collected batches, and broadcasts that information (e.g. avg_loss, avg_secondary_metric) back to the other processes

It is rather complex and perhaps too much code and hassle for what it is, but for me it was more the learning experience of how to work with multi-GPU and `gather` with, as a bonus, fast evaluation and testing.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Hey @BramVanroy  thank you for your reply and sorry for late reply. Unfortunately I am not sure if I understood what you suggested. Do you suggest to transfer predictions and gold labels of each bacth to the CPU  and then calculate metrics by using them if I want to evaluate my model during training ? 
As far as I see in most of the examples in the repo, it is okay to evaluate the model by using multi gpus once the training is over. They only do not suggest to eval it during training. There must be some reason for that ?  
 

I did not even know how to "evaluate models during training in single-gpu setting".
[PyTorch Bart] Split Bart
Author: patrickvonplaten
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


Comments:
[Seq2Seq Templates] Add forgotten imports to templates
Author: patrickvonplaten
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

I accidentally forgot to add this import in #9342.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


Comments:
[prophetnet] wrong import
Author: stas00
Pull request: yes

```
python -c "from apex.normalization import FusedProphetNetLayerNorm"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: cannot import name 'FusedProphetNetLayerNorm' from 'apex.normalization' (/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/apex/normalization/__init__.py)
```
It looks like this code has never been tested, so it silently fails inside try/except.

Discovered this by accident in https://github.com/huggingface/transformers/issues/9338#issuecomment-752217708

@patrickvonplaten, @LysandreJik 

note, prophetnet is missing from .github/PULL_REQUEST_TEMPLATE.md, .github/ISSUE_TEMPLATE/bug-report.md

Comments: