noamaanMulla-03/transformers-issues huggingface dataset:
ImportError: cannot import name 'is_flax_available' from 'transformers.file_utils'
Author: brendan-AI
Pull request: no

- `transformers` version: 3.4.0
- Platform: macOS-10.13.6-x86_64-i386-64bit
- Python version: 3.8.3
- PyTorch version (GPU?): 1.7.0 (False)
- Tensorflow version (GPU?): 2.3.0 (False)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


The problem arises when using trying to import:
from transformers import AutoTokenizer, AutoModel

Running on Jupyter notebook in conda virtual env
full error message below:
`---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-50-13d0bae188dc> in <module>
----> 1 from transformers import AutoTokenizer, AutoModel
      2 
      3 tokenizer = AutoTokenizer.from_pretrained("bashar-talafha/multi-dialect-bert-base-arabic")
      4 model = AutoModel.from_pretrained("bashar-talafha/multi-dialect-bert-base-arabic")

/opt/anaconda3/envs/py38/lib/python3.8/site-packages/transformers/__init__.py in <module>
     20 # Integrations: this needs to come before other ml imports
     21 # in order to allow any 3rd-party code to initialize properly
---> 22 from .integrations import (  # isort:skip
     23     is_comet_available,
     24     is_optuna_available,

/opt/anaconda3/envs/py38/lib/python3.8/site-packages/transformers/integrations.py in <module>
     57 
     58 from .file_utils import is_torch_tpu_available
---> 59 from .trainer_callback import TrainerCallback
     60 from .trainer_utils import PREFIX_CHECKPOINT_DIR, BestRun
     61 from .utils import logging

/opt/anaconda3/envs/py38/lib/python3.8/site-packages/transformers/trainer_callback.py in <module>
     24 from tqdm.auto import tqdm
     25 
---> 26 from .trainer_utils import EvaluationStrategy
     27 from .training_args import TrainingArguments
     28 from .utils import logging

/opt/anaconda3/envs/py38/lib/python3.8/site-packages/transformers/trainer_utils.py in <module>
     23 
     24 from .file_utils import is_tf_available, is_torch_available
---> 25 from .tokenization_utils_base import ExplicitEnum
     26 
     27 

/opt/anaconda3/envs/py38/lib/python3.8/site-packages/transformers/tokenization_utils_base.py in <module>
     30 import numpy as np
     31 
---> 32 from .file_utils import (
     33     add_end_docstrings,
     34     cached_path,

ImportError: cannot import name 'is_flax_available' from 'transformers.file_utils' (/opt/anaconda3/envs/py38/lib/python3.8/site-packages/transformers/file_utils.py)`


Comments:
It seems you have a mismatch between several `transformers` versions.

We generally recommend using virtual environments not relying on `conda`, as full `transformers` support is not yet available on `conda`. Can you reproduce using a pip virtual env?

```py
python -m venv .env
source .env/bin/activate
pip install transformers
```

that works in the virtual env, thank you!
Wrong files names in model list for "xlm-roberta-large-finetuned-conll03-german"
Author: padmalcom
Pull request: no

Hi, the names of the model files for "xlm-roberta-large-finetuned-conll03-german" are incorrect and have to model name as prefix.

Example:
xlm-roberta-large-finetuned-conll03-german-config.json instead of config.json



Comments:
Hi, what do you mean they're incorrect? Are you having issues loading the files in your `transformers` objects?

Hi, click on [https://huggingface.co/xlm-roberta-large-finetuned-conll03-german](https://huggingface.co/xlm-roberta-large-finetuned-conll03-german) and then on _List all files in model_ and move the mouse over each link. You will see that the files have a prefix (namely the name of the model). This is not allowed by the transformers api, since it expects to find e.g. config.json instead of xlm-roberta-large-finetuned-conll03-german-config.json. Hope this explaination helps.

~I cannot see the prefix in the link you've given, and~ I can correctly load the models in the library:

```py
from transformers import XLMRobertaModel
XLMRobertaModel.from_pretrained("xlm-roberta-large-finetuned-conll03-german")
```
works correctly.

The legacy models (e.g. `bert-base-cased`, this one too) had the prefix, but we've changed the approach since and only keep them that way for backwards compatibility.

I can confirm this is intended

Okay, I only experience this error when downloading the files and load from local storage instead of providing a name. But when this is working as intended, it is fine for me.
Changing XLNet default from not using memories to 512 context size following paper
Author: TevenLeScao
Pull request: yes

In #8317, we found out that calling `XLNetLMHeadModel.from_pretrained(mem_len=384)` still produced the FutureWarning that announces that the default configuration will change in 3.5.0, as the config first gets initialized with `mem_len=0` then `mem_len` gets changed. This PR moves the warning to the `forward` pass in the model to avoid this.

Comments:
Ha yeah I saw your message on Slack just afterwards, I'll just change the default :)

It's changed !

Merging now so that it's in v3.5.0!
Is there any Jupyter notebook or detailed example using BertGeneration or EncoderDecoderModel classes?
Author: kseh92
Pull request: no

I have been looking to do some seq2seq tasks in the huggingface-transformers using BertGeneration or EncoderDecoderModel classes.

But I only have ended up finding some simple examples described in the API documentation like below.

```
>>> import torch

>>> tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
>>> model = EncoderDecoderModel.from_encoder_decoder_pretrained('bert-base-uncased', 'bert-base-uncased') # initialize Bert2Bert from pre-trained checkpoints

>>> # forward
>>> input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0)  # Batch size 1
>>> outputs = model(input_ids=input_ids, decoder_input_ids=input_ids)

>>> # training
>>> outputs = model(input_ids=input_ids, decoder_input_ids=input_ids, labels=input_ids, return_dict=True)
>>> loss, logits = outputs.loss, outputs.logits

>>> # save and load from pretrained
>>> model.save_pretrained("bert2bert")
>>> model = EncoderDecoderModel.from_pretrained("bert2bert")

>>> # generation
>>> generated = model.generate(input_ids, decoder_start_token_id=model.config.decoder.pad_token_id)
```

Is there any Jupyter notebook or detailed example using BertGeneration or EncoderDecoderModel classes specifically? Even though I already know that these classes are released quite recently...

It would be a great help for me if I could find one. Thanks!

Comments:
Releasing in ~1 week - it's almost ready :-) 

Thanks for letting me know! :)

I've released two condensed notebooks as mentioned here: https://discuss.huggingface.co/t/leveraging-pre-trained-checkpoints-for-summarization/835/13?u=patrickvonplaten

Will also release a longer educational blog post in a bit.

https://huggingface.co/blog/warm-starting-encoder-decoder
Model versioning
Author: julien-c
Pull request: yes
Labels: name: Core: CLI

name: Documentation

name: model card

**Write-up about the context for this change, and the enabled features is at https://discuss.huggingface.co/t/announcement-model-versioning-upcoming-changes-to-the-model-hub/1914**

In short:

1. changes to the file downloading code used in `from_pretrained()` methods to use the new file URLs backed by huggingface-hosted git repos.
2. changes to the model upload CLI to create a model repo then be able to git clone and git push to it.


Comments:
LGTM!

When this PR is merged, how do I modify:
```
        data_cached = cached_path(
            "https://cdn-datasets.huggingface.co/translation/wmt_en_ro-tr40k-va0.5k-te0.5k.tar.gz",
            extract_compressed_file=True,
        )
```
to take advantage of this new API?  https://github.com/huggingface/transformers/blob/master/examples/seq2seq/test_bash_script.py#L27
This code was just merged yesterday.
Patch token classification pipeline
Author: LysandreJik
Pull request: yes

This PR patches issues found with the `TokenClassificationPipeline` since the merge of https://github.com/huggingface/transformers/pull/5970, namely not being able to load a slow tokenizer in the pipeline.

It also sets the `ignore_subwords` to `False` by default, as this does not work with the slow tokenizers. No release have been done since the introduction of that argument, so it is not a breaking change.

Comments:
Merging now to ensure the bug is patched in v3.5.0. Will address your comments in a future PR @Narsil.
sentencepiece==0.1.92 causing segmentation fault
Author: boy2000-007man
Pull request: no
Labels: name: dependencies

# 🐛 Bug

## Information

`transformers==2.9.1`
`torch==1.4.0`

Starting from today,
I notice newly released `sentencepiece==0.1.92` causing segmentation fault while calling torch functions.
Downgrade to `sentencepiece==0.1.91` solve it.


Comments:
@boy2000-007man  Hi, folk. Just curious about how do you find this bug? It costs me almost the whole day... Anyway, thank you so much!

OMG!!! Awesome Advice!!!! 

I spent a whole night to address the dependencies problems and almost lost my mind. This answer saved my life. Appreciate!

Thanks for this! Also curious how you worked this out - I've spent a whole day trying to figure this out!

Thanks so much you saved my day.

I was dreading the thought of having to dive into this issue with faulthandler and meticulously cross referencing dependencies with a working version....but this post just saved my night.  Thanks @boy2000-007man 

This seems like a new pytorch v1.4.0 incompatibility issue with the latest huggingface releases.  I'm assuming this may have been missed due to the focus on v1.5.0 support, but it seems like many people cannot make the jump to cuda 10.2/pytorch 1.5.0 currently, so this seems like a pretty big headache that should be addressed.

Closing this as solved by #5418

You are excellent!

same problem when use  sentencepiece==0.1.94

Having the same problem with sentencepiece==0.1.94

Cf #8199 we will remove the hard dependency on sentencepiece (replaced by the `tokenizers` library) in a coming release, probably end of next week.

Thank you a lot! You saved my day!
[examples] better PL version check
Author: stas00
Pull request: yes

`pkg_resources.require(f"{pkg}>={min_ver}")` does a great job of checking the minimal required versions at runtime, but I wasn't aware that it checks that the dependencies meet their requirements too! So I started getting a false alarm about needing `pytorch-lightning=1.0.4` when I already had a higher version. 

The problem was in:
```
$ python -c 'import pkg_resources; pkg_resources.require("pytorch_lightning>=1.0.4")'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pkg_resources/__init__.py", line 884, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pkg_resources/__init__.py", line 775, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (torch 1.8.0.dev20201106+cu110 (/mnt/nvme1/anaconda3/envs/main-38/lib/python3.8/site-packages), Requirement.parse('torch<1.8,>=1.3'), {'pytorch-lightning'})
```

Long story short, currently PL explicitly excludes pytorch-1.8 in its dependency list: https://github.com/PyTorchLightning/pytorch-lightning/issues/4596 - which leads to this problem. When I upgrade PL pip uninstalls `pytorch-1.8` - thanks, but no thanks - rtx-3090 doesn't work with pytorch < 1.8. So I install it back and now I get the failure above. Except in the current code it's masked by the `try/except` block which hides the actual problem. So this is not good.

This PR rewrites the check in a way that doesn't check whether the dependencies of the package in questions are in order, and only checks that the minimal version is correct.

@sshleifer  

Comments:
using multi_gpu consistently
Author: stas00
Pull request: yes

As discussed [here](https://github.com/huggingface/transformers/pull/8341#issuecomment-722705833) this PR replaces 
* `multiple_gpu`
* `multigpu`
with `multi_gpu` for consistency

There is no functionality change otherwise.

I did repo-wide:
```
find . -type d -name ".git" -prune -o -type f -exec perl -pi -e 's#(multiple_gpu|multigpu)#multi_gpu#g' {} \;
```

@LysandreJik, @sgugger 

Comments:
[No merge] TF integration testing
Author: LysandreJik
Pull request: yes

Adds integrationt tests for BERT, ELECTRA and Longformer to ensure that PR such as https://github.com/huggingface/transformers/pull/7605 do not impact the current state of models. RoBERTa not done because it's already done.

Patches a bug with the `ElectraForPreTraining` when batch size = 1

Comments:
@LysandreJik Is this PR done with your last changes in the tests?

Ah, I had forgotten about this. I'll rebase and ping you for review


should be good for review @jplu 

They're not the same as they don't rely on the full checkpoints but on some random tiny ones, to make the CI faster.
It does test every same aspect, however: the weights loading, the full inference, the expected results.
Pipeline for text generation
Author: r0levrai
Pull request: no

Hello,
* pipelines concise syntax and features are really nice, but there is none for text generation from left context
* `examples/run_generation.py` concise syntax (and some model-specific preprocessing) are really nice, but it is made for use via CLI and not from code

Any chance we see a text generation pipeline (optionally with some of the `run_generation.py` features) coming to 🤗 Transformers ?

Comments:
There is currently no text generation pipeline, but the `generate` method on both PyTorch/TensorFlow models is here for that purpose!

[Here's](https://huggingface.co/transformers/usage.html#causal-language-modeling) an example using that method for generating from left context :).

You are right, I thought there was a great deal of abstraction done by the `run_generation.py` script given its length, but it turns out except for a few things, it's just interfacing with the CLI. I will be fine with the vanilla `generate` function!

Thanks for the rapid answer :)

---

For future reference (hey there future me!), "a few things" are:
* tokenizer encoding+decoding
* careful seed initialization
* moving everything to cuda device
* stop token handling
* nice logging
* (padding text for some models)
* (capping generation length)

@LysandreJik Is there a way to generate text given context in a random position? For example, given a keyword 'window' I'd like to generate text that contains 'window', doesn't matter where. For example:

* she was looking out the window
* cleaning that window was a difficult task
Token Type IDs returned from the tokenizer for T5 don't work with special tokens
Author: dirkgr
Pull request: no

With `transformers-3.3.1`:

```
import transformers
t = transformers.AutoTokenizer.from_pretrained('t5-small')
t.encode_plus(["a"], ["b"], add_special_tokens=True, return_token_type_ids=True)
```

This results in
```
{'input_ids': [9, 1, 115, 1], 'token_type_ids': [0, 1], 'attention_mask': [1, 1, 1, 1]}
```

As you can see, the token type IDs don't align with the other outputs.

Comments:
The PR linked to this issue should fix it. However note that T5 does not use `token_type_ids`, so that we will simply return all 0.
Bug fix for apply_chunking_to_forward chunking dimension check
Author: pedrocolon93
Pull request: yes


Chunking should be in the chunking dimension, an exception was raised if the complete shape of the inputs was not the same rather than only the chunking dimension

# What does this PR do?
Fixes #8349


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@patrickvonplaten Incorporating the fix from the issue at https://github.com/huggingface/transformers/issues/8349
Let me know if you want a unit test or something! Ran all the tests too:
<img width="936" alt="Screen Shot 2020-11-07 at 3 29 26 PM" src="https://user-images.githubusercontent.com/5157240/98450932-1e35cb00-210f-11eb-855a-2457c44e243b.png">




Comments:
[model_cards] Update Italian BERT models and introduce new Italian XX…
Author: stefan-it
Pull request: yes
Labels: name: model card

Hi,

this PR updates some of our @dbmdz model cards for Italian BERT models:

* Add details about issues with vocab for BERT models
* Introduce new repository for downstream evaluations (NER and PoS tagging)

It also adds new model cards for our newly trained Italian XXL ELECTRA model (discriminator and generator part).

Comments:
T5Tokenizer: decode does not show special tokens
Author: jsrozner
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.4.0
- Platform: macOS-10.15.7-x86_64-i386-64bit
- Python version: 3.8.5
- PyTorch version (GPU?): 1.7.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: N/a
- Using distributed or parallel set-up in script? N/a

### Who can help
  examples/seq2seq: @sshleifer
 -->

## Information

Model I am using (Bert, XLNet ...): T5Tokenizer

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:
```python
from transformers import T5Tokenizer
input = "word  <pad> <unk> 
 
"
t5tokenizer = T5Tokenizer.from_pretrained('t5-small')
tokenized = t5tokenizer.batch_encode_plus([input], max_length=10, padding="longest", return_tensors="pt").input_ids
print(t5tokenizer.batch_decode(tokenized, skip_special_tokens=False, clean_up_tokenization_spaces=False))
```
IDs output: ` _word  <pad> <unk> 
 
`
decode output: `word  ⁇ `


## Expected behavior
The tokens should be shown in the decoded output, but everything except for the unknown token is dropped (no pad or EOS). 
`convert_ids_to_tokens` followed by `convert_tokens_to_string` also drops the tokens.
<!-- A clear and concise description of what you would expect to happen. -->


Comments:
T5: @patrickvonplaten I think you need to set `_additional_special_tokens`.

@jsrozner want to try to fix?

This is a duplicate of #5142 and will be fixed with the PR linked below. Thanks for reporting it - seems like multiple people were running into this issue!!!
Getting large alloc error while evaluating bert-base on NER task
Author: Sumegh-git
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information

Model I am using (Bert-base-multilingual-cased,):

Language I am using the model on (English):

The problem arises when using:
Evaluation

The tasks I am working on is:
My own custom dataset having the same as GermEval task format.

Running on **Colab**. 

I believe this is due to memory error. But why should there be memory **error while testing when things were fine during training **?

Max_seq_length during train was 128 and batch_size 8

04/14/2020 19:06:13 - INFO - transformers.modeling_utils -   loading weights file germeval-model/checkpoint-21750/pytorch_model.bin
04/14/2020 19:07:03 - INFO - __main__ -   Loading features from cached file ./cached_dev_bert-base-multilingual-cased_128
04/14/2020 19:07:05 - INFO - __main__ -   ***** Running evaluation  *****
04/14/2020 19:07:05 - INFO - __main__ -     Num examples = 22026
04/14/2020 19:07:05 - INFO - __main__ -     Batch size = 2
Evaluating:   0% 3/11013 [00:01<1:07:38,  2.71it/s]tcmalloc: large alloc 1110605824 bytes == 0x3e706000 @  0x7f207d0051e7 0x7f207a3995e1 0x7f207a3fde88 0x7f207a3fdfa3 0x7f207a49c098 0x7f207a49c8f4 0x7f207a49ca42 0x5678b3 0x5a067e 0x7f207a3e970d 0x50a8af 0x50c5b9 0x508245 0x50a080 0x50aa7d 0x50d390 0x508245 0x58958c 0x5a067e 0x7f207a3e970d 0x50a8af 0x50c5b9 0x508245 0x50a080 0x50aa7d 0x50d390 0x508245 0x50a080 0x50aa7d 0x50d390 0x508245


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I'm having the same issue. Training is fine but get this error when evaluate.