noamaanMulla-03/transformers-issues huggingface dataset:
[Benchmark] Pipeline for question answering
Author: dipanjannag
Pull request: no
Labels: name: wontfix

name: Core: Pipeline

# 🖥 Benchmarking `transformers`

## Benchmark

I'm trying to benchmark QA model with `bert-large-uncased-whole-word-masking-finetuned-squad`. But it seems it is extremely slow e.g. 3-4 sec for 1 question with 2 contexts.

I feel there is something I'm missing in pipeline.

## Sample Code:
```
def answer(self, contexts:List[str], question:str, **kwargs):
   ## tokenizer, model, pipeline all are cached in actual implementation 
  ## via [reify](https://docs.pylonsproject.org/projects/pyramid/en/latest/api/decorator.html) 
  ## so model loading is not a problem.
    tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', max_len=500)
    model = BertForQuestionAnswering.from_pretrained('bert-large-uncased-whole-word-masking- 
        finetuned-squad')
    pipeline('question-answering', model=model, tokenizer=tokenizer)
    pipeline_input = []
    for c in contexts:
        pipeline_input.append({
            'question' : question,
            'context' : c
        })
    answers = pipeline(pipeline_input)
```

## Set-up
CPU: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
memory: 16GB



Comments:
same experience with a 2080TI. It s like it s not batched...

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


As @pommedeterresautee mentioned, the samples are not batched. I've made a [simple modification](https://gist.github.com/vinicius-cleves/1b2d79a9665d165ac22451c225b2f8b0) to do batch inference on pytorch, but it didn`t seem to help much with the total processing time.
HuggingFace pipeline sentiment analysis giving wrong results.
Author: vishwa30
Pull request: no

I am just starting with hugging face and following the official doc. When using sentiment analysis pipeline I am getting incorrect output. I am not sure what's the reason behind it.

```
from transformers import pipeline
classifier=pipeline('sentiment-analysis')
text='This is just a statement'
a=classifier(text)
print(a)

```
Giving output as:
[{'label': 'NEGATIVE', 'score': 0.9583144783973694}]

I have changed the input with different sentences it is having problem with neutral statements but is able to predict the positive and negative statements, which have words like {"awesome","good","bad"}. 

Statements I have tried and respective output:
1. 'Today is thursday'                             {'label': 'POSITIVE', 'score': 0.987697184085846}
2. 'Give me my water bottle'                 {'label': 'NEGATIVE', 'score': 0.855629563331604}
3. 'Its raining outside'                            {'label': 'POSITIVE', 'score': 0.8293998837471008}
4. 'You are awesome'                            {'label': 'POSITIVE', 'score': 0.9998681545257568}
5. 'I hate you'                                        {'label': 'NEGATIVE', 'score': 0.9991129040718079}


Comments:
Hey @vishwa30 - what exactly do you mean by incorrect output? -> The model obviously doesn't always classify the input correctly. Since this issue doesn't seem to be a bug, could you maybe look in the forum whether people asked a similar question before and if not post this one there? :-)

https://discuss.huggingface.co/

@patrickvonplaten Sure! I will do that. Thanks!!
[FlaxBert] Fix non-broadcastable attention mask for batched forward-passes
Author: KristianHolsheimer
Pull request: yes

# What does this PR do?

Fixes #8790


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@mfuntowicz 
@avital 
@LysandreJik 


Comments:
@mfuntowicz @avital
I just fixed the bug that I was hitting. There might be other places that need this fix as well.

Wuuhu - first Flax PR :-). This looks great to me @KristianHolsheimer - think you touched all necessary files as well! 

@mfuntowicz - maybe you can take a look as well

**EDIT** I re-enabled GPU memory preallocation but set the mem fraction < 1/parallelism. That seemed to fix the tests. The problem with this is that future tests might fail if a model doesn't fit in 1/8th of the GPU memory.

---

The flax tests time out. When I ran the tests locally with `pytest -n auto`, I did notice OOM issues due to preallocation of GPU memory by XLA. I addressed this in commit 6bb1f5e600cd35c712f4f980699df7735b4f59eb.

Other than that, it's hard to debug the tests when there's no output.

Would it be an option to run these tests single-threaded instead?

I had the changes in another branche I'm working on, happy to merge this one and will rebase mine 👍.

Thanks for looking at it @KristianHolsheimer
[FlaxBert] Non-broadcastable attention mask in batched forward-pass
Author: KristianHolsheimer
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:
- Platform: Linux
- Python version: 3.8
- JAX version:
  - jax==0.2.6
  - jaxlib==0.1.57+cuda110
  - flax==0.2.2
- PyTorch version (GPU?): n/a
- Tensorflow version (GPU?): n/a
- Using GPU in script?: **yes** (cuda 11.0)
- Using distributed or parallel set-up in script?: **no**

### Who can help
@mfuntowicz 
@avital 
@LysandreJik 

## Information

I ran the script from the recent Twitter [post](https://twitter.com/huggingface/status/1331255460033400834):

![](https://pbs.twimg.com/media/EnmRwGDW4AA0j26?format=jpg)

The only thing I changed was that I fed in multiple sentences:

```python
from transformers import FlaxBertModel, BertTokenizerFast, TensorType

tokenizer = BertTokenizerFast.from_pretrained('bert-base-cased')
model = FlaxBertModel.from_pretrained('bert-base-cased')
# apply_fn = jax.jit(model.model.apply)

sentences = ["this is an example sentence", "this is another", "and a third one"]
encodings = tokenizer(sentences, return_tensors=TensorType.JAX, padding=True, truncation=True)
tokens, pooled = model(**encodings)
```
> ValueError: Incompatible shapes for broadcasting: ((3, 12, 7, 7), (1, 1, 3, 7))

See full stack trace: https://pastebin.com/sPUSjGVi


Comments:
Fixed typo in README.md of bert-base-greek-uncased-v1
Author: mdermentzi
Pull request: yes
Labels: name: model card

# What does this PR do?

The tokenizer called at the input_ids var of Example 2 is currently encoding text_1. This PR is changing the input to text_2.

Motivation and context for this change: I am considering using this model for a uni assignment and it took me a while to understand why the example code was yielding the wrong results. Hopefully, the next person who is eager to try these examples will not get confused by that typo.

- [x] This PR fixes a typo or improves the docs.

 documentation: @sgugger


Comments:
Thanks! Looks good to me, just pinging @iliaschalkidis for information/validation

Wow, thanks for the fix @mdermentzi. If I recall correctly, I was trying things in an initial scratch python script that was using a universal `text` variable back to back and I though it would be better to rename those in 3 different variables to make it clearer. It seems I was quite unwary... 

No worries @iliaschalkidis! Just a small typo. ;) Thank you for publishing this model. I am so happy I can play with it for my current uni project.
cache reuse
Author: Jiaxin-Wen
Pull request: no

I have downloaded wikitext by `load_dataset("wikitext", "wikitext-2-raw-v1")`, and get the cache file in `.cache/huggingface/datasets`,

then I try to copy the `huggingface/datasets` folder to the lab server to reuse but fails.

What's the proper way to reuse the cache downloaded on another pc?

Comments:
You should use [`save_to_disk`](https://huggingface.co/docs/datasets/package_reference/main_classes.html?highlight=save#datasets.DatasetDict.save_to_disk) and [`load_from_disk`](https://huggingface.co/docs/datasets/package_reference/loading_methods.html?highlight=save_to_disk#datasets.load_from_disk):

```python
import datasets
# Will download and preprocess the dataset
data = datasets.load_dataset("wikitext", "wikitext-2-raw-v1")
# Save it in a folder that you can copy
data.save_to_disk("PATH/TO/FOLDER")

# On the other machine - reload the ready to use dataset from the copied folder
data = datasets.load_from_disk("PATH/TO/FOLDER")
```

thanks!
Extend typing to path-like objects in `PretrainedConfig` and `PreTrainedModel`
Author: gcompagnoni
Pull request: yes

# What does this PR do?

In my experience, I often call the `from_pretrained` and `save_pretrained` methods of models and configurations with a path-like variable rather than a string. Since the paths are then used by various `os` functions, this works just fine: however, the relevant variables are typed as strings only, raising warnings when using an IDE :anguished: .

This PR extends the typing to `Union[str, os.PathLike]` when relevant inside `PretrainedConfig` and `PreTrainedModel`   methods. 
Since passing a path-like object is already tacitly supported in most cases, no significant changes to the code are necessary. In a few places, the relevant variable needs to be turned to a string in order to support functions such as `is_remote_url`.


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

Maybe (documentation): @sgugger



Comments:
Good idea!

This could be done for tokenizers as well, no? (the `from_pretrained` for tokenizers is in `tokenization_utils_base.py`)

I have extended the same modifications to the tokenizers, as suggested by @thomwolf , and to auto classes too.
Remove deprecated
Author: sgugger
Pull request: yes

# What does this PR do?

This PR removes old deprecated arguments and adjust tests/examples accordingly.
Co-authored with @LysandreJik.

Comments:
Hi, I noticed that https://github.com/huggingface/transformers/blob/master/examples/token-classification/run_ner_old.py#L294 and https://github.com/huggingface/transformers/blob/master/examples/token-classification/run_ner_old.py#L302 are still referencing is_world_master() instead of using is_world_process_zero(). I have made a change to run_ner_old.py locally and running the process again to see if it fixes the issue for me (AttributeError: 'Trainer' object has no attribute 'is_world_master')

We're not actively maintaining the old examples scripts anymore and they're still there for people using older versions of transformers. The good fix would be to add a minimum version to the README of that example.

I'm at loss here - which ones are old examples and which ones are new?

https://github.com/huggingface/transformers/issues/8792

If there should be an old version of something to be used with an old version of something - why not send users to the branch of that release that they want to use - they will end up with the version of examples that work for that release. And master examples should be working with master version of the code, IMHO.  Does it make sense?

If there are fixes to the old branch's examples, not pertaining to master, the fix can go into that branch.

Yes it all makes sense. Please be patient while I clean up the examples folder, as it's a long work. I promise this will all be clean when I'm done :-)
Use GPT to assign sentence probability/perplexity given previous sentence? 
Author: TalitaGroetzinger
Pull request: no

Hi!

Is it possible to use GPT to assign a sentence probability given the previous sentences? 
I have seen this code here, which can be used to assign a perplexity score to a sentence: 

https://github.com/huggingface/transformers/issues/473

But is there a way to compute this score given a certain context (up to 1024 tokens)? 



Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discusss.huggingface.co) instead?

Thanks!
[model loading] remove pointless log entries
Author: stas00
Pull request: yes

This PR removes 2 IMO-pointless log entries that literally say "all is well". 

Log entries are useful for debugging problems, but just add to the noise that make more difficult to see useful entries, when they state the obvious, no?

@sgugger, @LysandreJik 



Comments:
@LysandreJik, I can totally see your point.

But this one says the same thing twice:
```
            f"All model checkpoint weights were used when initializing {model.__class__.__name__}.\n"
            f"All the weights of {model.__class__.__name__} were initialized from the model checkpoint at {pretrained_model_name_or_path}.\n"
```

They aren't exactly the same but they say twice that there was no problem in loading the model

So as with your excellent example the following would be fitting:
```
"Loading model {name} at {path}"
(any exceptions go here)
"Model loaded"
```

Would you be open if changed this PR to follow this strategy?

They don't really say the same thing, I would see the first one as:

```
Checking if all checkpoints weights were used in the model ...
All checkpoint weights were used.
[...]
Checking if all weights of the model were initialized by the checkpoint ...
All model weights are initialized.
```
I think both serve a purpose:
1. Is the checkpoint meant for that architecture, or was it trained for another one.
2. Is the model perfectly initialized from that checkpoint, or will it require some fine-tuning.

The full current log for point 2 is:
``` 
               logger.info(
                    f"All the weights of {model.__class__.__name__} were initialized from the model checkpoint at {pretrained_model_name_or_path}.\n"
                    f"If your task is similar to the task the model of the checkpoint was trained on, "
                    f"you can already use {model.__class__.__name__} for predictions without further training."
                )
```

So that log's line 2+3 are still there. I didn't suggest to remove those.

But I feel that this is bordering on splitting hairs (from my side), so I will just let it be. 

Your explanation of its purpose makes sense, @LysandreJik. I'd have just tuned it up to make it more factual and less verbose ...

My issue is that I read those logs and not all of them feel like very readable to me...
suggest a numerical limit of 50MB for determining @slow
Author: stas00
Pull request: yes

This is a follow up to https://github.com/huggingface/transformers/issues/7250 which adds a guideline to when making a test `@slow` based on the download requirements if any.

The suggested value is >50MB, and we can adjust it later if it's too large or small.

Fixes: https://github.com/huggingface/transformers/issues/7250

@LysandreJik, @sgugger 

Comments:
`train.jsonl` file missing in MM-IMDb task
Author: prabhakar267
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Bert 

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: MM-IMDb
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Download the **raw** MM-IMDb dataset from http://lisi1.unal.edu.co/mmimdb/
2. Download `run_mmimdb.py` and `utils_mmimdb.py` from `/examples/contrib/mm-imdb`
3. Run the command given in [README.md](https://github.com/huggingface/transformers/blob/master/examples/contrib/mm-imdb/README.md#training-on-mm-imdb)

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

```
Traceback (most recent call last):
  File "run_mmimdb.py", line 614, in <module>
    main()
  File "run_mmimdb.py", line 555, in main
    train_dataset = load_examples(args, tokenizer, evaluate=False)
  File "run_mmimdb.py", line 339, in load_examples
    dataset = JsonlDataset(path, tokenizer, transforms, labels, args.max_seq_length - args.num_image_embeds - 2)
  File ".../mmimdb/utils_mmimdb.py", line 50, in __init__
    self.data = [json.loads(l) for l in open(data_path)]
FileNotFoundError: [Errno 2] No such file or directory: '.../mmimdb/dataset/train.jsonl'
```

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
+ `train.jsonl` should be present in the script directory

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:
- Platform: Linux
- Python version: Python 3.6.5
- PyTorch version (GPU?): `1.4.0` (yes)
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


It seems like this is the method to get the jsonl files: https://github.com/facebookresearch/mmbt/blob/master/scripts/mmimdb.py
[s2s-wip] ray instructions
Author: sshleifer
Pull request: yes
Labels: name: wontfix

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #7170 

### TODO
- test
- finish instructions
- modify ray_tune_cli to not hardcode all possible lightning args.
- probably more things


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Minor dead code clean-up
Author: guillaume-be
Pull request: yes

Hello,

I am not sure how sensitive you generally are about dead code in the repository. I have identified a few places with dead code, where I believe a clean-up would improve readability.

- removal of a couple of unused dropouts I came across in Albert and XLNet
- removal of an unused code block for relative attention shift for XLNet

## Who can review?

@LysandreJik , @TevenLeScao 


Comments:
Closing as it seems the changes were added in another PR
Shared vocabulary with EncoderDecoderModel
Author: bayanbatn
Pull request: no

# 🚀 Feature request
Currently, two separate models are instantiated as encoder/decoder when using this model class. It would be useful in a lot of fine-tuning applications (i.e. summarization) to share the same embeddings between the encoder / decoder classes -- is this something that could be supported with the library?

Comments:
I believe the `EncoderDecoderConfig` class has an argument `tie_encoder_decoder` which can be used to share weights between the encoder and decoder.

Is this what you're looking for?

@patrickvonplaten 

Thanks @LysandreJik; it's close enough to what I'm looking for. Closing this issue.

Hey @bayanbatn, 

We are currently working on a function (see #8224)  that automatically ties encoder and decoder word embeddings only in an automatic fashion...for now what one can do is to simply set it yourself via

```python
model.decoder.word_embeddings = model.encoder.word_embeddings
```
I want to use the Bert2GPT2 architecture,but my pretrained Bert and GPT2 have different vocabs,so what should I do for the vocabs? 
Author: wulaoshi
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:

Comments:
This model card gives an in-detail explanation of how to train, use a bert2gpt2 model: https://huggingface.co/patrickvonplaten/bert2gpt2-cnn_dailymail-fp16#bert2gpt2-summarization-with-%F0%9F%A4%97-encoderdecoder-framework

Let me know if this does not solve your errors.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
PegasusTokenizer: Newline symbol
Author: sshleifer
Pull request: no
Labels: name: wontfix

name: Core: Tokenization

name: pegasus

Ported models generate the `<n>` token at the beginning of sentences, whereas ours do not. The pegasus [original code](https://github.com/google-research/pegasus/blob/master/pegasus/ops/public_parsing_ops.py#L40) replaces `\n` newline symbol with `<n>`. `PegasusTokenizer` should probably do this. 

```python
_NEWLINE_SYMBOL = "<n>"
text = tf.strings.regex_replace(text, "\n", _NEWLINE_SYMBOL)
```



Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
is there a tokenizer only used whitespace for spliting chinese sentence?
Author: lw00245
Pull request: no
Labels: name: wontfix

i want to use bert masked language model to pre train for chinese sentence,  i have alweady splited chinese sentence into meaninful word, the data file like follows:

我  是  一个  队员
他  不是  一个 合格  的 老师
......

i only want to use whitespace split them, but BertWordPieceTokenizer will split them to character level , the final vocabulary like follows:
{'[SEP]': 3,
 '一': 7,
 '是': 15,
 '员': 12,
 '[CLS]': 2,
 '[UNK]': 1,
 '</S>': 6,
 '[MASK]': 4,
 '<S>': 5,
 '队': 19,
 '的': 17,
 '不': 8,
 '我': 14,
 '他': 10,
 '老': 18,
 '[PAD]': 0,
 '格': 16,
 '个': 9,
 '师': 13,
 '合': 11}
how to correct it ?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
[s2s] metrics.json is wrong on multigpu
Author: sshleifer
Pull request: no
Labels: name: wontfix

overwritten by last rank to save it.

@nateraw is there a way to check if my module `is_rank_zero` or some such?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.