noamaanMulla-03/transformers-issues huggingface dataset:
How can I get the exact position von answers?
Author: ahnz7
Pull request: no

I want to load a local model for Question and Answers task. I have copied the almost Code from Pipeline like follow:

model = MyBert()
checkpoint = torch.load('./training_mode_aim/checkpoint.pth.tar')
model.load_state_dict(checkpoint['model'])
model.eval()
#read the context in local file
texts = utils.readTxt()
question = Question
tokenizer = BertTokenizerFast.from_pretrained('')

all_answers = []
for text in texts[:1]:
    encoding = tokenizer(question,text,truncation = True,
                            max_length = 512,padding = True,stride = 256,
                            return_tensors = 'np',
                            return_overflowing_tokens=True,
                            return_token_type_ids=True,
                            return_offsets_mapping=True,
                            return_special_tokens_mask=True,)
    num_span = len(encoding['input_ids'])
    answers = []
    for span_idx in range(num_span):
        _,start_logits,end_logits = model(torch.tensor([encoding['input_ids'][span_idx]]),
                                        torch.tensor([encoding['attention_mask'][span_idx]]),
                                        torch.tensor([encoding['token_type_ids'][span_idx]]))
        with torch.no_grad():
            start_logits,end_logits = start_logits.cpu().numpy(),end_logits.cpu().numpy()
        undesired_tokens = np.abs(np.array(encoding['token_type_ids'][span_idx]) - 1) & np.array(encoding['attention_mask'][span_idx])
        undesired_tokens_mask = undesired_tokens == 0.0
        start = np.where(undesired_tokens_mask,start_logits, -10000.0)
        end = np.where(undesired_tokens_mask, end_logits, -10000.0)
        start = np.exp(start - np.log(np.sum(np.exp(start), axis=-1, keepdims=True)))
        end = np.exp(end- np.log(np.sum(np.exp(end), axis=-1, keepdims=True)))
        starts,ends,scores = decode(start,end,5,128)
        print('starts: {}, end: {}'.format(starts,ends))
        answers += [{
                'score':score.item(),
                'start':encoding.token_to_word(s),
                'end':encoding.token_to_word(e)}
                        for s,e,score in zip(starts,ends,scores)]
    answers = sorted(answers, key=lambda x: x["score"], reverse=True)[: 5]
    print(answers)

Then I got the relative positions for each subencodings. But finally I want to take the absolute positions of the answers. So someone know how to solve this problem?

Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discusss.huggingface.co) instead?

Thanks!

> Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
> Could you ask your question on the [forum](https://discusss.huggingface.co) instead?
> 
> Thanks!

okay, my bad
hf_argparser doesn't set the required flag on non-defaulted enums
Author: Craigacp
Pull request: no

## Environment info

- `transformers` version: 3.0.0-4.3.3
- Platform: macOS
- Python version: 3.9
- PyTorch version (GPU?): n/a
- Tensorflow version (GPU?): n/a
- Using GPU in script?: n/a
- Using distributed or parallel set-up in script?: n/a

### Who can help

I'm not sure who the owner is of hf_argparser.

## Information

We're using hf_argparser to parse our experiment config into dataclasses before training.

## To reproduce

Steps to reproduce the behavior:

1. Add an enum argument without a default to a dataclass
2. Parse the command line arguments without supplying the enum argument
3. Should have raised an exception and printed the usage, instead defaults the value to `None`.

## Expected behavior

It should raise an exception. The issue is on https://github.com/huggingface/transformers/blob/master/src/transformers/hf_argparser.py#L100, the if statement should have an else which sets `kwargs["required"]=True`, the same way line [134](https://github.com/huggingface/transformers/blob/master/src/transformers/hf_argparser.py#L134) does.

I can work up a patch if you agree this is an issue. I think it will also occur with anything that falls into [this branch](https://github.com/huggingface/transformers/blob/master/src/transformers/hf_argparser.py#L118) of the if too.


Comments:
I agree with your assessment and your proposed fix, so by all means, please suggest a PR! Thanks!
Multiple fixes in SageMakerTrainer
Author: sgugger
Pull request: yes

# What does this PR do?

This PR adds quite a few fixes to the `SageMakerTrainer` to make sure example scripts run fully. In particular it fixes:
- save made the training hanging forever
- predict didn't work
- evaluation required using `drop_last=True` which is not something anyone wants.

The goal is now to test a little bit more that functionality before merging the `SageMakerTrainer` into the main `Trainer` (otherwise one can't use model parallelism in seq2seq examples or QA example). The plan is to have them merged in the v4.5.0.

Comments:
Adding required flags to non-default arguments in hf_argparser
Author: Craigacp
Pull request: yes

Signed-off-by: Adam Pocock <adam.pocock@oracle.com>

# What does this PR do?

Fixes #10677. I didn't update the docs as I think this is the intended behaviour, but I can do if you think this change would be unexpected.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. Issue #10677.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

@sgugger

Comments:
No problem. I can't see what test failed in CircleCI as it wants to bind to my Github account and access private things in my org. Is there any way to see the failures without letting it into my account?
Seq2Seq Model with PreTrained BERT Model is Throwing Error During Training: ValueError("You cannot specify both input_ids and inputs_embeds at the same time")
Author: Ninja16180
Pull request: no

Hi,

I tried creating a seq2seq model using pretrained BERT model following your tutorials:
https://github.com/bentrevett/pytorch-sentiment-analysis/blob/master/6%20-%20Transformers%20for%20Sentiment%20Analysis.ipynb

https://github.com/bentrevett/pytorch-seq2seq/blob/master/1%20-%20Sequence%20to%20Sequence%20Learning%20with%20Neural%20Networks.ipynb

However during training, I am getting the following error:
```
AttributeError                            Traceback (most recent call last)

<ipython-input-63-472071541d41> in <module>()
      8     start_time = time.time()
      9 
---> 10     train_loss = train(model, train_iterator, optimizer, criterion, CLIP)
     11     valid_loss = evaluate(model, valid_iterator, criterion)
     12 

6 frames

/usr/local/lib/python3.7/dist-packages/transformers/models/bert/modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask, past_key_values, use_cache, output_attentions, output_hidden_states, return_dict)
    917             raise ValueError("You cannot specify both input_ids and inputs_embeds at the same time")
    918         elif input_ids is not None:
--> 919             input_shape = input_ids.size()
    920             batch_size, seq_length = input_shape
    921         elif inputs_embeds is not None:

AttributeError: 'Field' object has no attribute 'size'
```
I am sharing my code for your review in the following github repo:
https://github.com/Ninja16180/BERT/blob/main/Training_Seq2Seq_Model_using_Pre-Trained_BERT_Model.ipynb

Also, request you to kindly review the Encoder and Decoder classes which have been modified to incorporate pretrained bert embedding.

Thanks in advance!

Comments:
Hi @Ninja16180 

Could you please post a short code snippet to reproduce the issue? Thanks.

Hi Suraj,

I was able to find out the issue; there was a variable name I wrongly passed into the decoder class and hence the error.

Correction made:
`embedded = self.bert(sent2)[0] should be embedded = self.bert(input)[0] `

Thus closing this issue.
[examples s2s] AttributeError: 'MBartTokenizerFast' object has no attribute 'tgt_lang'
Author: stas00
Pull request: no

After this PR https://github.com/huggingface/transformers/pull/10205 This is still broken for other models:

```
python examples/seq2seq/run_seq2seq.py     --model_name_or_path facebook/mbart-large-en-ro      --do_train     --do_eval     --task translation_en_to_ro     --dataset_name wmt16     --dataset_config_name ro-en     --source_prefix "translate English to Romanian: "     --output_dir /tmp/tst-translation     --per_device_train_batch_size=16     --per_device_eval_batch_size=16     --overwrite_output_dir     --predict_with_generate     --max_train_samples 500     --max_val_samples 500
```

```
Traceback (most recent call last):
  File "examples/seq2seq/run_seq2seq.py", line 668, in <module>
    main()
  File "examples/seq2seq/run_seq2seq.py", line 469, in main
    train_dataset = train_dataset.map(
  File "/mnt/nvme1/code/huggingface/datasets-master/src/datasets/arrow_dataset.py", line 1120, in map
    update_data = does_function_return_dict(test_inputs, test_indices)
  File "/mnt/nvme1/code/huggingface/datasets-master/src/datasets/arrow_dataset.py", line 1091, in does_function_return_dict
    function(*fn_args, indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
  File "examples/seq2seq/run_seq2seq.py", line 450, in preprocess_function
    with tokenizer.as_target_tokenizer():
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/models/mbart/tokenization_mbart_fast.py", line 193, in as_target_tokenizer
    self.set_tgt_lang_special_tokens(self.tgt_lang)
AttributeError: 'MBartTokenizerFast' object has no attribute 'tgt_lang'
```

@patil-suraj, @sgugger 

Comments:
#10287 contains the fix.

Confirmed that it works, albeit the cl args changed so tested with:
```
PYTHONPATH=src python examples/seq2seq/run_translation.py     --model_name_or_path facebook/mbart-large-en-ro      --do_train     --do_eval      --dataset_name wmt16     --dataset_config_name ro-en       --output_dir /tmp/tst-translation     --per_device_train_batch_size=2     --per_device_eval_batch_size=2     --overwrite_output_dir     --predict_with_generate      --source_lang en_XX --target_lang ro_RO --max_val_samples 10  --max_train_samples 10
```
Tests run on Docker
Author: LysandreJik
Pull request: yes

This PR updates the GPU-based tests to run on docker images. This:
- Simplifies maintenance
- Horizontal scaling is made much simpler
- Environment can be managed with a single line change (the docker image) hello pytorch 1.3-1.7 tests!
- Setups a notification service to get alerted when scheduled tests fail.


Co-authored-by: Morgan <funtowiczmo@gmail.com>

Comments:
@stas00 I hear you regarding the installation of dependencies. We are installing every dependency except the main one (PT/TF) on the GPU machine, so I would say most errors are caught.
Furthermore, we are installing these exact dependencies on the CircleCI runs, so I believe we are testing that already on every commit.

If there is a scenario I am missing, please let me know and I will do my best to adjust.

All other comments have been adressed in the last three commits.

Ah, ok, for some reason I thought that if we are using a docker image then we can skip wasting time and resources on installing the same things million times a day and just run the tests right away, and in that case only an occasional test that installs from scratch will be needed. But perhaps this further speed up can be done in some future iteration.

Nothing more from my side.

Thank you, @LysandreJik
Zero-Shot-Classification: multi_class or multi_label?
Author: taufique74
Pull request: no

https://github.com/huggingface/transformers/blob/068df740bd73b95e9a1e233e47608df942fda9da/src/transformers/pipelines.py#L1048
Since we're allowing multiple labels to be true, shouldn't this be called `multi_label` instead of `multi_class`?

Comments:
cc @joeddav 

Yeah I think `multi_label` probably would have been better in retrospect given that many do seem to make this distinction between terms. Since it's already being widely used, though, I think we'll keep it as is for the moment. If it turns out to cause consistent confusion for people we can look at changing it later on.

Leaving a comment and hoping this to be reopened.

There are two major classification tasks (see for instance: https://towardsdatascience.com/journey-to-the-center-of-multi-label-classification-384c40229bff#:~:text=Difference%20between%20multi%2Dclass%20classification,the%20tasks%20are%20somehow%20related.):
A multiclass classification problem is a classification task with more than two classes, that makes the assumption that each sample is assigned to one and only one label: an animal can be wither a cat or a dog.

A multilabel classification assigns a set of target labels to each sample.  A text might have multiple categories, for instance politics, finance and education, or none of these.

The multi_class option here, works exactly opposite. You have to set it to multi_class=False to make it behave like multi class classification problem, and multi_class=True to make it multi label. It is really confusing.

Switching the behaviour would probably lead to even more confusion. My suggestion would be to depreciate "multi_class=True/False" and instead add the parameter "multi=label(default)/class. Probably not ideal, but less confusing.

@peregilk I think you're right, it's consistently been a little bit confusing. I don't think the parameter names need to **exactly** reflect the vernacular, but I do think we can deprecate `multi_class` and rename it to `multi_label` while keeping the behavior the same. I think that's the solution with the least confusion. I'll send a PR. The only case where it's not multi-class is when only a single label is passed, in which case the `multi_class` argument is treated as true since it doesn't make sense to softmax over a single label.

That sounds like a great solution. Thanks.
Rename zero-shot pipeline multi_class argument
Author: joeddav
Pull request: yes

Renames the `multi_class` argument to `multi_label` in the `ZeroShotClassificationPipeline` and adds a deprecation warning to the former. Typically, "multi-label classification" is used to refer to this type of classification (where each class is evaluated independently).

The name is changed in the zero-shot distillation script as well. Resolves #6668.

Comments:
Add minimum version check in examples
Author: sgugger
Pull request: yes

# What does this PR do?

This PR adds a minimum version check to all examples, in order to avoid the waves of issues created each time they use a functionality that was just released into `Trainer`. The script will immediately error if the version of Transformers does not match the required minimum version. At each release, a script will set that to the version released automatically (work in progress for a second PR with other release utils) so that the examples associated with one tag will require the minimum version of that tag.

The user can still remove that line to avoid the error (at their own risks). The error points out to:
- the instruction for a source install
- the examples README that now lists all examples folders with the various version tags.


Comments:
Image feature extractor design
Author: NielsRogge
Pull request: yes

# What does this PR do?

Here we can discuss how to design the `ImageFeatureExtractor` class, and the `ViTFeatureExtractor` subclass. 

The hierarchy looks as follows:
`FeatureExtractorMixin` -> `ImageFeatureExtractor` -> `ViTFeatureExtractor`. The `FeatureExtractorMixin` defines common properties among `SequenceFeatureExtractors` (for speech recognition) and `ImageFeatureExtractors` (for vision related tasks), namely saving utilities and the general `BatchFeature` class. 

Notes: 
- `ImageFeatureExtractor` is based on `SequenceFeatureExtractor`, but with some changes: renamed `max_length` to `max_resolution`, renamed `PaddingStrategy.LONGEST` to `PaddingStrategy.LARGEST` (to pad to the resolution of the largest image in a batch), renamed `PaddingStrategy.MAX_LENGTH` to `PaddingStrategy.MAX_RESOLUTION`. 
- Currently, this `ImageFeatureExtractor` class defines common properties among feature extractors for vision models, which are now `image_mean`, `image_std` and `padding_value`. Each concrete FeatureExtractor then provides values for these 3 attributes, and defines any additional attributes. 
- Currently, the `ImageFeatureExtractor` class only defines `pad` and `_pad` methods (which should be updated to work for 2D images), but I guess we can add general image transformation methods (such as resize, normalize), and maybe also a `__call__` method. These are now all defined in `ViTFeatureExtractor`. 

## Who can review?

@patrickvonplaten @patil-suraj @LysandreJik 




Comments:
Release utils
Author: sgugger
Pull request: yes

# What does this PR do?

This PR adds utilities to help with the release process and four make commands to use them easily:

1. `make pre-release` will do all the necessary steps prior to the commit with the release tag (put the right version in all the good places and clean the README from the references to the master documentation).
2. `make post-release` will do all the necessary steps after the release has been made (put the right dev version in all the good places and add the latest version in the deploy doc/doc navbar)
3. `make pre-patch` will do all the necessary steps prior to the commit with the release patch tag (put the right version in all the good places).
4. `make post-past` will do all the necessary steps after the patch release has been made and we are back on master (put the right dev version in all the good places and add the latest version in the deploy doc/doc navbar)

Comments:
Add DistributedSamplerWithLoop
Author: sgugger
Pull request: yes

# What does this PR do?

This PR adds a new distributed sampler that will provide a round multiple of the batch size samples on all processes by looping back at the beginning of the (shuffled) dataset. This is useful:
- for TPUs to avoid triggering a new XLA compilation for the last training batch
- for model parallelism to have batches of the same size on all processes

This PR also refactors some logic regarding the wold_size and process_rank in the `TrainingArguments`, as well as adds a test of the new `DistributedSamplerWithLoop`.

Tested on:
- single-GPU
- multi-GPU
- TPU
- SageMaker MP

Comments:
Remove old links to CDN
Author: sgugger
Pull request: yes

# What does this PR do?

This PR removes a few links left pointing to `https://cdn.huggingface.co` instead of `https://huggingface.co` (purely cosmetic since they are not actually used anymore, normally).

Comments:
those are not valid files though (check the url template for the other models)

Oopsie, thanks for flagging!
Online decoding for ASR
Author: arkadyark
Pull request: no

# 🚀 Feature request

Are there plans to implement online decoding for the speech recognition models such as wav2vec2 and XLSR? More specifically, to be able to receive audio in short chunks, and output partial transcripts as they become available.

## Motivation

Many use cases are covered by the current wav2vec2 model in the library, involving batch recognition of pre-recorded text. However for an online application that wanted to continuously recognize speech on a live input stream, this may not be sufficient.

Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discusss.huggingface.co) instead?

cc @patrickvonplaten 

Thanks!

Sure, no problem, sorry about that. There seems to be a typo in the forum link - for anybody reading this in the future here are the [forums:](https://discuss.huggingface.co/)
Why the RoBERTa's max_position_embeddings size is 512+2=514?
Author: kugwzk
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
When I see the code of Roberta, I have a question about the padding_idx = 1, I don't know very well. And the comment is still confused for me.

Comments:
What's your precise question?

> What's your precise question?

the self.padding_idx's meaning in modeling_roberta.py

It's the position of the padding vector. It's not unique to RoBERTa but far more general, especially for embeddings. Take a look at [the PyTorch documentation](https://pytorch.org/docs/stable/nn.html#embedding).

> It's the position of the padding vector. It's not unique to RoBERTa but far more general, especially for embeddings. Take a look at [the PyTorch documentation](https://pytorch.org/docs/stable/nn.html#embedding).

I know that, but I confuse about why there is 1 and the \<s\> is 0, is it ignore and why the max_position_embeddings size is 512+2=514?

Because that's their index [in the vocab](https://s3.amazonaws.com/models.huggingface.co/bert/roberta-base-vocab.json). The max_position_embeddings size is indeed 514, I'm not sure why. The tokenizer seems to handle text correctly with a max of 512. Perhaps someone of the developers can help with that. I would advise you to change the title of your topic.

https://github.com/huggingface/transformers/blob/ae50ad91ea2fedb64ecd2e7c8e2d0d4778dc03aa/transformers/tokenization_roberta.py#L84-L85

@LysandreJik can chime in if I’m wrong, but afaik `max_position_embeddings` is just the name of the variable that we use to encode the size of the embedding matrix. Max_len is correctly set to 512.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Answer here in case anyone from the future is curious: https://github.com/pytorch/fairseq/issues/1187

> Answer here in case anyone from the future is curious: [pytorch/fairseq#1187](https://github.com/pytorch/fairseq/issues/1187)

@morganmcg1 Tks for this, was getting all kinds of CUDA errors because i setted `max_position_embeddings=512`, now that i setted 514 it's running ok...