noamaanMulla-03/transformers-issues huggingface dataset:
Issues with MODEL_FOR_MASKED_LM_MAPPING.keys(), and transformer.utils.check_min_version()
Author: RasmusEdvardsen
Pull request: no

Hey,

I just recently wanted to pre-train on top of a BERT model, and ran into some issues. When I run `python run_mlm.py`,
I get the following error:
```
Traceback (most recent call last):
  File "run_mlm.py", line 46, in <module>
    from transformers.utils import check_min_version
ImportError: cannot import name 'check_min_version' from 'transformers.utils' (/PATH/TO/site-packages/transformers/utils/__init__.py)
```
https://github.com/huggingface/transformers/blob/d3d388b934ef515e96246ba643c924d675f6515d/examples/language-modeling/run_mlm.py#L46

After commenting out that line, and it's import (I know, shame on me), I get the following error:
```
Traceback (most recent call last):
  File "run_mlm.py", line 53, in <module>
    MODEL_CONFIG_CLASSES = list(MODEL_FOR_MASKED_LM_MAPPING.keys())
AttributeError: 'NoneType' object has no attribute 'keys'
```
https://github.com/huggingface/transformers/blob/d3d388b934ef515e96246ba643c924d675f6515d/examples/language-modeling/run_mlm.py#L54

Tried with python 3.7.10 and 3.8.3

Thanks

Comments:
Hello! Have you checked the ["Important note" at the top of the examples README](https://github.com/huggingface/transformers/tree/master/examples#important-note)? 

Did you get this error with a source install?

@LysandreJik that does the trick - I'll be more thorough in the future with my reading :)
[DeepSpeed] improve checkpoint loading code plus tests
Author: stas00
Pull request: yes
Labels: name: DeepSpeed

This PR further improves the DeepSpeed integration

* checkpoint resuming code has been cleaned up
* detailed checkpoint saving and resuming from checkpoint tests added
* a small reshuffle made in `test_trainer.py` to enable re-using helper functions in other test modules
* switched `test_trainer.py` to `TestCasePlus` so it's easier to deal with temp dirs during debug
* adjusted `init_deepspeed` to make a a deepcopy of the config dict passed to it, so that the user's copy isn't affected - needed at least for tests

Note that I made a failed attempt to load from resume point fatal under deepspeed. I'm not sure why the normal code just warns if a wrong path is passed. Unless I'm missing something, if a user expects to resume and it is not possible it should be fatal IMHO, so that they can correct their launching code.

@sgugger 

Comments:
Great, thank you for the feedback, @sgugger - I will add it separately https://github.com/huggingface/transformers/pull/10777
[trainer] make failure to find a resume checkpoint fatal + tests
Author: stas00
Pull request: yes

As a follow up to https://github.com/huggingface/transformers/pull/10760 this PR:
- makes a failure to find a valid checkpoint to resume from fatal - when an explicit `resume_from_checkpoint` was passed 
- extends `test_can_resume_training` to validate this change and also the boolean `resume_from_checkpoint` case.
- adds a small `test_can_resume_training` refactoring - so it's easy to see they are the same args on each invocation.

@sgugger 

Comments:
[Issue template] need to update/extend who to tag
Author: stas00
Pull request: yes

This PR 

* [x] adds an entry for what to do what to do when someone has model hub issues - thank you, @julien-c!

TODO/Input needed:

* [ ] need to update who to tag for `tensorflow` Issues

@LysandreJik 


Comments:
For broken models you would just tag the model author(s) – we'll add a feature to the model hub to tag someone in a conversation thread, but in the meantime you can use the Forum to ping them

That's a great idea to tag the model author - how would a user know the model author's corresponding forum username? I guess this is temporary so probably can be somehow figured out...

We use SSO so one's username on Forum is guaranteed to be one's username on hf.co

Ah, that's perfect then! Thank you!
Check copies blackify
Author: sgugger
Pull request: yes

# What does this PR do?

This PR update the check_copies utils to apply black when checking if a copy has diverged from the original when replacement happen. An example of the problem is given with the diff in `modeling_mobilebert.py` here, where the check copies could not be applied to whole class because of styling divergences.

It also fixes a bug where the check was not applied on functions after the end of the definition (it wasn't checking the function but was stopping at the first unindent when the closing parenthesis was). As a consequence, three files are changed because they diverged from the original function:
- modeling_m2m_100.py
- modeling_roberta.py
- modeling_speech_to_text.py

I'm not sure if the check should be removed on those or not (cc @patil-suraj )

Comments:
[examples s2s] run_seq2seq.py tweaks
Author: stas00
Pull request: no

Would it be possible to sync the new `run_seq2seq.py` with `./finetune_trainer.py` outputs?

Before:
```
2021-02-05 12:59:55 | INFO | __main__ | ***** train metrics *****
2021-02-05 12:59:55 | INFO | __main__ |   epoch = 1.0
2021-02-05 12:59:55 | INFO | __main__ |   train_n_objs = 60
2021-02-05 12:59:55 | INFO | __main__ |   train_runtime = 9.7768
2021-02-05 12:59:55 | INFO | __main__ |   train_samples_per_second = 6.137
2021-02-05 12:59:55 | INFO | __main__ | *** Evaluate ***
[INFO|trainer.py:1600] 2021-02-05 12:59:55,434 >> ***** Running Evaluation *****
[INFO|trainer.py:1601] 2021-02-05 12:59:55,434 >>   Num examples = 10
[INFO|trainer.py:1602] 2021-02-05 12:59:55,434 >>   Batch size = 1

2021-02-05 13:00:00 | INFO | __main__ | ***** val metrics *****
2021-02-05 13:00:00 | INFO | __main__ |   epoch = 1.0
2021-02-05 13:00:00 | INFO | __main__ |   val_bleu = 33.3125
2021-02-05 13:00:00 | INFO | __main__ |   val_gen_len = 50.1
2021-02-05 13:00:00 | INFO | __main__ |   val_loss = inf
2021-02-05 13:00:00 | INFO | __main__ |   val_n_objs = 10
2021-02-05 13:00:00 | INFO | __main__ |   val_runtime = 4.7266
2021-02-05 13:00:00 | INFO | __main__ |   val_samples_per_second = 2.116
2021-02-05 13:00:00 | INFO | __main__ | *** Predict ***

```
With the new script:
```
02/05/2021 13:00:41 - INFO - __main__ -   ***** Train results *****
02/05/2021 13:00:41 - INFO - __main__ -     epoch = 1.0
02/05/2021 13:00:41 - INFO - __main__ -     train_runtime = 1.33
02/05/2021 13:00:41 - INFO - __main__ -     train_samples_per_second = 3.008
02/05/2021 13:00:41 - INFO - __main__ -   *** Evaluate ***
***** Running Evaluation *****
  Num examples = 100
  Batch size = 32
02/05/2021 13:00:42 - INFO - __main__ -   ***** Eval results *****
02/05/2021 13:00:42 - INFO - __main__ -     epoch = 1.0
02/05/2021 13:00:42 - INFO - __main__ -     eval_bleu = 1.3059269919149237
02/05/2021 13:00:42 - INFO - __main__ -     eval_gen_len = 17.66
02/05/2021 13:00:42 - INFO - __main__ -     eval_loss = 5.084951400756836
02/05/2021 13:00:42 - INFO - __main__ -     eval_runtime = 0.7079
02/05/2021 13:00:42 - INFO - __main__ -     eval_samples_per_second = 141.261
```

As you can see:
1. the metrics numbers aren't rounded up
2. missing `*_n_obj` in metrics
3. logging is inconsistent.

   the old script has its own issues with log consistency, this one introduces its own inconsistencies.

   To make it easy to read the results - the user-targeted output should ideally be aligned on one side and not in 2 columns - i.e. left column date, etc. right column information.  In the new script `***** Running Evaluation *****` is missing the logger prefix.

Thank you!

@patil-suraj, @sgugger 

Comments:
this has been resolved in various PRs. It's looking even better than before.
[trainer] renaming cl args/ trainer attributes to be clear per-gpu vs total
Author: stas00
Pull request: no

As we started discussing here https://github.com/huggingface/transformers/issues/9801#issuecomment-767825869 perhaps we could have a design session where we look at all of the trainer cl args (and their class attribute counterparts) and see which of them contain ambiguity wrt per-gpu vs total (and perhaps other important renames where we find things are confusing).

The intention is to make the API more intuitive and minimize the number of time we introduce breaking changes, but to attempt to do that in one go as much as possible.

One such item we started to discuss is `--max_steps`, then @sgugger mentioned `--num_train_epochs` and there are probably others.

I also proposed to potentially entertain creating a back-compat module to minimize the breaking changes pain where it's possible - renames fall perfectly into this category. I wrote:

> In some previous projects for such things we also had a back-compat mode, which ones enabled supported a whole bunch of old ways until the user was ready to make the shift to the new code. Surely a rename of a cl arg could be easily supported by such feature. So here, instead of a deprecation cycle per item the approach is to keep anything old around but only if it's loaded from a helper module. So that the main code remains clean of deprecated things. This was in a different programming environment where it was developer, so I will have to think how to do the same here.


@LysandreJik, @patrickvonplaten, @sgugger 


Comments:
This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.

I'd love to get feedback on that. Thank you!

As it doesn't seem to resonate as a real need, I'm closing this one.
[fsmt] Expanding Positional Embeddings
Author: stas00
Pull request: no
Labels: name: wontfix

name: fsmt

splitting the comment from @sshleifer in https://github.com/huggingface/transformers/pull/7224#pullrequestreview-492155076 into this separate issue, as the PR hasn't introduced this potential issue. He writes:

> The expansion of embeddings may require a bit more care, but the comment below doesn't prevent merging this PR. You can just delete that logic later if it is bad.

# Expanding Positional Embeddings

```
if max_pos > self.weight.size(0): # recompute/expand embeddings if needed
```

> The reason I haven't autoexpanded the bart positional embeddings so far is that I wanted an error for long sequences that the model would translate poorly, instead of just poor performance. But if you can like concatenate a few en-ru examples and see that performance doesn't plummet it would be good. There is also a theoretical O(seq_len^2) theoretical cost associated with passing longer documents through transformers, so we may not want to encourage longer docs/instead write tooling that, for example, uses moses SentenceSplitter to chunk documents, pass them through the model, and rejoin the results correctly. If it just works with the auto expansion hack then I'm all for it.

@stas00: I shall investigate and will follow up when I got a chance to do so

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I was hoping that someone would have a need for that and then investigate, but since nobody asked this is still a may be.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


as there has been no interest in this, closing this for now.
[models website] search UI issues
Author: stas00
Pull request: no
Labels: name: Feature request

@sshleifer showed me a secret API to finding all fsmt models: https://huggingface.co/models?filter=fsmt

I have several why/how questions:

1. Why can't I do the same from the models page? Where does a user find access to the `filter=` key? It's the `Tags` dropdown, but there is no way to choose the tag I want - `fsmt` is not in the options.

2. How can I use the UI to combine keywords/tags so that I could get sub-sets, like https://huggingface.co/models?filter=fsmt&filter=allenai or perhaps https://huggingface.co/models?search=allenai&filter=fsmt?

3. I don't think `filter` can be repeated. If I reverse order: https://huggingface.co/models?filter=allenai&filter=fsmt I get different results from https://huggingface.co/models?filter=fsmt&filter=allenai - so it ignores the second key (this is sub-set related)

4. The left upper corner widget appears very unintuitive - it looks like a search input, it even says "Search", but it only does something (shows a dropdown list of choices) if the input matches something it finds behind the scenes. So if I type `fsmt` it just sits there quietly. I should be able to hit enter and it to tell me that it found nothing or something like that. Silence is not a great UI. 

5. Why doesn't that search widget in the left upper corner find `fsmt`? What fields does it search?

Thank you.

@julien-c 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Unstale - would be great to have some of these resolved.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
[model site] search UI: language: tags, directionality and filtering
Author: stas00
Pull request: no

I tried to use the models site to find which models I can use for translation of specific languages, and here are the issues I have encountered while doing that:

1. many HF created models aren't tagged for the languages they are trained for - e.g. t5: https://github.com/huggingface/transformers/issues/9314 - would it be possible to go over the HF-created main models and ensure they are clearly tagged with languages they were trained for? These get downloaded a lot, so putting a bit of metadata will go a long way of making user's life a bit easier.

2. the language tags presume bi-direction, but many models have been trained in one direction only - e.g. most wmt and most t5 models. Would it be helpful to support not just the language tags but also the directional language tags if they are one-way only?

   The t5 models are one direction only, so probably need one direction tags.  Not sure how that would work with Language tags in search UI. Perhaps it's enough to indicate that in README, but as the number of models grows being able to quickly filter what's needed will save a lot of user's time, so perhaps planning ahead would be useful. i.e. if I need to perform a FR to EN translation, the user may benefit from getting hits for only models that can do that.

3. wrt search UI - I don't understand how a handful of special language tags is selected - there are about a dozen of language tags in the search API dropdown, Malay is there when there are hardly any models trained on that language, but Russian which is 5th on the list of number of models is not there. 

   And those languages that are "favorite" aren't sorted... very strange. 

4. search UI 2: And to get to the language one wants which is not on the favorite list - one has to solve the puzzle:

   * select "See All languages", which goes to https://huggingface.co/languages 
   * hit on the list of models for that language, 
   * then filter by the model type by typing it in and then one has arrived.

   Surely, there must be an easier way to select a language filter that doesn't take 3 steps which aren't obvious at all
   
5.  Moreover if I want to select 2 languages that aren't on the favorite list, I'm out of luck, since it's not possible with the current API. It only works for the favorite list.

   And even if some of us know how to hack the URL and manually insert: https://huggingface.co/models?filter=ru,en - this is an OR operation, how do I do AND operation or I guess this is related to item 2 of this Issue - how do I filter by the to/from language.

All of these requests/questions are nice to have and none a showstopper. 

Thank you!
   
@julien-c 

Comments:
cc'ing @gary149 and @beurkinger 

Hmm, I just discovered I had 2 related issues opened some months back:
- https://github.com/huggingface/transformers/issues/8531
- https://github.com/huggingface/transformers/issues/7206
[model site] missing language tags for t5 models
Author: stas00
Pull request: no

Would it be possible to update core t5-* models' cards to include what languages they were trained on? Currently it says "en", which is very lacking.

e.g., see: 
* https://huggingface.co/t5-base
* https://huggingface.co/t5-small
* etc.

The core t5 models should somehow have hits with https://huggingface.co/models?search=t5&filter=de, but they don't. Probably because they aren't tagged with the language tags. So they aren't found.

From: https://s3.amazonaws.com/models.huggingface.co/bert/t5-small-config.json
it looks like: French/German/Romanian. But also it looks to only support one direction, so probably adding the following would be clear enough to the end user:

* en_to_fr
* en_to_ge
* en_to_ro

Thank you.

@patrickvonplaten


Comments:
I updated all 5 models to include fr/ro/de language tags.
Cannot import name swish from transformers.activations
Author: ardiantovn
Pull request: no

I have installed `transformers v4.4.1` and `tensorflow v2.4.1`. 

I tried to run
`from transformers.activations import gelu, gelu_new, swish`. 

I get the error like this:
`ImportError: cannot import name 'swish' from 'transformers.activations' (/Users/array/opt/miniconda3/lib/python3.7/site-packages/transformers/activations.py)`

Is there any solutions for this error?

Thank you🙏


Comments:
Hi! `swish` is not importable because it isn't available. `swish` is another name for `silu`, but arrived after it so the name you can use is `silu`:

```py
>>> from transformers.activations import silu
```

However, in our `ACT2FN` dict we have support for both `swish` and `silu`, so that you can do:
```py
>>> from transformers.activations import ACT2FN
>>> swish = ACT2FN["swish"]
>>> silu = ACT2FN["silu"]
```

Thank you
Fix bug in input check for LengthGroupSampler
Author: thominj
Pull request: yes

# What does this PR do?

This commit fixes a bug in the LengthGroupSampler where if
model_input_name is not set, the default value is None instead of
"input_ids"

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

I did not write a test for this, but if necessary I can. Neither this sampler nor the distributed version currently have test coverage for the ValueError this bug raises, but it might not be bad to have.

## Who can review?

@patrickvonplaten


Comments:
How to interpret fine-tuned model results and use model
Author: zakerytclarke
Pull request: no

Hello,

Apologies if this is the wrong forum to ask these kinds of questions, but I was unable to find this in the documentation.

I fine-tuned a seq2seq model on my custom dataset using the tutorial found here: https://github.com/huggingface/transformers/tree/master/examples/seq2seq

I am trying to find out the F1 and EM accuracy for the fine-tuned model, but am not sure how to interpret the output. I've attached a link to the training's output below:
https://github.com/zakerytclarke/transformers/tree/master/modelResults
```
{
    "epoch": 3.0,
    "eval_gen_len": 55.7429,
    "eval_loss": 2.063843250274658,
    "eval_mem_cpu_alloc_delta": 1998448,
    "eval_mem_cpu_peaked_delta": 638828,
    "eval_rouge1": 33.8505,
    "eval_rouge2": 13.1365,
    "eval_rougeL": 27.8332,
    "eval_rougeLsum": 31.5921,
    "eval_runtime": 119.8097,
    "eval_samples": 35,
    "eval_samples_per_second": 0.292
}
```
Can you point me to documentation about how to interpret these results and how I can load my fine-tuned model in order to evaluate it on a new piece of text?


Thanks for your help,
--Zak

Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discusss.huggingface.co) instead?

@patrickvonplaten @stas00 

Thanks!

@LysandreJik Thanks for pointing me in the right direction, I've moved the post over to the forum.
Allow `do_lower_case=True` for any tokenizer
Author: n1t0
Pull request: no
Labels: name: Feature request

# 🚀 Feature request

Extract the `do_lower_case` option to make it available for any tokenizer. Not just those that initially supported this, like the `BERT` tokenizers.

## Motivation

Sometimes we want to specify `do_lower_case=True` in the `tokenizer_config.json` of a custom tokenizer to activate the lowercasing. The problem is that this obviously works only for tokenizers based on one that originally used this option.

I think we should extract this feature to make it a shared one, that could be used with any tokenizer.

Example of a model that would need this described here: https://github.com/huggingface/transformers/issues/9518

## Special care points

- Make sure the `convert_slow_tokenizer` script also handles this, to activate the option in the resulting fast tokenizer.
- Maybe some other options could have the same treatment?

cc @LysandreJik @sgugger

Comments:
Discussed offline with @n1t0: our current decision is to wait for https://github.com/huggingface/tokenizers/issues/659 to be resolved before moving on with this issue. 
This is the better tradeoff as the alternative would imply duplicating a lot of logic in `transformers` that's already present but not exposed by `tokenizers`.
Typo in M2M100 model page
Author: xhluca
Pull request: no

Seems like there's a typo in the [m2m 100 page](https://huggingface.co/facebook/m2m100_418M):
```python
from transformers import M2M100ForConditionalGeneration, M2M100Tokenizer

hi_text = "जीवन एक चॉकलेट बॉक्स की तरह है।"
chinese_text = "生活就像一盒巧克力。"

model = M2M100ForConditionalGeneration.from_pretrained("faGreekook/m2m100_418M")
tokenizer = M2M100Tokenizer.from_pretrained("facebook/m2m100_418M")
```

Pretty sure it should be "facebook" instead of "faGreekook"

Comments:
Indeed! Pinging @patil-suraj 

urgh, my bad. Thanks for pointing it out. fixed!