noamaanMulla-03/transformers-issues huggingface dataset:
[Flax] Fix hybrid clip
Author: patil-suraj
Pull request: yes

# What does this PR do?

This PR fixes an issue with saving and loading `HybridCLIPConfig` which was causing issues when loading the model.

The config was expecting the `text_config_dict` and `vision_config_dict` arguments, but in the `to_dict` it was saved as `text_config` and `vision_config`. So the next time when `config` was loaded using `from_pretrained` the `model_type` key was missing from `text_config_dict` and `vision_config_dict` as we pop that in the init.

This PR renames `text_config_dict` and `vision_config_dict` to `text_config` and `vision_config` respectively. This is slightly breaking as we won't be able to load previously saved configs. A simple workaround for that is to re-initialize the config using
`from_text_vision_configs` method and resave them.

This PR also adds some usage examples for `FlaxHybridCLIP`.

Fixes #12513

(I should really add tests for this)

Comments:
Using huggingface Pipeline in industry
Author: Lassehhansen
Pull request: no

Hi,

I was wondering if there is anything to be aware of when using your sentiment-analysis pipeline for industry projects at work. Are there any limitations to what I can or cannot do?

Thank you for your always amazing service.

Lasse

Comments:
The pipeline is a simple wrapper over the model and tokenizer. When using a pipeline in production you should be aware of:
- The pipeline is doing simple pre and post-processing reflecting the model's behavior. If you want to understand how the pipeline behaves, you should understand how the model behaves as the pipeline isn't doing anything fancy (true for sentiment analysis, less true for token classification or QA which have very specific post-processing).
- Pipelines are very simple to use, but they remain an abstraction over the model and tokenizer. If you're looking for performance and you have a very specific use-case, you will get on par or better performance when using the model and tokenizer directly.

Does that answer your question?

I think my question was minded for whether the pipeline is free to use in industry, and or whether there is any security issues with using it in an automated work process for industry projects? Not so much minded for the bias of the model.

Thanks in advance
Lasse

For industry usage including security guidance etc I would recommend getting in touch with our Expert Acceleration Program at https://huggingface.co/support

Cheers

Perfect, I will try and do that. Thanks for the help
Is is possible to fine-tune GPT2 on downstream tasks currently?
Author: shizhediao
Pull request: no
Labels: name: wontfix

Is is possible to fine-tune GPT2 on downstream tasks currently?

Comments:
Yes we could add this. You mean tasks like GLUE or SQuAD?

> Yes we could add this. You mean tasks like GLUE or SQuAD?

Yes! exactly!
Please add this, thanks!

@thomwolf Are you still working on the code to finetune the GPT2 language model (not classification task)? Thanks.

@experiencor @thomwolf also curious about GPT2 LM finetuning issue, thanks!

We'll add an example for fine-tuning the models (probably refactor the Bert's one at the same time) this month.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


> We'll add an example for fine-tuning the models (probably refactor the Bert's one at the same time) this month.

Did you add the example ? looking for an example of fine tuning gpt-2 for downstream tasks.

+1. Also interested.
FlaxGPTNeo
Author: patil-suraj
Pull request: yes

# What does this PR do?

This PR adds the Flax version of GPTNeo. For local attention, it uses the fix proposed by @finetuneanon in #11630.

Thanks a lot, @finetuneanon for proposing the solution, it's especially important in JAX/Flax where we can't have dynamic shapes.

Official GPTNeo flax checkpoints are up on the hub and slow tests are passing.

Comments:
Edit readme
Author: SaulLu
Pull request: yes

# What does this PR do?

I have the impression that a tiny typo has crept into the readme.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
implementing tflxmertmodel integration test
Author: sadakmed
Pull request: yes

What does this PR do?
this PR implement an integration test for TFLxmertmodel as requested in [#9954](https://github.com/huggingface/transformers/issues/9954)

@LysandreJik

Comments:
I believe your version of the code quality tools isn't up to date - can you try running the following at the root of your clone: `pip install -e .[quality] -U` to update your tools?

Hi @LysandreJik  Yep It was my black version
[examples/flax] add adafactor optimizer
Author: patil-suraj
Pull request: yes

# What does this PR do?

This PR adds the Adafactor optimizer in all language modeling scripts. This enables fine-tuning large models like 1.3B GPTNeo. Without adafactor the current CLM script can't even load the optimizer states. With Adafactor and bf16 it should now be possible to fit 1.3B GPTNeo on a single v3-8.

With 1.3B GPT Neo, one should be able to fit
- 8 per device batch size with max length 512
- 2 per device batch size with max_length 1024

```
python examples/flax/language-modeling/run_clm_flax.py \
    --model_name_or_path EleutherAI/gpt-neo-1.3B \
    --dataset_name wikitext \
    --dataset_config_name wikitext-2-raw-v1 \
    --do_train --do_eval \
    --block_size 512 \
    --num_train_epochs 1 \
    --learning_rate 1e-5 \
    --per_device_train_batch_size 8  --per_device_eval_batch_size 8 \
    --dtype bfloat16 \
    --adafactor \
    --overwrite_output_dir \
    --output_dir ~/tmp \

Comments:
[Flax] Add FlaxMBart
Author: stancld
Pull request: yes

# What does this PR do?
This PR adds flax implementation of MBart.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@patrickvonplaten  @patil-suraj

Comments:
@patil-suraj Thank you a lot for your suggestions. I fixed the order of attention and normalization layers and some other minor bugs. Also added some additional copy statements.

I also changed the `shift_tokens_right` method as this one looks to be different for the MBart models as they don't have a single `decoder_start_token_id` in contrast to other Bart-like models. => This difference of having no `decoder_start_token_id`, however, currently leads to some issues within the `generate` method. (I'll try to have a look what can be done here)


@stancld I pushed a couple of commits to add the `layer_norm` in encoder and decoder. Now, all slow tests are passing.
@patrickvonplaten  could you please take a final look?
[trainer] add option to ignore keys for the train function too (#11719)
Author: shabie
Pull request: yes

# What does this PR do?

This pull request adds the option of ignoring certain output keys for evaluation during the training phase. As of now, this option is only available for `predict` and `evaluation` methods of the `Trainer` class which can only be called after the training.

Fixes #11719

Changes:

1. Add a new parameter to the `Trainer.train` function called `ignore_keys_for_eval`.
2. Pass this to the `ignore_keys` parameter of the `trainer.evaluate` function that is already called within the `trainer.train` function.
2. Add the parameter to the docstring.

Comments:
parameter `ignore_keys` of `trainer.predict` not accessible in `Trainer` or `TrainingArguments`
Author: shabie
Pull request: no

# 🚀 Feature request

The [`predict`](https://huggingface.co/transformers/main_classes/trainer.html#transformers.Trainer.predict) and [`evaluate`](https://huggingface.co/transformers/main_classes/trainer.html#transformers.Trainer.evaluate) methods of the Trainer class provide an excellent option of `ignore_keys`. Here is a small example:

```python
trainer.predict(dataset, ignore_keys=["ner_loss", "cls_loss", "ner_logits", "cls_logits"])
```

@sgugger 

This option is however, not accessible during the normal setup of defining `TrainingArguments` class nor the `Trainer` class so the a call to `trainer.train()` leads to errors during the mid-training evaluation.

## Motivation

I am unable to evaluate the model metrics on the validation set **during** the training to see if makes sense to continue.

## Your contribution

I am happy to make a PR if this is seen as a genuine problem. Like always, may be I am missing something.


Comments:
We could add an argument for this (like `ignore_keys_for_eval`) yes. Let me know if you want to tackle this!

I very much wanna do this and will get right down to it!  👍 Admittedly though, I am relatively new to making contributions.

Hi! As a short update: I will not be able to work on this until 23rd of June... so if anyone wants to pick it up good otherwise I need 3 more weeks.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

OK so I'd like to work on this :)

Sure! Ping me when you open a PR :-)
Necessary resources for training a (small/tiny) LM from scratch?
Author: brijow
Pull request: no

This is mostly a follow up question in regards to this hugging face blog post on training a LM (and a tokenizer) from scratch :
https://huggingface.co/blog/how-to-train

I think this may be an ideal approach to try out in my situation, but I'm wondering about cost and how much data I really need to train a LM from scratch on my domain-specific dataset? 

I'm quite new to the field and haven't read many papers on this subject as of yet, so I was hoping someone might be able to provide some ballpark estimates about computing resources required for training some small LM(s) from scratch? I'd like to obtain a fine-tuned (or trained from scratch) domain-specific LM to serve as a backbone for various downstream NLP tasks on my domain-specific text data. I have been experimenting with the fine-tuning LM approach, (i.e. fine-tuning BERT based models on MLM before performing task-specific fine-tuning), but I'm curious about the training from scratch option if I can get a rough idea the required compute resources / cost.

Thanks very much in advance for any help / tips on unpacking this question.

Comments:
Hi,

could you please ask this question on the [forum](https://discuss.huggingface.co/) rather than here? We like to keep Github issues for bugs/feature requests.

Thanks!

Sure, thanks @NielsRogge . Here is a [link to the post on the forum](https://discuss.huggingface.co/t/necessary-resources-for-training-a-small-tiny-lm-from-scratch/8139), cheers.
MLM training fails with no validation file(same as #12406 for pytorch now)
Author: Souvic
Pull request: yes

## Environment info

- `transformers` version: 4.9.0.dev0
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.10
- PyTorch version (GPU?): 1.9.0+cu102 (True)
- Tensorflow version (GPU?): 2.5.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes



## Information

Model I am using (Bert, XLNet ...): distilbert-base-cased

The problem arises when using: the official example scripts: (give details below)

The tasks I am working on is: MLM finetuning

## To reproduce

Steps to reproduce the behavior:

1. Just run the tensorflow  examples
2. python3 ./transformers/examples/pytorch/language-modeling/run_mlm.py\
 --model_name_or_path distilbert-base-cased \
 --output_dir ./g \
 --train_file "customdata.txt" \
3. The model fails with error message that no validation file is there.


## Expected behavior
it should use the validation split percentage parameter to divide the training set into training and eval samples.

Comments:
@sgugger @patil-suraj ..
Checks done..
Load Trainer state
Author: Aktsvigun
Pull request: no

# 🚀 Feature request

Hi,
I wonder if there is a method to load a saved state of the trainer, so that I can continue the training loop from where I started.
If I understand it right, I need to load the saved states of _model_, _optimizer_ and _scheduler_. 
## Motivation

However, it would be much more convenient to have all of this done in one method, when you only pass the `results` folder path to the method.

## Your contribution

In case there is not, I can submit the PR, since it does not seem too hard to be prepared.


Comments:
cc @sgugger 

How would this be different from
```
trainer.train(resume_from_checkpoint=True)
```
?

That's pretty similar, still, when one needs to use `trainer.predict` of a trained model, for instance.

You don't need the optimizer and scheduler states for this, just the model.

This makes sense, thank you! Closing the issue.

On Wed, 7 Jul 2021 at 16:33, Sylvain Gugger ***@***.***>
wrote:

> You don't need the optimizer and scheduler states for this, just the model.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/12529#issuecomment-875608286>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AIXZKXIKEBB7ZHSHCE7DJC3TWRJTTANCNFSM474BV33Q>
> .
>
Double check for attribute num_examples
Author: sgugger
Pull request: yes

# What does this PR do?

As pointed out by #12479, the `isinstance` check for `IterableDataset` and its subclasses does not look at the type, but whether the class implements some methods, but not all attributes. This PR adds a double check when necessary to avoid an `AttributeError`.

Fixes #12479

Comments:
Failure looks unrelated but circleCI is not letting me re-run the tests, so merging and watching master.
Remove tf.roll wherever not needed
Author: szutenberg
Pull request: yes

It was used in shift_right.
After this change TF code is more similar to Pytorch implementations
Also, TF graphs are optimized (one node less)

# What does this PR do?

This change optimizes TF graphs and code without modifying the math. 

Using roll is not necessary (probably most accelerators implement it as Slice + Concat).

## Who can review?

@patrickvonplaten, @LysandreJik

Comments:
@szutenberg Thanks for this PR! The changes look good - is there anything else you want to edit, or are you happy for us to merge it as is?

Hi @Rocketknight1 , 

Thanks for the review. The change is ready for merge as it is.

Could you have a look at: https://github.com/huggingface/transformers/pull/12332 ? It resolves issues with bf16. We can fix fp16 in a separate PR (I didn't have time to debug it but gave some hints in comments and I think that starting from debugging the inference graph would be a good idea).
Modify BERT encoder layers?
Author: calusbr
Pull request: no
Labels: name: New model

Hello I would like to modify the encoder layers of the BERT model, to insert FC and ReLu layers.

This idea allows you to reproduce the use of [Squeeze-and-Excitation Networks](https://arxiv.org/abs/1709.01507)

How to use an nn.module class to handle encoder outputs?

Example:

```
import torch.nn as nn
from transformers import BertModel

class CustomBERTModel(nn.Module):
    def __init__(self):
          super(CustomBERTModel, self).__init__()
          self.bert = BertModel.from_pretrained("bert-base-uncased")
          # add your additional layers here, for example a dropout layer followed by a linear classification head
          self.dropout = nn.Dropout(0.3)
          self.out = nn.Linear(768, 2)

    def forward(self, ids, mask, token_type_ids):
          sequence_output, pooled_output = self.bert(
               ids, 
               attention_mask=mask,
               token_type_ids=token_type_ids
          )

          # we apply dropout to the sequence output, tensor has shape (batch_size, sequence_length, 768)
          sequence_output = self.dropout(sequence_output)
    
          # next, we apply the linear layer. The linear layer (which applies a linear transformation)
          # takes as input the hidden states of all tokens (so seq_len times a vector of size 768, each corresponding to
          # a single token in the input sequence) and outputs 2 numbers (scores, or logits) for every token
          # so the logits are of shape (batch_size, sequence_length, 2)
          logits = self.out(sequence_output)

          return logits
```

Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co) instead?

Thanks!
Init pickle
Author: sgugger
Pull request: yes

# What does this PR do?

This PR is an alternative to #12552 and properly sets `_LazyModule` as a class used in all inits (no custom subclasses anymore) to make the `transformers` module picklable. It also cleans up nicely the inits.

The only downside is that new models started before this PR but not yet merged will need a rebase for the intermediate init.


Comments:
Don't stop at num_epochs when using IterableDataset
Author: sgugger
Pull request: yes

# What does this PR do?

Currently, when someone uses an `IterableDataset` inside the `Trainer`, the training loop stops after 3 iterations over the iterable dataset, this PR fixes that to just rely on `max_steps` (has to be set or there is an error at init).

Fixes #12499

Comments:
key error in BertQuestionAsnwering predict?
Author: search4mahesh
Pull request: no

Hi,

I am getting key error while using BertQuestionAsnwering predict?
I am breaking following loop after 10 iterations
for input_ids, input_mask, segment_ids, example_indices in tqdm(eval_dataloader, desc="Evaluating", disable=local_rank not in [-1, 0]):

Thanks
Mahesh

Error:


KeyError                                  Traceback (most recent call last)
<ipython-input-87-6ac2c26449fb> in <module>()
     41                   do_lower_case, output_prediction_file,
     42                   output_nbest_file, output_null_log_odds_file, verbose_logging,
---> 43                   version_2_with_negative, null_score_diff_threshold)

/content/run_squad.py in write_predictions(all_examples, all_features, all_results, n_best_size, max_answer_length, do_lower_case, output_prediction_file, output_nbest_file, output_null_log_odds_file, verbose_logging, version_2_with_negative, null_score_diff_threshold)
    473         null_end_logit = 0  # the end logit at the slice with min null score
    474         for (feature_index, feature) in enumerate(features):
--> 475             result = unique_id_to_result[feature.unique_id]
    476             start_indexes = _get_best_indexes(result.start_logits, n_best_size)
    477             end_indexes = _get_best_indexes(result.end_logits, n_best_size)

KeyError: 1000000088

Comments:
You found a solution?

@thomwolf , I made mistake in code, Repo code works just fine. Hence closed issue.
Thanks for this amazing repo :thumbsup:


How to solve this problem?


what was the solution ? Im seeing the same problem 

Hello! Do you mind opening a new issue with your problem?

Hi, I'm having the same error as described above. Is anyone able to post their solution?