Issue 1:
Feature request

🤗 Accelerate has a wrapper to help with distributed metric calculation (a tough problem!), and the no_trainer scripts should be updated to include it!

An example can be seen here, below is an example diff of what the integration would look like:

- predictions, references = accelerator.gather((predictions, batch["labels"]))
- # If we are in a multiprocess environment, the last batch has duplicates
- if accelerator.num_processes > 1:
- if step == len(eval_dataloader) - 1:
- predictions = predictions[: len(eval_dataloader.dataset) - samples_seen]
- references = references[: len(eval_dataloader.dataset) - samples_seen]
- else:
- samples_seen += references.shape[0]
+ predictions, references = accelerator.gather_for_metrics((predictions, batch["labels"]))

The list of available scripts to update include:

examples/pytorch/image-classification/run_image_classification_no_trainer.py

examples/pytorch/language-modeling/run_clm_no_trainer.py

examples/pytorch/language-modeling/run_mlm_no_trainer.py

examples/pytorch/multiple-choice/run_swag_no_trainer.py

examples/pytorch/question-answering/run_qa_beam_search_no_trainer.py

examples/pytorch/question_answering/run_qa_no_trainer.py

examples/pytorch/semantic-segmentation/run_semantic_segmentation_no_trainer.py

examples/pytorch/speech-pretraining/run_wav2vec2_pretraining_no_trainer.py

examples/pytorch/summarization/run_summarization_no_trainer.py

Motivation

This is a great first issue for someone who wants to learn how to use some of the latest bits in Accelerate and get an easy beginner contribution to the library 🤗

Your contribution

If you decide to pick up this issue, feel free to ping myself (@muellerzr), @sgugger, or @pacman100 to review 🤗

Issue 2:
🚀 Feature request

T5ForConditionalGeneration inherits generate from PreTrainedModel via GenerationMixin.

Currently in our docs if we write:

for more info see :meth:`transformers.T5ForConditionalGeneration.generate`

it doesn't get resolved in the html docs (it doesn't get a xref link, but leaves it as a text)

Neither transformers.PreTrainedModel.generate gets resolved - which is the super-class that has this method.

But since it's a mixin and actually comes from transformers.generation_utils.GenerationMixin the only way to get the cross-reference turned into a working link is to write:

for more info see :meth:`transformers.generation_utils.GenerationMixin.generate`

which is slow and complicated to figure out and non-intuitive. And this was just one example.

Both @sgugger and I tried to find a sphinx configuration to make it resolve transformers.T5ForConditionalGeneration.generate to transformers.generation_utils.GenerationMixin.generate and point to the right doc entry, but didn't succeed.

This could be related to the fact that we don't generate docs for inherited methods and only document them in the parent class. We want to keep it that way but to be able to cross-reference the inherited methods in the docstrings.

If you perhaps have an expertise in sphinx and could help us resolve this it'd be absolutely amazing!

T5 was just an example, this is needed for dozens of models.

If there is no way to currently do it you could also submit a feature request with sphinx and take a lead on following through for this new feature to be added.

Bonus:

As explained above we don't duplicate inherited methods docs, but it'd be really useful to list the inherited methods which didn't get documented in the sub-classes, e.g.: https://huggingface.co/transformers/model_doc/t5.html#tft5forconditionalgeneration could have a brief entry:

Inherited methods: generate, greedy_search, sample, beam_search, beam_sample, group_beam_search

with cross-references to the corresponding full doc entries in the super-class. Of course, while this could be created manually, it'd be very difficult to maintain - so we need to this to be autogenerated.

Thank you!

Related issues/prs:

#11049

#9202