Log continuously models with wandb
Author: borisdayma
Pull request: no

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

wandb integration currently logs last model (which can be the best by using `TrainingArguments.load_best_model_at_end`).

It would be great to allow continuous upload of model with appropriate aliases to versions.

Options would be:
    * `WANDB_LOG_MODEL = True` which just logs at the end as currently (not sure if we want to add scheduler and optimizer)
    * `WANDB_LOG_MODEL = 'all'` which logs continuously the model
    * `WANDB_LOG_MODEL = False` which does not log the model

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

Training can be very long and it would be so sad to lose a model :sob: 

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->

I can probably propose a PR but would love brainstorming on the ideal logic:
1. should we leverage `Trainer.save_model` (as currently) or `Trainer._save_checkpoint`
2. should we consider an artifact version as containing only the model & config or also containing optimizer and scheduler? Or should it actually be 2 separate artifacts?
3. if we leverage `on_save`, can we avoid the same current logic (fake trainer saving to a temporary directory that is then uploaded async) and just use an actual copy of what has been saved. We would just need the path or list of files that have been saved (should be straightforward)
4. If we log continuously the model, should we upload it only if it's improved (when `metric_for_best_model` is defined)? If that's the case, we'll need to be able to detect when that is the case. If that's not the case we'll still need to be able to know which one is the best.

Comments:
I'm realizing now it would be so helpful with xlsr trainings (lost models due to crash after very long training).

Would you have any input or suggestions @sgugger on how I could try to implement it?

Hi @borisdayma, I hadn't seen this issue. On our side we're more focused on how to continuously push the checkpoints to our own hub ;-).

That being said, we can definitely leverage the `on_save` event and just look for the last `checkpoint-xxx` folder, then push its content as artifact. If you have a tracking with `metric_for_best_model`, then you won't even have to look for the checkpoint, it will be in the state with `state.best_model_checkpoint`.
As for having one or separate checkpoints, I guess it really depends on what you think is best for WandB, you have more expertise than me here.

Thanks, it makes sense!
Actually I imagined you may probably have some interest in a similar logic for the model hub so I wanted to work on something that would be useful for everyone.

As a side note, models could also be stored on the model hub AND tracked by W&B (some people do it with ASW S3 for example). In this way, only checksums are actually stored and the files point back to the storage space so it could be complementary.

To do the same on the hub, my idea was to leverage the versioning system and just push the saved checkpoint every save with a commit message like "checkpoint step xxx". Ideally inside a Callback to avoid adding more stuff to the main training loop.
I'll try to focus on this next week and see what we can easily do!

Nice, if possible it would be cool to allow a logger method to be called right after you push your checkpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Still interested in working on it! Let me know when you have a hook for the model hub!

We have started working on it (the Trainer gained a push_to_hub API!) the last step missing is the continuous checkpointing, will try to work on that last part soon!

Awesome, if you can somehow have a hook after pushing to the hub (with access to the url maybe) then we could link link them to the runs.

The call to `Trainer.push_to_hub` returns the url of the commit to the model hub (it's not part of the train for now).

I really like where this is going!

Is the goal for `TrainingArguments.push_to_hub` to be eventually directly used by the `Trainer` or will it always be handled by the scripts?
Also would it be possible to save the return of `_push_to_hub` somewhere in the `Trainer` (that way that url could be used by wandb).

Yes, we can definitely save the URL somewhere! Would you like to make a PR with that?

I'm on another project that we will release soon right now but also plan to go back to the continuous integration after (should be in two weeks!)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.