Features proposals to simplify training Tensorflow model
Author: jplu
Pull request: no
Labels: name: wontfix

name: High-Level feature

name: TensorFlow

# 🚀 Feature request
Hello,

On my side I thought to implement some features to simplify the way we can train Tensorflow models (I think it can certainly be adapted to Pytorch as well), and I wanted to know if it might be useful for you. Here a non exhaustive list of features I have in mind:

1. Augment the tranining pipeline with some useful functions such as:
- An [LR finder](https://docs.fast.ai/callbacks.lr_finder.html) that will try to find the best LR for a specific dataset
- [Approach to help to set better hyperparameters](https://arxiv.org/abs/1803.09820)
- [Cyclical LR during training](https://arxiv.org/abs/1506.01186)
- Augment the config.json file of each model with specific training parameters (epochs, LR, batch size, GPUs, etc...) in order to better reproduce a specific training.

2. Modify a bit the `TFPretrainedModel` class in order to better handle:
- multiple GPU training
- Custom training loop
- Custom optimizer creation
- Gradient Accucmulation
- Add a checkpoint manager
- Handle Tensorboard

3. Modify few model classes to add custom loss computation such as for the NER as I have done in the TF example.

I don't know if it sounds interesting for you @thomwolf, @julien-c and @LysandreJik ?


Comments:
I think that we are trying to keep model logic and training logic very separate. So if you want to try to make the tf **examples** (where training happens) simpler, I'd recommend sending a PR that does that for `examples/run_tf_glue.py` or `example_run_tf_ner.py`, or even adding a new Tensorflow example for a task you care about.

Cool, thanks a lot for your reply!!

All these features are not only focus on Tensorflow, I have written Tensorflow because it is the framework I know, but I'm sure we can totally apply them on the Pytorch part as well.

I think I haven't been clear enough and this is my fault, sorry. What I meant is two kind of features:

The point 1. is specific to the [training part of the core pipeline](https://github.com/huggingface/transformers/blob/master/src/transformers/commands/train.py). I also do agree that modifying the config file can be confusing, maybe to create a separate config file specifically for training, I don't know, I'm still open to suggestions. My thought is to have some metadata on the training itself in order to be able to easily reproduce it without giving yourself the values to the network as parameters but just by sharing a file that we can upload to the models hubstore.

The point 2 is more like some utilities to simplify how to handle models.

The point 3 is to have something similar to the Pytorch models where the loss is directly computed in the forward method, I was thinking it could be a good idea to have the same facility for Tensorflow.

I have already started to work on 2 and 3 to see how it can be, and the pros/cons on the existing examples. I will certainly do a PR later this week or next week to see if you have any review on it.

(What I gonna say below is just my own opinion)
 
I'm suggesting all this because when I talk with most of my colleagues or friends (that are not very familiar with Deep Learning) they don't have the knowledge to create the NER example either in TF or in Pytorch, but would like to train a NER model for their work/project, same thing for simple text classification and they don't want to be bored by writing all this very technical code or to know the meaning of each parameter of the scripts. And in companies I think that more and more people want to train their model without having any ML knowledge.

Thanks, and I agree very much with your vision! Looking forward to your PR!


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.