[RFC] introduce `config.trained_precision`
Author: stas00
Pull request: no
Labels: name: WIP

# 🚀 Feature request

As we are discovering that `bf16`-pretrained models don't do well on `fp16` "regime" (and surely vice-versa), and some models are pre-trained in `fp32` and surely won't do well on either `bf16` or `fp16`, and the problem is going to grow as more `bf16`-supporting hardware comes out, I propose we start requiring that the model tells the user which mode it was pretrained under.

So I suggest we add `config.trained_precision` which currently would be one of `fp16`, `bf16`, `fp32`, `unknown`.

I haven't thought it through on how to derive this automatically during `save_pretrained`, but when porting checkpoints the porter can figure that out and manually set this in the conversion script.

For example, from what I understood gtp-neo if `bf16` for all but `2.7B` version, which is `fp32`.

@sgugger, @LysandreJik 

Comments:
Would that information not be better on the respective model cards? I think that's more on that side that it should go with the rest of the training setup.

1. It'd be difficult to enforce that it's being filled out
2. It'd be difficult to tell the user at run time that model A pre-trained in fp32, is attempted to be run in fp16 or bf16

But if we can't do it on config level at the very least it could be a required card entry (but I don't think anything is required in the cards).

How would you enforce it being filled out via the config? You would get the default for pretty much all models too: if a user is too lazy to fill the model card they will also be too lazy to fill the config.

I don't understand what you mean in 2, could you elaborate? Why would it be bad to fine-tune a model trained in FP32 in FP16 or bfloat16?

> How would you enforce it being filled out via the config? You would get the default for pretty much all models too: if a user is too lazy to fill the model card they will also be too lazy to fill the config.

Hmm, right. I was thinking about the tests, which can enforce the field existing in the config object but have no way to enforce the values.

> I don't understand what you mean in 2, could you elaborate? Why would it be bad to fine-tune a model trained in FP32 in FP16 or bfloat16?

It won't work out of the box and will require finetuning, which may not succeed if running into infs/nans. I suppose I was thinking more about inference, which won't work w/o finetuning first. If the model was pre-trained in mixed precision it can be used in fp16 inference, but this won't be the case if it was pretrained in fp32.

I think this feature would be welcome indeed and would save us a lot of trouble as we've seen in the past. Regarding whether we want to have this in the model card or in the configuration, I guess it really depends on whether we want to enforce that with errors or warnings.

I think the main point of having that field is to actually warn the user when they're doing inference with a model trained with a different precision, and to that extent, having it in the configuration makes more sense.

I also think the configuration is here to detail how a checkpoint is configured: how the architecture fits the weights (hidden size, layers) and how it should be used (model type, architecture). I think it would make sense to have this as a configuration field, as not knowing that can result in an unusable checkpoint in other environments. I think that's different from other training-related arguments, such as `gradient_checkpointing`, which don't really make sense once ported to a different environment.

Excellent! So that makes the 2 of us who think it would be most strategically placed in the model config.

So let's look at specifics. I can think of the following:

1. at conversion point - it'd be a responsibility of the porter to fill it out - but could also look at the imported state_dict first - perhaps the weights are already in non-fp32 (some models are saved in `.half()` so in this situation it could be derived automatically)

2. at `save_pretrained` - this is the difficult one. what do we set here? As `save_pretrained` has no way to determine how the model was trained.  So we will need to require the precision to be passed explicitly then? The Trainer can be adapted since it knows the precision, but for non-Trainer users will have to specify it explicitly.

3. rewriting history. what do we do about the thousands of models already on the hub? do a massive script that will push `config.trained_precision = unknown` and then over time start correcting this? at least for the main/popular models and problematic ones  - m?t5/pegasus/gpt-neo

any others cases that I missed?

what would be a good not too long keyword for this one? would `config.trained_precision` be not too long and clear enough?

I think the name is good. I would leave it to a default of `"unknown"` for all existing models, so that we don't have to add it anywhere (especially when we don't have the info). I would personally not try to guess it too much and only set that information when we have it from the people who trained the model.

For 2, I don't think we should try to guess it either when people are not using the `Trainer` and just focus on the trainer. We just need to add a `model.config.trained_precision = xxx` from the args and the env at the beginning of training, then the `save_pretrained` method, which also saves the config, will properly handle that.

For 3, I would only populate the popular models, for which we have the info.

> I think the name is good. I would leave it to a default of `"unknown"` for all existing models, so that we don't have to add it anywhere (especially when we don't have the info). I would personally not try to guess it too much and only set that information when we have it from the people who trained the model.

But we could require this new key for when new models are added. That's why I thought that if we were to massively rewrite the hub's config with `trained_precision = unknown` then we could start enforcing this new field.

> For 2, I don't think we should try to guess it either when people are not using the `Trainer` and just focus on the trainer. We just need to add a `model.config.trained_precision = xxx` from the args and the env at the beginning of training, then the `save_pretrained` method, which also saves the config, will properly handle that.

Yes! 

The only trouble here is that someone taking a model in fp32, training it for 10 steps in mixed precision doesn't quite qualify it for fp16.

----------------

I think the problem is that we can't make `save_pretrained` require this new field (for outside Trainer) since it'd be a breaking change.

And also the main event where this field needs to be set is when the model is ported from another system (since that's where the current problems all originate from). So how could we  at the very least enforce this in conversion scripts?


Made a [wiki post](https://discuss.huggingface.co/t/compiling-data-on-how-models-were-trained-fp16-fp32-bf16/5671) - hoping to gather more info via the community input, so that we can have enough data to do some initial seeding of this new field.


This PR is related - adding `config.torch_dtype` field: https://github.com/huggingface/transformers/pull/12316

I guess I can tackle this one next in line.


New development:

8-bit quantized models have arrived: https://github.com/huggingface/transformers/issues/14839 - need to make sure we don't load those in fp32!


> > How would you enforce it being filled out via the config? You would get the default for pretty much all models too: if a user is too lazy to fill the model card they will also be too lazy to fill the config.
> 
> Hmm, right. I was thinking about the tests, which can enforce the field existing in the config object but have no way to enforce the values.
> 
> > I don't understand what you mean in 2, could you elaborate? Why would it be bad to fine-tune a model trained in FP32 in FP16 or bfloat16?
> 
> It won't work out of the box and will require finetuning, which may not succeed if running into infs/nans. I suppose I was thinking more about inference, which won't work w/o finetuning first. If the model was pre-trained in mixed precision it can be used in fp16 inference, but this won't be the case if it was pretrained in fp32.

Hi, thanks for this nice analysis. Do you mean the weights of a model pre-trained in mixed precision (computation precision=bf16, **data/param=fp32**) can be loaded in bf16 during inference? Would that change the model weights and hurt inference performance? @stas00 stas00

It should work just fine, @LeoXinhaoLee - at least for the situation you are describing.

The problem usually emerges when a model is trained in bf16 but used in fp16.

> It should work just fine, @LeoXinhaoLee - at least for the situation you are describing.
> 
> The problem usually emerges when a model is trained in bf16 but used in fp16.

Thanks for your kind reply. Is this because fp32 and bf16 have the same range, so it will only lose something small after the decimal point when converting fp32 weights to bf16 for inference, but converting to fp16 may lead to overflow and significantly change weights? @stas00 

You will find this thread enlightening https://github.com/huggingface/transformers/pull/10956

Hopefully it'll address your question, @LeoXinhaoLee