Habana

/

gpt2

like 1

Follow

Habana AI 21

@olszd Could you add "use_torch_autocast": true in the Gaudi config please? Actually HMP was enabled since we have "use_habana_mixed_precision": true.

Besides, regression tests didn't pass because there is an issue with autocast when doing gradient checkpointing. So let's wait for this to be solved before merging this PR.

The issue with gradient checkpointing was solved, but default autocast is ~20% slower than HMP with custom ops. I tried autocast with custom ops, got similar speeds to HMP, but the loss is NaN. Let's wait for this PR to be merged before doing anything here.

@regisss I've updated the config, can we retest now and merge if the tests pass?

I updated the Gaudi config with custom bf16/fp32 op lists that give better throughput and similar accuracy, closing this one: https://huggingface.co/Habana/gpt2/blob/main/gaudi_config.json

I'm getting better results keeping the same custom ops (the current ones with HMP) for autocast than default autocast, so I'm going to update that directly.