NousResearch

/

Nous-Hermes-13b

like 428

Guys, what were the hyperparameters for the training job here? Also, did you do QLoRA or trained the full model in float16/float32?

It was mixed precision, dont know the hyperparams off the top of my head. 2-e5 was the lr I remember that

Gotcha. Does mixed precision mean it was trained using QLoRA, or LoRA? Did you use Cosine annealing schedule at all?

Gotcha. Does mixed precision mean it was trained using QLoRA, or LoRA? Did you use Cosine annealing schedule at all?

It means fp32/fp16, not lora or qlora