H2O-Danube-1.8B Technical Report
Authors: name: Philipp Singer; name: Pascal Pfeiffer; name: Yauhen Babakhin; name: Maximilian Jeblick; name: Nischay Dhankhar; name: Gabor Fodor; name: Sri Satish Ambati
Published: 2024-01-30 08:45:08

Abstract: We present H2O-Danube-1.8B, a 1.8B language model trained on 1T tokens
following the core principles of LLama 2 and Mistral. We leverage and refine
various techniques for pre-training large language models. Although our model
is trained on significantly fewer total tokens compared to reference models of
similar size, it exhibits highly competitive metrics across a multitude of
benchmarks. We additionally release a chat model trained with supervised
fine-tuning followed by direct preference optimization. We make H2O-Danube-1.8B
openly available under Apache 2.0 license further democratizing LLMs to a wider
audience economically.