Update README.md

README.md

@@ -9,16 +9,8 @@ language:
 
 <img src="https://allenai.org/olmo/olmo-7b-animation.gif" alt="OLMo Logo" width="800" style="margin-left:'auto' margin-right:'auto' display:'block'"/>
 
-# TODO
-* Update summary of Dolma 1.7
-* Remove installation requirements?
-* Evals pre and post annealing
-* details on annealing / accessing checkpoint (remove previous checkpoint instructions)
-
 # Model Card for OLMo 1.7-7B
 
-<!-- Provide a quick summary of what the model is/does. -->
-
 OLMo is a series of **O**pen **L**anguage **Mo**dels designed to enable the science of language models.
 The OLMo models are trained on the [Dolma](https://huggingface.co/datasets/allenai/dolma) dataset.
 We release all code, checkpoints, logs, and details involved in training these models.
@@ -56,8 +48,6 @@ A few revisions were lost due to an error, but the vast majority are present.
 
 ### Model Description
 
-<!-- Provide a longer summary of what this model is. -->
-
 - **Developed by:** Allen Institute for AI (AI2)
 - **Supported by:** Databricks, Kempner Institute for the Study of Natural and Artificial Intelligence at Harvard University, AMD, CSC (Lumi Supercomputer), UW
 - **Model type:** a Transformer style autoregressive language model.
@@ -69,8 +59,6 @@ A few revisions were lost due to an error, but the vast majority are present.
 
 ### Model Sources
 
-<!-- Provide the basic links for the model. -->
-
 - **Project Page:** https://allenai.org/olmo
 - **Repositories:** 
     - Core repo (training, inference, fine-tuning etc.): https://github.com/allenai/OLMo
@@ -83,8 +71,6 @@ A few revisions were lost due to an error, but the vast majority are present.
 
 ## Uses
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-
 ### Inference
 
 *Note: The OLMo models will shortly be included in Transformers.*
@@ -187,6 +173,16 @@ And for the 1B model:
 
 ### Data
 For training data details, please see the [Dolma](https://huggingface.co/datasets/allenai/dolma) documentation.
+**This model uses the new 1.7 version with more data sources, better deduplication, and quality filtering**.
+During the annealing phase we use a higher quality subset of Dolma with a linearly decaying learning rate to 0.
+
+### Staged training / annealing
+
+In contrast to OLMo 1.0, we trained OLMo 1.7 with a two-stage curriculum: 
+* In the first stage, we trained the model from scratch on the Dolma 1.7 dataset. We set a cosine learning rate schedule with a warmup of 2500 steps, a peak learning rate of 3e-4, and a cosine decay to 3e-5 after 3T tokens. We cut off this stage after 2T tokens, when the learning rate is still high. 
+* At this point we switch to the second stage, in which we train on a higher-quality subset of Dolma 1.7 (see below) for another 50B tokens, while linearly decaying the learning rate to 0. Our high-quality subset includes (1) using all available Wikipedia, OpenWebMath and Flan data, (2) removing Dolma CC, CC News, and Megawika, and (3) rebalancing remaining sources to achieve approximately equal proportions of each. See exact token counts and relative proportions of this second stage mix below.
+Both stages contribute equally to the final performance of the OLMo model. After the first stage, OLMo 1.7 already outperforms OLMo 1.0. The second stage consistently adds 2 to 3 points of performance on top.
+
 
 ### Architecture