Note 2/22/24 Type: PI, using hparam search for scale-factor per dimension 1. Observe high/low frequency inequality between the early/late hidden dimensions. Approach via progressive scaling over dimension (e.g. no scaling for dim 0 of 64, by 40, it's scaled down by ~30%) 2. Avoid scaling the first N tokens (avoid the attention sink) 3. Search for two parameters - (scale_i, window), with i being the hidden dimension #, scale-factor must monotonically increase as dimension # increases

Note 2/22/24 From author: I tend to view the contribution is data and data alone, not only the data composition but also the data scale. When comparing this work with https://arxiv.org/abs/2309.16039, note a foundamental difference is that we hypothesize that the long-context capability is already within the base model, and one only needs very light weight continue pretrain to unlock it, i.e. only use 5B data. This is a good news for research and open source.

Note In backlog

Note 2/5/24 Type: Survey Seem to mostly copy (sometimes verbatim) from surveyed work, does not include every RoPE trick PEs: Alibi, RoPE, Random PE (missing NoPE, T5, APE) RoPE tricks: Linear PE, YaRN, PoSE

Note In backlog

Note In backlog

Note 2/23/24 In this work, we take a step towards better context utilization in LCLMs. We focus on training data, keeping other components, such as the architecture and training objectives, unchanged. The broad question is how to organize training data to enhance long context capabilities? I think 1.5 uses a flavor of this technique to some extent. In particular to disambiguate groups of articles packed together. Likely uses custom seperators, attentions for each packed group of articles.

Note 2/23/24 U-Shaped Performance: Models are better at using information at the beginning (primacy bias) or end (recency bias) of the context. Performance drops when information is in the middle, indicating a limitation in handling long contexts. They likened this trend to the serial-position effect found in psychology.

Note 2/23/24 By simply reordering the pretraining data, this method offers a scalable way to significantly enhance the contextual reasoning abilities of language models. Instead of random documents, the models are trained on sequences of related documents. This simple change encourages the models to reason over longer contexts and learn relationships between documents, significantly boosting performance on tasks requiring contextual understanding.

Note 1. Ring Self-Attention for sequence-parallelism 2. Long context data engineering (incl generalization AND long-range retrieval acc) 3. ABF = 10M Very similar to LWM!