World Model on Million-Length Video And Language With Blockwise RingAttention
Authors: Hao Liu
Wilson Yan
Matei Zaharia
Pieter Abbeel
Venue: ICLR 2025 Poster (2025)
Keywords: long context
world model
attention
Primary area: generative models

Abstract: Enabling long-context understanding remains a key challenge in scaling existing sequence models -- a crucial component in developing generally intelligent models that can process and operate over long temporal horizons that potentially consist of millions of tokens. In this paper, we aim to address these challenges by providing a comprehensive exploration of the full development process for producing 1M context language models and video-language models, setting new benchmarks in language retrieval and new capabilities in long video understanding. We detail our long context data curation process, progressive context extension from 4K to 1M tokens, and present an efficient open-source implementation for scalable training on long sequences. Additionally, we open-source a family of 7B parameter models capable of processing long text documents and videos exceeding 1M tokens.

Published as a conference paper at ICLR 2025
WORLD MODEL ON MILLION-LENTTH VIDEO AND LANGUAGE WITH BLOCKWISE RINGATTENTION
Hao Liu" Wilson Yan" Matei Zaharia Pieter Abbeel
UC Berkeley
ABSTRACT
Enabling long-context understanding remains a key challenge in scaling existing sequence models crucial component in developing generally intelligent models that can process and operate over long temporal horizons that potentially consist of millions of tokens. In this paper, aim to address these challenges by providing a comprehensive exploration of the full development process for producing 1M context language models and video-language models, setting new benchmarks in language retrieval and new capabilities long video understanding. We detail our long context data curation process, progressive context extension from 4K to 1M tokens, and present an efficient open-source implementation for scalable training long sequences. Additionally, open-source family of 7B parameter models capable of processing long text documents and videos exceeding I M tokens.
INTRODUCTION
Enabling long-context understanding remains key challenge in scaling existing sequence models crucial step toward developing generally intelligent models that can process and operate over extended temporal horizons, potentially involving millions of tokens. Current modeling approaches predominantly limited to processing short sequences, whether in the form of language, images, or video clips (Brown al., 2020; Touvron al., 2023a;b; OpenAl, 2023; Brooks et al., 2024; Team et al., 2023). As a result, these models fall short when tasked with understanding complex, long-form language and visual contexts.
However, training models to process sequences that exceed millions of tokens is a significant challenge due to the high memory and computational costs, as well the lack of long-context data, In this work, we address these challenges by leveraging Blockwise Ring (Liu et al., 2024; Liu and Abbeel, 2023), technique that scales context without approximations or overheads, enabling efficient training on long sequences. We curate an extensive dataset of long-form videos and books from
Equual contribution. Email: hao, liubcs. bww.bberyy....c. adu, wi son1 wyweyynnneeeeee www.yy.keeey.............o. edu edu Code and models of Large World Model (LWM) are available at https//largeworddmoeell htw.://dddddooo.iiiih..o///.l github.io/lwm/.
Falcon 108
2K
GPT-4 18K)
LLaMA
SK
GPT-4 (32k)
32K
Claude
100K
LLM
GPT-4 Turbo
128k
GPT 01 Preview
128K
Gemini 1.0
L28K
LLaMA
128K
Gemini 1.5
MM
UyM (Durs)
10
Context Size (tokens)
Figure 1 Comparison of context size in state-of-the-art LLMs. Our model and concurrent work Gemini 1.5 both achieve 1M context size, significantly outperforming other LLMs..

---

Published as a conference paper at ICLR 2025
Gomini Pro
GPT-4
LIVI Text Chat IM (ours)
Retrieval Accuracy
0.8
0.6
0.4
0.2
13K
44k
75k
128K
220K
312K
204k
400k
SB8K
680K
772K
6844
95
1M
Needle Context Length
Figure 2 Retrieval comparisons against Gemini Pro and GPT-4. Needle retrieval comparisons against Gemini Pro and GPT-4 for each respective max context length 32K and 128K. Our model performs competitively while being able to extend to 8x longer context length. Note that in order to show fine-grained results, the x-axis is log-scale from 0- 128K, and linear-scale from 128K-1M.
public sources, covering a wide variety of activities and narrative structures. To address the scarcity of long- forg-form conversational datasets, developed model-based aunstion-answering technique, where a short-context model generates training data from books, significantly enhancing the model's chat capabilities over long sequences. To mitigate computational costs, gradually extended context size from an initial 4K tokens to IM tokens, achieving a cost-effective and scalable approach for long-context modeling.
Following this, we further train our long-context language model to incorporate visual modalities, such as image and video. Contrary existing popular languae models (Liu et al., 2023a; OpenAI, 2023; Chen et al., 2023a), we opt to additionally optimize next-token prediction losses for image and video (generation) with a VQGAN (Esser al., 2021) encoder. We encountered various challenges training on mixed modalities (video, image, text). To balance their unique characteristics sequential information, visual detail, and linguistic content we implement an efficient masked sequence packing strategy, as well introduce careful loss balancing to retain short context accuracy. This approach handles varying sequence lengths more effectively than standard methods. We also optimized the ratio of image, video, and text inputs in each batch, proposing empirically effective balance for cross- cooss-modllity learning. Since our model aims to model both textual and visual projections of the world through a large context window, drawing inspiration from prior work on world models (Brooks et al., 2024; Ha and Schmidhuber, 2018), we name our work as Large World Model (LWM).
Our contributions are threefold: (a) we train one of the largest context size transformers to date on long text documents and videos and achieved competitive results long video understanding and long context fact retrieval. (b) We discover a range of challenges associated with training on long sequences and propose solutions for them: masked sequence packing to effectively train with different sequence lengths and synthetic model model-generating
iresston-answerrig for effective attention. (c) We provide an open-source and optimized implementation for training with millions of tokens in context, well as family of Llama-based 1. context models capable of processing long documents LLMM-Text, LAM Text -Chat) ) and videos (LMM, LWM-Chat) of 1M tokens.
2 METHOD OVERVIEW
We train a large autoregressive transformer model with large context window of up one million tokens, building upon Llama2 7B (Touvron et al., 2023b). To achieve this goal, we implement a two-stage training strategy. In Stage (Section 3), we extend the context to 1M tokens using book-length texts. This is followed by Stage II (Section 4), where we conduct joint training on diverse long multimodal sequences, incorporating text-image data, text-video data, and book-length texts. Our model architecture is the standard autoregressive transformer design, as illustrated in Figure 3. For a comprehensive overview our training stages and the datasets employed, please refer to Figure 4.
2

---

Published as a conference paper at ICLR 2025
Image Tokens
Next Tokens
Next Image Takens
VQGAN
BPE Tokenizer
Autoregressive Transformer
A puppy running on a grassy lawn
Text Takens
Image Takens
Model Architecture. The LWM model is autoregressive transformer trained sequences of multimodal tokens. Each video frame is tokenized into 256 tokens using VQGAN, while text is processed using Byte-Pair Encoding (BPE) tokenizer. These tokens-both image and lext-are combined and input into the transformer to autoregressively predict the next token. The model can handle various input-output modalities, including text, image, video, and text-video pairs. To distinguish between images and text, special tokens <visions and </vision> are used for image and video frames, with <eof> and <eov> marking the end of these sequences. For simplicity, delimiters are not shown in the figure.
Figure 3
STAGE I: LEARNING LONG-CONTEXT LANGUAGE MODELS
This stage aims at first developing 1MM-Text and Text -Chat, a set of long-context language models learned by training on progressively increasing sequence length data, and modifying positional encoding parameters to account for longer sequence lengths (see Section 3. In Section 3.2, we show how to construct model-generated question-answeringg data for enabling long sequence conversations.
3.1 PROGRESSIVE TRAINING TOWARDS LONG CONTEXT
Learning long-range dependencies over sequences of millions tokens requires (I) memory efficient training to scale to such long sequences, as well as need to (2) compute efficient training to extend the context of our base language model. We outline our approach to these challenges, detailing our methods for training on long sequences, designs for efficiency and stability, and experimental setup.
Training on long sequences has become prohibitively expensive due memory constraints imposed by the quadratic complexity of attention weight computations. To address these computational limitations, we leverage recent advancements in scaling context window size, particularly Blockwise Ring RingAtteentin (Liu et al., 2024). This approach theoretically allows for an infinite context, bounded only by available devices. We further enhance performance by fusing it with FlashAttention (Dao al., 2022) using Pallas (Bradbury al., 2018) to optimize performance compared with using XLA compiler. Notably, with enough tokens per device device-already given- the communication cost during sequence parallelism is fully overlapped by computation, resulting in additional overhead.
For better efficiency, we adopt a training approach inspired by prior research on extending context (Jin al., 2023a), where model is trained on progressively longer sequence lengths, starting from 32K tokens and ending at 1M tokens in increasing powers two. Intuitively, this allows the model save compute by first learning shorter-range dependencies before moving onto longer sequences. For extending positional embeddings to longer contexts, we adopt a simple, scaled-up version of the approach explored in Rozière al. (2023), where the 0 parameter for RoPE (Su et al., 2024) scaled in proportion to the context length. We found this approach to be stable for extending positional embeddings with larger context lengths due to its simplicity, requiring the tuning of only a single hyperparameter. Specifically, we scale the 0 parameter for RoPE alongside increases in context window sizes the values are shown in Table The progressive training of growing context sizes shown in Figure 4.
We initialize from LLaMA-2 7B (Touvron et al., 2023b) as base language model and progressively increase the effective context length of the model across 5 stages: 32K. 128K, 256K, 512K, and IM. For each stage, we train on different filtered versions the Books3 dataset from The Pile (Gao et al., 2020). Table 6 details information about each training stage, such the number of tokens, total time, and the Books3 dataset filtering constraints. Each successive run is initialized from the prior sequence length.
3

---

Published as conference paper at ICLR 2025
Stage 1: LLM Context Extension
Stage 2: Vision-Language Training
Images: LAION-2B-ou COY0700M
Text-Vision (495B Tokens)
Doc Length 10k 100k
Tokens: 400B
Context: 82k
Exampless:
Examples 78K
Brown Dog
Rootbeer flost Popakctes
Rose petais shape
Imgga (4003)
100 trames 201
Uieer 450 mrrmas (1.20)
Text: Books3
Video (30 frames): Webvid OM InternVid1 0M (3M Subset)
Paames (2CEM)
Doe Length
10CK 230k
Context:
Text Data Tokans)
Tekene: 93B
Examples: 18M
Examples S2K
Billards, concentrated woman daaynng club.
Rostel 10L
200k 500 C300
Text: Books3
Video (100 frames): Valley-Instruct-73k Video-Instruct 100k
Enntro 5000
Doc Length 5.0kk
P0CK -500k
Tokere: 3.28
Context:
Exambless 175K
Towene: 108
Examples 37K
User What happenii sigwiingg video? Assistant: woman dealiing cards and chips at black jack table
Text: Booke3
Video (450 frames):: Valley-Instruct-75k Video-Insttuuct 100k
Doc Length 80CK 1M
Comeet:
Examples 3.5K
Tokems:
User: are some highlights
captires various ext ilarating moments cirt tiking, including bikers jumping inta the air,
Tokens:
Examples: 173K
Text: Books3
Video (4000 frames): Valley-Instruct-73k Video-Instruct 100k
Doc Length:
Context:
Tokens: 13
Examples: 0.8K
t- Do
User: Cauld you provide brief summary the empkyyee's actions? Assistant: the video, an employes prepares sub. After assembling the bread, ham, pepperoni. salami, and cheese, he toasts the sub the over...
Context:
1- 10mih
Tukenss 200M
Examples: 173K
Figure Curated dataset and training process with progressively increasing data length and complexity. The diagram outlines a two-stage training process. Stage extends text-based understanding using books datasets of increasing document lengths and token counts. Stage integrates vision- lasgua-language training. Pie charts display token distribution, showing that images and short-frame videos dominate visual data, while mid-length text examples lead the text corpus. 3.2 MODEL-GENERATTED QUESTION-ANSWERIN FOR EFFECTIVE CONTEXT
We construct simple anesttoon-answering dataset to develop long-context chat capabilities. First, we split documents from the Books3 dataset into fixed chunks of 1,000 tokens, feed each chunk into our short-context language model, and prompt it to generate a question-answer pair based on the content. To create longer examples (e.g., 32K tokens), the relevant question question-answer answer pairs toward the end of the sequence in chat format. The key intuition that the model must learn focus on any part of the context answer the questions, as the relevant information can appear anywhere within the sequence.
concatenate adjacent chunks and append
For chat fine-tuning, we train each model on a mix of the UltraChat conversation dataset (Ding et al., 2023) and our custom question-answeriig dataset, using approximately 7:3 ratio. We found crucial to pre-pack the UltraChat data to the training sequence length and keep these examples separate from our question-answertne generally contains a much higher proportion of loss tokens (due to densely packed, short questions in chat), whereas
data. This separation is necessary because UltraChat data
question-answe answering data has long questions in chat thus significantly lower percentage of loss tokens per sequence (< 1%). This difference arises from the long documents the given context of question-answering data, which are not included in loss calculations. Table 7
4

---

Published as a conference paper at ICLR 2025
provides further training details for each run. Notably, do not employ progressive training for any of the chat models; instead, we initialize them from their respective pretrained models at the same context length.
Summary:
Stage progressively increase sequence lengths using our curated dataset: starting with 32K tokens and gradually scaling up to IM tokens. Model-generated question-answering data aids learning effective long context.
3.3
LANGUAGE EVALUATION RESULTS
3.3. SHORT CONTEXT TASKS
Table presents comparative analysis between the Llama2-7B model with 4K context and its context-expanded counterparts, ranging from 32K to IM. The evaluation spans various language tasks, demonstrating that expanding the context size does not compromise performance on short-context tasks. In fact, the results suggest that models with larger context capacities perform equally well, not better, across these tasks. This evidence indicates the absence of negative effects from context expansion, highlighting the models' capability to adapt to different task requirements without losing efficiency in shorter contexts.
Table 1
Performance evaluation across language tasks, comparing Llama-2 7B (4K context window) and context-expanded variants of LWM-Text (32K to 1M). The results demonstrate that increasing context length does significantly degrade performance tasks with shorter contexts.
WWM-Text
Task / Metric
Llama-2 7B
32k
128k
256k
512k
1M
are arc_ je__hallenne/ace are arc_ hre__caallnggeecc_nnee norn
0.40
0.43
0.45
0.44
0.44
0.43
0.43
0.47
0.47
0.46
0.46
0.46
hellaswag/acc
0.57
0.57
0.57
0.57
0.56
0.57
hellaswag/acc_nom
0.77
0.76
0.76
0.76
0.75
0.75
mmlu
0.39
0.4
0.41
0.41
0.36
0.35
openbookqa/ace
0.32
0.33
0.31
0.32
0.33
0.30
openbookqa/acc_mmee _aor norm
0.44
0.44
0.44
0.43
0.41
0.41
3.3.2 RETRIEVAL TASK: SINGLE INFORMATION
We evaluate on the popular Needle A Haystack task (gkamradt, 2023) more specifically an version (ArizeAI, 2023) that finds and retrieves random numbers assigned to randomized cities from the context. Figure shows that we can scale to far larger contexts compared to the current best available LLMs. Figure 11 in Appendix shows nearly perfect retrieval accuracy the entire context of our M context model. Appendix C shows more single needle retrieval results for our other shorter context length models.
3.3.3 RETRIEVAL TASK: MULTIPLE INFORMATION
We additionally examine the performance of model more complex variant of the needle retrieval task by mixing in multiple needles, as well as trying to retrieve specific subset of them. Figure shows multi-needle retrieval results under different settings. Our model generalizes well when retrieving a single needle from multiple needles in context, with slight degradation when asked retrieve more than one needle. Table shows multi-needle comparisons, where our model is able to perform competitively or better than GPT-4 at retrieving one needle, or slightly lower performance when retrieving more than one needle. Furthermore, our model is also able to perform well and extend to longer context lengths of up to 1M tokens and far outperforms any recent shorter context baselines applies to longer sequence lengths through positional extrapolation techniques.. However, we note that we see degradation in accuracy while increasing the difficulty of the needle retrieval task, suggesting that there is still more room to improve on the 1M context utilization of our model. We believe that released model will provide a foundation for future work developing longer context models, as well as encourage more challenging benchmarks that contain difficult long-range tasks that require higher levels of synthesis, rather than pure fact retrieval.
5

---

Published as a conference paper at ICLR 2025
Table
Multi Needle in Haystack. denotes models after the completion of this paper.
Context Length
Model
2,
4,
N
Gemini Pro (02/23)
0.34
0.44
0.6
32K
GPT-4-1100
0.97
0.95
0.9
Llama-3. Beee-3..-8B-Ime
1-8B-Instruct*
0.87
0.95
0.93
1.0
1.0
0.97
Mistral-77BIIniiuutt00..33
0.98
0.85
0.83
LWM-Text-IM (Ours)
0.84
0.97
0.84
Gemini Pro (02/23)
128K
GPT-4-1 106
0.92
0.8
0.82
1amma-3.1-8B-nntteee
0.98
0.91
0.87
Swen2.5-7B-Instmuu
0.80
0.90
Mistral-7B-Insteee Instrncc--t..33
0.85
0.75
0,68
LWM-Text-IM. (Ours)
0.83
0.98
0.83
Gemini Pro (02/23)
IM
GPT-4-1100
Llama-3. 1lama-3.1-8B-1l
1-8B-Instruct*
0.27
0.32
0.18
Qwen2.5-7B. ----trtInttt
0.0
0.0
0.0
0.05
0.13
0.10
LWM-Text-IM (Ours)
0.67
0.84
0.69
N=4 / R=1
N / R=2
N=2 / R=2
1.0
0.8
1.5
Score
Score
0.2
0.2
1324
Context Length
Context Length
Context Length
Figure 5 Multiple needles retrieval task with LWM-IM. N is the number of facts in the context, and is the number of given facts model asked to retrieve.
3.3.4 EVALUATION ON LOFT
Table 3 Evaluations on some benchmarks in the LOFT dataset.
Setting: 512K Context
LWM (512K)
GPT-4o (128K)
Claude 3 Opus (200K)
Quora
0.38
0.23
0.37
NQ HotPotQA
0.37
0.22
0.37
0.72
0.21
0).32
We further evaluate our model coverage of the LOFT (Lee et al., 2024) dataset collection, we provides a more natural set of benchmarks that examine capabilities for long-context models in the context of document retrieval, and RAG. The benchmark includes tasks such duplication detection (Quora )), document retrieval (HotpotQA (Yang et al., 2018)), and retrieval- based question-ant -anweeing (NQ). Each dataset contains corpus of 1000s of documents, and the model is asked to retrieve set of document ids pertaining to its specific task (Quora, HotpotQA). For RAG (NQ dataset), the model asked to answer the question using the given context. Table 3 shows evaluations results 512K context length against various language model basclines.
Takeaway: Long context capability enables LWM to outperform state-of-the-arr text models at multiple benchmarks. This demonstrates effectiveness of our methods for enabling long context.
4 STAGE II: EXTENDING TO LONG-CONTEXT VISION-LANGUAGE
Our second stage aims to effectively joint train long video and language sequences. We will intro- duce architecture modifications for 1MM and L-M-Chat to incorporate vision input in Section 4.1.

---

Published as a conference paper at ICLR 2025
Training on varying sequence lengths is discussed in Section 4.2. The evaluation results are shown in Section 4.3. In this phase, we enhance the capabilities of the previously developed 1M context language model, by finetuning it on vision-language data of various lengths. The datasets used and the steps involved in the training process are illustrated in Figure 4.
4.1 ARCHITECTURAL MODIFICATIONS FOR VISION
We use the pretrained VQGAN (Esser et al., 2021) from aMUSEd (Patil al., 2024) that tokenizes 256 x 256 input images to 16 x 16 discrete tokens. Videos are tokenized by applying the VQGAN per-frame, and concatenating the codes together. generating, as well as knowing when to switch, we introduce mechanisms to mark the end of text generation / beginning of vision generation, and vice-versa. For defining the end of vision generation, we introduce new tokens, <eof> and <eov>, that represent end of frame (at the end of each video frame that is not the last video frame in the sequence), and end of vision (at the end of each single image, at the end of the last frame in video) boundaries respectively. For defining the end of text generation, we wrap the vision tokens with <vision> and < </visions (as text) text tokens. The model is trained with interleaved concatenations of vision and text tokens, and predicted autoregressively (see Figure 3).
order to distinguish between modalities when
4.2 TRAINING STEPS
We initialize from our LWM-Text-IM text model, and perform similar process of progressive training on a large amount of combined text-image and text-video data, with the exception that we do not additionally scale RoPE as it already supports up to 1M context. Table 8 shows details for each training stage, where the model is initialized from the prior shorter sequence length stage. For each stage, we train on the following data:
LWM 1x: We train on large set of text-image dataset comprising of a mix of LAION-2B-en (Schuh- mann et al., 2022) and COYO-700M (Byeon et al., 2022). The datasets were filtered to only include images with at least 256 resolution in total roughly IB text-image pairs. During training, concatenate the text-image pairs and randomly swap the order of the modalities to model both text-image generation, unconditional image generation, and image captioning. We pack text-image pairs to sequences of 1K tokens.
8X: We train on text-video dataset mix of VebVid10M (Bain et al., 2021) and 3M Intern- VidI0M (Wang et al., 2023) examples. Similar prior works (Ho al., 2022a;b; Villegas et al., 2022), we jointly train on both images and video with a 50-50 ratio of each modality. We pack images to sequences of 8K tokens, and 30 frame videos at 4FPS. Similar to image training, we randomly swap the order of modalities for each text-video pair.
1M: For the final 3 stages, we train a combined mix of chat data for each downstream task: (1) text-image generation, (2) image understanding, (3) tixt-video generation, and (4) video understanding. We construct simple version of text-image and text- video chat data by sampling random subsets of the pretraining data augmented with chat format. For image understanding, we using the image chat instruct data from ShareGPT4V (Chen et al., 2023a). Lastly, for the video understanding chat data, we use a combined mix of Valley- -hatthhTaaGPOOOO instruct data (Maaz et al., 2023). For all short
73K (Luo al., 2023) and context data (image generation, image understanding, video generation), we pack sequences to the training context length. During packing, we found crucial to mask out the attention so that each text-vision pair only attends to itself, as well as re-weighting losses to make computation identical to training non-packed+ + padding training regime. For video understanding data, we uniformly sample max number of frames to fit the training context length of the model if the video is too long. During training, We allocate 25% of each batch to each of the 4 downstream tasks.
For the first two stages of training (LWM-1K and LWM-8K), we additionally mix 16% of the batch to be pure text data from LaMA (Geng and Liu, 2023), as we found it beneficial to preserve language capabilities while training on vision data.
7

---

Published as a conference paper at ICLR 2025
hour YouTube compiaatio with 500 clips
person dressed riding?
GPT-4V: I'm sorry, provide assistance with request. X
Gemini Pro Vision: motorized scooter.
X
Video-LLaVA: LLaVA: The person dressed t-xx costume was riding truck
X
LWM (Ours): The dressed
matercycle.
jacket was girl on trampoline wearing"
GPT-4V: sequence provided does not include a definitive image gir on trampoline therefore, cannot describe the color of the jacket. X
Gemini Pro Vision: The girl on the trampoline was wearing green jacket.
X
Video-LLaVA: The girl the trampoline wearing black jacket.
(Ours): girl trampoline wearing a blue jacket.
Figure LWM excels answering questions about a 1-hour YouTube video. This figure compares LWM-Chat-IM with proprietary models like Gemini Pro Vision and GPT-4V, along with open-source models. The test involves answering questions based on an hour-long YouTube compilation containing over 500 video clips. LWM demonstrates superior performance in providing accurate answers requiring comprehension of extended video content.
Table 4 Long Video-MME Benchmark.
denotes models after the completion of this paper.
Method
Parameters
Frames
Medium (4min-15min)
Long (30min-60min)
Gemini 1.5 Pro*
Unknown
1800
74.3
67.4
GPT-4o*
Unknown
384
70.3
65.3
LLaVA- Video
72B
64
68.9
61.5
LaMA 2
72B
32
59.9
57.6
Long-I.LaVA
7B
64
51.4
45.4
Video-LLaVA
7B
38.1
36.2
LWM-1M
7B
S 1800
63.7
60.8
Summary: Stage II training incorporates image and video. Building on Stage it gradually increases sequence lengths of vision and text input. Importantly, we found our masked sequence packing and mixing synthetic and chat data crucial retain short context performance during progressive training. Appendix B shows ablations when not using training method on cnstuucionnfollowiii and text-image understanding benchmarks.
4.3
VISION-LANGUAGE EVALUATION RESULTS
4.3.1 LONG VIDEO UNDERSTANDING
Although vision-language model (Lin et al., 2023; OpenAI, 2023; Team et al., 2023) can ingest long videos, this commonly done by performing large temporal subsampling of video frames due to limited context length. For example, Video-LLa (Lin et al., 2023) is restricted to uniformly sampling frames from a video, no matter how long the original video may be. As such, models may lose more fine- fine-grainee grained temporal information that is important for accurately answering any questions about the video. In contrast, our model is trained on long sequences of 1M tokens, and as a result, can simultaneously attend thousands of frames of videos to retrieve fine-gra finenngained information over short time intervals. Table 4 shows long video evaluations on the Video-MME (Fu et al., 2024) benchmark, demonstrating our model as the best performing model among its size class. Figure 6 shows an example of our model correctly answering questions about long, 1-hour YouTube compilation consisting of more than 500 individual clips. Our baseline methods, on the other hand, generally have difficulty answering the questions due to a limited number of frames. More results are shown in Figure 18 and Appendix F.
8

---

Published as a conference paper at ICLR 2025
A black dog
An elephent under the sea
cube made of denim
of wine
A yellow and black cruising through a rainforest
Fireworks exploding sky
Waves crashing against the shore
Figure 7 LWM's ability to generate both static images and dynamic videos from text is shown. The top row illustrates image, while the bottom rows show video.
4.3.2 IMAGE UNDERSTANDING AND SHORT VIDEO UNDERSTANDING
We evaluate LWM on standard benchmarks for image and short video understanding, with results presented in Table 5. Our model performs comparably to baselines but falls short of state-of-the-ar (SOTA) models. This performance gap is not unexpected, given that SOTA models leverage vision backbones that have undergone extensive CLIP training (Radford al., 2021). In contrast, LWM utilizes discrete tokens from an off-the-shelf model (Patil et al., 2024). Discrete tokens result in greater information loss, particularly for OCR-like textual data, compared to continuous CLIP embeddings. Moreover, our model learns text-image alignment from scratch, while CLIP-based models benefit from large-scale pretraining. This work primarily focuses on long-context methodology, and we defer additional training to future work due to computational constraints. A straightforward approach to improving benchmark scores would be to incorporate CLIP embeddings as additional input. Despite not achieving SOTA scores on these short video benchmarks, we believe LWM provides valuable insights for future long-context language and video understanding and generation. The model's performance could be enhanced through additional training and minor modifications. We include qualitative image understanding examples in Appendix and qualitative video understanding examples in Appendix F.
4.3.3 IMAGE AND VIDEO GENERATION
Thanks to unified any-to-any architecture, our model can not only perform image/video captioning and question-ann answering but also generate images and videos from text. Figure 7 demonstrates examples of these capabilities. For autoregressive sampling, we employ classifier-fre freee guidance (Ho and Salimans, 2022) on the logits, similar to previous works (Yu et al., 2022; Gafni et al., 2022). In the unconditional branch, initialize each sequence with <bos><<<>> <vi s on? For additional image and video generation examples, please refer to Appendices H and I, respectively.
Takeaway: LWM excels in long video understanding by processing significantly more frames than previous state-of-the-ard resulting in better understanding. Morcover, its long-context enabled unified any-to-any architecture allows for versatile image and video and text understanding and generation.
Table 5
Image Understanding Benchmarks (left) and Video Understanding Benchmarks (right)
Method
Visual Token
VQAv2
GQA
SQA
Method
MSVD
MSRVTT
TGIF
MiniGPT-4
CLIP
30.8
25.4
VideoChat
56.3
45
34.4
Otter
CLIP
38.1
27.2
LLaMA-Adapte
54.9
43.8
InstructBLIP
CLIP
49.2
60.5
Video-LLaMA
51.6
29.6
LLaVA-1.5
CLIP
78.5
62.0
66.8
Video-Chat ChatGPT
64.9
49.3
51.4
IWW (ours)
VQGAN
55.8
44.8
47.7
LWM (ours)
55.9
44.1
40.9

---

Published as a conference paper at ICLR 2025
5 RELATED WORKS
Our research builds upon existing efforts to extend the context windows of language models, enabling them process more tokens (Chen al., 2023b; Tworkowski et al., 2023; Liu al., 2023c). These approaches often employ innovative extrapolation techniques to expand pretrained positional encodings, followed by model finetuning on longer context data. In contrast, our model takes straightforward approach by incrementally increasing 0 in RoPE positional encodings alongside expanding the training context window sizes, which we found to be effective. Additionally, there have been investigations into architectures that avoid modeling pairwise interactions, such as sparse attention and sliding window techniques (Child et al., 2019; Beltagy al., 2020). Prior research has explored sequence parallelization (Li et al., 2021; Korthikanti et al., 2022, inter alia), though it is not optimized for blockwise transformers or compatible with memory-effiiiet attention, both of which are critical for large context training. Our work further leverages large context transformer techniques (Liu et al., 2024; Liu and Abbeel, 2023) to capture exact pairwise interactions in extended sequences for enhanced performance. computation (Brandon et al., 2023; Li et al., 2023) offer room for further optimization. Concurrent developments like Gemini 1.5 (Reid et al., 2024) reach 1M tokens context size in language and video.
strategies, such as skipping causal masked
Additionally, our approach relates closely to advances in instruction tuning (Taori et al., 2023; Chiang al., 2023; Geng al., 2023), which focus finetuning models with conversational data boost their performance across diverse language tasks. We aim to extend these capabilities to the domain of long-sequence understanding in both video and language tasks. To achieve this, we extend the model's s context size by training on comprehensive datasets, including books and long videos, and finetune on model- generaeed question question-answering datasets to enhance its ability to handle extended conversational sequences.
Furthermore, our research draws from work on integrating vision capabilities into language mod- els (Liu et al., 2023b; Lin et al., 2023; Awadalla et al., 2023; Zhang et al., 2023; Jin et al., 2023b; Aiello et al., 2023). These efforts frequently utilize continuous embeddings (Radford et al., 2021; Li et al., 2022) to encode visual information into embeddings for inputting into language models. While these approaches benefit from CLIP's cross-modal understanding to encode textual information from images, their ability to predict text from visual input is limited, as is their capacity to learn from diverse lisual-language formats. In contrast, our autoregressive model, which processes "tokens in, tokens out," allows greater flexibility in modeling various formats, including image-text text-image, text-video, video-text, and pure formats like video, image, or text. Our method is compatible with these prior works, making it an interesting future direction combine continuous embeddings as input with discrete tokens and a long-context autoregressive model.
CONCLUSION
In conclusion, this paper tackles the critical challenge of enabling long-contex. understanding in sequence models, which is vital for developing generally intelligent systems capable of processing large temporal sequences. By exploring the development of IM context language and video-language models, the work sets new benchmarks in language retrieval and long video understanding. We outline approaches to data curation and progressive context extension, accompanied by an efficient open-source implementation for scalable training on long sequences. Moreover, open- family of 7B parameter models capable of handling over 1 M tokens in text and video.
Limitations. While this work successfully develop large large context of over 1M text and video tokens, and demonstrate promising results in processing hour-long videos and long documents, there are still some limitations that need to be addressed:
Improved tokenization and embedding. This work uses a vanilla image tokenizer for images and frame-by-frame tokenization for videos. Future work could explore video tokenization that takes time redundancy into account, as well as including continuous embeddings as input to enrich image understanding.
Limited scale. Our models use more tokens per parameter than Chinchilla's 's recommendation, but being much smaller than current large language models (100B+ parameters), our findings may not directly apply to them. Extrapolating to larger scales should be done cautiously, as different scaling behaviors could emerge at those larger sizes.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This project is supported in part by Office of Naval Research grant N00014-21-1-2769 and ARO MURI (2023) Neuro-Inspired Distributed Deep Learning. We thank Google TPU Research Cloud for granting us access to TPUs, and thank Google Cloud for granting us research credits for storage. Pieter Abbeel holds concurrent appointments as Professor UC Berkeley and an Amazon Scholar, This paper describes work performed at UC Berkeley and is not associated with Amazon.
REFERENCES
Emanuele Aiello, Lili Yu, Yixin Nie, Armen Aghajanyan, and Barlas Oguz. Jointly training large autoregressive multimodal models. arXiv preprini arXiv:2309. 15564, 2023.
ArizeAI. Needle in haystack pressure testing llms. https: btttp:://gitubbe wwt.huub..omAArieeeaaee 2023.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint 230.::2308.013000 2023.
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1728-176 1738, 2021.
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint Xiv::2000 05550,, 2020.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milhe and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL
http:://gtte com/ com///oole//////oo jax.
William Brandon, Aniruddha Nrusimha, Kevin Qian, Zachary Ankner, Tian Jin, Zhiye Song, and Jonathan Bagan-Kelleyy Striped attention: Faster ring attention for causal transformers. arXiv preprint Xr3iv:231 1.09431, 2023.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufci Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models world simulators. 2024. URL petttss:/oppent
com/research/
Hideo-genneattin seeeeratioonmodelss models - worldlrddd simulators.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Sae- hoon Kim. Coyo-700m: Image-text pair dataset. https::/gittiee cow/kakaobraii/
coyo-dataset, 2022.
Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4y: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023a.
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint M.Xiv:330611599 2023b.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality, March 2023. URL https:
//1msys. org/ htt///llg////// 2000:/22220000 -3-00-3--0 30 30-vviivvaa -vicuna/.
Rewon Child, Scott Gray, Alec Radford, and llya Sutskever. Generating long sequences with sparse transformers. arXiv preprint orXiv:/904 10509, 2019.

---

Published as a conference paper at ICLR 2025
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 95:16344-16359, 2022.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305.14233, 2023.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873-12883, 2021.
Facebook. Fully Sharded Data Parallel: faster Al training with fewer GPUs - tungineering.b.com fsdp// 2023. [Ac-
https: / eng wwe.ginnennnnnnnneee fb. com cessed 16- M0-M33.
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal ilms in video analysis. arXiv preprint TrXXi:2445521073 2024.
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make- a-scene: Scene-based text-to-image generation with human priors. European Conference on Computer Vision, pages 89-106. Springer, 2022.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace Hc, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2/01.0000 2020.
Xinyang Geng and Hao Liu. Openllama: An open reproduction of llama. URL: http:://github 2023.
Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song. Koala: A dialogue model for academic research. Blog post, April, 1, 2023.
gkamradt. Needle a haystack pressure testing Ilms, https: MiMlTestLNeeddeeelfffssttttttte /main, 2023. [Online; accessed - 20F2).
com/ gk amrrat/
David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv:1803.10122, 2018.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arX1v:2207.12598. 2022.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David Fleet, al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02300: 2022a.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models. arXiv preprint arXiv:2204.03458, 2022b.
Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Chia- Yuan Chang, and Xia Hu. Growlength: Accelerating Iims pretraining by progressively growing training length. arXiv preprint arXiv:2310.00570 2023a.
Yang Jin, Kun Xu. Liwei Chen, Chao Liao, Jianchao Tan, Bin Chen, Chenyi Lei, An Liu, Chengru Song. Xiaoqiang Lei, al. Unified language-vistie pretraining with dynamic discrete visual tokenization. arXiv preprint arXiv:2309.04660 2023b.
Vijay Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro. Reducing activation recomputation in large transformer models. 2022.
arXiv preprint arXiv:2205.05799
Jinhyuk Lee, Anthony Chen, Zhuyun Dai, Dheeru Dua, Devendra Singh Sachan, Michael Boratko, Yi Luan, Sébastien MR Arnold, Vincent Perot, Siddharth Dalmia, et al. Can long-context language models subsume retrieval, rag, sql, and more? arXiv preprint arXiv:2406. 13/2/, 2024.
12

---

Published as a conference paper at ICLR 2025
Dacheng Li, Rulin Shao, Anze Xie, Eric P Xing. Joseph E Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. Lightseq: Sequence level parallelism for distributed training of long context transformers. arXiv preprint arXfv:2310.03299 2023.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In International Conference on Machine Learning, pages 2288-129900 PMLR, 2022.
Shenggui Li, Fuzhao Xue, Yongbin Li, and Yang You. Sequence parallelism: Making 4d parallelism possible. arXiv preprint arXiv:2105.13120, 2021.
Bin Lin, Bin Zhu, Yang Ye, Munan Ning. Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122 2023.
Hao Liu and Pieter Abbeel. Blockwise parallel transformer for large context models. Advances in neural information processing systems, 2023.
Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near-
infinite context. International Conference on Learning Representations((CLRR)) 2024.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304. 200,223344008885, 2023b.
Xiaoran Liu, Hang Yan, Shuo Zhang, Chenxin An, Xipeng Qiu, and Dahua Lin. Scaling laws of rope-based extrapolation. arXiv preprint arXiv:2310.05209, 2023c.
Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Minghui Qiu, Pengeheng Lu, Tao Wang, and Zhongyu Wei. Valley: Video assistant with large language model enhanced ability, arXiv preprint arXiv22306.07207, 2023.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint 0:X1::2306.05424, 2023.
OpenAI. Gpt-4 technical report, 2023.
Suraj Patil, William Berman, Robin Rombach, and Patrick von Platen. amused: An open muse reproduction. arXiv preprint arXiv:2401.01800 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages PMLR, 2021.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schritwieserr, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu. Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308. /2900, 2023.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 15:25278-25249 2022.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Nearocomputing, 568: 127063, 2024.
13

---

Published as a conference paper at ICLR 2025
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpaca: A strong, replicable instruction-following model. Stanford Center for Research Foundation Models. https://erfm. stanford. edu/200.3/03/113aaapprr html, 36))7, 2023.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.1188) 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307 230.02207..0288,, 2023b.
Szymon Tworkowski, Konrad Staniszewski, Mikolaj Pacek, Yuhuai Wu, Henryk Michalewski, and Piotr Mitoš. Focused transformer: Contrastive training for context scaling. arXiv preprint arXiv:2307.03170, 2023.
Ruben Villegas, Mohammad Babacizadeh, Picter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual description. arXiv preprint 2rXiv:2210002399, 2022.
Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et al. Internvid: A large-scale video-text dataset for multimodal understanding and generation. arXiv preprint arXiv:2307. 00747,0692220 2023.
Zhilin Yang. Peng Qi. Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arxiv preprint 1xXiv:/809.09600 2018.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Bureu Karagol Ayan, al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint 17Xi::2206.1078, 2(3):5, 2022.
Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. alama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303.16199 2023.
Lianmin Zheng, Win Chiang. Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.0566 2023.
14

---

Published as a conference paper at ICLR 2025
A FURTHER DETAILS
Model Flops Utilization. We trained our models using TPUv4-1024, which is approximately equivalent 450 A100s, with batch size of 8M using FSDP (Facebook, 2023) and Blockwis- eRingAttention (Liu et al., 2024) for large contexts. Figure 8 shows the model FLOPS utilization (MFU) for each training stage. Blue color bars show language training and orange color bars show vision-language training. Our training achieves good MFUs even for very large context sizes.
LWM-Text Training MFU
37.5
36%
35.0
33%
30.0
29%
27.5
25.0
22.5
20.0
UWM text-32K
UWM 125K
LVM lek--2xx---KKK
Tex:-512K
LIWI text
Training Stage
LWM / LWM-Chat Training MFU
42%
MFU (%)
40
30%
35
30
28%
20
LWM-1K
LWM-Chat-32K WIM-Chat-128K
LWN-Chat-
Training Stage
Figure 8 High MFU training across sequence lengths. Model flops utilization (MFU) of each training stage for LWM-Text (top), and LWM LMM-Chat (bottom)
Training Loss Curves. Figure and Figure 10 show the training loss curves for each stage of training the language and vision-language models respectively.
Multi-Stage LWM-Text Training Loss Curve
2.2
2.0
1.8
1.6
32K
1.4
128K
1.2
256K
1.0
1M
0.8
Training Time (Days)
Figure Training progress over multiple days for LWM-Text. Train loss curve each training stage for LWM-Text models.
Training Hyperparameters. See Appendix ??
Scaling Inference. We additionally scale our inference code to support million-length sequences by implementing Ring Attention for decoding. Inference for such long sequences requires a minimum of v4-128 with a TPU mesh sharding of 32 tensor parallelism, and sequence parallelism (ring dimension). We perform inference in pure single precision, where additional improvements can be made through techniques in scalability such as quantization.
15

---

Published as a conference paper at ICLR 2025
Multi-Stage LWM LWM-Chat Training Loss Curve
5.5
5.0
4.5
4,0
3.5
1K
3.0
8K
2.0
128K
1.5
1M
1.0
Training Time (Days)
Figure 10 Training progress over multiple days for LMM. Train loss curve for each training stage for LWM and LWM-Chat models. Note that losses consist of a combination of losses different modalities, and may not be directly comparable across stages. The sharp peak in the middle of 1K training is due newly ineporating EOF and EOV tokens into the vision codebook.
Table
LWM-Text Training Stages
32K
128K
256K
512K
1M
Parameters
7B
7B
7B
7B
Sequence Length
215
217
2
219
220
RoPE
1M
10M
10M
25M
50M
Tokens per Batch
4M
4M
4M
4M
4M
Total Tokens
4.8B
12B
12B
3B
1.8B
Wall Clock
45h
83h
47h
58h
Compute (TPU)
v4-512
v4-512
v4-512
v4-512
v4-512
Doc Length
10K-100K
100K-200K
200K-500K
500K-1M
1M+
Table
LWM-Text- Training Details
128K
256K
512K
1M
Parameters
7B
7B
7B
7B
Sequence Length
217
218
2
22
RoPE 0
10M
10M
25M
50M
Tokens per Batch
4M
4M
4M
4M
Total Tokens
1.2B
1.2B
1.2B
1.2B
Wall Clock
6h
10h
20h
40h
Compute (TPU)
v4-512
v4-512
v4-512
v4-512
Table 8
LWM and LWM-Chat Training Stages
1K
8K
Chat-32K
Chat- 128K
Chat-1M
Parameters
7B
7B
7B
7B
7B
Sequence Length
210
213
215
217
220
RoPE
50M
50M
50M
50M
50M
Tokens per Batch
8M
8M
8M
8M
8M
Total Tokens
363B
107B
10B
3.5B
0.4B
Wall Clock
83h
32h
10h
6h
8h
Compute (TPU)
v4-1024
v4-1024
v4-1024
v4-1024
v4-1024
16

---

Published as a conference paper at ICLR 2025
B ABLATION STUDIES
B.1 MASKED SEQUENCE PACKING
As mentioned in Section 4.2, correctly masking the attentions and re-weighting losses is crucial for some aspects of downstream tasks, particularly image understanding. Table 9 shows comparison of our model with and without packing corrections. Naively packing shows large degradation in accuracy across image understanding tasks, We hypothesize naive packing degrades performance due to down-weighting text token answers which are shorter, which an important aspect for good image understanding benchmark performance.
Table 9
Ablation study comparing standard independent packing and our masked sequence packing mechanisms across three tasks. Results show that masked sequence packing significantly improves performance across all tasks.
VQAv2
SQA
POPE
Standard independent packing
48.3
34.8
62.5
Masked sequence packing (Ours)
55.8
47.7
75.2
B.2 MIXING SYNTHETIC AND CHAT DATA
We additionally evaluate the our model on MTT-Bench (Zheng et al., 2023) to test its conversation ability. Table 10 shows the MT-Bench scores of for each of models. Table 11 illustrates the relationship between the mix of chat and fact retrieval tasks and the performance on MT-Bench score and Needle Retrieval accuracy. As the proportion of chat increases and fact retrieval decreases, the MT-Bench score improves, indicating better chat performance measured by MTT-Benh.. Conversely, Needle Retrieval accuracy decreases, suggesting trade-off where increasing chat interaction capa- bilities may reduce the system's 's precision in retrieving specific information or 'needles' from input context. Across different context sizes, we found that the model supporting longer input sequences encounters slight decrease in MT--Bench score. We hypothesize that this is because we chose to train with fewer examples on longer sequence training and can be improved by simply training on more data. In addition, this trade-off may be resolved by acquiring higher quality long-context chat data that is closer to the chat distribution of the UltraChat dataset.
Table 10
Results on MT-Bench across different Table 11 Relationship between the mix of chat context sizes. Despite less training on longer se- quence lengths, they show only slight decrease in conversational ability.
and fact retrieval tasks and the performance on MT-Bench score and Needle Retrieval accuracy.
Chat QA Mix
MTT-Bench
Needle Ace
Model
MT-Bench
0% 100%
2.42
100%
LWM-Text-Chat-I
4.62
40% / 60%
4.14
100%
LWM-Text Text-Chat
5
70% / 30%
4.62
96%
LWM-Text- T
4.83
90% / 10%
5.1
55%
LWM- Text- Chat IM
4.19
100% 0%
5.8
31%
17

---

Published as a conference paper at ICLR 2025
C MORE SINGLE-NEEDLE RETRIEVAL RESULTS
Needle Retrieval (1M Context Model)
0%
1.0
11%
21%
16%
0.8
Depth Percent
325
37%
42%
0.6
47%
58%
51
6.3%
60%
79%
84%
89%
IK
S4K
106K
159k
211K
264k
316k
369
422K
474k
527k
579k
632k
685K
737k
790K
842K
895k
947k
1t
Context Length
Figure 11
Needle retrieval task using the WWM-Text-Chat- IM model. The model demonstrates
near-perfect retrieval accuracy across various positions within the 1M context window, reflected by consistently high scores at different depth percentages and context lengths.
Needle Retrieval (256K Context Model)
1.0
0%
11%
0.8
Depth Percent
22%
0.6
44%
0.4
67%
89%
28k
S4K
B1k
107K
134K
160k
187K
213K
240k
Context Length
Figure Single needle retrieval accuracy for the L.WM-Txx--hat--eeee achieves near- cear-perfect retrieval performance across various positions in the 256K context window, as shown by consistently high scores across all depth percentages and context lengths.
model. The model
Needle Retrieval (512K Context Model)
1.0
0%
11%
0.8
Depth Percent
33%
0.6
56%
0.4
78%
0.2
100%
K
58k
114K
171k
227K
284K
340k
397k
453k
510k
0.0
Context Length
Figure 13 Single needle retrieval accuracy for the LWM-Text Text-Chhtt-512K model. The model
demonstrates near-perfect retrieval performance across different positions within the 512K context window, indicated by consistently high scores at varying depth percentages and context lengths.
18

---

Published as a conference paper at ICLR 2025
D DETAILS OF THE MULTI-NEEDLE EVALUATIONS
We built on top of the original Needle in Haystack problem (see Section 3.3.2 for details), which was retrieve single magic number randomly inserted into some long context. In this case, magic numbers are associated with random cities ("The magic number for San Francisco is 2521233" - "What is the magic number for San Francisco?").. We extend the task by introducing two variables N and R, where N is the number of needles (magic numbers + different cities) randomly inserted into context, and numbers for San Francisco, .... and Bangkok?"", Correctness is computed by extracting out the numbers retrieved for each citics and checked with string matching.
the random subset magic numbers asked to retrieve ("What are magic
E MORE IMAGE UNDERSTANDING EXAMPLES
User: If you were a photographer looking to capture this location's essence, what time of day and weather conditions would you choose? Describe the reasons behind your choice.
User: Describe how this creative twist on classic work of art might be interpreted differently by various audiences.
Assistant: would choose the time of day to be during sunset. The image features wooden dock extending into calm lake, which perfect setting picturesque scene. The sunset casts a warm, golden light on the lake, enhancing the tranquility of the scene. Additionally, the weather conditions appear to be cool and cloudy, which complements the natural beauty the lake and the wooden dock. The combination these elements creates a visually appealing and calming atmosphere, which would make it an ideal time to capture the moment.
Assistant: In this creative twist on a classic work of art, the woman's face is transformed into dog's face, creating a visually striking and unexpected representation. The artwork could be seen as a commentary on the limitations of human understanding. well as the potential deeper connection and understanding between humans and animals.
serene and
Question answering based on image input using LWM. The assistant provides detailed responses to questions about capturing the essence serene lakeside scene and interpreting a creative twist classic artwork featuring dog.
Figure 14
19

---

Published as a conference paper at ICLR 2025
F
MORE VIDEO UNDERSTANDING EXAMPLES
Fomer
00:00:00
00:59:59
User: How many lemons were in the person's car?
GPT-4V: Sorry, can't help with identifying or making assumptions about the content in these images. X
Gemini Pro Vision: am not able to count the number of lemons in the person's car because cannot see any lemons in the video. X
Video-LLaVA: The video does not provide exact number of lemons the persons' car. X
LWM (Ours): There are three lemons in the person's car.
Figure LWM demonstrates video understanding over 1-hour video. The figure shows comparison of different AI systems responding to question about the number of lemons in person's 's car. While GPT-4V, Gemini Pro Vision, and Video-LLAVA fail to provide a correct response, LWM accurately identifies that there are three lemons in the car.
00:00:00
01:02:50
User: What animal was standing on a piano?
GPT-4V: I'm sorry, cannot provide assistance with these requests. X
Gemini Pro Vision: There is no piano in the video. X
Video-LLaVA: A dog was standing on a piano in the video. X
LWM (Ours): The animal standing on a piano was cat.
LWM demonstrates video understanding over --hou video. The figure compares AI systems' responses to the question 'What animal was standing on a piano?' While GPT-4V, Gemini Pro Vision, and Video-LLAVA provide incorrect or incomplete answers, LWM correctly identifies that the animal standing on the piano was a cat.
Figure 16
20

---

Published as a conference paper at ICLR 2025
03:46
User: What the video about?
Assistant: The video is about man who talks to the camera and shows with apples then proceeds pick apples and puts them into bow.
(a) The assistant answers questions about 4- 4-minute video man talks to camera shows tree of apples.
00:23
01:52
03:22
04:51
06:20
07:50
User: Who the person the video likely be?
Assistant:
The person in the video is likely to be politician or representative White House.
User: Where the video happening?
Assistant: The video is taking place in the Oval Office of the White House, where President Joe Biden is speaking the press
(b) The assistant answers a question regarding a 8-minute video president speaking to the press in the Oval Office.
00:00
00:06
00:15
00:20
00:27
00:34
User: Provide comprehensive description the video
running towards high dive in pool. As the giraffe reaches the dive, it jumps into the water, causing solace. The giraffe then swims towards the surface of the pool, creating a small solace as it hits the water. The giraffe's movements are graceful and elegant, showcasing its agility and grace.
Assistant: the video, giraffe
User: What is unusual about this video?
Assistant: The video unusual because it shows giraffe diving into a pool.
(c) The assistant answers question about 30-second video of giraffe..
Answering questions about videos using LWM. The assistant responds various user questions regarding different types of videos, ranging from a video about man picking apples to press briefing in the White House, and even humorous video of giraffe diving into pool.
Figure 17
21

---

Published as a conference paper at ICLR 2025
User: What the video about?
Assistant: The video about a man who changing a tire on a car. He demonstrates how change tire by using jack spare tire
(a) The assistant answers question about 5-minute video a man changing a car tire. The process involves using jack and a spare tire.
00:06
User: What is the video about?
Assistant: the video, person is riding dirt bike dirt trail. The trail is surrounded by trees and there are other people riding bikes in the background. The person is wearing helmet riding at high speed
User: What's the color the person's gloves?
Assistant: The person is wearing red gloves
(b) The assistant provides answers based 2-minute video of person riding dirt bike along forest trail. The rider helmet and red gloves, traveling high speed.
Figure 18 The system (LWM) successfully answers questions about video content.
G DETAILS OF QUALITATIVE VIDEO UNDERSTANDING EVALUATION
For qualitative evaluation of our videos, source various videos from YouTube that cover range of topics, such as ego-centric camera, how to videos, interviews, and animations. We evaluate all videos at IFPS, and sample uniformly our video can support at FPS. Videos are additionally resized and center cropped to 256 x 256 resolution before inputting into the model.
number of frames for videos that are longer than what
22

---

Published as a conference paper at ICLR 2025
H
MORE IMAGE GENERATION EXAMPLES
black dog
A blue colored pizza
cube made denim
glass wine
yellow and black bus cruising through rainforest
Oil painting of couple in formal attire caught rain without umbrelias
A couch in cozy living room
A carrol to the left of broccoli
Fisheye lens of turtle forest
blue colored dog
Stained glass windows depicting hamburgers and french fries
A pink
1999
cube made of brick
An elephant under the sea
A yellow book and red vase
A city skyline night
Figure 19
Images generation using LWM, showcasing various scenes and objects.
23

---

Published as a conference paper at ICLR 2025
MORE VIDEO GENERATION EXAMPLES 1
A bustling street in London with red telephones booths and Big Ben in the background
Fireworks exploding sky
Camera pans left to right mango slices sitting table
Slow motion flower petals falling on the ground
A boat sailing on a stormy ocean
A burning campfire in forest
Waves crashing against the shore
ball thrown in the air
Figure 20
Video sequences generated using LWM, showing various scenes.
24

---

Published as a conference paper at ICLR 2025
J TRAINING HYPERPARAMETERS
Table 12
LWM-Text Training Stages
32K
128K
256K
512K
1M
Parameters
7B
7B
7B
7B
7B
Initialize From
LLaMA-2 7B
Text-32K
Text- 128K
Text-256K
Text-512K
Precision
float32
float32
float32
float32
float32
Sequence Length
2
2
2
219
220
RoPE 0
1M
10M
10M
25M
50M
Tokens per Batch
4M
4M
4M
4M
4M
Total Tokens
4.8B
12B
12B
3B
1.8B
Total Steps
1200
3000
3000
720
450
LR Schedule
Constant
Constant
Constant
Constant
Constant
LR Warmup Steps
100
200
200
50
25
LR
x 10-5
4 x 10-5
4 x 10-5
4 x 10
x 10
Compute (TPU)
v4-512
v4-512 1,-1,8,1
v4-512
v4-512 1,-1,16,2
v4-512
Mesh Sharding
1,-1,4,1
1,-1,16,1
1,-1,16,4
Table 13
LWM- Chat Training Details
128K
256K
512K
1M
Parameters
7B
7B
7B
7B
Initialize From
Text- 128K
Text-256K
Text-512K
Text- 1M
Precision
float32
float32
float32
float32
Sequence Length
2
2
2
220
RoPE 0
10M
10M
25M
50M
Tokens per Batch
4M
4M
4M
4M
Total Tokens
1.2B
1.2B
1.2B
1.2B
Total Steps
300
300
300
300
LR Schedule
Constant
Constant
Constant
Constant
LR Warmup Steps
25
25
25
25
LR
4 x 10-5
10-5
10
10-3
Compute (TPU)
v4-512
v4-512
v4-512
v4-512
Mesh Sharding
1,-1,4,1
1,-1,8,1
1,-1,16,1
1,-1,16,2
Table 14 LWM L.WM/LWM-CHE Training Stages
1K
8K
32K
128K
1M
Parameters
7B
7B
7B
7B
7B
Initialize From
Text- 1M
1K
8K
32K
128K
Precision
float32
float32
float32
float32
float32
Sequence Length
210
213
2
2
220
RoPE
50M
50M
50M
50M
50M
Tokens per Batch
8M
8M
8M
8M
8M
Total Tokens
363B
107B
10B
3.5B
0.4B
Total Steps
45000
14000
1200
450
50
LR Schedule
Cosine
Cosine
Cosine
Cosine
Cosine
LR Warmup Steps
1000
500
100
50
5
Max LR
10-4
6 10
8 x 10-5
8 x 10-5
8 x 10-5
Min LR
6 10-5
6 10-5
8 10-5
8 10-5
8 10-5
Compute (TPU)
v4-1024
v4-1024
v4-1024
v4-1024
v4-1024
Mesh Sharding
1,-1,1,1
1,-1,1,1
1.-1.4.1
1.-1.8.1
1,-1,16,4
25


Reviews:
[{"review_id": "DcRdt5IzmK", "reviewer": "Reviewer_gZJr", "summary": "This paper proposes a novel model architecture and training process that achieves significant advancements in long-context modeling for developing large language and vision-language models capable of processing sequences up to 1 million tokens in length. The authors propose a two-stage training process - in the first stage, they progressively train a base language model on increasingly longer text sequences, while in the second stage, the models incorporate vision capabilities through joint training on image and video data. The consequent Large World Model(LWM) family demonstrates impressive performance on a few challenging tasks such as long context retrieval and long video understanding.", "questions": "1. When training the models of LWM-1K and LWM-8K, there was an 16% mix of the pure text data added from OpenLLaMA in the batch, how was it determined and how sensitive is the model's performance to this parameter?\n2. How did you achieve loss balancing across different tasks and contexts? Any specific strategy?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The paper addresses an important technical challenge in building foundational AI models, which is the long sequence understanding.\n2. This work achieves multiple state-of-the-art results across different training stages\n3. The open-sourced implementation and pre-trained models benefit and accelerate progress in the research community.", "weaknesses": "The model's suboptimal performance on image and short video understanding tasks and the choice of visual encoding constrain the general applicability of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nP1y8bIR4s", "reviewer": "Reviewer_QidU", "summary": "This paper introduces the Large World Model (LWM), a LLaMA-based model capable of processing sequences of up to 1 million tokens, combining language and video. The paper details the training datasets, progressive training stages, and evaluation results. It incorporates techniques such as RingAttention, Blockwise Transformers, and others for efficient training on million-length multimodal sequences.", "questions": "The core issue is the unclear relationship between the concept of a large world model and the multi-modal LLM with long context window presented in the paper. Additional suggestions include:\n1. Provide quantitative evaluations for image and video generation, as mentioned in Section 4.3.3.\n2. Explain the larger fluctuations in the purple curve compared to other curves in Fig. 9.\nTo improve the paper, the authors should:\n● Clearly define and discuss the concept of a \"large world model\" and its relationship to the proposed multi-modal LLM.\n● Conduct and present ablation studies to demonstrate the impact of individual components.\n● Provide more in-depth analysis of the interaction between text and video modalities during training.\n● Include comprehensive comparisons with other recent multimodal LLM works.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper provides extensive training details on scaling token sequences to 1M tokens, pushing the boundaries of context length for multimodal models.\n2. Empirical results demonstrate strong performance on challenging tasks like long video understanding and retrieval across 1M token contexts.\n3. The paper is well-structured and clearly written, with detailed explanations of the model architecture, training process, and evaluation results.", "weaknesses": "1. The title \"Large World Model\" is misleading. The paper primarily explores training multi-modal (text and video) large language models with long context windows, without adequately defining or discussing the concept of a \"large world model\" or its relationship to MLLMs.\n2. The paper fails to address fundamental questions about the necessity of long-context multi-modal LLMs for large world models and whether such models are sufficient for this purpose.\n3. The paper largely describes training datasets and stages, lacking in-depth ablation studies and analysis. It relies heavily on previously proposed components like blockwise RingAttention, making it more akin to an experimental record than a research paper.\n4. There is insufficient analysis of the potential synergies or conflicts between text and video in joint training. Do text and video in joint training mutually benefit each other, or do they have a negative influence on one another?\n5. The paper lacks comprehensive comparisons with recent multimodal LLM works.\n6. Despite claiming contributions in areas like masked sequence packing and loss weighting, the paper lacks ablation studies on those components", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lt5hzYxPR5", "reviewer": "Reviewer_zoBJ", "summary": "This paper introduces a training process that gradually extends LLMs from a limited context length of 4K tokens to an extremely long context of millions of tokens. The two-stage training process — where the first stage focuses on long-context expansion and the second stage addresses image and video understanding and generation — enables the models to support multimodal understanding and generation. The empirically feasible training process, which integrates understanding and generation capabilities into one extensive model using VQ, is quite useful. The evaluations, which involves both linguistic and multimodal tasks, are comprehensive.", "questions": "- For the retrieval task in section 3.3.3, what is the exact evaluation benchmark used for the Multi Needle in a Haystack?\n- What are the trainable parameters for each step in the stage 2?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "- The empirically feasible training process that integrates understanding and generation capabilities into one extensive model using VQ is quite useful, and the evaluations encompassing both linguistic and multimodal tasks are comprehensive.\n- The settings and hyperparameters for training and evaluation are presented in a meticulous manner.", "weaknesses": "- One major contribution of this paper is extending the context length of existing LLMs to extremely long through progressive training, which is a valuable practical implementation. However, aside from the fundamental support of RingAttention, the technical improvements in this study are marginal.\n- Building models that support both multimodal understanding and generation is interesting and meaningful. However, the simple incorporation of an off-the-shelf VQ model into the framework does not yield promising results. I believe that more in-depth analysis and improvements are needed in this direction.\n- There is a lack of quantitative evaluation for multimodal lengthy content understanding. For example, can we evaluate understanding of image details by increasing resolutions with lengthy vision tokens? Additionally, there are long video understanding benchmarks that could be used to assess the proposed LWM-1M.\n- Some crucial details in the main content of the paper, such as the evaluation data and the trainable parameters in each stage, are missing.\n- Compared to existing and up-to-date VLMs (e.g., LLaVA-NeXT, VILA, Qwen-VL, CogVLM), the performance of LWM on image understanding and video understanding benchmarks is relatively low.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wdwnTjF20g", "reviewer": "Reviewer_cWVz", "summary": "The paper scales the LLM context length up to 1M tokens by utilizing Blockwise RingAttention. They progressively increase the context length from 32K to 1M using book-length text data and video-language data. It uses discrete token representation of VQGAN to facilitate image/video generation tasks. It demonstrates competitive performance in long-context retrieval tasks. It also reports comprehensive experimental results in both image/video understanding and generation tasks.", "questions": "1. The three benchmarks used to report video performance is not long. It is better to show accuracy on long-video benchmarks such as VideoMME[1] and MLVU[2].\n2. In Table 3, how about comparing with existing open-sourced LLM supporting context length of 128K?\n3. The paper is the first to scale context length up to 1M. However, there is no baseline comparison in 1M context length. How about using linear scaling technique during inference stage to construct a naive baseline? E.g., scale 8x from a 128k model.\n\n[1] Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis. arXiv preprint arXiv:2405.21075.\n[2] MLVU: A Comprehensive Benchmark for Multi-Task Long Video Understanding. arXiv preprint arXiv:2406.04264.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "+ This paper is the first to scale sequence models up to 1 million tokens, pushing the boundaries of long-context processing in both language and multimodal (text-video) tasks.\n+ The model use VQGAN tokenization to enable image/video understanding as well as generation.\n+ In multi-needle retrieval tasks with a 128K context, the model achieves results comparable to or better than GPT-4, demonstrating its effectiveness in long-context retrieval tasks.", "weaknesses": "+ Lack of hour-long video benchmark evaluation, please refer to questions.\n+ No comparison with existing open-source LLMs in context length of 128K, for example Llama 3.1\n+ The usage of VQGAN decrease the image understanding ability as Table 4 shows.\n+ There is no quantitative result on image generation tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)