Analyse and comment on the following proposal:

***Empower Multimodal Language Models with Long Video Understanding***   

Long video understanding is a practically important topic. But in many long video understanding benchmarks, the results of most modern multimodal language models are close to random guess.

Their performance are mainly limited to the length of context window. As one frame takes nearly 200 tokens and most models only accept less than 32 frames in one like LLaVA-NeXT-Video, which is far way less in a 30 minutes’ video. To be more exact, in most cases, the model didn’t even see the scene that was queried.

The length of context window can be extended to longer without training via changing frequency base of RoPE or with position interpolation. However these methods will cause loss when accessing to longer position that are untrained. KV Cache compression methods and Q-former-like resamplers reduce the length of all the video tokens, but involve inductive bias and are hard to express all the information in a lossless way. Recently some work extend the length of multimodal language models by continuously training on long text corpus, which helps in training models with larger position embedding. But unlike the success achieved in pure language modality, the performance didn’t show much benefits in long-dependency video understanding on the increase of the number of accessed frames, which arouses the importance of modality alignment and splendid visual information that are specific in visual modality.

We plan to train multimodal language models continuously on long videos to extend the context length and improve its visual understanding ability. It is also an exploration on whether we can extend the context length purely by long videos and generated descriptions for multimodal language models in an acceptable cost. 

For the training process and related computational estimate, we keep the transformer architecture to achieve the best performance. We plan to do continual training on the open sourced LLaVA model, which is constructed by a visual encoder, a modal projector and a language model, and train all of the three modules to make the full usage of them to contribute to the final performance. We plan to make the frame per second rate becomes close to 1, as in traditional short video understanding evaluation. 

Currently, we are done with the training infrastructure to train a 7b model on 72k context length on 8x80G H800, using the strategy of sequence parallel, zig-zag ring-attention and parameter offloading. The time cost for 70M tokens is about 40 hours. Considering the experience from pure language modeling context extension, 500M tokens are enough for competitive long-distance dependency performance. We aim to extend the length even longer with designed offloading and reduce the disk I/O cost to make the training process even more efficient. 

As for training data,  Long videos are easy to access and collect more on the internet. We build a pipeline of collecting and filtering high quality long videos and generating description of the video according to its subtitles, makes it convenient and efficient to scale up the training corpus.

we manually select 6 dominant video categories, and ask generative model to propose a few subcategories for each of them, and propose multiply requesting query for each kind of subcategory. We manually examined every query to see if it can return diverse and popular videos from youtube official api and keep the qualified ones. Our ideal videos are supposed to be rich in visual motions and the subtitles are expected to describe the motion in the frame. After the human-integrated checking, we set some rules to automatically filter them based on meta data: the language should be English and the count of view and likes should pass our requirements. We requested about 1000 videos with duration of 20 to 30 minutes from each category passing the rules together with their subtitles. Then we filtered the video by the length of subtitles and the motion variance of the frames based on features represented by visual encoder. After that, we prompted gpt4o-mini to generate detailed and documentary-style description based on the subtitles.