Analyse the following proposal drafting, and based on your understanding, answer the following questions to apply for NVIDIA's funds. Generally you should make the project sound interesting and impactful, with visions about what is the outcome of the project, and what the outcome will lead to. Describe the method as it is reasonable. Brief and accurate response are expected. If you need more information beyond the above ones to answer the question, please let me know.

The proposal draft:
***Empower Multimodal Language Models with Long Video Understanding***   

Long video understanding is a practically important topic. But in many long video understanding benchmarks, the results of most modern multimodal language models are close to random guess.

Their performance are mainly limited to the length of context window. As one frame takes nearly 200 tokens and most models only accept less than 40 frames in one like LLaVA-NeXT-Video, which is far way less in a 30 minutes’ video. To be more exact, in most cases, the model didn’t even see the scene that was queried.

The length of context window can be extended to longer without training via changing frequency base of RoPE or with position interpolation. However these methods will cause loss when accessing to longer position that are untrained. KV Cache compression methods and Q-former-like resamplers reduce the length of all the video tokens, but involve inductive bias and are hard to express all the information in a lossless way. Recently some work extend the length of multimodal language models by continuously training on long text corpus, which helps in training models with larger position embedding. But unlike the success achieved in pure language modality, the performance didn’t show much benefits in long-dependency video understanding on the increase of the number of accessed frames, which arouses the importance of modality alignment and splendid visual information that are specific in visual modality.

We plan to train multimodal language models continuously on long videos to extend the context length and improve its visual understanding ability. It is also an exploration on whether we can extend the context length purely by long videos and generated descriptions for multimodal language models in an acceptable cost. 

For the training process and related computational estimate, we keep the transformer architecture to achieve the best performance. We plan to do continual training on the open sourced LLaVA model, which is constructed by a visual encoder, a modal projector and a language model, and train all of the three modules to make the full usage of them to contribute to the final performance. We plan to make the frame per second rate becomes close to 1, as in traditional short video understanding evaluation. 

Currently, we are done with the training infrastructure to train a 7b model on 72k context length on 8x80G H800, using the strategy of sequence parallel, zig-zag ring-attention and parameter offloading. The time cost for 70M tokens is about 40 hours. Considering the experience from pure language modeling context extension, 500M tokens are enough for competitive long-distance dependency performance. We aim to extend the length even longer with designed offloading and reduce the disk I/O cost to make the training process even more efficient. 

As for training data,  Long videos are easy to access and collect more on the internet. We build a pipeline of collecting and filtering high quality long videos and generating description of the video according to its subtitles, makes it convenient and efficient to scale up the training corpus.

we manually select 6 dominant video categories, and ask generative model to propose a few subcategories for each of them, and propose multiply requesting query for each kind of subcategory. We manually examined every query to see if it can return diverse and popular videos from youtube official api and keep the qualified ones. Our ideal videos are supposed to be rich in visual motions and the subtitles are expected to describe the motion in the frame. After the human-integrated checking, we set some rules to automatically filter them based on meta data: the language should be English and the count of view and likes should pass our requirements. We requested about 1000 videos with duration of 20 to 30 minutes from each category passing the rules together with their subtitles. Then we filtered the video by the length of subtitles and the motion variance of the frames based on features represented by visual encoder. After that, we prompted gpt4o-mini to generate detailed and documentary-style description based on the subtitles.

The questions for applying the funding:

1. Project Objective(s): A brief overview of the project. Clearly state the impact for NVIDIA and the impact for the partner.
2. Project Outcomes(s): Describe the outcomes (code, talks, publications) we expect to see as a result of supporting this project.
3. Project Approach: Describe how the project will be run, clearly identifying what will be done by the partner and what will be done by NVIDIA. As best you can, quantify the effort and indicate which teams or people at NVIDIA have agreed to that effort.
4. Project Timelines: Expected start date, milestones and durations.
5. Project Challenges: What risks or challenges does the project face? Are they in or out of NVIDIA’s control? How will they impact our commitment to the project? Are there reputational risks to NVIDIA – using early access HW or SW, research with obvious uses that are not aligned with our corporate brand?
6. Project Resources: What HW (on-prem, edge, cloud; GPU version) and SW (include all major libraries or platforms) will be needed? What of that is already present at the partner and what is expected to be procured?
7. Additional Information: What other information do we need to consider in approving or rejecting this project proposal?