Artemis:  Towards Referential Understanding in Complex Videos

Authors: jihao  qiu
yuan  zhang
xi  tang
lingxi  xie
tianren  ma
pengyu  yan
david  doermann
qixiang  ye
yunjie  tian
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-09 10:48:49

Abstract: Videos carry rich visual information including object description, action, interaction, etc., but the existing multimodal large language models (MLLMs) fell short in referential understanding scenarios such as video-based referring. In this paper, we present Artemis, an MLLM that pushes video-based referential understanding to a finer level. Given a video, Artemis receives a natural-language question with a bounding box in any video frame and describes the referred target in the entire video. The key to achieving this goal lies in extracting compact, target-specific video features, where we set a solid baseline by tracking and selecting spatiotemporal features from the video. We train Artemis on the newly established ViderRef45K dataset with 45K video-QA pairs and design a computationally efficient, three-stage training procedure. Results are promising both quantitatively and qualitatively. Additionally, we show that Artemis can be integrated with video grounding and text summarization tools to understand more complex scenarios. Code and data are available at https://github.com/NeurIPS24Artemis/Artemis.

Main paper: 
 introduction: 
 The past year has witnessed rapid progress of multimodal large language models (MLLMs) [33, 68, 40] , offering abundant abilities of open-world image understanding with language-based dialogues. In comparison, there are fewer studies on training MLLMs for video understanding, albeit videos are much more informative than still images. Existing video-based MLLMs [29, 61, 37, 31, 38] mostly focus on superficial dialogues in which the video is encoded holistically, inevitably lacking the ability to understand fine-level video contents, e.g., describing a user-specific target in the video.We are considering a new task called video-based referential understanding to compensate for the limitation. Specifically, we are interested in complex videos that span 20-30 seconds and the target performs multiple actions during this period. Given a video, the MLLM tries to answer a question like 'What is the target <region> doing in this video?' where <region> refers to a bounding box in any video frame. We argue that the task is not only challenging as it requires feature extraction, tracking, summarization, etc., but also important because it lays the foundation of finer-level video understanding. However, as shown in Figure 1 , existing MLLMs often fell short in What is the behavior of the <region> in this video?The man is carrying a suitcase and walking towards a car.The woman in red clothes walks to the piano and sits down.The girl is playing the piano.The girl is playing the flute in the video.A young girl, wearing a white dress and a red bow in her hair, displays her musical talent by playing a piece on the piano, captivating the audience with her performance.Why is this video strange?The video is strange because it shows a giraffe jumping into a swimming pool, which is not a common sight.What place is shown in the video?The video is taken from a helicopter flying over the city.The video shows the Statue of Liberty, which is a famous landmark in New York City.Can you describe the video? Sure, the video shows a girl performing a gymnastics routine on a balance beam. She starts by jumping onto the beam and then proceeds to perform a series of flips and tricks. The girl's movements are fluid and graceful, showcasing her impressive skills. As she finishes her routine, she jumps off the beam and lands gracefully on the mat. The girl's performance is a testament to the hard work and dedication required to master the art of gymnastics. Overall, the video is a captivating display of athleticism and skill that is sure to impress anyone watching.The man in suit is inspecting the car and opening the trunk.A man is seen throughout the video in different positions with a white car, possibly either giving directions, admiring or preparing to board it, and loading or unloading luggage. In the final scene, a police officer is seen standing next to the car, seemingly writing a ticket.The man in the white suit walks to the car, opens the door, and gets in.Where is the video taken from? How does the <region> move in this video? Artemis Figure 1 : Artemis' ability in video-based dialogue. Notably, Artemis excels particularly in videobased referring, outperforming the existing MLLMs including Merlin [57] and Video-LLaVA [31] lacking comprehensiveness and Osprey [59] suffering hallucination. this seemingly easy task because they were mostly trained for image-based referential understanding; as a result, they can only perceive the action in a single moment rather than that in an entire video 2 . This paper presents Artemis3 as a solid baseline for the above task. Artemis follows the generic design of modern MLLMs (i.e., visual instruction tuning [33] ), but encounters a challenge in finding sparse, target-related information from dense video data. A preliminary study shows that feeding raw video features into the MLLM results in computational inefficiency and training instability. To extract target-specific video features, we propose a simple yet effective solution that involves (i) tracking the target over time and (ii) selecting informative features from a long list of regions-of-interest (RoIs). The compactness of features makes it easier to train the MLLM. We design a three-stage training schedule where the MLLM gradually learns video-text alignment from coarse to fine. This efficient design requires only 28 hours (3 hours for the final stage) on 8× NVIDIA-A800 GPUs.To train and evaluate Artemis, we organize 7 existing video understanding datasets into the Video-Ref45K benchmark comprising 45K video question-answer pairs. To our knowledge, this is the first Video-based MLLMs. Compared to the large corpus of image-based MLLMs, there are fewer video-based MLLMs for at least two reasons. First, there are fewer paired video-text data, especially for instance-level video understanding. Second, the higher dimensionality of video data poses a greater challenge to efficiently encode videos into visual features and find useful features to answer the questions. Existing efforts include VideoChat [29] , Video-ChatGPT [37] , Video-LLaMA [61] , Video-LLaVA [31] , LanguageBind [67] , Valley [36] , etc.; most of them followed the paradigm of image-based MLLMs and some of them [37] proposed a more efficient video feature. Recently, there have been some preliminary studies for instance-level video understanding, e.g., LEGO [30] studied moment retrieval with the assistance of LLMs, and PG-Video-LLaVA [38] performed video grounding by employing off-the-shelf tracking and grounding modules. Merlin [57] studied videobased referring, but it was built upon three manually specified frames as visual input, incurring extra burden for users and also limiting the model's ability to understand long and complex videos. This paper aims to address the above two challenges, for which we set up a new formulation, establish a new benchmark named VideoRef45K, and present a solid baseline named Artemis.3 Artemis: A Baseline for Video-based Referential UnderstandingQuestion: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: The abstract and introduction provide a comprehensive overview of the background and motivation of this study, effectively outlining its main contributions, thus accurately reflecting the paper's scope and significance. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
limitations: 
 Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We primarily focused on discussing the limitations associated with this study in Sec. 5. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
problem formulation and data preparation: 
 A video can be represented in the raw form of V ∈ R T ×W ×H×C , where T , W , H, and C stand for the number of frames, width, height, and the number of channels, respectively. In the task of video-based referential understanding (a.k.a. video-based referring), the model receives a question in the form of 'What is the <region> doing in this video?', where the concrete class of the referred object (like man or dog) is not provided, and the <region> is supplemented by a bounding box B = (t; x 1 , y 1 , x 2 , y 2 ) in a frame t ∈ {1, 2, . . . , T }. The expected output is a sentence describing the target's action in the full video as detailed as possible (see Figure 1 for examples). Note that the proposed task requires a stronger ability beyond image-based referring and video captioning, The man in black clothes turns and walks to the door. mainly in the coverage and granularity of visual understanding. Specifically, the model is expected to produce complex action descriptions for different target <region> specified.We collect video data for referential understanding from 7 datasets, including HC-STVG [44] , VID-Sentence [10] , A2D Sentences [20] , LaSOT [18] , MeViS [16] , GOT10K [24] , and MGIT [23] . In total, there are 45K video QA pairs. We perform dataset-specific operations, including re-tracking (for HC-STVG and A2D-Sentences), clip cropping (for LaSOT and MGIT), and caption summary (for GOT10K), to convert them into the required form. Please refer to Appendix A for further details.The overall framework of Artemis, as illustrated in Figure 2 , follows the pipeline of visual instruction tuning [33, 68] where a multimodal large language model (MLLM) receives video features with a text prompt and produces the desired output. We denote the function as T out = f (F V , T in ), where T in and T out are input and output texts (in tokens) and F V is the set of features extracted from V.Compared to image-based referring, a clear difficulty of video-based referring arises from the high dimensionality of video data. Specifically, if we define F V as the set of dense video features (e.g., using a pre-trained visual encoder such as the CLIP ViT-L model [41] to extract frame-wise visual features for V), the features often contain highly redundant information due to the similarity of neighboring frames. This brings two-fold drawbacks: (i) extra complexity for the MLLM to deal with these vision tokens, and (ii) extra difficulty for the MLLM to locate useful information, which leads to a slower convergence. To overcome this issue, we decrease the input feature dimensionality by using various slices to replace F V , where each slice captures important yet complementary properties of the input video. Throughout this paper, we investigate three slices: the spatial, temporal, and target-specific video features.Spatial and temporal features. The extraction of spatial and temporal video features follows the design of Video-ChatGPT [37] . Given a video clip V ∈ R T ×W ×H×C , we use the CLIP ViT-L/14 visual encoder to cast it into frame-wise features, denoted as F frame ∈ R T ×W ′ ×H ′ ×D , where the number of frames remains unchanged, W ′ = W/s and H ′ = H/s are the down-sampled resolution (e.g., s = 14 for ViT-L/14) of the visual features, and D is the feature dimensionality.) Then, these features are fed into average pooling along the T axis (into the spatial features F S V ∈ R W ′ ×H ′ ×D ) and along the W ′ × H ′ plane (into the temporal features F T V ∈ R T ×D ), respectively.Target-specific features. F S V and F T V have focused on the spatial and temporal features but ignored the referred target which may move or change during the video. To offer a compromise feature that captures spatiotemporal features, we propose an RoI (region-of-interest) tracking and selection mechanism (detailed in Section 3.3) and obtain a list of RoIs (represented as bounding boxes) B = (B 1 , . . . , B M ), where M is the number of RoIs that are recognized by the algorithm to be important for referential understanding. We use the RoIAlign method [21] to extract visual features from each RoI, producing a set of target-specific features, F R V = (F R V,B1 , . . . , F R V,B M ). Instruction fine-tuning. When the video features are ready, we feed them with the text tokens into Artemis. The MLLM follows instruction fine-tuning through three steps, gradually acquiring the ability of video-based referring. The details are described in Section 3.4.Our goal is to extract compact features for video-based referring. The key lies in two factors, (i) completeness -locating the referred target in every video frame, and (ii) avoiding redundancy -not preserving too many features in the frames with similar semantics. We propose a simple solution upon RoI tracking and selection. As we shall see later, it offers a solid baseline for future work.Step 1: RoI tracking. We apply HQTrack [69] , an off-the-shelf tracking algorithm, to localize the RoI in each input frame. The pre-trained tracking model is not fine-tuned in the training phase. Given a RoI (a bounding box) in any video frame, the tracking algorithm outputs either a bounding box or nothing (e.g., if the target is occluded) in each of the remaining frames. This step outputs a raw list of RoIs denoted asB ′ = (B ′ 1 , . . . , B ′ M ′ )where M ′ can be close to the number of frames.Step 2: RoI selection. Feeding all tracked frames into the MLLM often incurs computational inefficiency and extra difficulties in model training. To avoid this, we select a subset from B ′ containing M < M ′ RoIs, with the goal being to preserve diverse visual features using a limited number of RoIs. In practice, we pre-defined the target number, M , and adopt the K-means algorithm to form M clusters from the original set of M ′ RoIs. The final RoI list, B, consists of a randomly chosen RoI from each cluster.Discussions. Finding representative RoIs belongs to a generic topic of feature selection. On one hand, one can set a simple baseline by performing random or uniform sampling from the original set B ′ . On the other hand, the information theory offers a general principle, i.e., maximize the diversity of RoIs throughout the selection procedure. As demonstrated in Section 4.1, random and uniform sampling algorithms frequently fail to capture semantic changes throughout complex videos. By contrast, the simple K-means clustering used in Artemis significantly increases the diversity (see Appendix D), ensuring representative video features. We conjecture that the effectiveness of feature selection is related to the quality of video features; with stronger video foundation models, more sophisticated feature selection algorithms can make a larger difference. We leave this topic to future research.The MLLM is built upon Vicuna-7B v1.5 [11] , an open-sourced LLM 4 . We use CLIP ViT-L/14 [41] to extract visual features. To feed these 1024-dimensional visual tokens into the LLM, we use a learnable, two-layer MLP (1024-4096-4096) to project the visual features into the 4096-dimensional language space. We always use the auto-regressive framework to train the MLLM.The training procedure of Artemis comprises three steps, (1) video-text pre-training, (2) video-based instruction tuning, and (3) video-based referring. The first two stages are similar to Video-LLaVA [31] but different training data are used. We set a unified template, User: <video-tokens> <instruction> Assistant: guiding the model to output the desired answer. Here, <video-tokens> contains the spatial and temporal video features (F S V and F T V , projected by MLP), and <instruction> contains the language tokens of the task description (see below).In the first stage, <instruction> has the form of 'Write a terse but informative summary of the following video clip.' and the model outputs the overall description of the video.The training data includes image-text and video-text pairs, using images as still videos. We use a subset of 558K LAION-CCSBU image-text pairs with BLIP [28] captions, sourced from CC3M [42] and refined by LLaVA [33] . Additionally, we use the 702K video-text pairs provided by Video-LLaVA [31] , derived from the 703K pairs constructed by Valley [36] using WebVid [3] . Only the MLP is trained (from scratch) in this stage, initializing the alignment of vision and language. The training elapses one epoch with a learning rate of 1 × 10 -3 , taking about 5 hours on 8×A800 GPUs.In the second stage, <instruction> contains specific task descriptions like 'Where is the person in the image?' and 'What is the person doing in the video?', and the model follows the instruction to produce the answer. The training data comprises the 665K imagetext instruction dataset from LLaVA-1.5 [33] and the 100K video-text instruction set from Video-ChatGPT [37] . Both the LLM and MLP are fine-tuned in this stage. The training elapses one epoch with a learning rate of 2 × 10 -5 , taking about 20 hours on 8×A800 GPUs.In the third stage, we use the curated VideoRef45K dataset to endow the model with the ability of video-based referring. The template is modified as follows,User: <video-tokens> <refer-instruction> <track-instruction> Assistant: Here, <refer-instruction> is formulated as 'What is the <region> doing during this video?' where the <region> token is replaced by the visual features extracted from the bounding box in the specified input frame, and <track-instruction> contains additional information, 'This is the tracking list: <region>, ..., <region>' where the <region> tokens are the target-specific features (F R V,B1 , . . . , F R V,B M , projected by a Linear) extracted from the selected RoIs 5 , and the number of <region> token is M . In this stage, we fine-tune the LLM (with LoRA [22] ), MLP and the RoI Align module. The training procedure elapses 3 epochs with a learning rate of 4 × 10 -5 , taking about 3 hours on 8×A800 GPUs.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Justification: The paper primarily focuses on multimodal large language models about video understanding using publicly available datasets that have undergone thorough validation. This study only serves as a foundation model for video-based referring, which is not directly applicable for practical scenarios until now. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
artemis is a strong baseline for video-based referential understanding: 
 Setting and metrics. We evaluate the ability of Artemis in video-based referring on the test set of HC-STVG [44] . The video and text data are pre-processed using the same method as in the training set. The test procedure uses the same instruction as in the third training stage and applies HQTrack [69] to localize the RoIs in video frames. We use the standard evaluation metrics including BERTScore [65] , BLEU@4 [39] , METEOR [4] , ROUGE_L [32] , CIDEr [50] , and SPICE [2] . Adapting existing MLLMs for video-based referring. Due to the limited availability of research for video-based referring, we compare our model to a few recent MLLMs trained for image-based or multi-frame based referring 6 . The image-based referring models include Osprey [59] , Ferret [56] ,What is this object's <region> behavior?The woman in black clothes gets out of the car and walks to the door of the building.The woman is walking down the sidewalk.What is this woman's <region> behavior?What is the <region>'s action in this video?The woman in red clothes walks to the man in black clothes and sits down.The woman, who is wearing a red dress, is sitting on the bed and massaging the man's feet.What is the woman's <region> action in this video? and Shikra [9] . For each video, we extract 5 key frames with RoIs produced by HQTrack and ask the trained model "What is the target <region> doing?" in the way the models are familiar with. Finally, we use GPT-3.5-Turbo to summarize the 5 answers into the overall description of the target. The multi-frame based reference model is Merlin [57] which receives 5 key video frames and RoIs and produces the overall description. The selection of key frames is consistent with Artemis. To compare with MLLMs that are trained for video understanding, such as Video-ChatGPT [37] and Video-LLaVA [31] , we follow [43] to draw a red rectangle to mark the referred object in each key frame of the video. Then, we feed the rendered video to the models and ask the question "What is the target indicated by the red rectangle doing?". Quantitative results, and necessity of native video-based referring. The numbers are summarized in Table 1 . Artemis outperforms other MLLMs in each single evaluation metric. Note that the advantage is significant for some metrics, e.g., BLEU4. Please refer to Figure 1 for representative examples. In Figure 9 (see Appendix C), we show the behavior of the methodology using a standalone LLM (e.g., GPT-3.5-Turbo) upon image-based referring outputs. The image-based models tend to describe individual moments rather than an entire video; based on these inputs, the LLM cannot realize video descriptions and is sometimes confused to hallucinate what never happens in the video. The comparison validates the necessity of training a native model (i.e., directly on the instruction data for video-based referring) like what Artemis has done. Equipping with such a fundamental ability of video understanding at a finer level, Artemis can perform even more complex video understanding tasks, as shown in Section 4.2.Qualitative results. We display several representative examples of video-based referring in Figure 3 . The output of Artemis is comprehensive (especially compared to other MLLMs, see Figure 1 ), often containing fine-grained actions of the target. This mainly concerns the compact video features extracted by the RoI tracking and selection algorithm that extracts key features for understanding.Ablative studies for target-specific video features. The key to extracting compact target-specific features lies in RoI tracking and selection. To validate this, we ablate two key parameters: the strategy of RoI selection and the number of preserved RoIs. In Table 2 , we define a baseline for region of interest. For each frame of a video containing an object of interest, we enclose the object's location with a red rectangle and encode the video using Video-ChatGPT. The system is then queried with "What is the object in the red rectangle doing in this video?". As illustrated in Table 2 , K-means clustering emerges as a simple yet effective approach for RoI selection, whereas random or uniform sampling fails to consistently capture representative RoIs. To validate the effectiveness of RoI features in representing the object, we replace the <region> features with visual features (the [CLS] token from CLIP-ViT-L/14) in key frames, a variation we refer to as "w/<track-instruction>". The performance decline compared to Artemis indicates that RoI features are of higher quality, as whole-frame features are more susceptible to background noise. Additionally, Table 3 demonstrates the importance of using multiple RoIs for understanding the full video content. While retaining more keyframes slows down both training and inference, it also introduces redundant information, leanding to a slight drop in performance metrics. Empirically, using 4 RoIs provides the optimal balance on the HC-STVG test set, although increasing the number of RoIs may be beneficial for more complex videos.From the information theory, we show that RoI tracking and selection improve informativeness (in terms of entropy) and diversity (in terms of frame-level difference) of the target-specific features in Figure 4 . As shown in Figure 5 , RoI tracking and selection gradually improve the comprehensiveness of the referring results. The woman is holding a gun and is in the process of shooting it.A woman walks into the room takes the gun out of the man's hand, and put it on the table.The woman in the red dress walks to the table, takes the gun, and put it in the man's hand.The woman in the skirt walks to the man and then stops.The woman in the brown dress walks to the man in the white shirt and stops.The woman in the white skirt turns and walks away.What does the <region> do during this video?How does the <region> act in the video? The man in black clothes raises his hand and shakes hands with the opposite man.The man in black clothes sits down and puts his hands on the table.The man in black clothes turns around and sits down.The man in the black suit walks to the door and opens the door.The man in black clothes turns his head and speaks to the woman in white, then turns his head again.The man in black clothes turns his head and speaks to the woman in white, then turns his head again, walks to the door and opens it, turns around and sits down, raises his hand and shakes hands with the opposite man, and finally sits down and puts his hands on the table.Prompt: I divided a video into 5 segments and described a specific object in each of these segments. Please connect these 5 descriptions in chronological order to form a complete sentence describing what the object did in the video. These are the 5 descriptions: The woman in black clothes walks to the table and sits down.A man in a gray suit sits with her.How does the man <region> act in this video?The man in grey clothes looks at the paper in his hand, then looks up at the woman in black. Multi-round video understanding with grounding. Multi-round dialogues, especially answering logically related chain-of-questions [46] , is an important yet challenging topic for MLLMs. In Figure 6 and Figure 14 in Appendix E, we show that Artemis's referential understanding ability can be combined with image-based grounding models (e.g., GroundingDINO [34] ) to answer multi-round chain-of-questions, where the entities mentioned in the video-based referring result is located and fed into the next round of video-based referring quest, allowing for more complex interactions.Long video understanding with text summarization. Target-centric understanding of long videos is a major challenge for existing video-based MLLMs. The difficulty mainly lies in extracting compact video features (to feed into the MLLM) and tracking the target throughout the long video. We offer a simple solution that first segments the video into shorter clips, applies Artemis for understanding these clips, and applies an off-the-shelf LLM (e.g., GPT-3.5-Turbo) for summarization. As shown in Figure 7 and Figure 12 in Appendix E, the final output offers a comprehensive understanding. To our knowledge, this function was not achieved by existing MLLMs.Video question answering. Lastly, we show that Artemis can perform general video question answering. We test the trained model on the Video-ChatGPT test set [37] and the other three benchmarks (i.e., MSVD-QA [8] , MSRVTT-QA [53] , and ActivityNet-QA [6, 58] ) where their training sets was not used to train Artemis. Results are summarized in Tables 4. Artemis shows competitive performance among a few recent MLLMs. These results inspire us that (i) an MLLM trained for finer-level video understanding can seamlessly transfer to coarser-level tasks, and (ii) extracting compact video features also benefits video question answering. 
conclusion: 
 This paper proposes a challenging setting for video-based referring and establishes an effective MLLM named Artemis. Compared to existing methods, Artemis can understand human intention from simpler inputs (a text prompt and a single-frame bounding box) and comprehensively describe the target's action in a complex video. At the core of Artemis is an RoI tracking and selection 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The proposed models are trained on benchmark datasets such as HC-STVG, VID-Sentence, and S2D Sentences, etc. These datasets have been extensively used in the computer vision community and have undergone comprehensive safety risk assessments. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: In the paper, we clearly specified the datasets and code sources used, and provided appropriate citations in the reference section. Additionally, we ensured transparency by including the original sources of any modified code files, making the changes traceable.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes]Justification: We have included the code, data, along with detailed usage instructions, in a anonymous link. After the review process is completed, we will make the code and data publicly available to the community.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This study does not involve any crowdsourcing experiments or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We organized 7 existing video datasets and performed a careful data curation procedure, resulting in the VideoRef45K benchmark, which comprises 45K video question-answer pairs. The 7 datasets include HC-STVG [44] , VID-Sentence [10] , A2D Sentences [20] , LaSOT [18] , MeViS [16] , GOT10K [24] , and MGIT [23] .HC-STVG is a movie clip dataset that provides tracking sequences and textual annotations describing the person's actions during a certain period. We use the training portion, which contains approximately 10K video clips as our training data. The validation portion, containing 3,400 video clips, evaluates Artemis's ability. The original tracking sequences in HC-STVG are of poor quality, so we use the off-the-shelf tracking model HQTrack [69] to regenerate the tracking sequences and remove some low-quality bounding boxes. To prevent tracking target deviation caused by cross-frame tracking, we select the first and middle frames with ground truth bounding boxes annotated by the HC-STVG dataset as the referring frames for HQTrack to generate tracking lists for the whole video. We then compare these two generated tracking lists with the HC-STVG annotations and exclude the frames with low IoU between the generated and annotated bounding boxes from the tracking lists.A2D Sentences provides tracking sequences and captions for different objects, but the tracking sequences are only 3 frames long. To address this, we use HQTrack to regenerate the sequences and obtain longer tracking frames, extending them to 20 frames.LaSOT provides a caption for an object along with its tracking sequence. However, LaSOT videos are usually long, and the captions for the entire video are generic. To address this, we extract three segments of 10 seconds each from the entire video for our training data.GOT-10K is a tracking dataset that provides tracking sequences of objects and their categories, actions, and adverbs describing the actions. We concatenate these elements to describe the object's action in the video, e.g., "bear is slowly walking."MGIT videos are typically long, with annotations indicating the object's actions at different time intervals. We extract these segments as our training data.For MeViS and VID-Sentence, we did not perform any special processing. We converted the mask annotations of MeViS into bounding boxes.Figure 8 shows some examples of VideoRef45K.The standing woman goes to the sofa and turns to sit down.The person standing next to the horse releases his hand and holds the rider, then takes the horse and walks away.Panda climbs up and then falls down. Zebra runs across the lake then runs back.A bicycle ridden by a person is running from left to right on the hillside. Black car operating forward and changing lanes. The total dataset composition is summarized in Table 5 . [16] 1644 4489 A2D Sentences [20] 3017 5359 LaSOT [18] 2640 7920 VID Sentence [10] 4045 7654 GOT-10K [24] 9090 9090 MGIT [23] 105 614Through the above processing steps, we obtained the VideoRef45K dataset. This dataset includes captions describing the actions of objects in videos, along with their corresponding tracking sequences.To utilize VideoRef45K, we created a template of questions to prompt the language model (LLM) to answer what the referred object did in a video. The template consists of '<refer-instruction>' and '<track-instruction>'. The '<refer-instruction>' is formulated as 'What is the <region> doing during this video?', and we utilized GPT-3.5-Turbo to generate the refer instruction template. The '<track-instruction>' contains additional tracking lists to help the LLM perceive the referred object, such as 'This is the region's tracking list: <region> ... <region>'. We created several '<track-instruction>' options as the track instruction template. During training, we randomly sampled a '<refer-instruction>' from the refer instruction template and a '<track-instruction>' from the track instruction template. The '<refer-instruction>' and the '<track-instruction>' are then concatenated as '<refer-instruction><track-instruction>' to formulate the text prompt.We report the detailed training hyper-parameters of Artemis in Table 6 . Existing multimodal language models (MLLMs) for video understanding rarely possess video-based referring capabilities. To address this limitation, we leverage image-based MLLMs (such as Osprey [59] and Ferret [56] ) to perform image referring on 5 keyframes within the video independently. Subsequently, we integrate these outputs using GPT-3.5-Turbo to achieve video-based referring, as depicted in Figure 9 . However, these models struggle to perceive the action and behavior of the given region of interest (RoI) as effectively as Artemis.A woman is standing in the image, positioned towards the right side. She is wearing a black shirt and appears to be looking at a man.A woman is standing in the image, positioned towards the right side. She is wearing a black shirt and appears to be looking at a man.A woman, possibly a mother, is standing next to a man. She is wearing a black shirt and appears to be looking at the man.A woman is standing next to a man. They appear to be in a pleasant and relaxed state, possibly enjoying a special occasion or a casual conversation.A woman, possibly a grandmother, is standing next to a man. She is wearing a black shirt and appears to be looking at the man.Throughout the video, a woman, possibly a mother or grandmother, wearing a black shirt, stands next to a man, smiling and engaging in a pleasant interaction, potentially enjoying a special occasion.The woman in black clothes walks to the man in the vest and stops.The region is a woman dressed in a black evening gown. Her attire suggests that she might be attending a formal event or a gathering. The woman's presence contributes to the elegance of the scene.The region is a woman. She stands in a doorway. The overall image appears to be set in a luxurious room with a man in a suit and a woman in a black dress.The image shows a formal event or gathering.The region is a woman. Her attire suggests that she is attending a formal event, perhaps a dinner or a party. The woman's dress, which is worn at such events, contributes to the elegance of the occasion.The region is a woman who seems to be looking away.It appears that she is involved in a conversation that doesn't require her to face the camera. Her body language and suggest she might be listening something.The region is a woman who seems to be looking away.It appears that she is involved in a conversation that doesn't require her to face the camera. Her body language and suggest she might be listening something.Throughout the video, a woman dressed in a black evening gown attends a formal event or gathering, standing in a doorway and adding elegance to the scene. She is seen listening or observing, contributing an element of mystery and intrigue to the overall image. To better demonstrate the effectiveness of our method, we conducted the following experiments. Firstly, we computed the attention values between different RoI tokens and temporal tokens. We observed that the tracking tokens added through tracking compensated for the weak perception aspect of the initial video tokens, as illustrated in Figure 10 . Subsequently, as depicted in Figure 4 (top), we calculated the information entropy before and after adding the tracking list. Upon adding the tracking list, the overall amount of RoI information fed into the MLLM increased by 20.3%. Additionally, we computed the inter-frame differences of the boxes chosen from the tracking list. As shown in Figure 4 (bottom), the K-means clustering method selects RoIs with greater differences than random and average selection. This enables the MLLM to better perceive the action changes of RoIs throughout the entire video.Figure 11 shows more examples of Artemis's outputs of video-based referring. Figure 12 shows more examples of target-centric video understanding with text summarization. Figure 13 showsTracking RoI Tokens What is the behavior of the <region> in this video?Temporal TokensThe woman in white clothes turns her head and glances at the man behind her, then turns her head back.What is the <region> doing during this video?The woman in the black dress takes a sip of wine and puts the glass down.How does the <region> move in this video?The man in the black suit walks to the table and sits down. Artemis's failure cases, revealing the limitation of Artemis. Figure 14 shows mores example of combining Artemis with off-the-shelf grounding model, e.g. GroundingDINO to answer multi-round chain-of-questions conveniently. The man in the vest goes to the table and sits down.The man in the red tie turns and walks to the door.The man in the red tie turns and walks up the stairs.The man in the tie turns around and walks to the woman in the hat.The man in the red tie turns around and walks to the woman in the hat, then he turns and walks up the stairs, followed by turning again and walking to the door, and finally, the man in the vest goes to the table and sits down. What is the motion of the <region> in this video?What is the <region> doing during this video?The man in white clothes opens the door and gets out of the car.The man in blue clothes comes down the stairs and turns to the left. NeurIPS Paper ChecklistOne can also use an image-based model for frame-wise referring and call an LLM for text summarization, but, as shown in Appendix C, this method often fails to understand the intrinsic logic in complex videos.The name refers to Artemis' ability to track prey and select pivotal hunting moments.A stronger LLM (e.g., with a larger number of parameters) brings marginal improvement, because at the current stage, video understanding does not rely on strong language modeling abilities.During the training phase, we randomly select a frame with an annotated region bounding box as the input and employ the tracking module to locate the bounding box of the referred object in the sampled frames. During inference, we track the given region to generate the tracking list.The comparison against these methods is not totally fair because they have not been trained for video-based referring. We mainly use the comparison to claim that video-based referring is important and challenging, yet image-based MLLMs cannot do it well. 


Decision: The paper originally received borderline scores leaning towards acceptance. The authors provided a detailed rebuttal and the paper has been extensively discussed by the reviewers. Reviewers CDQu and xzsp raised some concerns about comparisons to the previous works and missing baselines (in particular for what concerns the video-language models) and the contributions of the tracked ROIs. The authors successfully addressed the issues and both the reviewers where satisfied by the rebuttal and increased their scores to 7:Accept. The AC carefully revised all the material and there has been also some further discussion with the reviewers. Overall this is a solid submission and the authors are encouraged to follow and implement all the reviewers' suggestion in the final revision of the paper.

Reviews: review: paper_summary: paper_summary: The paper proposes to bring fine-grained understanding to multimodal LLMs (MLLMs) by introducing video-based referential understanding task. The paper motivates with the drawbacks of current image- and video-based MLLMs, and the need for region-specific features to answer region-specific questions . The proposed approach expands the set of video-level features with target region-specific features (box-level prompts) via tracking (HoITrack), alignment (RoIAlign) and selection (clustering). The paper ablates the effectiveness of tracking and selection criteria for improved performance.
strength_weakness: strength_weakness: - A fair baseline
    - It’s great that the paper provides comparison with image-based MLLMs by extending them to videos and using an LLM to obtain video-level answers
    - As for the multi-frame approach, it appears that MERLIN is the closest baseline approach. And there seems to be some intersection of pre-training data used for Artemis and MERLIN that includes GOT10K, LaSOT, MeViS
    - But the evaluation of the video-based referring ability is done on the test set of HC-STVG (lines 197-198), and the train set of HC-STVG is included in the pre-training data for Artemis but not for MERLIN
    - Since MERLIN was not trained or fine-tuned on HC-STVG, it does not seem to be a fair, apples-to-apples comparison that MERLIN be evaluated on the test set of HC-STVG
    - On top of that, datasets have biases which includes different label spaces, scene and setting (e.g. HC-STVG is collected from movies), annotation gathering (which can also result in different caption distribution, and hence biasing the eval metrics)  
- Ablation
    - Assuming “w/o” in Table 2 means that there is no RoI selection (not mentioned or defined in the text), which I’m assuming to mean that there is no <track-instruction>. If the above assumption is true, “w/o” sets up a strong baseline (even better than MERLIN)
    - In any case, it seems that a careful ablation study is missing
        - (1) w/o <track-instruction>
        - (2) w/ <track-instruction> but where <region> features are not RoI features but the key-frame features. This is to establish whether the improvement is brought forth by key-frame selection or region-level features specifically   
- Visualization and sanity check post-training
    - Do the authors have an example of a video with multiple different regions? Mainly this is to inspect how the model response changes with selection of different regions in the same video, and whether it doesn’t degenerates to the same response?
    - Less important, but do the authors have an impression on if the video is reversed, does the caption change?
 
- Human evaluation
    - Lastly, to substantiate the effectiveness of the approach, did the authors think about a human evaluation study on accuracy / relevance of predicted captions to the video-question pair?
        - This could also be done within your group and with anonymized predictions (meaning the human evaluator doesn’t know what prediction is from what model)


## Minor
- A bit more about the architectural details would have been great (at least in supplemental), especially the tokenization process and whether / how start-end tokens for <instruction> were used
- Do the authors have some statistics on the object category of <region> in the test set of HC-STVG?
    - This is mainly to identify what biases the models are dealing with, and whether those biases skew heavily in one direction. For example, "person" may be the majority category
    - Similarly, any statistics of actions being performed and asked in questions?
- Lastly, do the authors have a quantitative breakdown of performance to understand where the model fails?
questions: questions: See Weaknesses
limitations: limitations: Yes
paper_summary: paper_summary: This paper introduces Artemis as a robust solution for the video-based referential understanding task. This task involves analyzing complex videos, each spanning 20–30 seconds, where the target performs multiple actions. Given a video, the Multimodal Large Language Model (MLLM) attempts to answer questions such as "What is the target <region> doing in this video?" with <region> referring to a bounding box in any video frame. Artemis follows the general design principles of modern MLLMs, such as visual instruction tuning. To extract target-specific video features, Artemis employs a straightforward yet effective approach involving (i) tracking the target over time and (ii) selecting informative features from a comprehensive list of regions of interest (RoIs). The training of Artemis consists of three stages, with the first two stages being similar to LLaVA. For the final stage, this paper introduces the VideoRef45K benchmark, comprising 45,000 video question-answer pairs, with box-level prompts and answers for complex videos. Experiments demonstrate the promising performance of Artemis across various quantitative metrics, including BERT score, BLEU, and more.
strength_weakness: strength_weakness: 1. As far as I know, Multi-Object Tracking (MOT) is far from satisfying in accurately tracking target regions, particularly in challenging scenarios such as motion blur and occlusion. Although this paper mentions these issues in its limitations section, it does not discuss them in detail. My concern is that the method performs well on the presented benchmarks because the scenarios to be tracked are relatively simple. It would be beneficial if the authors could provide the performance of HQTrack on these benchmarks or offer examples showing the method's efficacy in complex scenarios (e.g., multi-person tracking).
2. This paper does not delve deeply into learning the temporal relationships among tracked Regions of Interest (RoIs). Therefore, the temporal encoding knowledge of these RoIs mainly derives from HQTrack. Since the improvement of this model in video-based referential understanding primarily stems from its better comprehension of temporal dynamics, it raises the concern that the benefit may come from HQTrack rather than the model itself.
3. Comparisons with other models (e.g., Shikra, Merlin) seem somewhat unfair, as these models do not utilize the dynamic knowledge from HQTrack. A fairer comparison setting would enhance the validity of the results.

I am willing to upgrade my rating, if the authors can address my above concerns.
questions: questions: I am a bit surprised that MLLMs for video referring and grounding have not been explored yet. As I am not very familiar with referential understanding, I look forward to more feedback from other reviewers. If this paper does not overlook any important references, I am willing to upgrade my rating.
limitations: limitations: Please refer to weakness.
paper_summary: paper_summary: This paper introduces Artemis, a video-language model for video-based referential understanding. It can describe a target referred by a bounding box in a video. A referential video dataset named VideoRef45K is collected to train the model. Artemis is evaluated on HC-STVG benchmark and outperforms baselines adapted from image-based referring models. Besides referential understanding, Artemis can also perform general video question answering, and serve as a component in multi-round and long-form video understanding.
strength_weakness: strength_weakness: 1. The RoI selection step of Artemis clusters the object bounding boxes from different frames. However, the clustering algorithm only considers the bounding box coordinates but does not take the visual content in the bounding boxes into account. In some cases, the bounding box of an object may remain unchanged for a long time but the object state keeps changing, e.g., a person standing at a certain location performs a series of actions. Clustering these frames together can compressing them into one would lose valuable information.
2. The proposed Artemis is built based on video-language models Video-LLaVA and Video-ChatGPT. However, these video-language models are not used as baselines in the experiments. Although they are not originally developed for video referring, there is a simple approach to adapt them for the task. As suggested by [1], directly drawing a circle or a rectangle on an image can help VLMs focus on the indicated object. Therefore, one can adapt the video-language models for video referring by drawing the object bounding boxes on the video frames and ask the model "What is the target indicated by the red rectangle doing?". Artemis should be compared with this simple baseline to demonstrate the effectiveness of the RoI feature branch in its model architecture.

[1] Shtedritski et al. What does CLIP know about a red circle? Visual prompt engineering for VLMs. ICCV 2023.
questions: questions: 1. Does the target-specific features in \<region\> tokens include the positional information of the bounding boxes?
limitations: limitations: Limitations are discussed in the paper.

Comments: title: Thanks
Thanks
comment: We are happy that our response addressed your question. We appreciate your support for our work.
The authors' feedback has addressed my concern, I choose to raise my rating.
We are delighted that our response addressed your question. We appreciate your support for our work.
I read all the reviews and the authors' responses to them. The responses to my comments are satisfactory and I highly appreciate the additional experiments. Therefore, I raised my rating to 7 (Accept).