Artemis:  Towards Referential Understanding in Complex Videos
Authors: Jihao Qiu
Yuan Zhang
Xi Tang
Lingxi Xie
Tianren Ma
Pengyu Yan
David Doermann
Qixiang Ye
Yunjie Tian
Venue: NeurIPS 2024 poster (2024)
Keywords: Video Referring
Multimodal
RoI Selection Mechanism
Primary area: machine_vision
TL;DR: This paper proposes a challenging setting for video-based referring and establishes an effective MLLM named Artemis.

Abstract: Videos carry rich visual information including object description, action, interaction, etc., but the existing multimodal large language models (MLLMs) fell short in referential understanding scenarios such as video-based referring. In this paper, we present Artemis, an MLLM that pushes video-based referential understanding to a finer level. Given a video, Artemis receives a natural-language question with a bounding box in any video frame and describes the referred target in the entire video. The key to achieving this goal lies in extracting compact, target-specific video features, where we set a solid baseline by tracking and selecting spatiotemporal features from the video. We train Artemis on the newly established ViderRef45K dataset with 45K video-QA pairs and design a computationally efficient, three-stage training procedure. Results are promising both quantitatively and qualitatively. Additionally, we show that Artemis can be integrated with video grounding and text summarization tools to understand more complex scenarios. Code and data are available at https://github.com/NeurIPS24Artemis/Artemis.

Artemis: Towards Referential Understanding in Complex Videos
Jihao Qiu!"
Yuan Zhang
Xi Tang
Lingxi Xie
Tianren Ma
Pengyu Yan?
David Doermann
Qixiang Ye
Yunjie Tian
Uniieersity of Chinese Academy of Sciences
University at Buffalo
[qiujiahao19 19, zhangyuan192 tangxi19, matianren18, tianyunjie 19) mails. pyan4@buffalo..o doermann @ buffallo.du 19880xxc@gmail.com
qxye@ucas.ac.oom
Abstract
Videos carry rich visual information including object description, action, interac- tion, etc., but the existing multimodal large language models (MLLMs) fell short in referential understanding scenarios such as video-based referring. In this paper, present Artemis, MLLM that pushes video-based referential understand- ing to a finer level. Given a video, Artemis receives a natural- latural- question with bounding box in any video frame and describes the referred target in the entire video. The key to achieving this goal lies in extracting compact, target- specific video features, where
solid baseline by tracking and selecting spatiotemporal features from the video. We train Artemis on the newly established VideoRef45K dataset with 45K video-QA pairs and design a computationally ef- ficient, three-stage training procedure. Results are promising both quantitatively and qualitatively. Additionally, we show that Artemis can integrated with video grounding and text summarization tools to understand more complex scenarios. Code and data are available https: :// titt........ com/ com/qiuji ihao19/ Artemis
Introduction
The past year has witnessed rapid progress of multimodal large language models (MLLMs) 33 68
[40,, offering abundant abilities of open-world image understanding with language-based dialogues. In comparison, there fewer studies on training MLLMs for video understanding, albeit videos
much more informative than still images. Existing video-based MILMS (29 61 37 31 38 mostly focus superficial dialogues in which the video is encoded holistically, inevitably lacking the ability to understand fine-level video contents, e.g., describing a user-specific target in the video.
We considering
task called video-based referential understanding compensate for the limitation. Specifically, we are interested in complex videos that span 20-30 seconds and the target performs multiple actions during this period. Given video, the MILMM tries to answer question like 'What is the target <region> doing in this video?' where <region> refers to a bounding box in any video frame. We argue that the task is not only challenging feature extraction, tracking, summarization, elc., but also important because it lays the foundation of finer-level video understanding. However, shown in Figure] [ existing MLLMS often fell short in
it requires
Equal contribution. i Corresponding Author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Video-based Referring
What the behavior
How does the <region> in this video?
<region> in Uhis video?
The man the white suit walks to the car, opens the gets in.
The woman red clothes walks to the piano and sits down.
The man is carry ing a sultcase and walking lowards
The gid playing the piano.
The man suit inspecting the car and opening the trunk.
E
The playing the flute in the video.
man throughout the vidoo in ditterent positions with white
A young girl, wearing white dress and red bow ber heir,, displays her musical talent by playing piece on piano, caplivaling audicecce with her performance.
possibly either giving
directions, admiring or preparing to board and loading or unloading luggage. In the flnal scene, police olficer seen standing next the seemingly writing a ticket.
Long Video Description
Video Reasoning
you describe the video?
the vidoo shows girl performing gymnastics routine on balance beam. She starts by jumping onto the beam and then proceeds perform series of flips and tricks. The girl's movements are fluid and graceful, showcasing impressive skills. As finishes hor routine, she jumps off the beam and lands gr accefully on the mat. The girl's testament the hard work and dedication required to master the art gy gymnastics. Overall, the video is coptiv ating display athleticism and skill that is sure to impress anyone watching.
J0
Why is this video strange?
The video strange bocause shows giraffe
jumping into swimming pool, which common sigbl.
?>
Video QA
Wheee video from?
performance
video is laken from helicopter flying over the city.
What, place shown the video?
The video shows the Statue Liberty, which is famous landmark in New Y ork City.
Figure 1: Artemis' ability in video-based dialogue. Notably, Artemis excels particularly in video- based referring, outperforming the existing MLLMs including Merlin [57] and Video-LLaVA [31] lacking comprehensiveness and Osprey [59] suffering hallucination.
this seemingly casy task because they were mostly trained for image-based referential understanding: as result, they can only perceive the action in a single moment rather than that in entire video
This paper presents A as a solid baseline for the above task. Artemis follows the generic design of modern MLLMs (i.e., visual instruction tuning [33]), but encounters challenge in finding sparse, target-related information from dense video data. A preliminary study shows that feeding raw video features into the MILM results in computational inefficiency and training instability. To extract target-specific video features, we propose simple yet effective solution that involves (i) tracking the target time and (ii) selecting informative features from long list of regions-of-interee (Rols). The compactness of features makes it casier to train the MLLM. We design three-stage training schedule where the MLLM gradually learns video-text alignment from coarse fine. This efficient design requires only 28 hours (3 hours for the final stage) on 8x NVIDIA-A800 GPUs.
To train and evaluate Artemis, organize 7 existing video understanding datasets into the Video- Ref45K benchmark comprising 45K video question- -nnwwer pairs. To our knowledge, this is the first
an LLM text summarization, but, as shown in Appendix(( C this method often fails to understand the intrinsic logic complex videos. "The name refers Artemis' ability track prey and select pivotal hunting moments.
2 "nne can also use an image-based model frame-wiss referring

---

benchmark with box-level prompis and answers spanning complex videos. Experiments show the promising results of Artemis in a wide range of quantitative metrics including the BERT score, BLEU, elc.. Qualitatively, Artemis also shows a clear advantage in the comprehensiveness of description meanwhile avoiding hallucination (see Figure [ for examples). Beyond the ability of video-based referring. Artemis serves an important building block for complex video understanding, where we integrate Artemis with off-the-shelf video grounding and text summarization tools for interactive video-based dialogue and long video understanding, respectively. We expect our work to shed light on upgrading MLLMs for fine-level and interactive video understanding.
2 Related Work
Large language models (LLMs) and multimodal LLMss (MLLMs). LLMs [5 15 45 12
64
49|66) [I] have opened a new era of AI, demonstrating the potential to deal with various language- based understanding and generation tasks. To unleash the power of LLMs for visual understanding, the computer vision community has been working on aligning language and vision data in the same feature space [41). There are mainly two lines of research, where the internal adaptation methods 1 integrated cross-attention within an LLM for visual-language alignment, and the external adaptation methods [27] 14 33 trained extra modules for this purpose. As a result, the vision
foundation models, especially vision transformers 17 35 41 48 47 666 25), have been upgraded
into MLLMs 33 46
63
26
which gain the ability of language-guided visual understanding.
MLLMs for referring and grounding. MLLMs can be integrated with instance-level visual understanding tasks, allowing the models to (i) respond to questions targeted at specific regions of the image and (ii) identify regions corresponding to the contents in the dialogue - these functions main ways to integrate these functions into MLLMs, differing from each other in how the positional
referred to as visual referring 633 7 and grounding 440 34] respectively. There
information processed. The explicit methods 40 52 introduced extra tokens encode positions, while the implicis methods (9 (1] used natural language to represent positions. Recently, there are also efforts [46] that used LLMs to call external vision modules for more flexible instance-level understanding quests.
Video-based MLLMs. Compared to the large corpus of image-based MLLMs, there are fewer video-based MLLMs for at least two reasons. First, there are fewer paired video-text data, especially for instance-level video understanding. Second, the higher dimensionality of video data poses greater challenge to efficiently encode videos into visual features and find useful features to answer the questions. Existing efforts include VideoChat (29], Video-ChatGPP (37] Video-LLLMM [6]],
Video-LLL VA (3T. LanguageBind (67). Valley [36. etc.; most of them followed the paradigm of
imsge MILMSs and some of them 37 proposed more efficient video feature. Recently, there have been some preliminary studies for instance-level video understanding, e.g., LEGO [30] studied moment retrieval with the assistance of LLMs, and PG-Video-LLa VA (38) performed video grounding by employing off-the-shelf tracking and grounding modules. Merlin (57) studied video- based referring, but it was built upon three manually specified frames as visual input, incurring extra burden for users and also limiting the model's 's ability to understand long and complex videos. This paper aims to address the above two challenges. for which we set up a new formulation, establish new benchmark named VideoRef45K, and present a solid baseline named Artemis.
3 Artemis: A Baseline for Video-based Referential Understanding
3.1 Problem Formulation and Data Preparation
video can be represented in the raw form of V E BTXWXHXC where T. W. H, and C stand for the number of frames, width, height, and the number of channels, respectively. In the task of video-based referential understanding (a.k.a video-based referring), the model receives a question in the form of 'What is the <region> doing in this video?', where the concrete class of the referred object (like man or dog) is not provided, and the <region> supplemented by a bounding box B- (4; ,,, ;,, U2, U2) in frame E ... T). The expected output is a sentence describing the target's ss action the full video as detailed as possible (see Figure [ for examples). Note that the proposed task requires stronger ability beyond image-based referring and video captioning,
3

---

Artemis Response
Selection
Selection
The black clothes and walks the door.
ARTEMIS
Multimodal Large Language Model
User Query
MLP
Linear
What does <tregion> do this video?
qpata Teatume
spreet-spoclto features
Aiggmeet
Rol Treoking & Aligmment& ROI Selection
CLIP-L/14
Rol Tracking
Figure 2: Left: the overall framework of Artemis, where an MILM receives text prompt together with spatial, temporal, and target-specific video features, and produces the answer. Right: the Rol tracking and selection mechanism to generate target-specific features. We use different IDs to show the clustering result. This figure is best viewed in color.
mainly in the coverage and granularity of visual understanding. Specifically, the model is expected to produce complex action descriptions for different target <region> specified.
We collect video data for referential understanding from 7 datasets, including HC-STVG 44 VID-
Sentence [0], A2D Sentences [20] LaSOT [8]. MeViS [16), GOTIOK (24], and MGIT (23]. In
total, there are 45K video QA pairs. We perform dataset-specific operations, including re-tracking (for HC-STVG and sentennnen,,,, clip cropping (for LaSOT and MGIT), and caption summary
(for GOT1OK), to convert them into the required form. Please refer to Appendix A for further details.
3.2 Overall Framework and Visual Features
The overall framework Artemis, as illustrated in Figure 2 follows the pipeline visual instruction tuning [33 68) where multimodal large language model (MLLM) receives video features with text prompt and produces the desired output. We denote the function as Tout = vFv, Tin). where Tm, and Tout are input and output texts (in tokens) and Fy is the set of features extracted from V.
Compared to image-based referring, a clear difficulty of video-based referring arises from the high dimensionality of video data. Specifically, if we define Fv as the set of dense video features (e.g.. using a pre-trained visual encoder such as the CLIP ViT-L model [1] to extract frame-wise visual features for V), the features often contain highly redundant information due to the similarity of neighboring frames. This brings two-fold drawbacks: (i) extra complexity for the MILM to deal with these vision tokens, and (ii) extra difficulty for the MILMM to locate useful information, which leads to a slower convergence. To overcome this issue, we decrease the input feature dimensionality by using various slices to replace Fv. where each slice captures important yet complementary properties of the input video. Throughout this paper, we investigate three slices: the spatial, temporal, and target-specific video features.
Spatial and temporal features. The extraction of spatial and temporal video features follows the design of Video-ChatGin
[3]. Given video clip V I Txxxixxx we use the CLIP ViT-L/14 visual encoder to cast it into frame-wise features, denoted as Firame E TT'W' x xD. where the number frames remains unchanged, W = W/s and H' = H/s are the down-sampled resolution (e.g., 14 for ViT-L/14) of the visual features, and D is the feature dimensionality.) Then, these features are fed into average pooling along the T axis (into the spatial features F e IR W' H and along the W' x H' plane (into the temporal features FT E T))x)), respectively.

---

Target-specific features. FV and FV have focused on the spatial and temporal features but ignored the referred target which may move or change during the video. To offer a compromise feature that captures spatiotemporal features, we propose an Rol (region-of-interee
tracking and selection mechanism (detailed in Section (..3 and obtain list of Rols (represented as bounding boxes) B = i,...... BM), where M is the number of Rols that are recognized by the algorithm to be important for referential understanding. We use the RolAlign method [21] to extract visual features from each Rol, producing set of target-specific features, FV =FF V. F... VVB..)
Instruction fine-tuning, When the video features are ready, we feed them with the text tokens into Artemis. The MILM follows instruction fine-tuning through three steps, gradually acquiring the ability of video-based referring. The details are described Section 3.4
3.3 Rol Tracking and Selection
Our goal is to extract compact features for video-based referring. The key lies in two factors, (i) completeness locating the referred target in every video frame, and (ii) avoiding redundancy not preserving too many features in the frames with similar semantics. We propose a simple solution upon Rol tracking and selection. As we shall see later, it offers a solid baseline for future work.
Rol tracking. We apply HQTrack [69], an off-the-shelf tracking algorithm, to localize the Rol in each input frame. The pre-trained tracking model is not fine-tuned in the training phase. Given a Rol (a bounding box) in any video frame, the tracking algorithm outputs either a bounding box or nothing (e.g., if the target is occluded) in each of the remaining frames. This step outputs raw list of Rols denoted as B' = (B',.... B'rr)) where M' can be close to the number of frames.
Step 1:
Step 2: Rol selection. Feeding all tracked frames into the MILM often incurs computational inefficiency and extra difficulties in model training. To avoid this, we select subset from B' containing M < .M' Rols, with the goal being to preserve diverse visual features using limited number of Rols. In practice, we pre-defined the target number, M, and adopt the K-means algorithm to form M clusters from the original set of M' Rols. The final Rol list, B, consists of randomly chosen Rol from each cluster.
Finding representative Rols belongs to generic topic of feature selection. On one hand, one can set a simple baseline by performing random or uniform sampling from the original set B3.. On the other hand, the information theory offers a general principle, i.e., maximize the diversity of Rols throughout the selection procedure. As demonstrated in Section 4. random and uniform sampling algorithms frequently fail to capture semantic changes throughout complex videos. By contrast, the simple K-means clustering used in Artemis significantly increases the diversity (see Appendix D)). ensuring representative video features. We conjecture that the effectiveness of feature selection is related to the quality video features; with stronger video foundation models, more sophisticated feature selection algorithms can make a larger difference. We leave this topic to future research.
Discussions.
3.4 Model Architecture and Training
The MILM is built upon Vicuna-7B v1.5 M1 open-sourced LLM We use CLIP VIT-L/14 41 to extract visual features. To feed these (024-dimensional visual tokens into the LLM, we use a learnable, two-layer MLP (1024-4096-409) to project the visual features into the 1096-dimensi language space. We always use the regressee framework to train the MLLM.
The training procedure of Artemis comprises three steps, (1) video-text pre-training, (2) video-based instruction tuning, and (3) video-based referring. The first two stages are similar to Video-LLaVA 31 but different training data are used. We set unified template.
User: <video-tokens> <instruction> Assistant:
guiding the model to output the desired answer. Here, <video-tokens> contains the spatial and temporal video features ( and F projected by MLP), and <instruction> contains the language tokens of the task description (see below).
In the first stage, <instruction> has the form of 'Witt a terse but informative summary of the following video clip." and the model outputs the overall description of the video.
A stronger LLM (e.g., with a larger number of parameters) brings marginal improvement, because at the current stage, video understanding does not rely on strong language modeling abilities.
5

---

The training data includes image-text and video-text pairs, using images as still videos. We use subset of 558K LAION-CCSBL image-tett pairs with BLIP [28] captions. sourced from CC3M 42] and refined by LLaVA 33
Additionally, we use the 702K video-text pairs provided by Video- LLaVA [1,, derived from the 703K pairs constructed by Valley [36] using Web Vid [3] Only the MLP is trained (from scratch) in this stage, initializing the alignment of vision and language. The training elapses one epoch with a learning rate of 10-3, taking about 5 hours on 8x A800 GPUs.
In the second stage, <instruction> contains specific task descriptions like 'Where is the person in the image?' and 'What is the person doing in the video?', and the model follows the instruction to produce the answer. The training data comprises the 665K image- text instruction dataset from LLaVA-11. 1.5 [33] and the 100K video-text instruction set from Video- ChatGPT [37] Both the LLM and MLP are tine-tuned in this stage. The training elapses one epoch with learning rate of x 10-5, taking about 20 hours on 8x A800 GPUs.
In the third stage, we use the curated VideoRef45K dataset to endow the model with the ability of video-based referring. The template modified as follows,
User: <video-tokens>> crefer-instruction Here, <tefer-instruction> is formulated 'What is the <region> doing during this video?' where the <region> token is replaced by the visual features extracted from the bounding box in the specified input frame, and <track-instructions contains additional information, 'This is the tracking list: <region>, , are the target-specific features (F V.B, Fl V.B"' projected by Linear) extracted from the selected Rols5] and the number of <region> token is M. In this stage, we fine-tune the LLM (with LoRA (22)), MLP and the Rol Align module. The training procedure elapses epochs with learning rate of x 10-5, taking about 3 hours on 8 A800 GPUs.
strack-instruction
Assistant:
<region>' where the <region> tokens
4 Experiments
4.1 Artemis Is Strong Baseline for Video-based Referential Understanding
Setting and metrics. We evaluate the ability of Artemis in rideo-based referring on the test set of HC-STVG [44). The video and text data training set. The test procedure uses the same instruction as in the third training stage and applics HQTrack [69] to localize the Rols in video frames. We use the standard evaluation metrics including
pre-procsseed using the same method in the
BERTScore 65 BLEU@4 4 339. METEOR 4 ROUGE L 62] CIDEr (50). and SPICE 2
Table 1: A comparison of video-based referring metries on the HC-STVG test set. 1: We use 5 key frames while using 8 frames leads to worse results.
Method
BERT Score
BLEU@4
METEOR
ROUGE L
CIDEr
SPICE
Osprey 59
0.8698
0.7
12.0
18.0
1.2
15.6
Ferret- 33
56
0.8632
0.5
10.2
17.0
1.2
11.2
Shikra- 7B
9
0.8742
1.3
11.5
19.3
3.1
13.6
Vidco- Maco-CChaGGPT
37
0.8718
1.3
10.1
20.2
5.5
11.7
Vidco-LLaVA 31
0.8639
1.7
9.8
20.8
2.6
9.1
Merlin 57
0.8829
3.3
11.3
26.0
10.5
20.1
Artemis (Ours)
0.9135
15.5
18.0
40.8
53.2
25.4
Adapting existing MLLMs for video-based referring, Due to the limited availability of research for video-based referring, we compare our model to a few recent MLLMs trained for image-based or multi-frame based referring [ The image-based referring models include Osprey [59] Ferret [56]].
DDuiing the training phase, we randomly select a frame with an annotated region bounding box as the input and employ the tracking module to locate the bounding box of the referred object in the sampled frames. During inference, track the given region generate the tracking list.
'The comparison against these methods is not totally fair because they have not been trained for based-based referring. We mainly use the comparison to claim that video- based referring is important and challenging, yet image-based MLLMS cannot do it well.

---

What this object's <region> behavior?
The woman in black clothes gets out of the car and walks to door of building.
Artemis
What this woman's <region> behavior?
The woman walking down the sidewalk.
Merlm
What the <region>'s action this video?
The red clothes walks the in black clothes and sits down.
Artemis
Jo
What is the woman's 's <region>> action in this video?
The woman, who is wearing a red dress, is sitting on the bed and massaging the man's feet.
Merli
Figure 3: Artemis and Merlin for video- based referring. Note that Merlin needs the semantic class of <region> be provided while Artemis does In each case, the orange rectangle indicates the input <region>, blue rectangles are the tracked Rols, and yellow stars label the selected Rols. Red and green texts indicate incorrect and correct answers, respectively. This figure best viewed in color.
and Shikra 99. For each video, we extract key frames with Rols produced by HQTrack and ask the trained model "What is the target <region> doing?" in the way the models are familiar with. Finally, we use GPT-3.5-Turbo to summarize the answers into the overall description of target. The multi-frame based reference model is Merlin (57) which receives key video frames and Rols and produces the overall description. The selection of key frames consistent with Artemis. To compare with MLLMs that are trained for video understanding, such as Video-LLaVA (II we follow [43] to draw red rectangle to mark the referred object in each key frame of the video. Then, feed the rendered video the models and ask the question "What is the target indicated by the red rectangle doing?".
[37] and
Quantitative results, and necessity of native video-based referring.
Inter-Prame Roll Information
Enbropy
10.268
The numbers are summarized Table [ Artemis outperforms other MLLMs in each single evaluation metric. Note that the advantage is significant for some metrics, e.g., BLEU4. Please refer Figure for representative examples. In Figure (see Appendix C). show the behavior of the methodology using a standalone LLM (e.g., GPT-3.5-
Tracking
Tracking
78.45
Turbo) upon image- based referring outputs. The image-based models tend to describe individual moments rather than an entire video; based these inputs, the LLM cannot realize video descriptions and is sometimes confused to hallucinate what never happens in the video. The comparison validates the necessity of training native model (i.e,, directly on the instruction data for video-based referring) like what Artemis has done. Equipping with such fundamental ability of video understanding at finer level, Artemis can perform even more complex video understanding tasks, as shown in Section 4.2
72.88
83.98
Unitumm Random K-moons
Figure 4: Rol manipulation increases the informative- ness and diversity of Rols. See Appendix D for details.
Qualitative results. We display several representative examples of video-based referring in Figure The output of Artemis is comprehensive (especially compared to other MLLMs, see Figure D often containing -nn-grri actions of the target. This mainly concerns the compact video features extracted by the Rol tracking and selection algorithm that extracts key features for understanding.
Ablative studies for target-specific video features. The key to extracting compact target-specific features lies in Rol tracking and selection. To validate this, we ablate two key parameters: the strategy of Rol selection and the number of preserved Rols. Table we define baseline for region of interest. For each frame of video containing object of interest, enclose the object's location
7

---

with red rectangle and encode the video using Cheo--hhttPPT. The system is then queried with "What the object in the red rectangle doing this video?". As illustrated in Table 2 K-means clustering emerges
simple yet effective approach for Rol selection, whereas random or uniform sampling fails to consistently capture representative Rols. To validate the effectiveness of Rol features in representing the object, replace the <region> features with visual features (the [CLS]] token from CLIP-ViT--/144 in key frames, a variation we refer to as decline compared to Artemis indicates that Rol features are of higher quality, whole-frame features are more susceptible to background noise. Additionally, Table demonstrates the importance of using multiple Rols for understanding the full video content. While retaining more keyframes slows down both training and inference, it also introduces redundant information, leanding to a slight drop in performance metries. Empirically, using 4 Rols provides the optimal balance on the HC-STVG test set, although increasing the number of Rols may be beneficial for more complex videos.
The performance
From the information theory, we show that Rol tracking and selection improve informativeness (in terms of entropy) and diversity (in terms of frame-level difference) of the target-specific features in Figure 4 As shown in Figure 5, Rol tracking and selection gradually improve the comprehensiveness of the referring results.
Table 2: Ablation on different Rol selection methods. Results are reported on HC-STVG.
Table 3: Ablation on the number of selected Rols. Results reported HC-STVG.
BLFU184
METEOR
ROUGE
CIDEr
SPICE
Rol
Bert Score
BLEU@4
METEOR
ROUGE_L
buseline
wio
13.9
11.2
13.9
16.9
16.3
16.9
39.1
18.2
42.1
43.7
23.2
23.1
21.4
2
0.9113
0.9114
13.9
14.6
16.9
17.5
39.0
39.5
43.5
43.3
23.7
23.6
Unitcemly
14.2
17.2
39.4
44.5
23.6
Randomly
14.3
17.1
40.0
46.5
26.2
6
0.9122
14.3
16.9
39.5
46.2
23.9
Chusering
14.6
17.4
40.2
47.2
23.9
0.9110
14.0
17.0
39.0
43.1
23.6
W hat does the <region= do during this vidoo?
Only input queried Rol
woman holdings gun and the process of shooting
Tracking and uniformly choosing Rols
A woman walkss the room takes the gum out. table.
hand, and put
Tracking & Clustering choose 4 Rois
The woman in the red dress walles to the table. takes the gui, and put in the man't 's hand.
How does the <regian> act the video?
Only input the queried Rol
The woman skirt walks man then stops
Tracking and randomly choosing Rols
The woman brown dress walks to man the white shirt and stops
The woman white skirt and wallss away.
Tracking & Clustering choose Rols
Figure 5: How Rol tracking and selection gradually improves the quality of video-based referring. In each example, the orange rectangle indicates the input <region>, blue rectangles the tracked Rols, and green and yellow stars label the uniformly sampled and K-means selected Rols, respectively. Red and green texts highlight the incorrect and correct outputs. This figure is best viewed in color.

---

The man in black clo- thes turns his head and speaks to the woman in while, then lums
The man in the black suit walks the door and opens the door.
The man in black clothes turns around and sits down.
The in black clothes raises his hand and shakes hands with the opposile man.
The man in black clothes sits down and puls his hands the table.
head again.
[]]
[2]
[3]
[4]
P Prompt: divided a video into 5 segments and described a specific object in each of these segments. Please connect these 5 descriptions in chronological order to form complete senlence describing what the object did in the video.
T Thess are the 5 descriptions: [1] [2] [3] [4] [5]
The man in black clothes turn his head and speaks to the woman in white, then turns his head again, walks to the door and opens it, lums aro und and sits down, rasses his hand and shakes hands with the opposite man, and finally sits down and puts his hands on the table.
Figure 7: Example of long video understanding. We apply Artemis to output descriptions for segmented video clips and integrate them using an LLM (GPr-3.5-Turbo in this example).
4.2 Artemis Is a Building Block for Complex Video Understanding
With a strong ability of video-based referring, Artemis serves as a building block that strengthens the existing video-based MLLMs in complex video understanding.
Multi-round video understanding with grounding. Multi-round dialogues, especially answering logically [46]. is an important yet challeng- ing topic for MLLMs. In Figure 6 and Figure 14 in Ap- we show that Artemis's 's referential understanding ability can be combined with image-based grounding mod- els (e.g., GroundingDINO (34) to answer multi found -faus-of-questions, where the entities mentioned in the video-based referring result is located and fed into the next round of video-based referring quest, allowing for more complex interactions.
0 wommn black clothes walks the table and is down.
What does the cregion> do im this video?
lated chain-of-questtone
pendix E
Who with her?
A met in grey suit sits with bee.
How does the <reqion> thiyfidno??
Ggo
The gney clothes looks paper in hand, then looks up woman
Long video understanding with text summarization. Target-centric understanding of long videos is major chal- lenge for existing video-based MLLMs. The difficulty mainly lies in extracting compact video features (to feed into the MLLM) and tracking the target throughout the long video. We offer a simple solution that first segments the video into shorter clips, applies Artemis for understanding these clips, and applies an off- off-the-shelf LLM (e.g., GPT-3.5-Turbo) for summariza- tion. As shown in Figure and Figure T2 in Appendix F. the final output offers a comprehensive understanding. To our knowledge, this function was not achieved by existing MI..Ms.
Figure 6: An example of multi-round, video-based referring by integrating Artemis with GroundingDINO [34].
Video question answering. Lastly, we show that Artemis can perform general video question answering. We test the trained model on the Video-ChatGPT test set [37] and the other three benchmarks (i.e., MSVD-QA [8). MSRVTT-QA [53], and ActivityNe--QA 66 (88) where their training sets was not used to train Artemis. Results are summarized in Tables Artemis shows competitive performance among few recent MLLMs. These results inspire us that (i) an MLLM trained for finer-level video understanding can seamlessly transfer leveel tasks, and (ii) extracting compact video features also benefits video question answering.
5
Conclusion
This paper proposes a challenging setting for video-based referring and establishes an effective MILMM named Artemis. Compared to existing methods, Artemis can understand human intention from simpler inputs (a text prompt and single-frame bounding box) and comprehensively describe the target's 's action in complex video. At the core of Artemis is an Rol tracking and selection

---

Table 4: Left: Video QA on Video-ChatGPP. Metries: correctness (CN), detail orientation (DO),
contextual understanding (CU), temporal understanding (TU), consistency (CC). Right: Zero-shot video QA on MSVD-QA, MSRVTT-QA, and ActivityNet-QA. Metrics: accuracy (Acc.), score (Sc.).
Method
CN
DO
CU
TU
Method
MSVD
MSRVTT
ActivityNet Sc.
Acc.
Sc.
Acc.
Sc.
Ace.
Video-Chat
29
2.23
2.50
2.53
1.94
2.24
FrozenBiLM
322
24.7
I.aMMAAdapter
62
19
2.03
2.32
2.30
1.98
2.15
Video Chat A9
56.3
2.8
45.0
2,5
26.5
2.2
Video-LLaMA 66]
1.96
2.18
2.16
1.82
1.79
LLaMA-Adaper
02
54.9
3.1
43.8
2.7
34.2
Video-ChaIGPT
2.40
2.52
2.62
1.98
2.37
Video- Lalc
61
51.6
2.5
29.6
1.8
12.4
1.1
Valley-v3 36
2.43
2.13
2.86
2.04
2.45
Video-ChatGPT
37
64.9
3.3
49.3
2.8
35.2
2.7
Valley [6
60.5
3.3
51.1
2.9
45.1
3.2
Artemis (Ours)
2.69
2.55
3.04
2.24
2.70
Artemis (Ours)
72.1
3.9
56.7
3.2
39.3
2.9
mechanism to extract compact video features. Artemis shows advantages in video-based referring in VideoRef45K and transfers the ability to general video understanding, including being integrated with other modules for more complex tasks. We hope that Artemis can serve solid baseline to facilitate the research in fine-level video understanding.
First, Artemis relies on tracking algorithm to generate the Rols; however, the tracking algorithm may produce inaccurate results and can confuse Artemis- see an example Figure (top) in Appendix E Second, Artemis also suffers from the issues of general video-basee understanding, such as the visual content accurately see an example in Figure 13 (bottom) where Artemis accurately predicts the movement of the target but reverses the temporal order.
Limitations.
aliasing problem, which can affect the model's ability to describe the
Acknowledgments
This work supported by National Natural Science Foundation of China (NSFC) under Grant No.62225208 and CAS Project for Young Scientists Basic Research under Grant No. YSBR-117.
References
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-2333 2022.
[2] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould, Spice:
Semantic propositional image caption evaluation. In Computer Vision-EECC 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part 14, pages 382-398. Springer, 2016.
[3] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint vidco and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1728-1738, 2021.
[4] Satanjeev Banerjee and Alon Lavic. Metcor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the act workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65-72, 2005.
[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learners. Advances in neural information processing systems, 13:1897-1 2020.
[6] Fabian Caba Heilbron, Victor Escorcia, Bemard Ghanem, and Juan Carlos Niebles. Activitynet: A aagee-scale video benchmark for human activity understanding. Proceedings of the ieee conference on computer vision and pattern recognition, pages 961 1109700 2015.
[7] Chi Chen, Ruoyu Qin, Fuwen Luo. Xiaoyue Mi, Peng Li, Maosong Sun, and Yang Liu. Position Enhaaced Visual Instruction Tuning for Multimodal Large Language Models. arXiv preprint 30oe:223881333777 August 2023.
[8] David Chen and William Dolan. Collecting highly parallel data for paraphrase evaluation. Proceedings of the 49th annual meeting of the association for computational linguisties: human language technologies. pages 190-200. 2011.
10

---

[9] Keqin Chen, Zhao Zhang, Weili Zeng. Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic. arXiv preprint arXiv:2306. 15795, July 2023.
[10] Zhenfang Chen, Lin Ma, Wenhan I Luo, and Kwan-Yea K Wong. Weakly- spatio-temporally grounding natural sentence in video. arXiv preprini arXiv: 02..:0299.0 2019.
[11] Wei-Liin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chathot impressing gpt-4 with 90%* chatgpt quality. See https://vvicna. Imsys. org (accessed 14 April 2023), 2023.
[12] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint wrXiv:2204.02311 2022.
[13] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetume language models. arXiv preprint X2 11416, 2022.
[14] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards generas vision-language models with instruction luning. arXiv preprint arXiv:2305, 06500, 2023.
[15] Jacob Devlin. Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint 1810::881000480: 2018.
[16] Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, and Chen Change Loy. Mevis: A large-scale benchmark for video segmentation with motion expressions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2694-2705 2004, 2023.
[17] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold. Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition at scale. arXiv preprint arXiv:2010.1199 2020.
[18] Heng Fan, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Hexin Bai, Yong Xu, Chunyuan Liao, and Haibin Ling. Lasot: high-quality benchmark for large-scale single object tracking. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5374-5383, 2019.
[19] Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He. Xiangyu Yue, et al. Llama-adapter v2: Parameter-efficient visual instruction model, arXiv preprint arXiv:2304. 15010, 2023.
[20] Kirill Gavrilyuk. Amir Ghodrati, Zhenyang Li, and Cees GM Snoek. Actor and action video segmentation from a sentence. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5958-500 1958-5966,20 2018.
[21] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969. 2017.
[22] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen Yuanzhi Li, Shean Wang. Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106. 2021.
[23] Shiyu Hu, Dailing Zhang. Xiaokun Feng, Xuchen Li, Xin Zhao, Kaiqi Huang, et al. A multi-modal global instance tracking benchmark (mgit): Better locating target complex relationship. Advances in Neural Information Processing Systems, 36, 2024.
and causal
[24] Lianghua Huang, Xin Zhao, and Kaiqi Huang. Got-10k: A large high-diversity benchmark for generic object tracking in the wild. IEEE transactions pattern analysis and machine intelligence, 43652:1562- 1577, 2019.
[25] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitchead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment Anything. arXiv preprint 2304:22304.02443, April 2023.
[26] Xin Lai, Zhuotao Tian. Yukang Chen. Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia, LISA: Reasoning Segmentation via Large Language Model. arXiv preprint 2308 00692, August 2023.
[27] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping Language-Image Pre- training with Frozen Image Encoders and Large Language Models. arXiv preprint 12X.522301.1259, May 2023.

---

[28] Jumnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-imag image tre-training for unified vision-language understanding and generation. In International Conference Machine Learning, pages 2288-12990, PMLR, 2022.
[29] KunChang Li, Yinan He, Yi Wang. Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang. Limin Wang, and Yu Qiao. Videnchat: Chat-centric video understanding, 2024.
[30] Zhaowei Li, Qi Xu, Dong Zhang, Hang Song, Yiqing Cai, Qi Qi. Ran Zhou, Junting Pan, Zefeng Li, Van Tu Vu, Zhida Huang, and Tao Wang. Groundinggptlanguuge enhanced multi- grounding model, 2024.
[31] Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-Ilava: Learning united visual representation by alignment hefore projection, 2023.
[32] Chin- Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74- 81, 2004.
[33] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual Instruction Tuning, arXiv preprint 20Xiv:2.304008483 April 2023.
[34] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang. Chunyuan Li, Jianwei Yang. Hang Su, Jun Zhu, al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection, arXiv preprint WxXi:2303.05449 2023.
[35] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[36] Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Da Li, Pengcheng Lu, Tao Wang, Linmei Hu, Minghui Qiu, and Zhongyu Wei. Valley: Video assistant with large language model enhanced ability, 2023.
[37] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models, 2023.
[38] Shehan Munasinghe. Rusiru Thushara, Muhammad Maaz, Hanoona Abdul Rasheed, Salman Khan, Mubarak Shah, and Fahad Khan. Pg-video-Ilaval Pixel grounding large video-language models, 2023.
[39] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318, 2002.
[40] Zhiliang Peng, Wenhui Wang. Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding Multimodal Large Language Models to the World. arXiv preprint arXiv:2306. 14824, July 2023.
[41] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PMLR. 2021.
[42] Piyush Sharma, Nan Ding, Scbastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pages 2556-2565, 2018.
[43] Aleksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi. What does clip know about a red circle? visual prompt engineering for vlms. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 1 1987711997, 2023.
[44] Zongheng Tang, Yue Liao, Si Liu, Guanbin Li, Xiaojie Jin, Hongxu Jiang. Qian Yu, and Dong Xu. Human- centric mettio-temporl video grounding with visual transformers. IEEE Transactions on Circuits and Systems for Video Technology, 32(12):8238 8249, 2021.
[45] Romal Thoppilan, Daniel De Freitas, Jamic Hall, Noam Shazcer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint 2201::2200.82333 2022.
[46] Yunjie Tian, Tianren Ma. Lingxi Xie, Jihao Qiu, Xi Tang, Yuan Zhang, Jianbin Jiao, Qi Tian, and Qixiang Ye. Chatterbox: Multi-round multimodal referring and grounding. arXiv preprint 133 2024.
12

---

[47] Yunjie Tian, Lingxi Xie, Jihao Qiu, Jianbin Jiao, Yaowei Wang. Qi Tian, and Qixiang Ye. Fast-itpn: Integrally pre-trained transfommer pyramid network with token migration. IEEE Trunsactions on Pattern Analysis and Machine Intelligence, 2024.
[48] Yunjie Tian, Lingxi Xie, Zhaozhi Wang, Longhui Wei, Xiaopeng Zhang, Jianbin Jiao, Yaowei Wang. Qi Tian, and Qixiang Ye. Integrally Pre- Transformer Pyramid Networks. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18610-1866 IEEE, June 2023.
[49] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1 13971, 2023.
[50] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus- image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1166-4575,200 2015.
[SI] Weihan Wang, Qingsong Lv. Wenmeng Yu, Wenyi Hong, Ji Qi. Yan Wang, Junhui Ji, Zhuoyi Yang. Lei Zhao, Xixuan Song, et al. Cogylm: Visual expert for pretrained language models. arXiv preprint 1xXiv:331.003777 2023.
[52] Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng. Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks, arXiv preprint vrXiv:2305.11177 2023.
[53] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 5288-5296, 2016.
[54] Shiyu Xuan, Qingpei Guo, Ming Yang, and Shiliang Zhang. Pink: Unveiling the Power of Referential Comprehension for Multi modal LLMs. arXiv preprint arXiv:2310.0055: October 2023.
[55] Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Zero-shot video question answering via frozen bidirectional language models. Advances in Neural Information Processing Systems, 35:124-141, 2022.
[56] Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang. Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and Ground Anything Anywhere at Any Granularity. arXiv preprint arXiv:2310.0700,, October 2023.
[57] En Yu, Liang Zhao, Yana Wei, Jinrong Yang, Dongming Wa, Lingyu Kong, Haoran Wei, Tiancai Wang. Zheng Ge. Xiangyu Zhang, and Wenbing Tao. Merlin:empowering multimodal Ilm with foresight minds, 2023.
[58] Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yucting Zhuang, and Dacheng Tao. Activitynet-ga: dataset for understanding complex web videos via question answering, 2019.
[59] Yuqian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xinjic Luo, Chi Qin, Lei Zhang, and Jianke Zhu. Osprey: Pixel understanding with visual instruction tuning, 2024.
[60] Aohan Zeng, Xiao Liu, Zhengxiao Du Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang. Yifan Xu, Wendi Zheng, Xiao Xia, et al. Glm-130h: An open bilingual pre-trained model. arXiv preprint 22000444 2022.
[61] Hang Zhang, Xin Li, and Lidong Bing. Video-llama: instruction-tuner audio-visual language model for vidco understanding. 2023.
[62] Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303. 16199, 2023.
[63] Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo. GPT4Rol: Instruction Tuning Large Language Model on Region-of-Interest arXiv preprint arXiv:2307.0300 000000007.00 July 2023.
[64] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan Mona Diab, Xian Li. Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint 220:2220500006. 2022.
13

---

[65] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXtv:1904.090 09675, 2019.
[66] Xiaosong Zhang, Yunjie Tian, Lingxi Xie, Wei Huang. Qi Dai, Qixiang Ye, and Qi Tian. Hivit: A simpler and more efficient design of hierarchical vision transformer. In The Eleventh International Conference Learning Represemtations, 2022.
[67] Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang. Yatian Pang. Wenhao Jiang, Junwu Zhang, Zongwei Li, Wancai Zhang, Zhifeng Li, Wei Liu, and Yuan. Languagebind: Extending video-language pretraining to n-modality by language saaseag--baeed semantic alignment, 2024.
[68] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint 2rX1:2304.11599. 2023.
[69] Jiawen Zhu, Zhenyu Chen, Zeqi Hao, Shijie Chang. Lu Zhang. Dong Wang. Huchuan Lu, Bin Luo, Jun Yan He, Jin-Peng Lan, et al. Tracking anything in high quality. arXiv preprint arXiv:2307.1397 2023.
14

---

A Data Curation
We organized 7 existing video datasets and performed a careful data curation procedure, result- ing in the VideoRef45K benchmark, which comprises 45K video question-answer pairs. The datasets include HC-STVG [44), VID-Sentence 100, A2D Sentences
20 LaSOT 18] MeViS [16).
GOT1OK 24), and MGIT
23
HC-STVG is movie clip dataset that provides tracking sequences and textual annotations describing the person's actions during a certain period. We use the training portion, which contains approximately 10K video clips as our training data. The validation portion, containing 3,400 video clips, evaluates Artemis's s ability. The original tracking sequences in HC-STVG are of poor quality, so we use the off-the-shelf tracking model HQTrack [69] regenerate the tracking sequences and remove some low-quality bounding boxes. To prevent tracking target deviation caused by cross-frame tracking, select the first and middle frames with ground truth bounding boxes annotated by the HC-STVG dataset as the referring frames for HQTrack to generate tracking lists for the whole video. We then compare these generated tracking lists with the HC-STVG annotations and exclude the frames with low IoU between the generated and annotated bounding boxes from the tracking lists.
A2D Sentences provides tracking sequences and captions for different objects, but the tracking sequences are only 3 frames long. address this, we use HQTrack to regenerate the sequences and obtain longer tracking frames, extending them to 20 frames.
LaSOT provides caption for object along with its tracking sequence. However, LaSOT videos usually long, and the captions for the entire video are generic. To address this, we extract three segments of 10 seconds each from the entire video for our training data.
GOT-10K is a tracking dataset that provides tracking sequences of objects and their categories, actions, and adverbs describing the actions. We concatenate these elements to describe the object's action in the video, egg, "bear is slowly walking."
MGIT videos are typically long, with annotations indicating the object's actions at different time intervals. We extract these segments as our training data.
For MeViS and VID-Sentence, we did not perform any special processing. We converted the mask annotations of Me ViS into bounding boxes.
Figure shows some examples of VideoRef45K.
The person standing next to the horse releases his hand and holds the rider, then takes the horse and walks away.
standing woman goes
Panda climbs up and then falls down.
Zebra runs across lake then runs back.
bicycle ridden by a person is running from left to right the hillside.
Black car operating forward and changing lanes.
Figure 8: Some examples of VideoRef45K.
The total dataset composition is summarized in Table
15

---

Table 5: Curated datasets for Video-based Referring.
Dataset
Video Clips
Q&A Pairs
HC-STVG 44
10105
10105
MeViS [6
1644
4489
A2D Sentences 20
3017
5359
LaSOT [8]
2640
7920
VID Sentence 10
4045
7654
GOT-IOK 24
9090
9090
MGIT 23
105
614
VideoRef45K
30646
45231
Through the above processing steps, we obtained the VideoRef45K dataset. This dataset includes captions describing the actions of objects in videos, along with their corresponding tracking sequences. To utilize VideoRef45K, we created template of questions prompt the language model (LLM) to answer what the referred object did in a video. The template consists "<track-instructionn" The during this video?', and we utilized GPT-3.5-Turbe to generate the refer instruction template. The "<track-instruction>> contains additional tracking lists help the LLM perceive the referred object, such as 'This is the region' s tracking list: <region> created several <<rrack-instruction>> options as the track instruction template. During training, domly sampled '<reffr-instruutiion>> from the refer instruction template and 'ttrak- -iatckiisttuutton>>> from the track instruction template. The <<refer-intrructio>>> and the strack-instruction> are then concatenated as <rreerrisstuciioooccraaiiiiiiiie
prefer-instruction>> and is formulated 'What is the <region> doing
<regions reggion>.. ? We
to formulate the text prompt.
B Implementation Details
We report the detailed training hyper-parameters of Artemis in Table 6
Table 6: Training syper-parameten of Artemis.
Configuration
Pre-training
Instruction Tuning
Referring Instruction Tuning
VIT init.
CLIP-L/14
CLIP-L/14
CLIP-L/14
LLM init.
Vicuna-7B-vI.55
Vicuna-7B-vI.5
Artemis-Finetune Artemis-Finetune
Projection init.
random
Artemis-Pretrain
Image resolution
224
224
224
Video feature length
356
356
356
LLM sequence length
2048
2048
2048
Optimizer
AdamW
AdamW
Adam
Peak learning rate
x 10-3
2 x 10-5
4 x 10
Minimum learning rate
0
10
Learning rate schedule
cosine decay
cosine decay 0
constant
Weight decay
0
0
LoRA rank
None
None
16
Number input trackbox
None
None
Number choosen bhox
None
None
Training steps
4927
5979
142
Global batch size
256
128
48
Numerical precision
6
bfloat16
float16
C Image-based MILMS for Video-based Referring
Existing multimodal language models (MLLMs) for video understanding rarely possess video-based referring capabilities. To address this limitation, we leverage image-based MILMs (such as Os- prey [59] and Ferret (56)) to perform image referring on keyframes within the video independently. Subsequently, integrate these outputs using GPT-3.5-Turbe to achieve vidco-basee referring, as
16

---

depicted in Figure 9 However, these models struggle to perceive the action and behavior of the given region of interest (Rol) as effectively as Artemis.
woman is standing in the image, positioned towards the right side. She is wearing black shirt and appears to be looking at a man.
woman standing in the image, positioned towards the right side. She is wearing black shirt and appears to be looking at man.
woman, possibly mother, standing next to She is eeaing black shirt and appears to be looking at the man.
A woman is standing next to man. They appear to be in pleasant and relaxed state, possibly enjoying a special occasion or a casual conversation.
A woman, possibly grandmother, is standing next to man. She wearing a black shirt and appears to be looking at the man.
Digerry
Throughout the video, woman, possibly mother or grandmother, wearing black shirt, stands next a man, smiling and engaging in pleasant interaction, potentially enjoying a special occasion.
GPI-ES
The region is woman dressed in black evening gown. Her attire suggests that she might be attending formal event or gathering. The woman's presence contributes the elegance of scene.
I The region is woman. She stands in doorway. The overall image appears be set a luxurious room with a man a suit and woman in black dress. The image shows formal event gathering.
The region is woman. Her attire suggests that she is attending formal ent,, perhaps a dinner or party.
The region is a woman who be looking away. appears that she is involved in conv ervettttion that doesn't tt require her to face the camera. Her body language and suggest she might be listening something.
The region is a woman who seems to be looking away. appears that she involved in conversed conversaiion that doesn't 't requii her to face the camera. Her body language and suggest she might be listening something.
EXITIC
The woman's 's dress, which is worn at such events, contributes to the elegance the occasion.
I
Throughout the video, a woman dressed in black wening gown attends formal event or gathering, standing in a doorway and adding elegance to the soene. She is seen listening or observing. contributigg an element of mystery and intrigue to the overall image.
GPT-S5
The woman black clothes walks the man vest stops.
Armazia
Figure 9: A comparison of video-based referring between image-based based MILMS and Artemis, GPT- 3.5-Turbo is used to integrate the 5 independent outputs from the image-based MLLMs.
D How Rol Tracking and Selection Improve the Results
To better demonstrate the effectiveness of our method, we conducted the following experiments. Firstly, computed the attention values between different Rol tokens and temporal tokens. We observed that the tracking tokens added through tracking compensated for the weak perception aspect of the initial video tokens, illustrated in Figure [0] Subsequently, depicted in Figure 4 (top), calculated the information entropy before and after adding the tracking list. Upon adding the tracking list, the overall amount of Rol information fed into the MILMM increased by 20.3%. Additionally, computed the inter-frame differences of the boxes chosen from the tracking list. As shown in Figure (bottom), the K-means clustering method selects Rols with greater differences than random and average selection. This enables the MILMM to better perceive the action changes of Rols throughout the entire video.
E
More Qualitative Examples
Figure TT shows more examples of Artemis's outputs of video-based referring. Figure T2 shows
more examples of target-centric video understanding with text summarization. Figure 13 shows
17

---

Initial Rol Token
Tracking Rol Tokens
Temporal Tokens
Figure 10: Attention map between Rol tokens and temporal tokens.
is behavior <regian=in this video?
The woman in white clothes tums her head glances at the man behind her, then tums ber head back.
ls <region>dooi doing during video?
The woman in the black dress takes sip of wine and puts the glass down.
How does the <regian> in this video?
rovi move
The man in the black swit walks to the table and down.
Figure 11: Examples of video-based referring generated by Artemis.
Artemis's failure cases, revealing the limitation of Artemis. Figure 14 shows mores example of combining Artemis with off-the-shelf grounding model, e.g. GroundingDINO to answer multi-round chain-of-questioos conveniently.
18

---

15
105
185
275
305
556
The man the tie tums around and walks to the woman in the hat.
The man the red tie turns and walks up the stairs.
The man in the tie turns and walks to the door.
The man in the vest. goes to the table and sits down.
Prompt: divided a video into 4 segments and described a specific object in each of these segments.
Please connect these descriptions chronological order to form a complete sentence describing what the object did the video. These the descriptions: [1] [2] [3] [4]
The man in the red tie turns around and walks to the woman in the hat, then he turns and walks up the stairs, followed by tuming again and walking to the door, and finally, the man in the vest goes to the table and sits down.
Figure 12: The example of long video understanding generated by Artemis.
W hat is the mation of the <region> in this video?
The man in white clothes opens the door and gets out of the car.
W hatt the <region> doing during this video?
The in blue clothes comes down the stairs and turns the left.
Figure 13: Failure cases of Artemis. Top: The tracking module generates inaccurate Rol list, misleading Artemis's understanding. Bottom: Spatial-temporal aliasing in video hinders Artemis to perceive objects.
19

---

How does the <region> act this video?
The woman in the blue coat opens the door and gets out of the car.
W ho also the car?
A man is also in the car.
W hat is the action of the man <region>
this video?
The man in the car turns his head and looks the woman the blue coat, then his head again.
GroundingDINO
is the motion of <region> this video?
The in the red tie walks up the stairs.
ho stands with the man?
woman stands with the man.
How does the woman <region> move in this video?
(
The woman in the hat turns and walks the stairs.
Grounding INO
What is in the woman's hand?
The woman is holding a bag in her hand.
W hat is the color of the bag?
The color of the bag is white.
W is the <region> do in this video?
The woman in white clothes turns around and sits on the bed.
o
Wh also this mom?
man also this room.
hat action the man <region> perform in this video?
The man in short sheeves es goes to the bed and sits down.
GroundingD
Figure 14: Examples of multi- multi-round video understanding with grounding generated by Artemis.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction provide a comprchensive overview of the background and motivation of this study, effectively outlining its main contributions, thus accurately reflecting the paper's 's scope and significance.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We primarily focused on discussing the limitations associated with this study in Sec. 5
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
21

---

Answer: [Yes]
Justification: The paper includes the theoretical result. Notably, it covers the results of information entropy and inter-frame Rol difference, ensuring completeness and accuracy in theoretical presentation.
Guidelines:
The answer NA means that the paper does not include theoretical results, All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All information regarding the key contribution of this paper including architec- tural, data, and experimental configurations, have be fully disclosed.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
22

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
The supplementary material submitted with the manuscript includes scripts necessary to faithfully reproduce the main experimental results and the anonymous link includes the code and data. Instructions for running the code are also provided within the anonymous link.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
Cooeeee//ddSSSbbbbmca popoddSSbmisssooPoo//...o ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
The paper specifies detailed experimental configurations in Section 4.1 in main manuscript and Section[A]an and Section B in Appendix, providing readers with essential information to comprehend the results.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Generally, papers about MLLMs do not include error bars, and we have found that the MILM's training quite stable with little variation across multiple runs. However, we have provided the code, hyperparameters, and random seeds used our experiments to facilitate the reproducibility of our findings.
Guidelines:
23

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: All experiments were carried out on an 8x A800 GPU server, as detailed at Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: After carefully reviewing the referenced document, we certify that the research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethies.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]

---

Justification: The paper primarily focuses on multimodal large language models about video understanding using publicly available datasets that have undergone thorough validation. This study only serves as a foundation model for video-based referring, which is not directly applicable for practical scenarios until now.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The proposed models are trained on benchmark datasets such as HC-STVG. VID-Sentence, and S2D Sentences, etc. These datasets have been extensively used in the computer vision community and have undergone comprehensive safety risk assessments.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: In the paper, we clearly specified the datasets and code sources used, and pro- vided appropriate citations the reference section. Additionally, we ensured transparency by including the original sources of any modified code files, making the changes traceable.
25

---

Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license ge.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We have included the code, data, along with detailed usage instructions, in anonymous link. After the review process is completed, will make the code and data publicly available to the community.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This study does not involve any crowdsourcing experiments or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
26

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No crowdsourcing experiments research with human subjects were involved in this study. All experiments were conducted using code and GPU servers.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "4cRJiF676R", "reviewer": "Reviewer_U8HM", "summary": "The paper proposes to bring fine-grained understanding to multimodal LLMs (MLLMs) by introducing video-based referential understanding task. The paper motivates with the drawbacks of current image- and video-based MLLMs, and the need for region-specific features to answer region-specific questions . The proposed approach expands the set of video-level features with target region-specific features (box-level prompts) via tracking (HoITrack), alignment (RoIAlign) and selection (clustering). The paper ablates the effectiveness of tracking and selection criteria for improved performance.", "questions": "See Weaknesses", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The paper is well-written, clear, and the motivation for target-specific features is well presented", "weaknesses": "- A fair baseline\n    - It’s great that the paper provides comparison with image-based MLLMs by extending them to videos and using an LLM to obtain video-level answers\n    - As for the multi-frame approach, it appears that MERLIN is the closest baseline approach. And there seems to be some intersection of pre-training data used for Artemis and MERLIN that includes GOT10K, LaSOT, MeViS\n    - But the evaluation of the video-based referring ability is done on the test set of HC-STVG (lines 197-198), and the train set of HC-STVG is included in the pre-training data for Artemis but not for MERLIN\n    - Since MERLIN was not trained or fine-tuned on HC-STVG, it does not seem to be a fair, apples-to-apples comparison that MERLIN be evaluated on the test set of HC-STVG\n    - On top of that, datasets have biases which includes different label spaces, scene and setting (e.g. HC-STVG is collected from movies), annotation gathering (which can also result in different caption distribution, and hence biasing the eval metrics)  \n- Ablation\n    - Assuming “w/o” in Table 2 means that there is no RoI selection (not mentioned or defined in the text), which I’m assuming to mean that there is no <track-instruction>. If the above assumption is true, “w/o” sets up a strong baseline (even better than MERLIN)\n    - In any case, it seems that a careful ablation study is missing\n        - (1) w/o <track-instruction>\n        - (2) w/ <track-instruction> but where <region> features are not RoI features but the key-frame features. This is to establish whether the improvement is brought forth by key-frame selection or region-level features specifically   \n- Visualization and sanity check post-training\n    - Do the authors have an example of a video with multiple different regions? Mainly this is to inspect how the model response changes with selection of different regions in the same video, and whether it doesn’t degenerates to the same response?\n    - Less important, but do the authors have an impression on if the video is reversed, does the caption change?\n \n- Human evaluation\n    - Lastly, to substantiate the effectiveness of the approach, did the authors think about a human evaluation study on accuracy / relevance of predicted captions to the video-question pair?\n        - This could also be done within your group and with anonymized predictions (meaning the human evaluator doesn’t know what prediction is from what model)\n\n\n## Minor\n- A bit more about the architectural details would have been great (at least in supplemental), especially the tokenization process and whether / how start-end tokens for <instruction> were used\n- Do the authors have some statistics on the object category of <region> in the test set of HC-STVG?\n    - This is mainly to identify what biases the models are dealing with, and whether those biases skew heavily in one direction. For example, \"person\" may be the majority category\n    - Similarly, any statistics of actions being performed and asked in questions?\n- Lastly, do the authors have a quantitative breakdown of performance to understand where the model fails?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the contributions of our work. We deeply appreciate your constructive comments and have provided point-to-point responses below. Further comments are welcomed.\n\n**Q1:** *A fair baseline.*\n\n**A1**: During the rebuttal, we fine-tuned Merlin on the same data (*i.e.* VideoRef45K) using LoRA (same as Artemis). We extracted 5 key frames with a bounding box for each clip, aligning with Artemis which uses 5 RoI tokens.\n\nResults are shown in the following table. Fine-tuning brings a significant improvement to Merlin, but the metrics are still lower than Artemis. The reason is two-fold. (1) Artemis introduces RoI tracking, clustering, and selection to obtain accurate localization, so that extracted visual features are of higher quality. (2) Merlin only sees 3--8 frames, while Artemis' encoding method (see Lines 116--122) preserves richer information.\n\n|Method|BLEU@4|METEOR|ROUGE_L|CIDEr|SPICE|\n|:---:|:---:|:---:|:---:|:---:|:---:|\n|Merlin|3.3|11.3|26.0|10.5|20.1|\n|Merlin (ft)|9.7|14.2|35.7|35.1|21.9|\n|Artemis (ours)|15.5|18.0|40.8|53.2|25.4|\n\nWe will add them to the paper.\n\n**Q2:** *Ablation.*\n\n**A2**: There are misunderstandings. Explanations below (we will update the notations).\n\nFirst of all, \"w/o\" is **not** the baseline, but indicates the option that utilizes the RoI features to encode the referred object based on Video-ChatGPT. That said, \"w/o\" is a part of the proposed method, which reports higher performance than Merlin.\n\nTo facilitate a more intuitive comparison, we added a new baseline. Given an object of interest, we enclosed its location in each frame with a rendered red rectangle, encoded the video using Video-ChatGPT, and asked \"What is the object in the red rectangle doing in this video?\". As shown, this \"baseline\" achieves slightly lower results than that of \"w/o\".\n\nWe also added a new option named \"w/ <track-instruction>\", where the <region> features were replaced with visual features (the `CLS` token of CLIP-ViT-L/14) in the key frames. As shown, there is a performance drop compared to Artemis. This is because the RoI features are of higher quality unlike the whole-frame features impacted by background.\n\n|Method|BLEU@4|METEOR|ROUGE_L|CIDEr|SPICE|\n|:---:|:---:|:---:|:---:|:---:|:---:|\n|baseline|11.2|16.3|34.9|23.8|21.4|\n|w/o|13.9|16.9|39.1|43.7|23.2|\n|w/ \\<track-instruction\\>|13.9|16.9|38.2|42.1|23.1|\n|Uniformly|14.2|17.2|39.4|44.5|23.6|\n|Artemis (Ours)|15.5|18.0|40.8|53.2|25.4|\n\n**Q3:** *Visualization and sanity check.*\n\n**A3**: Figure 15&19 (`attachment`) shows how Artemis produces different (and correct) answers for different <region>s, as long as tracking is correct. Figure 16 shows how Artemis produces correct descriptions for regular and reversed videos, *i.e.* the woman is walking *down* the stairs in the regular video, and walking *up* the stairs in the reversed video. We will add them to the paper.\n\n**Q4:** *Human evaluation.*\n\n**A4**: We randomly selected 100 videos from the HC-STVG test set. For each case, we applied Artemis and the fine-tuned Merlin (see **A1**) to produce the answers, and asked a person to evaluate their quality (video and ground-truth are provided). The evaluator is **not** aware of which answer is from which model. The evaluator gives a score of 1--5 to each answer (1=worst and 5=best). On average, Artemis and fine-tuned Merlin score 3.36 and 2.65, respectively. Artemis wins Merlin in 53/100 cases, and loses in 21/100 cases. We will add them to the paper.\n\n**Q5:** (Minor) *More architectural details.*\n\n**A5**: We will add them to the paper.\n\nArtemis consists of three components: a visual encoder (CLIP-ViT-L/14), a large language model (Vicuna-v1.5-7b), and a RoI feature extraction module. The RoI feature extraction module utilizes the visual features of 4 layers of the visual encoder to extract RoI features and passes them to a linear layer to obtain the RoI token.  \n\n\nFor text tokenization, we use Vicuna-v1.5's built-in tokenizer. To insert the video and RoI features into Q&A, we use \\<image\\> as a placeholder for video features, with its ID=-200 indicating the position of video features. Similarly, we use <bbox> for ROI tokens with ID=-500. No additional tokens like <image-start> and <image-end> are used.\n\n**Q6:** (Minor) *Statistics on object category/actions.*\n\n**A6:** All referred objects in the HC-STVG test set are humans (referred to as different nouns like man/woman). In the training set (VideoRef45K), there are other categories (*e.g.* animals/vehicles) -- see Figure 8 in Appendix A; the actions of these objects are much simpler compared to humans in HC-STVG, so we chose HC-STVG test set to challenge Artemis and others.\n\nWe used SpaCy to extract and count the action types in the HC-STVG test set. The distribution of 384 actions is shown in Figure 17 (`attachment`).\n\n**Q7:** (Minor) *Breakdown of failure.*\n\n**A7:** We defined five error types:\n* Temporal perception error, *e.g.* output is \"going down\", ground-truth is \"going up\".\n* Incomplete action error, *e.g.* only mentioning \"walking\" but omitting \"turning\" in the ground-truth.\n* Action recognition error, *e.g.* output is \"running\", ground-truth is \"jumping\".\n* Object recognition error, *e.g.* output is \"woman\", ground-truth is \"man\".\n* Multi-object interaction error, *e.g.* output is \"woman gave sth. to man\", ground-truth is \"man gave sth. to woman\".\n\nWe provided the output and ground-truth of each failure case to GPT-3.5 to get the error type. Statistics are shown in Figure 18 (`attachment`). Most frequent failures come from incomplete action and object recognition, albeit RoI tracking has alleviated them to some extent. Besides, object recognition errors mainly happen in the interacting object, *e.g.*, \"the man touches the woman's face\" is misrecognized as \"the man touches the man's face\". This indicates limited improvement of RoI tracking on interacting objects. This enlightens a future direction.\n\nWe will add them to the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sz6B39i2Bv", "reviewer": "Reviewer_CDQu", "summary": "This paper introduces Artemis as a robust solution for the video-based referential understanding task. This task involves analyzing complex videos, each spanning 20–30 seconds, where the target performs multiple actions. Given a video, the Multimodal Large Language Model (MLLM) attempts to answer questions such as \"What is the target <region> doing in this video?\" with <region> referring to a bounding box in any video frame. Artemis follows the general design principles of modern MLLMs, such as visual instruction tuning. To extract target-specific video features, Artemis employs a straightforward yet effective approach involving (i) tracking the target over time and (ii) selecting informative features from a comprehensive list of regions of interest (RoIs). The training of Artemis consists of three stages, with the first two stages being similar to LLaVA. For the final stage, this paper introduces the VideoRef45K benchmark, comprising 45,000 video question-answer pairs, with box-level prompts and answers for complex videos. Experiments demonstrate the promising performance of Artemis across various quantitative metrics, including BERT score, BLEU, and more.", "questions": "I am a bit surprised that MLLMs for video referring and grounding have not been explored yet. As I am not very familiar with referential understanding, I look forward to more feedback from other reviewers. If this paper does not overlook any important references, I am willing to upgrade my rating.", "limitations": "Please refer to weakness.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is well-written and easy to follow. The authors present a straightforward solution to address video-based referential understanding, a relatively unexplored research area so far as shown by the authors. The method adopted is simple yet aligns well with the paper's motivation. Additionally, the experimental section is robust and well-executed.", "weaknesses": "1. As far as I know, Multi-Object Tracking (MOT) is far from satisfying in accurately tracking target regions, particularly in challenging scenarios such as motion blur and occlusion. Although this paper mentions these issues in its limitations section, it does not discuss them in detail. My concern is that the method performs well on the presented benchmarks because the scenarios to be tracked are relatively simple. It would be beneficial if the authors could provide the performance of HQTrack on these benchmarks or offer examples showing the method's efficacy in complex scenarios (e.g., multi-person tracking).\n2. This paper does not delve deeply into learning the temporal relationships among tracked Regions of Interest (RoIs). Therefore, the temporal encoding knowledge of these RoIs mainly derives from HQTrack. Since the improvement of this model in video-based referential understanding primarily stems from its better comprehension of temporal dynamics, it raises the concern that the benefit may come from HQTrack rather than the model itself.\n3. Comparisons with other models (e.g., Shikra, Merlin) seem somewhat unfair, as these models do not utilize the dynamic knowledge from HQTrack. A fairer comparison setting would enhance the validity of the results.\n\nI am willing to upgrade my rating, if the authors can address my above concerns.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the contributions of our work. We deeply appreciate your constructive comments and have provided point-to-point responses below. We hope our responses address all your concerns, and further comments are welcomed.\n   \n**Q1:** *As far as I know, Multi-Object Tracking (MOT) is far from satisfying in accurately tracking target regions, particularly in challenging scenarios such as motion blur and occlusion. Although this paper mentions these issues in its limitations section, it does not discuss them in detail. My concern is that the method performs well on the presented benchmarks because the scenarios to be tracked are relatively simple. It would be beneficial if the authors could provide the performance of HQTrack on these benchmarks or offer examples showing the method's efficacy in complex scenarios (e.g., multi-person tracking).*\n\n\n**A1:** Good suggestion! Indeed, none of the existing MOT algorithms are close to satisfactory. The original tracking results in HC-STVG were produced by SiamRPN, an early video tracking model with lower accuracy. Although we upgraded the tracking algorithm to HQTrack, it may still fail to track all the objects, especially in complex scenarios.\n\nDuring the rebuttal, we delved into the test set of HC-STVG and found several examples of multi-person tracking in complex scenarios. We show the tracking and video-based referring results in Figure 19 in the `attachment`. As shown, HQTrack sometimes fails to track the object throughout the video clip -- due to the unavailability of ground-truth labels, we cannot quantitatively compute its accuracy. Regarding the referring results, Artemis produces correct descriptions as long as the tracked boxes are accurate. Interestingly, even when the object is missing in some frames, Artemis can (sometimes) produce correct descriptions based on the visual information from other frames. \n\n    \nWe will add these examples and analysis to the final paper.\n\n\n**Q2:** *This paper does not delve deeply into learning the temporal relationships among tracked Regions of Interest (RoIs). Therefore, the temporal encoding knowledge of these RoIs mainly derives from HQTrack. Since the improvement of this model in video-based referential understanding primarily stems from its better comprehension of temporal dynamics, it raises the concern that the benefit may come from HQTrack rather than the model itself.* \n\n\n**A2:** Thanks for the question. We totally agree that incorporating richer temporal information and/or knowledge is beneficial for video understanding.\n\n(1) Indeed, our work did not introduce an extra module to formulate the temporal information of the tracked RoI, *e.g.* how it changes throughout the video clip. During the rebuttal, we investigated more examples and found that the model has acquired a preliminary ability to describe the temporal patterns of a video -- see Figure 16 in the `attachment`, where Artemis produces reversed descriptions (a woman walking *down* and *up* the stairs) when the input video is played through the regular and reversed directions. This implies that the MLLM can learn extra temporal knowledge beyond the tracked RoIs. Such abilities may stem from the self-attention module of the MLLM that summarizes the sequential visual features.\n\n(2) An important discovery of our work is that making  proper use of temporal knowledge can largely boost the accuracy of video-based referring. We show a preliminary solution (*i.e.* using an off-the-shelf tracking algorithm to compensate the temporal knowledge), and our efforts reveal a future direction to equip the MLLMs with this ability (*e.g.* one can prompt MLLMs to track the referred objects). Respectfully, we believe that introducing a tracking algorithm (*e.g.* HQTrack) and making the system work is part of our technical contribution.\n\nWe will explore stronger solutions in the future. The above discussions will be added to the final paper.\n\n   \n**Q3:** *Comparisons with other models (e.g., Shikra, Merlin) seem somewhat unfair, as these models do not utilize the dynamic knowledge from HQTrack. A fairer comparison setting would enhance the validity of the results.* \n\n\n**A3:** Good question! Actually, in evaluating Shikra and Merlin for comparison, we used the same set of tracked bounding boxes (produced by HQTrack) to compute the required visual information: for Merlin, visual features of the entire image were fed into the MLLM together with the bounding box information (in texts); for Shikra which cannot process multiple images simultaneously, both the images and tracked bounding boxes were provided. Therefore, both Shikra and Merlin made use of the dynamic knowledge from HQTrack and thus the comparison was fair to our best efforts.\n    \nWe will add the above clarification to the final paper.\n\n   \n**Q4:** *I am a bit surprised that MLLMs for video referring and grounding have not been explored yet. As I am not very familiar with referential understanding, I look forward to more feedback from other reviewers. If this paper does not overlook any important references, I am willing to upgrade my rating.*\n\n\n**A4:** Here we offer some information for your reference. To the best of our knowledge, two related papers exist prior to our work, namely, PG-Video-LLaVA and Merlin, both of which have been cited in the original submission (see Lines 78--81 in Section 2). PG-Video-LLaVA used off-the-shelf detectors to perform grounding, but the model itself did not have the ability to perform fine-grained video understanding. Merlin studied video-based referring, but it required three manually specified input frames, incurring extra burden for users. Additionally, we also validate the advantage of Artemis over Merlin, the direct competitor.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XuzFDXRBMz", "reviewer": "Reviewer_xzsp", "summary": "This paper introduces Artemis, a video-language model for video-based referential understanding. It can describe a target referred by a bounding box in a video. A referential video dataset named VideoRef45K is collected to train the model. Artemis is evaluated on HC-STVG benchmark and outperforms baselines adapted from image-based referring models. Besides referential understanding, Artemis can also perform general video question answering, and serve as a component in multi-round and long-form video understanding.", "questions": "1. Does the target-specific features in \\<region\\> tokens include the positional information of the bounding boxes?", "limitations": "Limitations are discussed in the paper.", "rating": 7, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 4, "strengths": "1. A video-based referential understanding dataset, VideoRef45K, is established. It facilitates the development of the area, providing referential pretraining data.\n2. The design of target-specific feature branch in the model architecture is well-motivated.\n3. Although no video-based referring model exists, the paper adapts image-based referring models to video as baselines. The adaption method is quite reasonable.\n4. Artemis outperforms the adapted image-based baselines significantly on HC-STVG.\n5. Artemis can still perform general video question answering and achieve better performance after training on the video referring task. This demonstrates that video referring can boost the reasoning capability of video-language models.\n6. Combined with existing video-language models, Artemis can perform multi-round video understanding with grounding and long-form video understanding.", "weaknesses": "1. The RoI selection step of Artemis clusters the object bounding boxes from different frames. However, the clustering algorithm only considers the bounding box coordinates but does not take the visual content in the bounding boxes into account. In some cases, the bounding box of an object may remain unchanged for a long time but the object state keeps changing, e.g., a person standing at a certain location performs a series of actions. Clustering these frames together can compressing them into one would lose valuable information.\n2. The proposed Artemis is built based on video-language models Video-LLaVA and Video-ChatGPT. However, these video-language models are not used as baselines in the experiments. Although they are not originally developed for video referring, there is a simple approach to adapt them for the task. As suggested by [1], directly drawing a circle or a rectangle on an image can help VLMs focus on the indicated object. Therefore, one can adapt the video-language models for video referring by drawing the object bounding boxes on the video frames and ask the model \"What is the target indicated by the red rectangle doing?\". Artemis should be compared with this simple baseline to demonstrate the effectiveness of the RoI feature branch in its model architecture.\n\n[1] Shtedritski et al. What does CLIP know about a red circle? Visual prompt engineering for VLMs. ICCV 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the contributions of our work. We deeply appreciate your constructive comments and have provided point-to-point responses below. We hope our responses address all your concerns, and further comments are welcomed.\n    \n**Q1:** *The RoI selection step of Artemis clusters the object bounding boxes from different frames. However, the clustering algorithm only considers the bounding box coordinates but does not take the visual content in the bounding boxes into account. In some cases, the bounding box of an object may remain unchanged for a long time but the object state keeps changing, e.g., a person standing at a certain location performs a series of actions. Clustering these frames together can compressing them into one would lose valuable information.* \n\n\n**A1**: Thanks for the question. We **did** take visual contents within the bounding box into consideration. Specifically, we computed a token for each RoI which contained the visual features extracted from the bounding box (from the same visual encoder, *i.e.* a pre-trained CLIP ViT-L/14 model) and used the token for clustering. We will add the implementation details to the final paper.\n\n    \n**Q2:** *The proposed Artemis is built based on video-language models Video-LLaVA and Video-ChatGPT. However, these video-language models are not used as baselines in the experiments. Although they are not originally developed for video referring, there is a simple approach to adapt them for the task. As suggested by [1], directly drawing a circle or a rectangle on an image can help VLMs focus on the indicated object. Therefore, one can adapt the video-language models for video referring by drawing the object bounding boxes on the video frames and ask the model \"What is the target indicated by the red rectangle doing?\". Artemis should be compared with this simple baseline to demonstrate the effectiveness of the RoI feature branch in its model architecture.*\n> [1] Shtedritski et al. What does CLIP know about a red circle? Visual prompt engineering for VLMs. ICCV 2023.\n\n\n**A2**: Thanks for the suggestion! During the rebuttal, we evaluated Video-LLaVA and Video-ChatGPT for video referring using the method you proposed. Specifically, we followed [1] to draw a red rectangle to mark the referred object in each key frame of the video -- please note that tracking is required to mark the object in most frames. Then, we fed the rendered video to the models and asked the question \"What is the target indicated by the red rectangle doing?\" Results are summarized in the following table. One can see that, even with the help of an offline tracking algorithm, both Video-LLaVA and Video-ChatGPT report significantly lower scores compared to Artemis. This validates the effectiveness of Artemis' design. We will add these contents to the final paper.\n\n| Method | BLEU@4 | METEOR | ROUGE_L | CIDEr | SPICE |\n| :---: | :---: |:---: |:---: |:---: |:---: |\n| Video-ChatGPT | 1.3 | 10.1 | 20.2 | 5.5 | 11.7 |\n| Video-LLaVA | 1.7 | 9.8 | 20.8 | 2.6 | 9.1 |\n| Artemis (Ours) | 15.5 | 18.0 | 40.8 | 53.2 | 25.4 |\n    \n**Q3:** *Does the target-specific features in <region> tokens include the positional information of the bounding boxes?*\n\n**A3**: Yes. We used positional encoding to incorporate the coordinates of the bounding boxes. Therefore, the <region> token contains the positional information.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank reviewers for their meticulous work and the insightful comments provided to us. 

All reviewers acknowledged the novelty and contributions of the proposed approach (Artemis). 

**Reviewer U8HM&CDQu:** The paper is **well-written and clear** and the **motivation is well presented**.

**Reviewer CDQu:** The **method is simple yet aligns well with the paper's motivation**. The **experimental section is robust and well-executed**. 

**Reviewer xzsp:**  A video-based referential understanding dataset, VideoRef45K, is established. **It facilitates the development of the area**, providing referential pretraining data.  The design of target-specific feature branch in the model architecture is **well-motivated**. Combined with existing video-language models, Artemis can perform multi-round video understanding with grounding and long-form video understanding.

The major concerns and suggestions lie in the **fair comparison with other models** (**U8HM**, **CDQu**), **more ablative studies** (**U8HM**), **more model comparisons** (**xzsp**), the **influence of the tracking model** in Artemis (**CDQu**), and more examples and **implementation details** (**U8HM**, **CDQu**). 
    
During the rebuttal, we have carefully considered reviewers' every feedback and provided more experiments and ablation studies, as suggested. We believe these point-to-point responses can address reviewers' concerns and further enhance our work. We also include a PDF file (referred to as the `attachment`) with additional experimental results to support our responses.

Meta-review: The paper originally received borderline scores leaning towards acceptance. The authors provided a detailed rebuttal and the paper has been extensively discussed by the reviewers. Reviewers CDQu and xzsp raised some concerns about comparisons to the previous works and missing baselines (in particular for what concerns the video-language models) and the contributions of the tracked ROIs. The authors successfully addressed the issues and both the reviewers where satisfied by the rebuttal and increased their scores to 7:Accept. The AC carefully revised all the material and there has been also some further discussion with the reviewers. Overall this is a solid submission and the authors are encouraged to follow and implement all the reviewers' suggestion in the final revision of the paper.
Decision: Accept (poster)