Egocentric Vehicle Dense Video Captioning

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 09:01:31

Abstract: Typical dense video captioning mostly concentrates on third-person videos, which are generally characterized by relatively delineated steps among events as seen in edited instructional videos. However, such videos do not genuinely reflect the way we perceive our real lives. Instead, we observe the world from an egocentric viewpoint and witness only continuous unedited footage. To facilitate further research, we introduce a new task, Egocentric Vehicle Dense Video Captioning, in classic first-person driving scenario. This is a multi-modal, multi-task project for a comprehensive understanding of untrimmed, egocentric driving videos. It consists of three sub-tasks that focus on event location, event captioning, and vehicle state estimation separately. For the purpose of accomplishing these tasks, it is necessary to deal with at least three challenges, those are extracting relevant ego-motion information, describing driving behavior and understanding the underlying rationale, as well as resolving the boundary ambiguity problem. In response, we devise corresponding solutions, encompassing a vehicle ego-motion learning strategy and a novel adjacent contrastive learning strategy, which effectively address the aforementioned issues to a certain extent. We validate our method by conducting extensive experiments on the BDD-X dataset, all of which show promising results and achieve new state-of-the-art performance on most metrics, which proves the effectiveness of our approach.

Main paper: 
 introduction: 
 Dense Video Caption (DVC) is a branch of video understanding that aims to locate and describe all events within an untrimmed video [30, 40, 56, 66, 70, 74] . Leveraging DVC allows for more efficient video processing, like detailed content retrieval and intelligent surveillance. Several related research has been conducted in some specific scenarios, such as procedural human activities [4] and instructional videos [58, 73] , all these efforts have seen remarkable capabilities.However, the majority of DVC researchers focus on analyzing exocentric, or third-person videos, yet investigations into an egocentric, or first-person, viewpoint remain rare. Different from the former, which commonly narrates details about other objects within the camera lens as shown in Figure 1 (b), this view primarily concerns the movements of camera wearer [8, 17, 26, 34, 51] itself. In fact, it is the egocentric perspective that precisely reflects the authentic and natural manner in which human beings and autonomous agents observe their surroundings [8, 17, 35, 60, 61] . Besides, it fundamentally affects how we understand and engage with our environment on a daily basis, by influencing our perception [39, 49] , decisions [22, 46, 69] and interactions [36, 41, 59] in the complexities of the world around us [54, 69] . Consequently, this gives rise to Egocentric Dense Video Captioning (Ego-DVC). In contrast, Ego-DVC is characterized by its unique competence to learn motion changes of the view from untrimmed egocentric videos, which corresponds closely to real-life experiences and is thus more practical and meaningful. There are plenty of potential applications for this subject, among which, driving is a classic domain.For the facilitation of investigation, we present a new task, Ego Vehicle Dense Video Caption. It is devoted to driving scenarios, where a camera is mounted at a specific location on the vehicle, to capture multiple variable ego-motion information and the evolving landscape as the vehicle moves. Ideally, the information recorded should implicitly include the locations of various driving behaviors, corresponding descriptions and rationales, as well as vehicle states, hence we propose such a task. As shown in Figure 1 (a), given a sequence of first-person driving frames, we are tasked with three sub-tasks. a) Event Location, aiming to identify all driving events that take place in the video, while also simultaneously pinpointing the precise start and end timestamps for each. b) Caption generation, intending to describe the actions of all detected events in natural language and provide their contextually relevant rationale. c) Vehicle State Estimation(VSE), attempting to estimate specific vehicle states for the whole video, including velocity and steer, these states describe the fundamental motion patterns of the vehicle, for example, we might infer a vehicle is slowing down if we detect that its velocity values are consistently decreasing during a period of time.Upon a profound analysis, we suppose our task is confronted with at least three distinct challenges:Ego-Motion Information: For egocentric videos, ego-motion information is embedded in the dynamic changes of the camera lens. However, in the complex road environment, irrelevant objects and deterministic signals [62] for driving are always intricately intertwined and subtly changing, making it difficult to extract effective motion representation.Description and Rationale: Analyzing the cause is often more difficult than describing the problem, this also applies to our task. As shown in 1(b), the reason the car slows down is that it encountered a red light, which is not easy to analyze because the accurate rationale(traffic light) that truly corresponds to the description in driving scenarios often appears trivial and is prone to be overlooked, it requires a thoroughly probe into the videos.Boundaries Ambiguity: Compared to typical DVC, the egocentric videos are always continuous, unedited real-life footage. Additionally, those captured by vehicle-mounted cameras predominantly feature the road, with the majority of the view occupied by it and only several impalpable changes occurring, especially in suburbs. As shown in Figure 1 (b), frames near the event boundary are very similar thus causing ambiguous, yet traditional DVC in Figure 1 (a) generally does not encounter this issue. This feature obstructs the precise identification of the boundary between two adjacent driving behavior events.To address these challenges, we intentionally designed corresponding strategies. Concerning the first two challenges, we introduce a vehicle ego-motion learning strategy, it integrates a pretrained extractor and VSE module, the former function on extracting ego-motion features incorporating driving-decision, and the VSE module is applied to strengthening the representation with specific motion values(vehicle state). Regarding the third one, we devise an adjacent contrastive learning(ACL) strategy that enhances event representation by performing contrastive learning among the three modalities of adjacent events. This approach is capable of reducing the ambiguity of event boundaries and thereby distinguishing them more clearly.To summarize, our main contributions are three-fold:(1) We introduce the Ego Vehicle DVC task, allowing for a detailed multimodal comprehension of untrimmed egocentric driving videos.(2) We pioneered incorporating an ego-motion information learning strategy in DVC. Besides, we design an adjacent contrastive learning strategy for event representation learning.(3) We conducted extensive experiments on the BDD-X dataset, achieving state-of-the-art results in most metrics, thereby demonstrating the effectiveness of our approach. 
related work 2.1 egocenertic vision: 
 Egocentric vision, providing a distinctive and intuitive perspective on human interactions with the environment [8, 35, 36, 41, 59, 61] , is thriving increasingly. In contrast to traditional tasks, which typically process well-defined exocentric videos curated by photographers, egocentric videos possess their unique features that remain underexplored, such as view changes [20] , even the currently popular large language model (LLM) still performs poorly on this issue [8] . To delve deeper, a wide range of related topics are gradually emerging and attracting the attention of researchers.Egocentric human-object interaction(EGO-HOI) is a vital task in this field, it primarily concentrates on the interactions between hands and objects from an egocentric viewpoint [9, 17, 53, 64] . Some research attaches importance to hand pose estimation and object-centric representations [1, 64] , others strive to learn reasoning and indirect reference through question-answering on realworld egocentric footage [25, 26] , and further works on captioning egocentric videos by cross-view transfer learning from exocentric sources [21, 63] . EGO-HOI paves the way for nuanced communication between humans and external entities, while it primarily focuses on specific targets, lacking a comprehensive understanding of the overall scenario.Egocentric Visual Perception, which generally serves as the eye of the entire autonomous system, is a crucial part of this topic. It is widely applied in applications such as Virtual Reality (VR) and Augmented Reality (AR), where a fundamental task involves locating the 3D positions of multiviewed visual queries in complex scenarios [17, 37] . Embodied AI, a prevalent topic at present, is also inextricably linked to this technology. [13, 54, 72] endeavors to master comprehensive 3D scene understanding skills, enabling real-world embodied agents to execute commands effectively. Autonomous driving is another typical application of egocentric visual perception, studies such as [19, 38, 47] attempt to plan vehicle action based on surroundings and achieve impressive outcomes, despite the understanding of the underlying rationale remains elusive. Furthermore, Several works consider detecting regions pertinent to driving decisions [12, 28, 45] , although these methods are useful for those familiar with traffic laws, they may confuse ordinary users. Therefore, [27, 29, 65, 68] strive to comprehend vehicle motion with rational and straightforward natural language, while they limit to a short-term video with one main behavior which is not consistent with reality.Dense video captioning (DVC) is a multi-task project that requires identifying events and generating captions for them. Originally, this task predominantly employed two-stage methodologies [23, 24, 30, 52, 57, 70] , starting with the temporal localization of events, followed by their captioning. This paradigm heavily depends on the performance of the first stage, consequently, many have begun to consider the end-to-end one-stage approach [6, 7, 10, 31, 40, 44, 48, 55, 56, 66] , intending to reach mutual improvement by coordinating the interaction of jointly training two sub-tasks. Learning from Bert [11] , [70, 74] design mask mechanisms for the interaction between the two modules separately. The emergence of DETR [5] brought fresh prospects to the task, based on which Deformable-DETR [75] , [56] employs a set prediction scheme to elegantly parallelize the two sub-tasks. Building on this foundation, [55] introduces contrastive learning to enhance event representation through the contrast between events and captions. Given the information embedded in the audio tracks of these videos, [23, 24, 66 ] take a unique view by extracting the audio features from the videos, thereby significantly improving performance.However, the methods designed for edited and exocentric videos fall short of meeting our task's requirements, which focused exclusively on analyzing unedited, audio-free, continuous, real-world driving footage from a first-person perspective. In Driving scenarios, the views are typically dominated by the road surface, and present minimal variation in the surrounding environment, leading to ambiguous distinctions between events and complicating event localization. What's more, the inherent nature of egocentric videos concerning dynamic view changes, adding another layer of complexity to video analysis. 
methodology: 
 In this work, we focus on locating all driving behavioral events, captioning behavior descriptions and rationales, as well as estimating vehicle state values throughout the entire ego-vehicle video. Figure 2 provides a graphical illustration of our comprehensive framework. Initially, we pre-train a vehicle ego-motion extractor(Sec 3.1), leveraging which frame features will be captured and fed into a DETR-based architecture and be further amplified by VSE module (Sec 3.1) at the end of the encoder, the decoder will generate several event representations, we then introduce a novel adjacent contrastive learning (Sec 3.2) strategy to enhance semantic representation of these event and finally generate all descriptions and rationales.In this section, our objective is to introduce the vehicle ego-motion learning strategy. It comprises two parts, the first part involves pretraining a vehicle ego-motion extractor by self-supervised learning, while the second part enhances ego-motion representation through supervised VSE. Vehicle Ego-motion Extractor. In this part, we try to achieve an extractor that allows us to map the raw frame input to a compact representation containing basic ego-motion information, which is essential for our three sub-tasks. Considering that under normal circumstances, driving decisions should be consistent with actual behavior, we attempt to incorporate driving-decision awareness, expecting to assist in extracting appropriate motion representation while also taking into account the crucial visual cues based on the current scenario. Following the design of PPGeo [62] , our pre-training progress consists of two stages.Self-supervised photometric reconstruction. Photometric Reconstruction aims to reconstruct the scenario by learning photometric differences, or more specifically, standard color constancy between frames. There exists a prevalent method that enables the model to translate input pixels into ego-motion and detailed scene architecture as well as estimating camera intrinsics.We follow [16, 71] to perform photometric reconstruction by jointly training a PoseNet and DepthNet across two frames. PoseNet is designed to estimate the 6-DoF ego-motion and camera intrinsics between consecutive frames, and DepthNet predicts the depth map in the meantime. We employ a ResNet and MPViT to serve them separately. Assuming that we want to reconstruct the t-th frame 𝐼 𝑡 from 𝐼 𝑡 -1 . the pixel-wise color constancy can be reconstructed as follows:EQUATIONhere 𝐼 𝑡 ′ is the reconstruction of frame 𝑡, proj() indicates the operation with which we project original pixels space of 𝐼 𝑡 -1 into predicted 2D coordinates making use of depth map 𝐷 𝑡 -1 from and relative pose 𝑇 𝑡 -1→𝑡 between 𝐼 𝑡 -1 and 𝐼 𝑡 . Afterward, utilizing bilinear interpolation, we sample values to create 𝐼 𝑡 ′ through <> operation. As for camera intrinsics 𝐾, we consider it a constant value and assess it by calculating the average of 𝐾 𝑡 -1 and 𝐾 𝑡 predicted from relevant frames. Align with [16, 71] , We calculate the loss according to the following formula:EQUATIONhere L 𝑝𝑒 represents photometric loss comprised of structural similarity index measure(SSIM) and 𝐿 1 term:EQUATIONL 𝑠 represent disparity smooth-ness loss:L 𝑠 = 𝜕 𝑥 𝑑 * 𝑡 𝑒 -|𝜕 𝑥 𝐼 𝑡 | + 𝜕 𝑦 𝑑 * 𝑡 𝑒 -|𝜕 𝑦 𝐼 𝑡 | (4)where 𝑑 * 𝑡 is the mean-normalized inverse depth map. Vehicle ego-motion extractor. After the preceding phase, we will obtain a DepthNet and a PoseNet. This PoseNet can capture relative motion differences between two adjacent frames. In fact, what we really need is the "difference" on a certain frame, it essentially means learning the driving policy, that is to say, performing suitable driving behavior based on current observation. To bridge this gap, we follow the methodology outlined in [62] , wherein 𝐼 𝑡 is removed and retain only one 𝐼 𝑡 -1 as input. In addition, we freeze In Adjacent Contrastive Learning, these three modalities enhance each other within the same event(shown in the same color), while simultaneously weakening their adjacent events(yellow and pink), but ignoring interaction with temporally distant events(yellow and green).the DepthNet, reinitialize the parameters of PoseNet, and conduct retraining of the entire model as the preceding phase once again. Ultimately, a new PoseNet will be obtained with the assistance of the well-trained DepthNet, referred to as the vehicle ego-motion extractor. It enables the acquisition of not only ego-motion representations but also critical decision-making information, that is, the rationale.In this part, we intend to introduce VSE, a special module that plays a dual role within our framework. The major function is to carry out the task of vehicle state estimation. Simultaneously, since the vehicle state is, in essence, a concrete form of ego-motion, VSE and vehicle ego-motion representation can be mutually beneficial throughout this supervised learning process.Assuming that an ego-vehicle video 𝑉 , consisting of 𝑁 frames, is labeled as {(𝑣 1 , 𝑠 1 ), (𝑣 2 , 𝑠 2 ), ..., (𝑣 𝑀 -1 , 𝑠 𝑀 -1 ), (𝑣 𝑀 , )}, here 𝑣 𝑡 denotes the velocity at the 𝑡-th timestamp, 𝑠 𝑡 denotes the steering value between the 𝑡-th and (𝑡 + 1)-th records. It is important to notice that the collected vehicle states do not always align with frame sequences. In practice, they are recorded at fixed intervals, rather than frame by frame, therefore 𝑀 is usually less than 𝑁 . We will feed into 𝑁 frames and extract features through the vehicle egomotion extractor, after transformer encoder, the features 𝑓 1 , 𝑓 2 ...𝑓 𝑁 will be applied to estimate velocity and steer of the ego-vehicle throughout the duration as follows:EQUATIONGiven 𝑀 records associated with 𝑁 frames, we utilize linear interpolation to downsample 𝑁 features to 𝑀. Note that we use multi-scale features with CNN, to accumulate them we apply max pooling results in 𝑓 1 , 𝑓 2 ...𝑓 𝑀 , then we map them into scalars as follows by MLP:EQUATIONEQUATIONHere 𝑣 ′ 𝑖 represents estimated velocity, 𝑠 ′ 𝑖 represents steer between 𝑓 𝑖 and 𝑓 𝑖+1 .Finally, We calculate the loss using Mean Square Error:L 𝑚𝑠𝑒 = 1 𝑀 𝑀 ∑︁ 𝑖=1 (𝑣 𝑖 -𝑣 ′ 𝑖 ) 2 + 1 𝑀 -1 𝑀 -1 ∑︁ 𝑖=1 (𝑠 𝑖 -𝑠 ′ 𝑖 ) 2 (8)In this section, we will introduce our contrastive learning strategy particularly devised for driving events location. The core concept involves applying three types of modalities to adjacent driving behavioral events. In this task, we regard an untrimmed ego vehicle video 𝑉 as a set 𝐸 = {𝑒 𝑛 |𝑒 𝑛 = (𝑙 𝑛 , 𝑐 𝑛 ), 𝑛 = 1, 2, ..., 𝑁 }, where 𝑙 𝑛 = (𝑙𝑠 𝑛 , 𝑙𝑒 𝑛 ) defines the time location for event 𝑛, starting at time 𝑙𝑠 𝑛 and ending at time 𝑙𝑒 𝑛 , 𝑐 𝑛 = (𝑑 𝑛 , 𝑟 𝑛 ) provides the caption for event 𝑛, with 𝑑 𝑛 offering description of driving behavior, 𝑟 𝑛 offering its rationale. For convenience, We define a new set 𝐸 𝑛 consisting of event 𝑛 and its adjacent events as follows:𝐸 𝑛 = {𝑒 𝑖 ∈ 𝐸|(𝑙𝑒 𝑖 ≥ 𝑙𝑠 𝑛 ∧ 𝑙𝑠 𝑖 ≤ 𝑙𝑒 𝑛 )∨ (|𝑙𝑒 𝑖 -𝑙𝑠 𝑛 | ≤ 𝜀 ∨ |𝑙𝑠 𝑖 -𝑙𝑒 𝑛 | ≤ 𝜀)} (9)where 𝜀 is a constant, defining the temporal distance threshold of adjacent events. As seen in Figure 2 , the representations of frame sequences flow into the Event Encoding module and are encoded into 𝑀 (𝑀 ≤ 𝑁 ) predicted driving behavioral events. Sequencely, after a typical Hungarian Match, they will be matched with ground truth. To deepen discriminability, we introduce an additional head for event representation learning. This enables us to project the predicted events into 𝑀 semantically representations 𝐸𝑝 = {𝑒𝑝 1 , 𝑒𝑝 2 , . . . , 𝑒𝑝 𝑀 }. Ideally, we consider they should satisfy at least the following three criteria:• They should fully encapsulate the vehicle ego-motion information of the corresponding behavioral events. • Any two adjacent behavioral events should be clearly distinguished. • Any two behavioral events that are distant in time should not affect each other.Taking these three considerations into account, we devise a novel contrastive learning method. Previous methods usually calculate the loss in a global range, even extending to a training batch [18, 43] attribute to the variety of their data pairs. On the contrary, we merely confine ourselves to an adjacent range. This is because the scope of behaviors observed in ego-vehicle videos is generally limited and prone to repetition. A vehicle described as "The car turns right at an intersection" during a certain period may likely be described by the same sentence again before long, this phenomenon is so common in vehicle scenarios that previously prevalent contrast learning would result in a significant decline in effectiveness. However, we can make sure that the driving behaviors of the neighboring events certainly contain obvious differences, otherwise they would not be divided into two events. Our strategy consists of the following three parts.The information in egocentric driving videos is not confined to the frame sequences alone, it exists in the associated caption as well. We posit that the essential motion insights from both mediums ought to be consistent. With the prior knowledge of relevant textual features, we can enhance the semantics of events and achieve cross-modal alignment between video and caption content. Clip [43] is a standard work related to this idea. Unfortunately, as a generation task, it is impossible to access caption representation before generating it, which leads to a deadlock situation.To break this deadlock, we introduced a pre-trained text encoder to encode all captions(𝑑 𝑛 or 𝑟 𝑛 ) into 𝐶. After aligning, we project 𝐸𝑝 and 𝐶 into a shared space and calculate the cross-modal cosine similarity matrix between the projected embeddings as 𝜔 𝑒𝑐 ∈ R 𝑀 ×𝑁 , we will calculate our adjacent event-caption contrastive loss as follows:L 𝑒𝑐 = - 𝑁 ∑︁ 𝑛=1 logexp(𝜔 𝑒𝑐 (match(𝑛), 𝑛)/𝜏) 𝑍 𝑒𝑐 𝑛 (10) here 𝑍 𝑒 𝑛 is a modified normalization factor:EQUATIONwhere match() devotes the matching operate from 𝐸 to 𝐸𝑝, 𝜏 signifies a temperature ratio.Fundamentally, the generation of driving videos is attributed to the driver issuing signals for steering and velocity based on a certain observation, therefore, the information entailed in the vehicle states should also maintain consistency with the behavioral events. As a result, we further implemented cross-modal semantic alignment between vehicle motion and event. This approach enhances the semantic representation of events by incorporating readily captured motion information.Just as in Event-Caption Contrastive Learning, we adopt a motion encoder to encode the ground truth vehicle motion state values of current events into a representation resulting 𝑆, After aligning and projecting into a joint space, we calculate the cross-modal cosine similarity between 𝐸𝑝 as 𝑆 as matrix 𝜔 𝑒𝑠 ∈ R 𝑀 ×𝑁 . Same as before, contrastive loss L 𝑒𝑠 will be derived in the same manner.We present the last contrastive Learning between motion and caption, aiming for a mutual complementarity of information between the two modalities, and further indirectly enhancing event representation.Differing slightly from the former two components, the score matrix will result in a square matrix 𝜔 𝑚𝑐 ∈ R 𝑁 ×𝑁 , the motioncaption contrastive loss is determined using the following formula:EQUATION)where 𝑍 𝑚𝑐1 𝑛 calculate the normalization factor vertically of 𝜔 𝑚𝑐 , we can also acquire L 𝑚𝑐2 with another 𝑍 𝑚𝑐2 𝑛 normalized horizontally, then the loss is expressed as follows:EQUATIONFinally, our complete adjacent contrastive learning loss L 𝑐𝑙 is signified as following expression:EQUATION)Where 𝛼 + 𝛽 + 𝛾 = 1, and 𝛼, 𝛽, 𝛾 are three trainable paramaters. Principally, as the Trimodal Adjacent Contrastive Learning module depicted in Figure 2 , we utilize the three modalities within a certain event to achieve cyclical contrastive learning, thereby enabling mutual enhancement among them and meeting the first point of the criteria we mentioned earlier. Furthermore, our method, by limiting the scope of contrast, not only weakens the semantic relevance of adjacent events but also concurrently avoids the influence of temporally distant events, satisfying the second and third points of the criteria, thus ultimately achieving the purpose of distinctly distinguishing the representation of adjacent events. 
experiments: 
 Dataset. To the best of our knowledge, there are no datasets perfectly aligned with the requirements of our task, we attempt to In BDD-X, the GPS information is collected at 1Hz using the same equipment, suggesting the camera intrinsics are identical, that's the reason we can estimate a single group of this parameter during the self-supervised photometric reconstruction stage at Sec 3.1. Velocities are recorded directly by the GPS, while for steering, we apply the course message (angle relative to geographic true north) as pseudo-values between two consecutive GPS records. Implementation Details. To pre-train the vehicle ego-motion extractor, we use ResNet34 and MPViT as the PoseNet and DepthNet respectively. For each stage, it takes about 5 days on 8 Tesla V100 GPUs to train for 20 epochs with batch sizes of 32 and 64.During training, we employ a frozen Roberta model as the text encoder and a BiLSTM model for the vehicle states encoder; Following PDVC [56] , our method is based on a deformable-DETR with two encoder-decoder layers of 512 dimensions and uses LSTM-DSA [56] serve as event caption head to generate captions; Contrastive learning is used only in training and ignored at the inference stage, we set the temperature 𝜏 to 0.1, temporal threshold 𝜀 to 3; All events will be sorted by their confidence scores, and the number of predictedk events will be decide according to a CounterHead. We set the batch size to 2 and trained for 30 epochs in a Tesla V100 using Adam with a learning rate of 0.0001 and a weight decay of 0.0001. Evaluation metrics. As a multi-task model, We evaluate our method in three aspects: 1) For VSE, we employ root mean squared error (RMSE) and threshold accuracies 𝐴 𝜏 . It calculates the ratio of test samples that have prediction errors smaller than a predefined threshold 𝜏, which we set at multiple levels: {0.1, 0.5, 1.0, 5.0}. 2) For events location, We calculate the average precision and average recall for IoU thresholds set at {0.3, 0.5, 0.7, 0.9} and their harmonic mean, the F1 score. 3) For captioning, we follow the widely utilized evaluation tool provided by ActivityNet Challenge 2018 [15] adopting BLEU4(B4) [42] , METEOR(M) [2] , ROUGE_L(R) [33] and CIDEr(C) [50] to measure matched pairs between generated caption and ground truth across IOU thresholds of {0.3, 0.5, 0.7, 0.9}. Taking into the quality of the story of the whole drive video, we additionally use SODA_c(S) [14] for an overall evaluation.Since there is currently no task that aligns completely with ours, we primarily compare event location and dense caption with current DVC tasks. We compare five approaches on the BDD-X dataset using their official codebases. MT [74] is the first one to utilize a transformer in this field, we pick up the top 30 events for indicator calculation; ESGN [40] offers a reinforcement learning approach, however, a proposal method is needed to extract candidates in advance, we adopt ActionDetection-DBG [32] instead of its original SST [3] and select top 100, noting that the official codebase solely contains the event sequence generation stage, we only compare in this subtask. UEDVC [70] transforms event-location into a sequence generation problem and proposes three pre-training tasks to effectively reinforce the correlation between sub-tasks. GVL [55] is slightly similar to our method, utilizing global contrastive learning to strengthen event representation, besides, it utilizes a semanticaware label assignment mechanism to improve recall. PDVC [56] initially introduces detr into this task, presenting an elegant end-toend approach. There are also several remarkable works [23, 24, 66] focus principally on audio features, which are not included in BDD-X, so we do not consider comparing with them. For the sake of a fair comparison, we all use the ResNet34(R34) structure as the basic feature extractor, we should keep in mind the vehicle ego-motion extractor(VEM) is also from R34. In addition, we omit vehicle states(vs) and keep only captions, which means discarding the vehicle state estimation task and motion-related contrastive learning module. Event location performance. Table 1 exhibits the effectiveness of our approach and other several state-of-the-art methods. It indicates that we achieve the best results in terms of precision and F1 score, with respect to recall, MT receives the highest score, the reason lying in its inherent emphasis on recall over precision. We witness that although both are from PDVC and applying contrastive learning, the precision of GVL is lower than PDVC in the BDD-X dataset, while the situation is reversed in common datasets such as ActivityNet [4] and [73] . The reason lies in that the modules designed in GVL are better suited for one-to-one style datasets, where events in the video directly match the caption. For datasets like BDD-X, where there is a one-to-many relationship and multiple driving events correspond to the same caption, this approach is not applicable, whereas our method overcomes this defect. Dense caption performance.Table 2 shows the impact on caption generation, namely, description and rationale generation in our task, compared with several previous works. we can observe that our method sets new state of the art on all metrics. From an overall perspective, the effectiveness of rationale generation is much lower than that of description generation, which aligns with our speculation that analyzing the reasons for a phenomenon is much more challenging than describing it.Adjacet Contrastive Learning. We conducted extensive experiments to assess the impact of our ACL approach. As illustrated in Table 3 , which shows its influence on driving behavior event location. Ground truth vehicle state and caption(description and rationale) are two optional types of potential impact factors of this approach, in addition to the essential event features modality. We also contrast two ways of integrating losses across different modality pairs, direct addition and learnable weighted combination. Although it does not significantly impact the recall rate compared to the approach without ACL, it does substantially improve precision by around 10.33%, resulting in an increase of around 5.01% in F1. This is because our method, designed among adjacent events, can enhance the boundary awareness capability of the model, making it more distinguishable from neighboring events.To verify the necessity of contrastive learning within the vicinity, we conducted additional experiments to confirm the impact of a temporal distance 𝜀. As depicted in Figure 3 , we adjusted 𝜀 from 1s to 40s, with 40s representing engagement in comparative learning over the entire range of the video. The precision observed in the line graph shows an initial rise followed by a subsequent decline. This pattern emerges because, in driving scenarios, we can only ascertain differences between adjacent driving events; However, it remains uncertain whether two temporally distant driving behaviors are distinct. Therefore, a large 𝜀 may impair results by contrastive learning method. Theoretically, although ACL does not affect caption generation, it can indirectly influence it by affecting event location. Given this possibility, we continue to carry out experiments on the change of text generation capability. Table 4 demonstrates our deduction, we can see our method achieves the best scores in most indicators. Vehicle Ego-motion Learning. We conducted experiments to investigate the impact of different feature extractors on the VSE task. In Table 5 , we compared ResNet34 (pre-trained on ImageNet 1k), CLIP (pre-trained on 400 million image-text pairs), and I3D (pretrained on Kinetics 400) with our VEM extractor trained from on an initialized ResNet34 (VEM r34 ). Among these, ResNet34, CLIP, and VEM r34 were applied to image frames, while I3D is a video feature extractor. As shown in Table 5 , our VEM r34 features performed significantly better than the other two image encoders. The RMSE of steer and velocity decreased by 6.90% and 28.57% respectively compared with the original Resnet34. Moreover, we achieve results comparable to those of the video encoder I3D, even though I3D outperforms ours when the threshold is relatively low. This overall result demonstrates the effectiveness of our method in capturing vehicle ego-motion representation.Table 6 illustrates the effect of vehicle ego-motion learning strategy, namely extractors and VSE module, on event location and caption. A driving event should contain a trend-oriented behavior, which is closely related to ego motion. Our vehicle ego-motion feature is not only informative regarding ego-motion but also contains driving decision-making, in other words, vehicle behavior rationale information, which exactly meets the requirement. In addition, as a component of our vehicle ego-motion learning strategy, the VSE task is also beneficial to event location and text generation. In table 6, the average of event location with VEM r34 almost surpasses all others, especially its counterpart, Resnet34, achieving 3.91%,6.96%, 5.28% for Recall/Precision/F1 respectively. Simultaneously, the caption task also achieves substantial enhancement. Despite the fact that rationale generation is still inferior to description, the improvements in rationale are much larger than Description. For instance, compared with Resnet34, the description sees an increase of about 14.78% with a rationale of 20.01%, demonstrating that our method indeed facilitates the model in mining explanatory information. 
conclusion: 
 This paper presents the Ego Vehicle DVC, involving a multi-modal task with three sub-tasks targeting the investigation of dense video captioning in real-life first-person driving scenarios. Due to its distinctive observational viewpoint, this task comes with its unique challenges. We develop a strategy for learning vehicle ego-motion and a novel adjacent contrastive learning for boundary ambiguity. Extensive comparisons and ablation experiments demonstrate the effectiveness of our proposed method. We suppose this topic is of great significance, whereas there are still some barriers hindering further research, a prominent problem is related to the dataset. The duration of the BDD-X dataset is relatively short, the descriptions and rationales lack diversity, and The recording frequency for vehicle states is too low. We may consider contributing a higher-quality dataset to this topic in the near future.  
 
 Appendix:  


Decision: After thoroughly checking reviews and rebuttal, I believe this study is worse to accept.

Reviews: review: paper_summary: paper_summary: In this paper, author propose a dense video captioning framework for first-person view driving videos. The problem statement is described and previous works in related fields are discussed. The method is evaluated on the BDD-X dataset and compared to various baselines on different sub-problems.
strength_weakness: strength_weakness: - The methodology and the model architecture are well described.
- The number of baselines chosen for comparison is sufficient. The experimental setting itself is extensively detailed.
- The paper has the merit of dealing with a partially unexplored problem.
limitations: limitations: - The fact that the problem is not well established makes comparison with existing works difficult.

- "However, the majority of DVC researchers focus on analyzing exocentric, or third-person videos, yet investigations into an egocentric, or first-person, viewpoint remain rare." This is a very strong assumption. There are some wide dataset of first-person videos used in active research (e.g. Ego4D). If this affirmation is relative to driving videos only, that should be better explicited.

- Formatting is somehow incosistent. References are sometimes spaced ("...two-stage methodologies [23 , 24 , 30..."), sometimes not ("...Learning from Bert[11]..."). Some subsection titles, for instance "Vehicle Ego-motion Extractor.", have a newline after, some other have not. This is not a methodological limitation of course but it hinders reading.

- Why is SSIM in Equation 3 multiplied by alpha/2 instead of just alpha? Even if this is a common practice and not proposed by the authors, a note explaining the reason behind the choice could be useful.

- Will you release the subset of BDD-X with additional video state data used for experiments? That's essential to prove reproducibility.

- Table 1 shows that the proposed method has low performance in terms of recall. Is there any task-specific reason to overlook recall performance? This is an important reason to discuss as the drop is consistent. More in general, a summary of method limitations should be presented in a dedicated paragraph.

I recognize the authors merits on dealing with an unconventional task. However, this brings to some non-trivial limitations in the evaluation which limits the impact of this work and make me doubt it is suitable for a top conference such as ACMMM, but I may be convinced otherwise.
paper_summary: paper_summary: 1. The authors proposed a new task, Egocentric Vehicle Dense Video Captioning (Ego Vehicle DVC), including three sub-tasks: Event Location, Caption Generation, and Vehicle State Estimation. The dataset is adapted from BDD-X.

2. For the Ego Vehicle DVC task, an ego-motion information learning strategy and an adjacent contrastive learning strategy are introduced.
strength_weakness: strength_weakness: S1: The proposed Ego Vehicle DVC task is novel and challenging.

S2: Utilizing the existing ego-vehicle video dataset BDD-X for the Egocentric Vehicle Dense Video Captioning task is clever, and it saves a lot of data collection effort.

S3: The proposed method achieves SOTA results at most of the metrics.
limitations: limitations: W1: While leveraging the existing ego-vehicle video dataset is innovative, and the proposed method achieves state-of-the-art performance on it, the BDD-X dataset's quality is lacking. Issues such as short video durations and a lack of diversity impact the reliability of the method's performance.

W2: The proposed method exhibits a low recall on the Event Location task.

W3: The text contains too many typos.
paper_summary: paper_summary: The paper proposes a new task, Egocentric Vehicle Dense Video Captioning, aimed at understanding untrimmed, first-person driving videos through event location, event captioning, and vehicle state estimation. Unlike traditional dense video captioning, which focuses on third-person videos, this approach addresses the challenges of extracting relevant ego-motion information, describing driving behavior, and resolving boundary ambiguities. The method, validated on the BDD-X dataset, employs vehicle ego-motion learning and adjacent contrastive learning strategies, achieving new state-of-the-art performance on most metrics.
strength_weakness: strength_weakness: The paper's primary strength lies in applying dense video captioning (DVC) to first-person driving scenarios. This could be crucial for applications like development of semi-autonomous cars. This technology could be particularly relevant for companies focused on semi-self-driving vehicles, as it can help determine whether a driver or an AI agent is at fault in adverse situations.
limitations: limitations: 1. Sorry to say, but the writing requires significant revision. There are numerous grammatical errors, and many sentences lack clear transitions and context.

    - The motivation in the abstract is not sufficiently relevant to highlight the importance of the work. The motivation feels lacking when transitioning from the first few sentences to the discussion on DVC on vehicles.

    - The paper lacks confidence in its claims, as seen in Lines 105-106 and in the conclusion sections.

    - Concepts like boundary ambiguity are not well positioned, despite their apparent importance to the authors. They require more context and explanation.

    - The discussion of vehicle state in terms of velocity seems trivial. More meaningful states would include being stopped, speeding, slowing down, hard braking, or sudden acceleration.

    - Although the paper hints that the ego-view is from the driver's perspective, it seems to be from the car's perspective. Including egocentric information from the driver's perspective would add more value to the work.

2. Lack of novelty in terms of algorithm

    - The paper does not introduce new algorithms; it is purely application-based.

3. Lacking motivations and explanations

    - Several scenarios in the paper lack sufficient motivation, such as the use of photometric reconstruction. The necessity of this approach is not explained.

    - The comparison between (a) and (b) in Figure 1 is misleading. The author's point about boundary ambiguity when comparing these figures is unclear.

    - The presented metrics, Recall and Precision, are insufficient to demonstrate the performance of this nature. Analyzing recall shows that the proposed model does not perform well. The author has not highlighted the importance of individual metrics.

The paper requires significant revision.
paper_summary: paper_summary: This paper proposes a novel dataset for dense video captioning on a vehicle's egocentric view, where not only event description but also rationale of the event are the target of mention. The authors also provide pre-training for ego-motion comprehension and a contrastive loss specially designed for this task, which contains three targets of mention. The experimental results show the challenge of this dataset and where we are currently at with this problem.
strength_weakness: strength_weakness: 1. A dataset with unedited video footage is much more practical than one with edited web videos. This reviewer agrees that this dataset is novel and significant for the dense video captioning task.
1. The captioning target includes rationale, which enhances explainability of the decision and matches to autonomous driving applications.
1. The initial attempt of pre-training and adjacent contrastive learning seem reasonable and helpful for dataset followers.
1. The attempt achieves the best performance among baselines.
limitations: limitations: 1. **[absence of Vid2Seq in the baseline]** Vid2Sec [66] (CVPR2023) is the State-of-the-art DVC method, but it is not in the baselines. The latest one in the current baselines [55] is an arXiv paper and is not authorized enough. This reviewer recommends testing Vid2Seq for better benchmarking of this new dataset.

2. **[Unclear vocabulary diversity in the dataset]** No statistics for the vocabulary diversity are provided. Such statistics are important to understand the difficulty of the task.