A Motion-aware Spatio-temporal Graph for Video Salient Object Ranking
Authors: Hao Chen
Yufei Zhu
Yongjian Deng
Venue: NeurIPS 2024 poster (2024)
Keywords: Video Salient Object Ranking
Spatio-temporal Graph
Video Retargeting
Primary area: machine_vision

Abstract: Video salient object ranking aims to simulate the human attention mechanism by dynamically prioritizing the visual attraction of objects in a scene over time. Despite its numerous practical applications, this area remains underexplored. In this work, we propose a graph model for video salient object ranking. This graph simultaneously explores multi-scale spatial contrasts and intra-/inter-instance temporal correlations across frames to extract diverse spatio-temporal saliency cues. It has two advantages: 1. Unlike previous methods that only perform global inter-frame contrast or compare all proposals across frames globally, we explicitly model the motion of each instance by comparing its features with those in the same spatial region in adjacent frames, thus obtaining more accurate motion saliency cues. 2. We synchronize the spatio-temporal saliency cues in a single graph for joint optimization, which exhibits better dynamics compared to the previous stage-wise methods that prioritize spatial cues followed by temporal cues. Additionally, we propose a simple yet effective video retargeting method based on video saliency ranking. Extensive experiments demonstrate the superiority of our model in video salient object ranking and the effectiveness of the video retargeting method. Our codes/models are released at [https://github.com/zyf-815/VSOR/tree/main](https://github.com/zyf-815/VSOR/tree/main).

A Motion-aware Spatio-temporal Graph for Video Salient Object Ranking
Hao Chen!.2 Yufei Zhu and Yongjian Deng²
School of Computer Science and Engineering, Southeast University, Nanjing, China Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary
Applications (Southeast University), Ministry Education, China.
College of Computer Science, Beijing University of Technology, Beijing, China fhaochen303, 220232213)&seu. wetesseeeuuuuueeeeeeeeeeeeeeeeee cn, yjdeng@bjut. cn
Abstract
Video salient object ranking aims to simulate the human attention mechanism by dynamically prioritizing the visual attraction objects in scene over time. Despite its numerous practical applications, this remains underexplored. In this work, we propose graph model for video salient object ranking. This graph simultaneously explores multi-scale spatial contrasts and intra-finter-instance temporal correlations across frames to extract diverse spatio spatio-temporal saliency cues. It has two advantages: 1. Unlike previous methods that only perform global inter-frame contrast or compare all proposals across frames globally, we explicitly model the motion of each instance comparing its features with those in the same spatial region in adjacent frames, thus obtaining more accurate motion saliency cues. 2. We synchronize the spatio-temporal saliency joint optimization, which exhibits better dynamics compared to the previous stage- wise methods that prioritize spatial cues followed by temporal cues. Additionally, we propose a simple yet effective video retargeting method based on video saliency ranking. Extensive experiments demonstrate the superiority our model in video salient object ranking and the effectiveness of the video retargeting method. Our codes/models are released http:///iihbbbooomyyyysssSSSStmh
in single graph for
1 Introduction
Lin's
Ours
GT
Figure 1: Previous methods (e.g., I] in video SOR primarily focus on capturing temporal correlations between global features across frames, resulting in limited ability effectively model temporal saliency. These methods tend to emphasize objects with prominent static saliency cues (i.e., the man right), whereas our proposed model places greater emphasis instance-wise temporal correlation, resulting in more accurate results (i.e. the walking woman left).
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Salient Object Ranking (SOR) 2 is to mimic the visual priority coding mechanism 7 in the human visual system to rank the attractiveness of objects in a scene. It has a large range of applications such as image caption
g 9
image compression [1O] 11] and image retargeting [12 13 Previous
research primarily focuses on ranking salient objects in static images. However, Video Salient Object
Ranking (VSOR) 1
14 holds specific potential for applications such as video summarization and
surveillance. It also presents distinct challenges due to the changing saliency ranking of objects within a sequence. Nonetheless, this area remains understudied.
SVSNet [4] is a pioneering work in Video Salient Object Ranking (VSOR) that leverages the proportion of eye fixation points on each object to infer the relative saliency ranking among objects. The model employs frame-level LSTM to capture global temporal information for saliency detection and fixation prediction. However, this approach depends on the availability of eye fixation labels, which are often absent. Moreover, relies on an external object detector [15] to generate bounding boxes for computing fixation density, rendering the network non end-to-end. Additionally, it lacks instance- wise temporal modeling.
Recently, Lin T propose an end-to-end solution by integrating the object detector into the VSOR model for joint training. The VSOR results are obtained by initially modeling the correlations among all objects in each frame to perform individual image SOR. Subsequently, these static image SOR results are combined by considering inter-frame correlations to produce the final outcome.
Although this approach eliminates the dependency on eye fixations and achieves and-to-end video SOR, it still has a significant issue: as shown in the 1st row in Figure 1 it directly builds correlations among all instances across frames without explicitly differentiating instance-wise motion, thus remaining agnostic to instance-level motion saliency.
As the human visual system naturally focuses more on moving objects, our core motivation is to explicitly explore the inter-frame variation for each instance. Considering the lack of annotations for object tracking, we propose simple and effective method to capture instance-level motion information. shown in the 2nd row in Figure | since the absolute positions of objects between two frames do not change drastically, we can approximate the instance motion by comparing the features of instance at the same position across frames. Additionally, we double the area of the bounding boxes for comparison to reduce errors caused by camera motion.
Spatial saliency in a frame is also indispensable for VSOR. Drawing inspiration from 22. we also conduct multi-scale contrast contrast-between an instance and its larger local region, the global image, and another instance-in order to explore diverse spatial saliency cues in graph neural network (GNN).
The following question is how to combine and optimize the temporal and spatial graphs. In previous work M. a sequential combination of spatial and temporal graphs is adopted, but in our opinion, this approach lacks adaptability and may exhibit bias towards one side. For instance, when initially aggregating spatial information, the instance features contain spatial saliency signals, which can hinder the extraction of temporal features. Therefore, we propose to jointly optimize the various spatial and temporal relations within a unified graph. This joint graph facilitates more comprehensive and dynamic interactions among instances and frames, consequently enhancing the fusion sufficiency of diverse saliency cues and improving the ranking results.
As an emerging field, we also explore its application scenarios. The ranking of salient objects in videos reveals the varying degrees of attention and trajectory shifts towards different regions during video playback. This information happens be crucial for the task of video retargeting [6 12). Therefore, we propose method that applies video saliency object ranking to video retargeting. Our method is straightforward. We assign different weights salient objects based on their saliency rankings, and then locate the saliency centroid of the current frame as the center for retargeting. Experimental results demonstrate that our proposed simple method effectively preserves the regions of interest with greater accuracy and comprehensiveness while mitigating background interference. Additionally, it exhibits better inter-frame continuity.
In summary, our contributions are as follows:
(1) Introducing a novel unified spatial-temporal graph that explicitly integrates motion-aware temporal correlation and adaptively integrate spatial and temporal saliency cues.
(2) Proposing simple yet efficient video retargeting method based on VSOR.
2

---

(3) Conducting experiments that validate the efficacy of our temporal saliency cues, the strategy for fusing spatial and temporal cues, and the superior performance of our VSOR and retargeting models compared to state-of-the-art methods.
2 Related Work
2.1 Salient Object Ranking on Images
Islam (3] first notice that it is difficult for people form unified judgment about which object is the most salient in scene. Therefore, they propose the problem of relative saliency, namely saliency ranking, and use stage-wise refinement model to gradually detect objects with different saliency levels. However, the use of pixel-level saliency ranking often results in varying saliency values for different pixels in the same instance.
To address this problem, a substantial amount of subsequent work [4] 5 2 6 has proposed instance- level saliency ranking tasks. Siris 4 infer the saliency ranking by modeling the human attention transfer mechanism with a combined bottom-up and top-down attention framework. Considering the influence of low-level features such the position and size of objects instance saliency, Fang S proposed a PPA module to model position information. Liu 2 utilize graph neural networks to comprehensively consider the spatial correlations, including the contrast between instances, the distinctiveness of instance in its local/global contexts, as well as prior knowledge of categories.
These methods achieve good results in saliency ranking static images, but for videos, the motion of objects greatly affects their saliency order. Therefore, existing saliency ranking models for images cannot achieve good results and specific designs towards the inter-frame relations are required.
2.2 Video Salient Object Detection
Another topic related to our task is video salient object detection. Unlike static image salient object detection, it [1] needs to jointly consider the different saliency cues generated by the motion information across frames and static appearance. Li [19] use optical flow to model temporal features and combine spatial-temporal relationships through attention mechanisms. Fan (0 further highlight the attention shift problem resulted from motion and propose a recurrent model to model temporal dynamics and attention-shift jointly. Liu (T consider single-frane appearance information,
long/short-tenn -eer motion and apatial-tempote cues jointly in transformer.
However, these works focus detecting the most salient object from video, while salient object ranking task is more complicated and with more temporal dynamics. Also, these works fails to explore the instance-wise motion cues, which is crucial change saliency ranking results.
2.3 Video Salient Object Ranking
Wang 14 first introduce the video salient object ranking task and propose a multi-task network SVSNet. However, SVSNet only utilizes LSTT to capture global temporal features, ignoring the instance-level features that are indispensable to saliency ranking. Furthermore, the utilization of extra detectors SVSNet leads to the model not being end-to-end. Lin M1 address these problems by proposing an end-to-end architecture, where the relationships between objects and temporal information are combined assign rankings each object. However, the temporal level, Lin only use global features as saliency cues, ignoring the motion information of each instance.
Instead, we optimize the multi-scale spatial contrasts, inter-frame instance relations, and instance- wise local motion at the same time, and the resulting network is equipped with comprehensive saliency cues and enjoys better fusion adaptivity.
3 Method
3.1
Overview
As shown Figure 2 our proposed VSOR framework firstly segments instances based on Mask R-CNN (22). and then enhance instance features through attention mechanisms [ and introduce
3

---

Spatisl-teporal Graph Reasoning
Objjcct Desector
Attention
Saliency
Position Emiolling
Spatial Correlation Modeling
Object
Instance feature
SOR Ressik
T
Ubject Detector
Pool
Temporal Corvelation Modeling
Local context intance
Spatial Correlation Modeling
Temporal Correlation Modeling
Global Interaction
Trajectery-mise Contrast
f Global context of current frame
Local
Local
Contrast
Instance Interaction
Contrast
Locul context adjacent frames
Instance Interaction
Figure 2: The main architecture of our model. Firstly, we obtain instances of each frame through an object detector and extract features of each instance using an attention mechanism and position embedding. Then, we utilize a spatial-temporal graph reasoning network to fuse spatial-temport saliency cues, ultimately obtaining saliency scores and final results.
Subsequently, A spatial-temporal graph neural network is crafted to jointly optimize multi-scale spatial and temporal contrast cues to generate integrated instance-level saliency features. Finally, fully connected neural network employed to infer saliency scores. which, combined with the results of instance segmentation, yield the final saliency ranking map. This model can be trained end-to-end.
position and scale saliency prior using PPA 5
3.2 Unified Spatial-temporal Graph for VSOR
After obtaining the detection results through Mask R-CNN, instance features fi are introduced into the Spatial-temporal Graph Reasoning model. Our graph reasoning model includes views: spatial correlation modeling and temporal correlation modeling. In spatial correlation modeling, draw inspiration from 2 introduce the interaction between instances S the comparison between instances and their local contextual features S as well as the comparison between instances and global contextual features S We denote local contextual features as f/. where means the instance i, and global features are represented as f9, In temporal correlation module, we introduce the motion-aware contrast 7 and instance interaction T (1-1,4+1) means the motion features in frame - and + of instance i.
3.2.1 Spatial Correlation Modeling
Interaction between instances. The competition among multiple instances for saliency highlights the importance of inter-instance interaction saliency ranking. Therefore, we establish edges between instances to capture their correlations and the aggregation function is represented
N
AND
a W.ff
a N ReL" FATTA
(I)
1=1
where N means the number of instances and of, is the attention weight modulating the edge between instance and j, and Wr. U and V" are projection matrixes. || means connection between features.
Local contrast and global interaction. It clear that the saliency of an instance is influenced by both its local and global contexts (2] To capture the local contextual feature f we expand the bounding box of instance by doubling its size. For the global contextual features, we divide the image into 3 3 and utilize average pooling to extract each feature.

---

3.2.2 Temporal Correlation Modeling
Instance interaction. Similar to the interaction between instances in the spatial correlation module, the interaction between instances in different frames can also have an impact on the saliency of instances. Hence, we begin temporal correlation modeling by establishing edges between instances in adjacent frames and instances in the current frame. And the aggregation function is represented as:
Ne-1
Σ
Wa5::
0
(2)
Ni-1 + No+1
where N;-1 No+1 means the total number of instances in frame and + and ofy is the attention weight modulating the edge between instance and j.
Motion-wise contrast. Merely constructing temporal connections by modeling instance interaction is not enough to model temporal saliency, as the human visual system tends to prioritize moving objects. Hence, it is necessary to introduce independent construction of motion representations for each instance. To achieve this, for instance i, we double its bounding box and apply ROIAlign to extract features within this region the previous and next frames and represent them as f!-1 and f!+1 By comparing the instance features with local contextual features f and f(+1 in adjacent frames, are able model the saliency from the motion trajectory for instance i. The aggregation function is represented as:
Note Σ
o Waft: j=t-1441
d ReLU
3.2.3 Spatial-temporal Fusion
Another important issue is how to integrate spatial-temporal cues. An intuitive idea is to aggregate spatial and temporal correlation modeling results in a serial manner done in [1. However, this sequential integration strategy may embed spatial clues in the instance features, which is counter- productive extracting temporal saliency. To solve this problem, we propose unify spatial and temporal correlation modeling and optimize them jointly.
Drawing inspiration from the graph updating methods outlined in [2], we extend the overall graph to parallel subgraphs to stabilize the learning process and enrich the learned node interaction connections. Each subgraph independently learns node interactions. As result, the instance feature can be represented as:
A =
(
Whiika Nys
Whenhh
3.3 Loss Function
Our loss function can be computed as:
Ldet LSOR
(5)
Laet Lidss + Liozz Lemask is defined in Mask R-CNN [22] be used to detect the salient instance, where Letes Lhoas Lenask denote the classification loss, regression loss and mask loss respectively. LsoR is the ranking loss proposed by [2]:
C² LsOR Σ B,log(1 + exp((-s,) + s32)))
(6)
where B, is the dynamic loss weight and 841, 942 saliency scores predicted by our network.
5

---

Experiment
4.1 Experimental Setup
Dataset collection and annotation. Wang [4] propose a video saliency ranking task and the first dataset RVSOD, but they only focus on frame-level saliency and ignore the instance-level saliency. Therefore, RVSOD lacks instance-level annotations to evaulate VSOR models. To address this limitation, we utilize the manually annotated masks provided RVSOD obtain instance masks and assign the saliency ranking score to each instance based on the distribution of fixation points. In this way, the instance-level annotations are generated.
However, the RVSOD dataset suffers from two notable limitations: (a) a lack of complex scenes, and (b) most scenes consist of only one salient instance, rendering it unsuitable for evaluating the ability of salient object ranking (SOR). To address these issues, we utilize the video saliency detection dataset DAVSOD (20] extract saliency ranking results, thereby making more appropriate for video saliency ranking tasks. In order to fully utilize the DAVSOD, we classify the scenes in DAVSOD into three categories: (a) all video frames contain only one salient instance, (b) some video frames contain one salient instance, and (c) all video frames contain multiple salient instances. We discard the scenario described in (a) and proceed divide the remaining scenes into training and testing sets in 4:1 ratio for both scenarios (b) and (c). The generation of the instance-level annotations is the same as the RVSOD. The process of generating salient ranks and the statistical analysis of DAVSOD are presented in the supplementary material in section A.1
Evaluation metrics. SA-SOR and Mean Absolute Error (MAE) are used to evaluate the perfor- mance of our model in ranking and segmentation. SA-SOR can reflect both segmentation and ranking performance at the same time which make appear more reliable than the original SOR 33 or the SSOR 44) metric. MAE serves as reliable indicator of segmentation quality.
Implementation details. We adopt the training strategy outlined in B3, utilizing the pre-trained Mask R-CNN learned on the MS-COCO 2017 training split. The settings remain consistent throughout the experiment, with the box head solely dedicated classifying objects and non-objects. Stochastic gradient descent (SGD) is employed to optimize the loss function. To facilitate training, we implement warm-up strategy, commencing with an initial learning rate of 5e-3. At the 420,000 and 500,000 steps, the learning rate is reduced by a factor of 10.
Once the training of our detection branch is complete, we proceed to incorporate graph module and fine-tune the entire network using two separate datasets RVSOD and DAVSOD. Throughout this process, set the batch size to and the maximum iteration count to 200,000. For optimization, we utilize the Adam optimizer with an initial learning rate of 5e-6. At the 80,000th and 150,000th steps, the learning rate is reduced by factor of 10. Our model is implemented using Py PyTorch and all experiments are conducted on NVIDIA RTX4090.
4.2 Compared to State-of-the-art Methods
Table 1: Quantitative results on the generated datasets. The bigger the SA-SOR, the better the performance of ranking. The smaller the MAF, the better the performance of segmentation. The best results are indicated bold. The method by Lin [ for the results RVSOD.
is not yet open-source, so we only asked them
RVSOD
DAVSOD
Method
SA-SOR
MAE
SA-SOR
MAE
Fang
5
0.350
0.0984
0.157
0.0760
Liu 2
0.563
0.0728
0.179
0.0639
SOR
PSR 24
0.405
0.074
SeqRank 25
0.512
0.0761
DCFNet [26
0.1180
0.0684
VSOD
SCOTCH [27
0.1230
0.1126
Lin [
0.560
0.0745
VSOR
Ours
0.603
0.0698
0.207
0.0626

---

Image
Salient Instance
Liu's
Lin
Ours
GT
Figure Visual results with our model and other sota methods. The scenarios are chosen from respectively.
RV SOD/ /aclionclipless00055 and RVVOOD
/uclioonciipees00000 /
Quantitative results. We compare our proposed network with six other state-of-the-ase salient object ranking methods, including four image-based SOR methods, two video salient object detection methods and one available video-based SOR method. As shown in Table [ our model largely outperforms previous methods on both SA-SOR and MAE, indicating advantages of our model in modeling and fusing spatial-temporal saliency. We discuss the reasons for poor results on DAVSOD in the supplementary material in section A.2
Visual results. We also perform visual comparisons to verify the advantages of our method. As shown Figure 3 the results of highlight those objects with distinguished static saliency cues (e.g.. the green and red instances) such as larger size, closer distance or clearer appearance, while the blue instance with less static saliency cues yet rich motion cues is ranked the least salient, as this method only considers multi-scale spatial contrast and instance relation in an individual image. Although the method of []] additionally takes the temporal cues into account, its rankings are also biased towards static saliency cues it simply models temporal relations across the global representation of each frame, ignoring the instance-wise correspondence moving across frames.
In contrast, our method explicitly considers the motion cues of the instance and perform instance-level spatial and temporal reasoning in a unified graph. Our method successfully distinguish instances with less static saliency cues but semantically rich motion cues (c.g., the blue instances should be the central character).
4.3 Ablation Study
Table 2: Ablation study for our temporal interaction. Our baseline is the Liu's model without person prior. Then we consider three temporal features: global-aware temporal information(GTRMM instance-aware features(TRMM), and motion aware features(MTRMM, Our final method combines the above three temporal features simultaneously.
SA-SOR
Method
RVSOD
DAVSOD
Baseline( w/o TRM
0.563
0.179
Baseline GTRM
0.585
0.186
Baseline + ITRM
0.591
0.194
Baseline + MTRM
0.587
0.195
Ours(joint)
0.603
0.207
Effectiveness on the temporal interaction. We firstly study the advantages of our motion-awan temporal correlation modeling. Specifically, the model that excludes temporary relationship modeling serves as the baseline. Then, we have explored three types of temporary relationship modeling, including GTRM (global-aware temporal relation modeling), ITRM (instance-aware temporal relation modeling) and our motion-awan aware temporal relation modeling (MTRM). For GTRM, utilize the

---

Image
Salient Instance
w/o TRM
w/ GTRM
W/MTRM
Ours
GT
Figure 4: Visual results ablation study. GTRM means global-aware temporal relation modeling and ITRM means instance-aware temporal relation modeling. is obvious that our motion-aware features can effectively model the motion information of instances. The scenarios are chosen from RV /ott///tnnnnlptrai0000000 and DAV Sel//seeeteeDD/e 0674 respectively.
/
global features of the preceding and subsequent frames as temporal saliency cues. ITRM and MTRM represent instance interaction and motion- wise contrast the method section respectively. Table 2 illlustrates the large improvement by adding the motion-awan temporal interaction. It's obvious that either method greatly improves the ranking performance the model compared to the baseline. Moreover, our proposed motion-aware feature is significantly superior to traditional temporal modules. Therefore, our final method combines the above three temporal features simultaneously.
In Figure 4 we observe that, when performing graph reasoning based purely on multi-scale spatial contrasts, the ranking shown in the 3th row tends to favor static saliency cues (e.g.. larger-sized individuals those in closer proximity). Despite introducing global inter-frame contrast ("w/GTRM") and frter- cross-instance contrast ("w/ITRM") can capture some temporal cues, they are still unable to effectively model instance motion. However, by incorporating correlation modeling into the graph, our method can effectively identify instances with noticcable motion cues, leading to successfully highlight of motion instance and superior performance.
motion-awart temporal
Table 3: Quantitative comparison by varying the bounding box sizes.
SA
SOR
Expand Scale
RVSOD
DAVSOD
1x
0.6092
0.2421
2x
0.629
0.246
3x 0.5900
0.2374
4x
0.5684
0.2393
Effectiveness on the local context size in motion-aware temporal relation modeling. Without access to ground truth tracking annotations, it becomes quite challenging to model the motion trajectories of each individual instance accurately. Therefore, motion-aware temporal relation modeling are proposed to evaluate the temporal saliency of the instance. However, as the instance moves and the camera shifts, simply projecting the bounding box onto adjacent frames not enough to fully capture the changes in the instance. As shown in Table double sized bounding box achieved the best results, while excessive expand scale can easily introduce excessive background noise and degrade the results.

---

Table ablation study for our fusion methods. Our baseline is the Liu's [2] model without person
prior. Then we merge spatio-temporal relationships in three different orders.
SA-SOR
Method
RVSOD
DAVSOD
Baseline( w/o TRM)
0,563
0.179
thaniillthee-temporal
0.571
0.184
Temporal-then-spatial
0,590
0.183
Ours(joint)
0.603
0.207
Effectiveness on the unified spatial-temporal reasoning. Another question we want to investigate is how make joint decisions based spatial and temporal saliency cues. An intuitive choice to forming the spatial and temporal graph separately and combine two graphs in a stage-wise manner. Hence, involve two fusion variants for comparison. a) ppatial-then-temprral start by aggregating three consecutive frames' spatial correlation graph independently and then fusion the temporal information using temporal correlation graph. b) remporal-then-spaaee aggregating temporal correlation graph through the temporal context and then aggregate the spatial correlation graph with spatial context. c) Our joint spatial-temper
start by
graph reasoning: It takes all features as inputs simultancously and model their correlations adaptively for reasoning.
As shown in Table two stage-wiss reasoning strategies achieves similar improvement compared to the baseline that merely considers spatial ppatial-based reasoning, while our joint spatial-tempon reasoning strategy achieves much higher improvement than them, demonstrating method's better adaptivity and sufficiency in combining spatial and temporal saliency cues.
4.4 Application to Video Retargeting
Input videos
Results from VSOR
Saliency center generation
Cropping region
Smoething for time series
Instance saliency center
Image saliency center
Figure 5: Our method for video retargeting. We generate instance-level saliency information including bounding box, mask and ranks based on our VSOR model. Thus, we get the cropping center according to instance-level saliency information.
With the development of multi-media technology, people can watch videos on different devices such as smartphones and tablets. Video retargeting adjusts the videos to different aspect ratios to suit different endpoints. The purpose of image retargeting is to preserve visually salient areas for better visual experience. Based on this, image retargeting has been extremely successful in the past through methods such seam carving (18)31), cropping 35 and warping (36.39] However, simply applying these methods to video retargeting will lead to severe temporal inconsistency that greatly affect the human visual experience. Some works [12 13 have been proposed to alleviate temporal inconsistency. For example, Zhu [12 utilize a dynamic spatial-temporal buffer to reduce temporal inconsistency, while Apostolidis [3 utilizes cropping for video retargeting based on saliency detection [40]. However, their method only distinguishes between foreground and background, ignoring the non-uniform saliency across instances in the foreground, thus usually resulting in biased retargeting to backgrounds or objects with less saliecncy.
Based on our VSOR, we can incorporate the saliency of instances into retargeting, thereby more adaptively preserving important semantic concepts multi-object scenes. As shown Figure 5 based on the rank of each salient instance with our VSOR model, We assign weights each instance to get the cropping center. The center instance with rank, (the higher number means the more salient the instance i) is represented (z,, v1). Next based on the ranks of instances, generate the

---

cropping center (Ze, Nc) with the equation:
(Fer Mc) rank; x (xi v:))
rank;)
Then we follow the smart VidCrop 1 to generate the crop windows for each frame and utilize the LOESS 41 to overcome visual mutations caused by camera shake.
Image
2:3
(a)
(b)
(c)
(a)
(c)
Figure 6: Comparison of retargeting, (a) is the method of seam carving [2], (b) is smartVideoCrop 13 (c) is our methods. The yellow box in (a) shows the artifacts after retargeting.
Our retargeting results are shown in Figure 6 we retarget the videos with the aspect ratio as 2:3 and 4:5 respectively. We compare method with seam-carving based method proposed by Zhu [12) and crop-based method [13.. It is evident that our approach offers superior visual experience compared the other methods. For instance, while the method in [12] detects all salient instances, it causes significant distortion to those instances, which negatively impacts the video quality. In contrast, Apostolidis's 's method [3] sometimes fails crop all the salient instances gets distracted by the background information, resulting in inaccurate cropping windows. This indicates that our VSOR provides more precise semantic importance for the frames, allowing for more adaptive and reasonable results. Additional examples are available in the supplementary material in section A.3
5 Conclusion
In this work, we propose a novel unified spatial-temporal graph that explicitly integrates trajectory- wise temporal correlation and adaptively integrate spatial and temporal saliency cues. Specifically, the first step involves extracting the features of each instance through an object detector and a feature enhancement module. And then spatial-temporal graph reasoning network to fuse spatial-temporal saliency cues, ultimately obtaining saliency scores and final results. Finally, we applied our VSOR model video retargeting and achieved impressive results.
Limitations. In our model, a detector is used extract instance features. However, in different scenarios, the same object may exhibit different levels of saliency. For example, we should detect tree a desert but not in a forest. For non-salient objects, although our model assigns them lower saliency ranking, we ideally do not want them to be detected at the first. In the future, we aim to design a saliency-sensitive detector to further improve performance.
Acknowledgement
This work supported in part by the National Natural Science Foundation of China (NSFC) under Grant 62102083, Natural Science Foundation of Jiangsu Province under Grant BK20210222, the National Natural Science Foundation of China (NSFC) under Grant 62261 160576, 62203024 and 92167102 and the R&D Program of Beijing Municipal Education Commission (KM2023 10005027).
10

---

References
[1] Jiaying Lin, Huankang Guan, and Lau RynsonWH. Rethinking video salient object ranking. arXiv preprint arXiv:2203. 17257, 2022.
[2] Nian Liu, Long Li. Wangbo Zhao, Junwei Han, and Ling Shao. Instance- relative saliency ranking with graph reasoning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11):8321-85 8337, 2021.
[3] Md Amirul Islam, Mahmoud Kalash, and Neil DB Bruce. Revisiting salient object detection: Simultancous detection, ranking, and subitizing of multiple salient objects. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7 7142 7150, 2018.
[4] Avishek Siris, Jianbo Jiao, Cary KL Tam, Xianghua Xie, and Rynson WH Lau. Inferring attention shift ranks of objects for image saliency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12133-12143 2020.
[5] Hao Fang, Daoxin Zhang, Yi Zhang, Minghao Chen, Jiawei Li, Yao Hu, Deng Cai, and Xiaofei He. Salient object ranking with posttion-preserved Conference on Computer Vision, pages 16331-16344 2021.
preserved attention. In Proceedings of the IEEE/CVF International
[6] Xin Tian, Ke Xu, Xin Yang. Lin Du Baocai Yin, and Rynson WH Lau. Bi-directional object-context prioritization learning for saliency ranking. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5882-5891, 2022.
Nicole C Rust and Marlene R Cohen. Priority coding in the visual system. Nature Reviews Neuroscience, 23(6):376 388, 2022.
[8] Kelvin Xu, Jimmy Ba. Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International Conference on Machine Learning, pages 2048-2057. PMI.R, 2015.
[9] Songtao Ding, Shiru Qu, Yuling Xi, and Shaohua Wan. Stimulus- and concept- analysis for image caption generation. Nenrocomputing, 598:520-530, 2020.
[10] Li- Li-Heng Chen, Christos G Bampis, Zhi Li, Andrey Norkin, and Alan C Bovik. Proxiqa: A proxy approach to perceptual optimization of learned image compression. JEEE Transactions on Image Processing, 30:360-373, 2020.
[11] David Tellez, Geert Litjens, Jeroen Van der Laak, and Francesco Ciompi. Neural image compression for gigapixel histopathology image analysis. IEEF Transactions on Puttern Analysis and Machine Intelligence, 43(2):567-578, 2019.
[12] Zhu Chuning. Fast video retargeting based seam carving with parental labeling. arXiv preprint 19X1v:1903033180. 2019.
[13] Konstantinos Apostolidis and Vasileios Mezaris. A fast smart-cropping method and dataset for video retargeting. In 202/ IEEE International Conference on Image Processing (ICIP), pages 2618-2622. IRRE, 2021.
[14] Zheng Wang, Xinyu Yan. Yahong Han, and Meijun Sun. Ranking video salient object detection. In Proceedings of the 27th ACM International Conference on Multimedia, pages 873-881, 2019.
[15] Shaoqing Ren, Kaiming Hc, Ross Girshick, and Jian Sun. Faster r-cnn: Towards cal--time object detection with region proposal networks. Advances in Neural Information Processing Systems, 28, 2015.
[16] Sung In Cho and Suk-Ju Kang. Temporal incoherence- video retargeting using foreground aware extrapolation. IEEE Transactions on Image Processing, 29:4848 4861, 2020.
[17] Hyunwoo Nam, Dubok Park, and Kangwon Jeon. Jitter-robust video retargeting with kalman filter and attention saliency fusion network. In 2020 IEEE International Conference on Image Processing (ICIP), pages 858 862. IEEE, 2020.
[18] Christel Chamaret and Olivier Le Meur. Attention-based video reframing: Validation using eye-tracking. In 2008 19th International Conference on Pattern Recognition, pages 1-4, IEEE, 2008
[19] Haofeng Li, Guanqi Chen, Guanbin Li, and Yizhou Yu. Motion guided attention for video salient object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7274-7283, 2019.
11

---

[20] Deng-Ping Fan, Wenguan Wang. Ming-Ming Cheng, and Jianbing Shen. Shifting more attention to video salient object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8554-8564, 2019.
[21] Nian Liu, Kepan Nan, Wangho Zhao, Xiwen Yao, and Junwei Han. Learning complementary spatial- temporal transformer for video salient object detection. IEEE Transactions Neural Networks and Learning Systems, 2023.
[22] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE International Conference on Computer Vision, pages 2961-2969, 2017.
[23] Sanghyun Woo, Jongchan Park, Joon-Young Young Lec, and In So Kweon. Cbam: Convolutional block attention module. In Proceedings of the European Conference on Computer Vision, pages 19, 2018.
[24] Chengxiao Sun, Yan Xu, Jialun Pei. Haopeng Fang, and He Tang. Partitioned saliency ranking with dense pyramid transformers. In Proceedings of the s1 ACM International Conference on Multimedia, pages 1874 1883, 2023.
[25] Huankang Guan and Rynson WH Lau. Seqrank: Sequential ranking of salient objects. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 1941-1949, 2024.
[26] Miao Zhang, Jie Liu, Yifei Wang. Yongri Piao, Shunyu Yao, Wei Ji, Jingjing Li, Huchuan Lu, and Zhongxuan Luo. Dynamic context-sensitive filtering network for video salient object detection. In Proceedings of the IEEE/CVF international conference on computer vision. pages 1553-1563, 2021.
[27] Lihao Liu, Jean Prost, Lei Zhu, Nicolas Papadakis, Pietro Liò, Carola-Bibiane Schönlich, and Angelica I Aviles-Rivero. Scotch and soda: A transformer video shadow detection framework. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 10449-10458, 2023.
[28] Yanfei Peng, Jing Wang, Xiaoxuan Liu, and Shengjie Gong. Seam carving algorithm based on partitioning. In 2023 IEEE International Conference on Control, Electronics and Computer Technology, pages 1173- 176. IEEE, 2023.
[29] Srishti Sharma and Yatharth Piplani. Comparative analysis of seam carving in images. In Proceedings of the Second International Conference on Information Management and Machine Intelligence, pages 139 199-146.555 Springer, 2021.
[30] Shai Avidan and Ariel Shamir. Seam carving for content-aware image resizing. In Seminal Graphics Pupers: Pushing the Boundaries, Volume 2. pages 609-617. 2023.
[31] Mahdi Hashemzadeh, Bahareh Asheghi, and Nacer Farajzadeh. Content-aware image resizing: An improved and pressevvigee seam carving method. Signal Processing, 155:233-244 2019.
[32] Li-Qun Chen, Xing Xie, Xin Fan, Wei- Ying Ma, Hong-Jiang Zhang, and He-Qin Zhou, A visual attention model for adapting images on small displays. Multimedia Systems, 9:353-364, 2003.
[33] Hao Liu, Xing Xie, Wei- Ying Ma, and Hong-Jiang Zhang. Automatic browsing of large pictures mobile devices. In Proceedings of the eleventh ACM International Conference on Multimedia, pages 148-155, 2003.
[34] Anthony Santella, Maneesh Agrawala, Doug DeCarlo, David Salesin, and Michael Cohen. Gaze-based interaction for semi-automaticc photo cropping. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 771-780, 2006.
[35] Bongwon Suh, Haibin Ling, Benjamin B Bederson, and David W Jacobs. Automatic thumbnail cropping and its effectiveness. In Proceedings of the 16th annual ACM Symposium on User Interface Software and Technology, pages 104, 2003.
[36] Yu-Shuen Wang, Chiew-Lan Tai, Olga Sorkine, and Tong-Yee Lee. Optimized scale-and-stretch for image resizing. In ACM SIGGRAPH Asia 2008 popers. pages 2008
[37] Yanwen Guo, Feng Liu, Jian Shi, Zui-Hua Zhou, and Michael Gleicher. Image retargeting using mesh parametrization. IEEE Transactions on Multimedia, 1 165)8856-867 2009.
[38] Bing Li, Yiming Chen, Jinqiao Wang, Ling- Yu Duan, and Wen Gao. Fast retargeting with adaptive grid optimization. In 20/1 IEEE International Conference Multimedia and Expo, pages 1-4, IEEE, 2011.
[39] Pierre- Yves Laffont, Jong Yun Jun, Christian Wolf, Yu- Wing Tai, Khalid Idrissi, George Drettakis, and Sung- Eui Yoon. Interactive content- aware zooming, In Graphics Interface 2010 Conference, pages 77-87. Canadian Information Processing Society Toronto, Ont., Canada, 2010.
12

---

[40] Richard Droste, Jianbo Jiao, and J Alison Noble. Unitied image and video saliency modeling, In Computer Vision-ECCH 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part V 16. pages 415- Springer, 2020.
[41] William S Cleveland and Susan J Devlin. Locally weighted regression: approach to regression analysis by local fitting, Journal of the American Statistical Association, 83(403).596 610, 1988.
13

---

A Supplemental Material
A.1 Details of Dataset Generation
Data collection
Mask generation for each instance
Rank instances based on fixation maps
videos with single instance
videos with multiple instances
Figure 7: Flowchat for dataset collection and annotation.
Table 5: Statistical analysis of DAVSOD dataset for video salient object ranking. Semi valid indicates that some video frames the scene only have one salient instance, valid indicates that the scene video frame contains multiple salient instances, and invalid indicates that the video frame only contains one salient instance.
Number of scenes semi valid
Number of images
Dataset
valid
invalid
valid
Training set Test set
53
40
2469
9067
13
9
723
2364
As shown in Figure 7 We select videos with varying object numbers and then determine the saliency rank by the given instance masks and comparing the given fixation numbers for each instance. Table 5 represents the statistical analysis of DAVSOD dataset for video salient object ranking. Semi valid indicates that some video frames in the scene only have one salient instance, valid indicates that the scene or video frame contains multiple salient instances, and invalid indicates that the video frame only contains one salient instance.
A.2 Analysis for Poor Results on DAVSOD
Table 6: Analysis for DAVSOD test set.
Category
SA-SOR
mAP
Proportion
Example
(a) hard to detect
-0.07
0.50
5/22
select_ -05ct, select_ (0eect_00,,,, select_0572
(b) low quality of labeling
0.16
0.44
7/22
select_0600 Select_0343 select_055 0577
(c) others
0.45
0.70
10/22
We perform an in-depth study of the DAVSOD test set in Table 6 The results in the table below classify scenes by challenge, proportion, and examples, and report ranking (SA-SOR) and detection (mAP) performance using our model trained on the full DAVSOD training set. Analysis of these results reveals two key reasons for the low performance:
a) Severe occlusion among multiple objects or objects with very small sizes, making it difficult to successfully detect all salient objects (e.g., the instructor and the person skydiving with blocked each other are perceived as a single entity in the select 0557 video). Similar data includes: select 0208, select 0557, select delect_0590____ 3,, select_ 0257 and @elect_05722
b) Severe variance salient objects between adjacent frames: The DAVSOD salient object annota- tions are based on subjective eye fixations from multiple testers, which exhibit increased variance as the number of objects increases. This results inconsistent and unreliable SOD and ranking labels, fixations shift significantly across frames in scenes with diverse objects. Compared to RVSOD,
14

---

DAVSOD contains much more such scenes with multiple objects, where the saliency of individual objects can flicker between salient and non-salient adjacent frames (e.g.. the bull and person in the select_0000 bullfighting video). Similar data includes: select_ select_0000 select_0297 select_0653, select select_0577, select _044 and select_06i
A.3 More Visual Results of Video Retargeting
Figure The results of the seam-carving based method
12
smart-
vidco-crop
Ours
Figure 9: The results of the cropping based methods. The upper part the smart-video-crop and the lower part is our model. The red boxes in the images represent the crop area.
KK
As shown in Figure 8 the seam carving based method can easily cause image distortion which will result in extremely poor visual experiences. While in Figure 9 cropping based methods solve this problem. However, previous work [13 utilized image-level saliency for cropping, making the model is sensitive to noise and resulting in inaccurate cropping of salient objects. On the contrary, our method adopts instance-level saliency, keeping the salient object the center of the cropping box, thereby greatly improving the human visual experience.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: the introduction section we introduce the main contributions of our work and summarize them at the end.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Justification: We discusse the limitations of our work in the conclusion section 5 and propose future directions for our efforts.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
16

---

Answer: [NA]
Justification:
Our work does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: In the method section B architecture and introduce the implementation details in the experimental section??.
provide detailed description of our model
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
17

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We will provide the corresponding data and code after the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //iiss.cc/pill wwtpcccppppppppeeeeeeeeeeeeeeeee httpe//gggeeee/eeeeubbissssssss /Cc////oddiiisssseeeeeee ssiiiiiisiyyyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We introduce details at the beginning the experiment section ??.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Each experiment requires substantial amount of resources and time, do not conduct a statistical analysis.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
18

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have specific instructions in the implementation details of the experiment section ??.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp//e//eaausssccccc/////hchht
Answer: [Yes]
Justification: We have carefully read the NeurlPS Code of Ethics and ensured compliance with
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We apply the VSOR model video retargeting, which is advantageous in the context of today's 's advanced multimedia technology.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
19

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We ensure that the assets we use are licensed.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
20

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will provide the corresponding assets after the paper is accepted.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "Rsr2mN5Qde", "reviewer": "Reviewer_KoDU", "summary": "This paper proposes a video salient object ranking approach based on spatio-temporal graph, leveraging instance trajectories and spatio-temporal saliency cues to improve SOR accuracy. Experiments demonstrate the superiority of the proposed model.", "questions": "How do you obtain the saliency rankings and instance masks for the DAVSOD database?\n\nWhy set the batch size to 1 during training?", "limitations": "From my point of view, the \"Instance interaction\" only considers temporal correlations of instances bettwen two adjacent frames, which may be insufficient for modelling long-range temporal dependencies.", "rating": 7, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "Originality: The proposed approach has some original aspects.\n\nQuality: The overall quality of the paper is average.\n\nClarity: Certain sections of the paper lack clarity.\n\nSignificance: The proposed approach is significant to some extent.", "weaknesses": "● It seems that the proposed approach is an temporal extension version of [3], I cannot see much new insights from this paper.\n\n● In the introduction section, the motivation of introducing GNN for video salient object ranking is not well described, why using GNN in both spatial and temporal dimension of videos? The authors should briefly describe why and how you construct GNNs for SOR in this section.\n\n● Experiments are insufficient; some typical SOR approaches are not compared.\n\n● The approach for approximating instance trajectories may not be sufficiently accurate or generalizable in real-world scenarios, as it merely compares the features of an instance at the same spatial position across different frames.\n\n● Some missing related works:\n\n  ○ [1] Qiao et al. HyperSOR: Context-aware graph hypernetwork for salient object ranking\n\n  ○ [2] Guan et al. SeqRank: Sequential Ranking of Salient Objects\n\n● Reference：\n\n  ○ [3] Liu et al. Instance-level relative saliency ranking with graph reasoning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "###1 Weakness\n\n**1.** _It seems that the proposed approach is an temporal extension version of [1]._  \n**Response**  \nUnlike [1] that focuses on spatial saliency cues for static images, we focus on two new key problems for video SOR: 1) Modeling diverse temporal saliency cues, especially instance-level motion variations; 2) A unified graph to optimize spatial and temporal cues jointly. Additionally, we propose a simple yet effective VSOR-based video retargeting method, which significantly improves the performance of retargeting in preserving the key semantic information from the original video content.\n\n\n**2.** _The motivation of introducing GNN for video SOR, why using GNN in both spatial and temporal dimension of videos? How to construct GNNs for SOR._  \n**Response**  \n1) As noted in prior work [1], the intricate interaction and competition relationships among different object instances are the most critical cues for inferring their relative saliency.\n2) Similarly, optimizing the delicate balance between the cooperative and competitive relationships of spatial and temporal saliency cues is the key to accurately determining the overall saliency priority in videos.\n3) However, these complex relational dynamics cannot be effectively captured by traditional CNNs. In contrast, GNNs offer a powerful framework to explicitly model diverse relationships and optimize their combination for joint saliency inference by learning the edge connections among nodes.\n4) The features for different spatial scales (i.e., each instance, its local context, its global context) and its local contexts in adjacent frames are treated as nodes. Edges are then constructed among them to build multi-scale spatial relations and instance-wise temporal correlations. By optimizing the various edges, the GNN can adaptively combine diverse spatial-temporal relationships and saliency cues for joint inference. \n\n\n**3.** _Insufficient comparison to some typical SOR approaches._  \n**Response**  \n1) Our primary focus is on modeling temporal saliency cues and effectively combining them with spatial saliency information. Therefore, we believe directly comparing our approach to image-based SOR methods is not the most pertinent way to highlight these key contributions.\n2) However, we do include such a comparison in Table R1, which demonstrates that our method achieves significant performance improvements over traditional SOR methods, even they adopt a more powerful backbone (e.g., SwinTransformer in SeqRank) than ours (ResNet50). This clearly showcases the effectiveness of our temporal saliency modeling and the proposed spatial-temporal fusion strategy.\n\n\n**4.** _The approach for approximating instance trajectories may not be sufficiently accurate or generalizable in real-world scenarios, as it merely compares the features of an instance at the same spatial position across different frames._  \n**Response**    \n1) Our core motivation is to measure the magnitude of motion for individual object instances across adjacent frames, in order to infer instance-level temporal saliency in accordance with the human attention mechanism. Therefore, while tracking or re-identifying each object instance can be helpful, it is not our original motivation and necessarily a strict requirement for our task.  \n2) In fact, instances that move quickly and fail to be accurately tracked will consequently exhibit large inter-frame feature contrast, and thus obtain a high saliency priority (see Fig. R1 in the PDF). Conversely, even if we successfully track a rigid instance (e.g., a fast-moving football), directly comparing the appearance features between two completed instances will result in a high similarity score and low saliency. Hence, the key to determining temporal saliency lies not in the tracking itself, but in the quantification of instance-level motion dynamics.   \n3) Additionally, without access to ground truth tracking annotations, it becomes quite challenging to model the motion trajectories of each individual instance accurately. Instead, our approach focuses on directly quantifying the magnitude of motion at the instance level, which can effectively capture the temporal saliency cues even in the absence of precise tracking information.  \n\n\n**5.** _Some missing related works._  \n**Response**  \nThank you for providing these related works. Both are image-based SOR models, and therefore focus on exploring spatial saliency cues. Specifically, Qiao et al. investigate the influence of scene context on saliency ranking by constructing a new dataset with varying contexts and building a hypergraph to model diverse spatial relations. Guan et al. formulate the image-based SOR as a sequential and continuous process.  \nUnlike these approaches that focus on spatial cues, our core motivation lies in modeling the temporal saliency cues and the adaptive combination of spatial and temporal ones. We will involve more related works in the manuscript to enhance the literature review.\n\n\n\n###2  Question\n\n**1.** _How do you obtain the saliency rankings and instance masks for the DAVSOD database?_  \n**Response**  \nThe DAVSOD dataset provides us with eye fixation distributions and instance-level masks. We determine the saliency level of different instances based on the total number of fixation points assigned to each, i.e., an instance with more total fixation points will be assigned a higher saliency priority.\n\n\n**2.** _Why set the batch size to 1 during training?_  \n**Response**   \nThis setting inherits the configuration from [1] without any additional tuning, for the sake of fair comparison.\n\n###3 Reference\n\n[1]  Instance-level relative saliency ranking with graph reasoning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FM5CLqJQlG", "reviewer": "Reviewer_kDtj", "summary": "This paper introduces a graph model for the video salient objects ranking task. Distinguishing itself from prior research, this study incorporates instance trajectory modeling to amplify temporal saliency cues. Additionally, the authors present a cohesive optimization approach that seamlessly integrates spatial and temporal saliency cues in an adaptive manner. Also, a VSOR-based video retargeting method is introduced, demonstrating notable advantages over existing techniques. Experimental results underscore the exceptional performance of the proposed spatial-temporal graph for VSOR, as well as the effectiveness of the accompanying video retargeting model.", "questions": "- I am interested in understanding the testing time performance of both the proposed graph model and the video retargeting model. This information would provide valuable insights into the practicality and efficiency of the approach.\n- I am curious about how the proposed ITRM and TTRM contribute to the salient object detection and ranking processes. A comparison involving representative samples would be insightful, showcasing the specific benefits of these modules in inferring salient objects.\n- Regarding the determination of local context size in line 141, the authors mention doubling the size. However, more details on the criteria or heuristic used to determine this size would be beneficial for a complete understanding.\n- In Figure 3 and Figure 4, it would be helpful to specify which dataset the samples were chosen from. Additionally, in Figure 2, a brief explanation of the different colors or line types used to represent various elements or metrics would improve clarity.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- In general, the presentation of the methodology is clear, concise, and easy to follow, making it accessible to readers.\n- The literature review is thorough and well-analyzed, providing a solid foundation for introducing instance trajectory cues as a reasonable motivation. \n- The proposed method effectively addresses the limitations of previous temporal saliency modeling approaches, demonstrating its well-grounded nature. The trajectory-wise temporal graph introduced is highly effective, and the joint optimization strategy for integrating spatial and temporal graphs seems to be both reasonable and adaptive. \n- The authors comprehensively consider various spatial and temporal saliency cues, employing an integration strategy that fully explores VSOR, resulting in significant experimental improvements.\n- The proposed video retargeting method based on VSOR is intriguing. The concept of utilizing salient object ranking to determine the cropping center is both intuitive and innovative, offering a simple yet effective solution.", "weaknesses": "- I would appreciate seeing some failure cases and their corresponding analyses for the proposed method. Including these would provide a more comprehensive understanding of the properties and limitations of the approach.\n- While the proposed video retargeting method outperforms previous crop-based solutions in localizing the saliency center and preserving key semantics, it's important to acknowledge that crop-based approaches inherently struggle when salient objects are dispersed across different areas, potentially leading to the loss of foreground regions.\n- To clarify the dataset collection and annotation process in Section 4.1, presenting a flowchart would be a helpful addition.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "###1 Weakness\n\n**1.** _Some failure cases._  \n**Response**  \nAs shown in Fig. R2, the final saliency inference is highly influenced by the object detector performance. Poor detector output leads to inaccurate instance features and wrong saliency ranking.\nIn future, we plan to address this by two ways:\n1) Adapting the detector to incorporate saliency priors, enabling joint optimizing detection and saliency estimation. \n2) Exploring a unified model that learns detection and saliency estimation simultaneously, allowing the components to benefit from each other's signals.\n\n**2.** _Limitation of crop-based approaches when salient objects are dispersed across different areas, potentially leading to the loss of foreground regions._  \n**Response**  \nYes, the cropping-based method does have the common limitation of being constrained by the original image aspect ratio. However, from a practical business implementation perspective, the cropping approach is the safest option as it faithfully preserves the content of the original image without introducing any distortions.\nThis is an important consideration, as users may become uncomfortable with visual distortions introduced by more aggressive techniques like seam carving. Maintaining the integrity of the original image can help avoid potential user criticism or dissatisfaction, which is a crucial factor for real-world deployment.\n\n**3.** _The flowchart for dataset collection and annotation._  \n**Response**  \nThank you for your suggestion. The flowchart can be found in Fig. R3 in the one-page PDF, and we will add it in the final version if the paper is published. We select videos with varying object numbers and then determine the saliency rank by the given instance masks and comparing the given fixation numbers for each instance.\n\n\n\n###2  Question\n\n**1.** _Testing time._  \n**Response**  \nFor video salient object ranking, our model can process 20 frames per second on an RTX 4090 GPU. In practical applications, increasing the batch size can further improve the inference speed. The inference time for the retargeting part is 820 frames per second.\n\n**2.** _How do the proposed ITRM and TTRM contribute to the salient object detection and ranking processes._  \n**Response**   \nThe ITRM explores global temporal cues by comparing all instances in adjacent frames, while the TTRM further captures instance-wise motion cues by comparing each instance and its local context in adjacent frames to approximate the instance trajectory.\n\nAs shown in Figure 4 in the manuscript, comparing the results of the fifth and sixth rows, we can see that the ITRM highlights an instance distinguishing from all instances in adjacent frames (e.g., the person with a large size in the first example), but tends to overlook instances without a distinctive appearance but exhibiting rich motion patterns (e.g., the blurry person with a small size in the first example).\n\nIn contrast, by capturing instance-wise temporal motion cues, our proposed TTRM method better highlights instances with significant temporal motion, even if they are smaller in size and undistinguished in appearance. This suggests that our TTRM is more effective at capturing the nuanced saliency cues associated with object dynamics and movement.\n\n**3.** _Determination criteria of local context size._   \n**Response**  \nWe set it empirically without experimental analyses. To further validate this design choice, we conducted ablation experiments to test the impact of different scaling factors, examining 1x, 2x, 3x, and 4x scaling.\n\nThe experimental results presented in Table R2 demonstrate that doubling the bounding box area, effectively a 2x scaling, achieved the best overall performance. This suggests that this level of spatial context expansion was the most beneficial for accurately capturing the relevant saliency cues, without introducing too much extraneous information.\n\n**4.** _Specify which dataset the samples were chosen from in Figure 3 and Figure 4. A brief explanation on different colors or line types in Figure 2._   \n**Response**  \nThe scenes in Figure 3 are taken from the RVSOD dataset, while the scenes in Figure 4 are from both the RVSOD and DAVSOD datasets. In Fig. 2, the different colors represent different instances.\n\nThank you for your suggestions, and we will make improvements based on your feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VllMRtxLKS", "reviewer": "Reviewer_vz4h", "summary": "This paper proposes a graph-based video salient object ranking method. It introduces a spatial-temporal graph to integrate trajectory-wise spatial and temporal saliency cues. Based on VSOR, this paper proposes a video retargeting method to adjust the videos to different aspect ratios adaptively. Extensive experiments also demonstrate the effectiveness of the video salient object ranking method and the video retargeting method.", "questions": "The writing of this paper needs to be further improved. Some concepts are not clearly illustrated. For example, the paper mentioned trajectory features often (e.g., title, abstract, method) but did not explain where and how it was generated.", "limitations": "Yes, the authors clearly illustrate the limitations of the paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. A graph-based model is proposed for video salient object ranking.\n2. This paper synchronizes the spatio-temporal saliency cues in a single graph for joint optimization to facilitate video saliency ranking.\n3. Based on VSOR, this paper proposes a simple but efficient retargeting method.\n4. Experiments and ablation studies validate the effectiveness of the method and its components.", "weaknesses": "1. In Figure 4, the ablation visual results in the first and second examples are very similar. I cannot see the effectiveness of different ablation models according to these visual results. \n2. The main purpose of retargeting is to find the correct window for each frame. The seam carving [12] can find better windows for the salient instances. Even if some distortions exist in these instances, they can be easily solved by a post-processing method. For example, we can first get the window position (x1,y1,x2,y2) by calculating the left top and right bottom points, then crop the original image according to the window position, which can eliminate the distortions in these salient instances. By the way, the cropped regions in Figure 5 are inconsistent with Figure 6. \n3. Some symbols are confusing, e.g., the symbol (||) in Equation (4).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "###1 Weakness\n\n**1.** _Similar ablation visual results in the first and second examples in Figure 4. I cannot see the effectiveness of different ablation models according to these visual results._  \n**Response**  \n1) Yes. The methods 'w/o TRM', 'w/ GTRM', and 'w/ ITRM' will indeed share similar result, highlighting the instances whose static appearance cues are very distinctive (e.g., the person with a large size and close to the camera in the first example), as they all lack instance-wise motion modeling. 2) However, when it is difficult to differentiate the saliency by static appearance alone (e.g., the instances in the second example), these methods will show different performance. As shown in the last two columns,, 'w/o TRM' and 'w/ GTRM' fail to identify the person as salient, due to their reliance on spatial cues or rough temporal cues by comparing two frames globally. 'w/ITRM' achieves improvement on additionally identifying the person by eliminating the background influence and considering inter-frame cross-instance contrast. However, they are still unable to effectively model instance-wise motion. As a result, they give a wrong saliency ranking. \n3) In contrast, by incorporating our trajectory-aware temporal correlation modeling into the graph, our method ('Ours' in Fig. 4) can effectively identify instances with noticeable motion cues, leading to the successful highlighting of motion instances and accurate saliency ranking.\n\n**2.** _The main purpose of retargeting is to find the correct window for each frame. The seam carving [12] can find better windows for the salient instances. Even if some distortions exist in these instances, they can be easily solved by a post-processing method. For example, we can first get the window position (x1,y1,x2,y2) by calculating the left top and right bottom points, then crop the original image according to the window position, which can eliminate the distortions in these salient instances. By the way, the cropped regions in Figure 5 are inconsistent with Figure 6._  \n**Response**  \nI may not fully understand your method, but I'm concerned the traditional seam-carving approach is semantic-agnostic, without instance-level awareness or semantic prioritization. It simply calculates image gradients and crops low-gradient regions, which could inadvertently remove uniform areas within instances. As a result, I'm worried it may be difficult to reliably retain all key semantics when determining the cropping window solely by optimizing the top-left and bottom-right coordinates.\n\n**3.** _Some symbols are confusing, e.g., the symbol ($\\|\\|$) in Equation (4)._  \n**Response**  \nThe symbol ($\\|\\|$) in Equation (4) represents feature concatenation.\n\n\n\n###2  Question\n\n**1.** _Unclear concepts. Where and how were the trajectory features generated._  \n**Response**  \nThanks for your suggestion and we will detail some key concepts to reduce any confusion. For the modeling of trajectory features, we project the absolute position of the current frame's instance onto the adjacent frames to capture changes in motion over time. To account for potential camera movement and drastic changes in the scene, we doubled the size of the instance's bounding box when performing this projection. This enhancement is intended to improve the model's robustness and ability to reliably track objects, even in the face of significant contextual variations. An instance moving fast will consequently obtain a low inter-frame similarity score between features $f_i$ and $f_i^{t+1}$ and consequently get a large saliency score as expected.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kmGlfUIUM3", "reviewer": "Reviewer_TupH", "summary": "This work proposes a trajectory-aware spatial-temporal graph for video salient object ranking. The proposed model includes a spatial correlation graph and a temporal correlation graph. Unlike previous VSOR methods, this work suggests to modeling instance-level temporal relations. They conduct experiments to demonstrate the advantage of their method, and also show the effectiveness of their method in video retargeting task.", "questions": "1. How does the model benefit from \"Trajectory-wise Contrast\" if the motion is too dynamic? For example, in Figure 2, the persons are moving away from their original position and go out of the expanded bounding box at t+1 frame. Then, $f_1$ cannot track the future trajectory by comparing $f_1$ and $f_1^{t+1}$, as the person goes outside of the bounding box. If the model fails to track the object, how does the model benefit from \"Trajectory-wise Contrast\"?\n\n2. It would be beneficial if the authors included studies to investigate how the bounding box area impacts VSOR performance. Whether a larger bounding box is more friendly for faster motion?\n\n3. What does the \"||\" mean in Eq. 1, Eq. 2, Eq. 3 and Eq. 4? Does it mean concatenation? It would be better to specify this just after Eq. 1. \n\n4. Why the numerical results in table 1 are significantly different from those reported in [1]?\n\n5. The Eq. 2 and Eq. 3 are confusing and inconsistent with Figure 2 ($T^i, T^t$). $f_{j}^{t}$ means $j^{th}$ instance at frame $t$, and $f_j$ mean $j^{th}$ instance at current frame. If so, why $f_j$ in Eq. 2 doesn't include a superscript? The $f_*$ in Figure 2 ($T^i$) also misses some superscripts. Does $h_{N_{i}^{t_i}}$ aggregate information from $f_i$ in Eq. 2?\n\n6. How long does it take to train the model?", "limitations": "The authors have discussed the limitations, and I agree that there is no potential negative societal impact of their work.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "1. The idea of this work is clear. Unlike previous VSOR methods, which focus on frame-level temporal relations, this work proposes to explore the instance-level temporal relations.\n\n2. They apply their VSOR model to the video retargeting task and achieved good results.", "weaknesses": "1. The authors claim that they **explicitly** model the motion trajectories of each instance (L8-9,L42,L69). However, the proposed method doesn't support this claim as the proposed method neither track each instance nor re-identify the same instance. Instead, the proposed method aggregates local contextual features from the same position across frames to **approximates** the instance trajectory. This claim might be not accurate.\n\n2. Lack comparison with latest SOR methods. VSOR is highly related to image-based SOR. In table 1, the compared image-based methods are [2,5] (2021), however, some latest methos are available, such as, \"Bi-directional Object-context Prioritization Learning for Saliency Ranking\" (2022-CVPR), \"Partitioned Saliency Ranking with Dense Pyramid Transformers\" (2023-ACM MM) and \"SeqRank: Sequential Ranking of Salient Objects\" (2024-AAAI).\n\n3. There is a real lack of experimental results to substantiate its advantages. In Table 1, only three methods are presented: two image-based salient object recognition (SOR) methods from 2021 and one video salient object recognition (VSOR) method from 2022. To strengthen the evaluation, it would be prudent to compare the proposed method with more approaches from related fields. For instance, consider video salient object detection methods like “Shifting More Attention to Video Salient Object Detection” (2019 CVPR) and “Dynamic Context-Sensitive Filtering Network for Video Salient Object Detection” (2021 ICCV). Additionally, exploring video shadow detection methods such as “SCOTCH and SODA: A Transformer Video Shadow Detection Framework” (2023 CVPR) could provide valuable insights.\n\n4. Why the SA-SOR scores of DAVSOD is much worse than those in RVSOD? Need explanation. In addition, it would be better to include a naive solution as comparison, since the SA-SOR score is very low. For example, a naive solution is to rank the instances according to their sizes to see how well the model outperforms such naive solution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "###1 Weakness\n\n**1.** _Inaccurate claim of \"explicitly model the motion trajectories of each instance.\"_  \n**Response**  \nYes, thanks for pointing out this problem. Actually, according the the human visual mechanism, our motivation is to measure the magnitude of instance motion across adjacent frames to infer instance-level temporal saliency. Therefore, tracking or re-identifying each instance is helpful but not necessary for our task as instances moving fast and fail to tracking will consequently obtain a large inter-frame contrast score and consequently large saliency priority. Also, without tracking labels, it is difficult to model the motion trajectories of each instance accurately. Replacing 'model the motion trajectories' with 'approximate the motion trajectories' seems a more accurate claim.\n\n**2.** _Lack comparison with latest image SOR methods._  \n**3.** _Lack comparison with latest image SOR methods._  \n**Response to 2 and 3**  \n1) Since our primary focus is on modeling temporal saliency cues and effectively combining them with spatial saliency information, image-based SOR methods are not the most pertinent way to highlight these key contributions. Therefore, we choose the most related video SOR works to showcase these advancements.\n2) While video SOD typically highlights the most salient region without explicit instance modeling, video SOR detects more objects to rank their saliency, often relying on a detector for object localization. This results in significant differences in the inference stage.\n3) However, we do include a comparison in Table R1, which demonstrates that our method achieves significant performance improvements over traditional image-based SOR and video SOD methods. This clearly showcases the effectiveness of our temporal saliency modeling and the proposed spatial-temporal fusion strategy.\n\n**4.** _Why the SA-SOR scores of DAVSOD is much worse than those in RVSOD?_  \n**Response**  \nThe reason stems from the varying difficulty between the two datasets. As shown in Fig. R4, DAVSOD has low resolution and low-quality appearance, making it challenging to detect all objects successfully. The missing objects consequently result in low SA-SOR scores. \n\n\n###2  Question\n\n**1.** _How does the model benefit from \"Trajectory-wise Contrast\" if the motion is too dynamic?_  \n**Response**  \nAs shown in Fig. R1 in the one-page PDF, an instance (e.g., the person in the red box) moving rapidly and outside the local context will consequently obtain a large inter-frame contrast score between the features of this instance and its local context in adjacent frames (i.e., $f_i$ and $f_i^{t+1}$). This will result in the instance receiving a higher saliency score, as expected.\n\n\n\n**2.** _Whether a larger bounding box is more friendly for faster motion?_  \n**Response**  \n1) As mentioned previously, an instance moving rapidly and outside the local context will consequently obtain a low inter-frame similarity between $f_i$ and $f_i^{t+1}$, and thus receive a high saliency score as expected.\n2) In our view, a bounding box that is too large will cause the instance-level contrast to become local-global contrast, increasing the risk of introducing another instance into the comparison. This can generate confusing instance temporal motion cues.\n3) The comparison shown in Table R2 verifies our argument. A larger bounding box fails to achieve better performance. \n\n\n**3.** _Some issues in our writing._  \n**Response**  \nThe symbol ($\\|$) in Eq. 1-4 represents feature concatenation.\n\n\n**4.** _Why the numerical results in table 1 are significantly different from those reported in [1]?_  \n**Response**  \nRegarding Lin [1], it mentioned that annotators were invited to re-annotate the data, resulting in differences between our dataset and Lin's dataset. As they refused to release their labels, we retrained their model on our version of the dataset for a fair comparison. The results from other papers are also based on our reproduction of their open-source code, ensuring that the experimental results are authentic and reliable.\n\n**5.** _How long does it take to train the model?_  \n**Response**  \nWe set the batchsize to 1, and training for approximately 20 hours to achieve convergence on a RTX 4090 GPU.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
The figures and tables can be seen in PDF.

Meta-review: This paper proposes a graph-based video salient object ranking method. It introduces a spatial-temporal graph to integrate trajectory-wise spatial and temporal saliency cues. It also presents a cohesive optimization approach that seamlessly integrates spatial and temporal saliency cues in an adaptive manner.  The presented experiments demonstrate the advantage of the proposed method, and also show the effectiveness in video retargeting task. Synchronizing the spatio-temporal saliency cues in a single graph for joint optimization is appreciated to facilitate video saliency ranking.  Main concerns raised by the reviewers were clarification of the novelty, insufficient experiments, failure case discussion, and writing. The authors’ rebuttal resolved the concerns to convince the reviewers. During post-rebuttal discussion, in-depth analysis of low performance on DAVSOD was suggested. Using “motion-aware” instead of “trajectory-aware” was also suggested because the proposed method does not accurately track the instance.  The author addressed these concerns adequately.  The reviewers unanimously support this paper in the end.  The paper should be accepted, accordingly.  All the discussion with the reviewers should be properly incorporated in the final version.
Decision: Accept (poster)