Instance-Level Panoramic Audio-Visual Saliency Detection and Ranking

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-05 16:39:40

Abstract: Panoramic audio-visual saliency detection is to segment the most attention-attractive regions in 360° panoramic videos with sound. To meticulously delineate the detected salient regions and effectively model human attention shift, we extend this task to more fine-grained instance scenarios: identifying salient object instances and inferring their saliency ranks. In this paper, we propose the first instance-level framework that can simultaneously be applied to segmentation and ranking of multiple salient objects in panoramic videos. Specifically, it consists of a distortion-aware pixel decoder to overcome panoramic distortions, a sequential audio-visual fusion module to integrate audio-visual information, and a spatio-temporal object decoder to separate individual instances and predict their saliency scores. Moreover, owing to the absence of such annotations, we create the ground-truth saliency ranks for the PAVS10K benchmark. Extensive experiments demonstrate that our model is capable of achieving state-of-the-art performance on the PAVS10K for both saliency detection and ranking tasks. The code and dataset will be released soon.

Main paper: 
 related work 2.1 video salient object detection: 
 Video salient object detection (VSOD), an extension of image saliency detection, aims to segment the most eye-catching objects in a video sequence. Traditional VSOD methods are usually based on handcrafted features, such as motion boundaries [27] , center prior [15] , long-term point trajectories [25] , etc. Recently, some works employ either ConvLSTM [32] , 3D convolution [16] or attention mechanisms [8, 12] to better establish the temporal relation in consecutive frames. Furthermore, optical flow is introduced to capture motion saliency clues and enhance spatio-temporal representation. With this strong prior, optical-flow-based models [14, 49] can easily locate the representative salient objects and have achieved impressive performance in VSOD. However, all the abovementioned methods rely on only visual data for saliency detection, neglecting audio hints that also significantly attract human attention under realistic scenarios. Besides, this is contrary to what psychological studies [23, 24] have found out, i.e., human attention is sensitive and susceptible to both visually salient objects and sonic-emitting entities.By imitating human perceptual mechanisms in complicated audiovisual scenes, audio-visual salient object detection (AVSOD) is to locate and segment video regions that are salient in both visual and audio sources simultaneously. The main challenge of AVSOD is how to appropriately fuse audio-visual information and learn consistency across modalities. DAVE [34] adopts a straightforward deep-learning-based plain fusion strategy, which takes audio-visual features as input and then concatenates them before saliency prediction. [5] utilize de-convolution operation for the audio-visual alignment, where 1D audio vector is expanded to be the same size as 2D visual counterpart. This strategy views the audio cues as auxiliary information, with corresponding spatial locations being emphasized through embedded semantic consistency. Recent works [35, 45] The overall framework for instance-level panoramic audio-visual saliency detection and ranking. Firstly, the video sequence is divided into T segments, and we use two encoders to extract visual and audio features, respectively. Then, a distortionaware pixel decoder integrates multi-stage visual features while reducing panoramic distortions via a bi-projection point sampling and a feature aggregation module. Next, we perform audio-visual fusion using the audio-visual spatial activation module and audio-visual instance alignment module sequentially. Finally, a spatio-temporal object decoder takes initial learnable queries and the above cross-modal representations to generate instance-level saliency maps and rank scores.mismatching, and enable strong interaction between visual and audio. Moreover, [47] use the self-attention mechanism to build relationships between the visual pixels and the audio signals. This benefits video segmentation with accurate auditory entities while ignoring those spurious sound sources that do not actually emit acoustic signals. [45] propose a new panoramic audio-visual salient object detection (PAVSOD) task, aiming at segmenting the salient objects in 360°panoramic videos. To support the proposed task, they collect the first benchmark PAVS10K for PAVSOD and present a baseline model CAV-Net for object-level panoramic saliency prediction by bi-linear fusion and conditional variational auto-encoder. Different from CAV-Net that focuses on detecting pixels belonging to salient regions without considering individual instances, we propose the first pixel-wise instance-level AVSOD method. Instead of just detecting salient regions, our model also distinguishes individual object instances within them. This is essential for real-world applications that require finer distinction.Audio-visual localization [28, 30] aims to locate the regions of sounding objects within the visual frame. Previous methods typically tackle the task through self-supervised or weakly-supervised learning to explore the correlations between audio and visual features, where the goal is to predict the coarse heatmap or bounding box of the sounding object. As a complex extension to the AVL task, audio-visual segmentation [19, 22] is a more challenging task as it requires more fine-grained pixel-level shape description besides localization. Specifically, Zhou et al. [48] focus on multi-stage fusion of audio-visual features to facilitate supervised segmentation tasks on their AVSBench dataset, predicting the probability of each pixel in the image belonging to the sounding object. More recently, CATR [18] proposes a decoupled audio-visual transformer that combines audio and video features from temporal and spatial dimensions, capturing their combinatorial dependence. Additionally, they introduce a set of audio-constrained learnable queries to select which object is being referred to segment. In this work, we focus on saliency detection, which imitates human attention and predict subjects' fixations, instead of querying sounding objects. That is, audio cannot be viewed as the necessary and sufficient condition for judging saliency, so this query-based segmentation paradigm is not suitable for saliency detection and ranking tasks. 
methodology: 
 Given an input video sequence containing both visual and audio tracks, we first split it into 𝑇 non-overlapping visual and associated audio segment pairs {X V 𝑖 , X A 𝑖 } 𝑇 𝑖=1 . For each visual snippet, we sample a fixed number of frames. As shown in Fig. 2 , we apply an image encoder, ResNet [10] , to extract multi-level visual features f V ∈ R 𝑇 ×𝐻 ×𝑊 ×𝐶 on video frames. For each audio snippet X A , we encode it into a feature vector f A ∈ R 𝐷 using the first seven 1D convolutions layers of SoundNet [1] . Then, a distortion-aware pixel decoder (Sec. 3.1) is proposed to integrate multi-level visual features while reducing panoramic distortions via a bi-projection point sampling and a feature aggregation module. Next, our model performs audio-visual fusion using the audio-visual spatial activation module and audio-visual instance alignment module sequentially, as detailed in Sec. 3.2. Lastly, a spatio-temporal object decoder (Sec. 3.3) takes initial queries and the above cross-modal features to generate the final salient object instance segmentation masks and the corresponding rank scores. The equirectangular, commonly used in panoramic videos, is susceptible to geometrical distortions, particularly in the north and south polar regions. To overcome this problem, we present a distortionaware pixel decoder (DPD) for pixel-wise omnidirectional vision tasks following the three loop steps: 1) sample a small set of initial neighbor points on the cubic and equirectangular domains for each pixel; 2) aggregate all neighborhood features to form the pixel's output feature across multi-scale feature maps; 3) dynamically adjust the position of sampling points with learnable position bias. By aggregating relative neighbor points, our model mitigates panoramic distortions while learning intricate patterns and local region representation, thereby improving performance in panoramic dense prediction tasks. 3.1.1 Bi-Projection Neighbor Point Sampling. Given the input image feature map f V , let 𝑝 𝑖 be a pixel in the feature map, namely central point, and 𝑝 𝑖 𝑗 refers to its neighbor point. We adopt a biprojection neighbor point sampling scheme to find neighbor points around each central point by leveraging two projections: equirectangular (ER) and cubemap projections. Specifically, cubemap projection avoids distortion but incurs discontinuity at the cube boundary, whereas ER projection incorporates a complete field-of-view but introduces distortion. To mitigate distortions and ensure continuity simultaneously, we sample neighbors on both the cubic and ER domains. For ER sampling, we directly select the eight nearest neighbor pixels of each pixel on the ER projection as neighbor points. The process is given as follows:EQUATIONwhere 𝑎 and 𝑏 are the sampling step size. For cubemap sampling, it contains the following steps: i) equirectangular-to-cube transformation. Let the side length of a cube map be 𝑤. As the fieldof-view (FoV) of each face is 90 • , each face can be treated as a perspective camera with a focal length of 𝑤 2 , and they all share the same center point in the world coordinate system. Due to the fixed viewing direction in cubemap projection, a rotation matrix 𝑅 ℎ can represent the extrinsic matrix of each camera. For a pixel 𝑝 𝑖 on equirectangular map, we can transform it into the coordinate on the certain cube face ℎ by the following mapping: EQUATIONwhere 𝜃 and 𝜙 represent the longitude and latitude of point 𝑝 𝑖 on the sphere, respectively. The range of 𝜃 spans from -𝜋 to +𝜋, while the range of 𝜙 spans from -0.5𝜋 to +0.5𝜋. The x, y, and z components of vector 𝑞 𝑖 are represented as 𝑞 𝑥 𝑖 , 𝑞 𝑦 𝑖 , and 𝑞 𝑧 𝑖 , respectively. ii) uniform sampling on the cube map. Similar to ER sampling, we select the eight nearest neighbor pixels of each central pixel in the perspective domain, as shown in Fig. 3 . iii) cube-to-equirectangular transformation. All these neighbors are projected back to the ER domain. Given a neighboring point 𝑝 𝑖 𝑗 on a specific face ℎ, we can perform a coordinate transformation to map it onto the ER projection using the following mapping:EQUATIONTake 15 × 30 feature map as an example, Fig. 8 visualizes all the central points and their neighbor points that are projected from cube map to equirectangular map.3.1.2 Multi-scale Distortion-Aware Aggregation. As illustrated in Fig. 4 , image feature maps F V are flattened and reshaped into a sequence, and then a linear projection is applied to produce query Q and value V, inspired by [50] . Next, we sample two groups of neighbor points for each central point via the bi-projection neighbor point sampling strategy, and combine them as initial sampling points p ij . The Q is passed into two linear layers for calculating neighbor scores A after a softmax operator and the neighbor offsets Δp. The initial neighbor points p ij and Δp are added together to obtain the final neighbor points. By dynamically adjusting the spatial distribution of sampling points, our model can capture various geometric characteristics and perceive more complicated patterns. Moreover, we adopt bi-linear interpolation on V to generate neighbor features and then aggregate them using neighbor scores A.Lastly, a linear projection is used to output distortion-aware visual features. Note that we perform distortion-aware aggregation on multi-stage features with resolution 1/8, 1/16, and 1/32 at once.Our objective is to fuse visual and audio features, thereby retrieving salient objects from both modalities. To achieve this, we propose a sequential multi-modal fusion scheme with two components: 1) an audio-visual spatial activation module to embed entangled audio information into visual features and activate all sounding regions; 2) an audio ProtoNet to unmix dense ambisonics into a set of prototypes, and then an audio-visual instance alignment module to assign them into individual visual instances and capture instanceaware audio-visual dependence by introducing contrastive learning and cross-modal transformer, respectively.As shown in Fig. 2 , audio-visual spatial activation module (AV-SAM) takes audio features f A and distortion-aware image features f D from DPD as inputs. Specifically, f A is first spatially duplicated 𝑇 𝐻𝑊 times and projected to the same size as f D . Then we add the above features of two modalities and apply 3 × 3 convolution to locate sounding regions, which can be formulated as:EQUATIONIn this way, 1D audio features are treated as auxiliary information and correlated to each spatial location of the visual counterpart. This simple multi-modal fusion strategy can well retain 2D spatial structure and activate all sounding objects.While AV-SAM can exploit audio information and build inter-modal relations, the interaction between visual and audio is still weak and ambiguous. To better learn the matching relationship between each visual object and sounds in complicated scenarios, we propose an audio ProtoNet and an audio-visual instance alignment module (AV-IAM), as depicted in Fig. 2 . In audio ProtoNet, we first use an MLP to transform audio feature f A into multiple implicit sub-variables (referred to as audio prototypes) p A ∈ R (𝑇 ×𝑁 ) ×𝐶 , where 𝑁 is the number of the object query. Next, an original transformer encoder [6] is adopted to perform self-attention on 𝑇 𝑁 audio prototypes as: In AV-IAM, audio prototypes pA and AV-SAM features f SA are passed as input to an audio-visual transformer, where the query is p A and the key/value is f SA . Consequentially, the audio-visual transformer produces audio-conditioned image attention and a-to-v representations in the audio stream, so as to associate each audio prototype with different sounding objects:EQUATIONEQUATIONIn order to ensure the disentanglement of audio representations and align the corresponding audio prototypes and pixel features, we introduce an audio-visual contrastive learning objective after the audio-visual transformer. To be specific, we measure the similarity between a-to-v representations f AV and query features f Q from spatio-temporal object decoder by calculating cosine distance between them in embedding space, where the cosine distance function is defined as 𝑠 (𝑢, 𝑣) = 𝑢 𝑇 𝑣/∥𝑢 ∥∥𝑣 ∥. We would like to enforce the consistency between modalities by pulling close the query features whose category is the salient object with its corresponding audio prototypes. To increase the heterogeneity between audio prototypes, we push away the above audio representations. Therefore, by implementing the NT-Xent loss [3] , our audio-visual contrastive loss between f AV and f Q can be formulated as follows:EQUATIONwhere ∀𝑛, ∀𝑘 ∈ {1 . . . 𝑁 }. 𝜏 is a temperature co-efficient and 𝑁 is the number of object query. For an audio prototypef Q n , the cor- responding (f Q n , f AV n) is regarded as a positive pair, and the rest 𝑁 -1 audio prototypes constitute negative examples. We intend to maximize the similarity of positive pairs while minimizing the similarity of negative pairs in embedding space. This process enables our model to disengage dense audio signals into unique sounding embeddings. After that, a visual-audio transformer treats f SA as query and a-to-v representation as key/value, and computes crossattention on them to retrieve the audio prototype for each pixel. This instance-aware alignment method allows our model to embed disentangled audio representations into individual visual objects and further guide the proposed instance-level saliency detection and ranking tasks. The process can be defined as:f VA = 𝐶𝑟𝑜𝑠𝑠𝐴𝑡𝑡𝑛(f SA , f AV , f AV ) (8)Spatio-temporal object decoder (STOD) aims to predict salient object instance masks for each frame. Motivated by [38] , we also initialize a fixed number of learnable positional embeddings, termed object queries. Assuming the model decodes 𝑁 instances per frame, the total number of object queries for 𝑇 frames becomes 𝑇 × 𝑁 , which corresponds to the size of the audio embeddings. All queries are fed into STOD along with the AV-IAM features. Similar to [4] , STOD consists of two successive attention modules: 1) masked selfattention is performed among all queries from 𝑇 frames to integrate temporal information; 2) cross-attention is computed between object queries and v-to-a features from the visual-audio transformer to generate query features f Q over space. Ultimately, two linear classifiers, i.e., class head and rank head, determine whether each object query is a salient object and output the corresponding rank order. For mask prediction, a dot product followed by a sigmoid activation is applied between the query features and DPD's features to produce the final masks.In the training phase, we assign a unique ground truth for each instance prediction. The matching process can be done through one-to-one bipartite matching strategy. To find the best assignment of a prediction to ground truth, we uniformly sample a set of points on the predicted mask and then construct a cost matrix. Given a matching, each prediction is supervised with a saliency classification loss and a mask loss. The former is binary cross-entropy loss and the latter is composed of a focal loss and a dice loss. The final loss function can be formulated as:EQUATION 
experiments 4.1 dataset and evaluation metrics: 
 We evaluate the performance of our method on the PAVS10K dataset [45] , comprising 40 training videos totaling 5,796 frames and 27 testing videos with 4,669 frames. These videos are randomly selected from 67 4K-resolution equirectangular videos (each of 30 seconds), following an approximate 6:4 ratio. Instance-level pixelwise masks have been provided in the original PAVS10K dataset. For the PAV-SOR, we use the rank labeling method proposed in [31] to generate our ground-truth saliency rank annotations. Quantitative results on 6 widely used evaluation metrics are reported: F-measure, E-measure, S-measure, MAE, SOR, and #images used.We use PyTorch and follow CAV-Net [45] setting for instance-level PAV-SOD and PAV-SOR. The shorter side of all frames is resized to 480 during training and evaluation, without relying on multi-scale or any other data augmentation strategies. Our model is optimized by AdamW algorithm with weight decay 0. By setting the default video clip length as 2 and batchsize as 2, the training of the entire framework starts with an initial learning rate of 1𝑒 -4 on a single NVIDIA Quadro RTX 6000 GPU.To verify the effectiveness of our model on PAV-SOD, we compare our model with the state-of-the-art methods, including Image SOD (I.): CPD-R [41] , SCRN [42] , F3Net [39] , MINet [26] , LDF [40] , CSFR2 [7] , GateNet [46] , Video SOD (V.): COSNet [20] , RCRNet [43] , PCSA [9] , 3DC [21] , RTNet [29] , Panoramic Image SOD (PI.): FANet [13] , and Panoramic Audio-Visual SOD (PAV.): CAVNet. The quantitative results are shown in Table 1 . For main metrics, our model has obvious improvement among various approaches. Specifically, it reaches 43.2% on F 𝛽 , 69.9% on S 𝛼 , and 74.0% on E 𝜖 , surpassing the second-best ones with 10.7%, 6.6%, and 4.2%, respectively. When introducing ranking supervision, our model exhibits superior results, suggesting that the newly incorporated rank prediction branch can augment the performance of PAV-SOD task. Additionally, our model is the first one that addresses instancelevel saliency detection while the existing ones are object-level.The qualitative results are shown in Fig 6 . We compare our model with CAVNet on PAVS10K dataset and the result indicates that our model performs better in various challenging cases based on the following fundamental observations: 1) locate the correct salient objects from the multi-instance scenario (e.g., singing persons in the 3 𝑟𝑑 and 4 𝑡ℎ rows); 2) predict the accurate segmentation mask closer to ground truth in severe distortions (e.g., skiing persons in the 6 𝑡ℎ row); 3) detect very small objects (e.g., running dogs in the 2 𝑛𝑑 row). We attribute this to the strong distortion resistance of our DPD and the effective audio-visual fusion of AV-SAM and AV-IAM. Additionally, it is highlighted our method is the first instance-level saliency detection network, and the novelty lies in its ability to differentiate different instances. To verify the effectiveness of our model on PAV-SOR, we compare our model with ASSR [31] and PSR [33] . Table 2 shows that our model outperforms other recent approaches for all measurements. In panoramic images with 360°omnidirectional view, MAE is dominated by the background region, owing to the small proportion of the foreground region. From the 1 𝑠𝑡 row in where 𝐺 and 𝑆 denote ground truth and the prediction, and F and B indicate the foreground and background region of ground truth, respectively. We present our qualitative result in Fig. 7 . Compared to PSR [33] , our model produces almost-the-same segmentation masks with regard to the ground truth while correctly predicting ranking orders. 3 provides a comprehensive validation of the efficacy of our proposed distortionaware pixel decoder (DPD). In the 2 𝑛𝑑 row, we introduce DPD, which samples neighbor points only on the equirectangular projection and then aggregates the corresponding neighbor features. The comparison between the first two rows indicates that the integration of local information significantly bolsters the visual feature representation. Nonetheless, the equirectangular map suffers from inherent distortions. The information from rectangular and regular neighbor priors may occasionally deviate from the true spatial arrangement. To address this, we propose cubemap neighbor point sampling, as depicted in Fig. 3 . Back-projecting the sampled neighbor points onto the equirectangular map allows us to derive a set of distortion-aware neighbor points. This strategy can effectively capture complex shapes and alleviate the adverse effects of distortions inherent in panoramic images. The results presented in the 3 𝑟𝑑 row underscore the significant performance boost. While the cubemap projection circumvents panoramic distortions, it will induce discontinuity at points near the cube's boundary. Remarkably, by concatenating and jointly utilizing the results derived from both projections, we manage to achieve an optimal outcome, as evidenced in 4 th row of Table 3 . Take a 15 × 30 feature map as an example, Fig. 8 visualizes all the central points and their neighbor points that are projected from cube map to equirectangular map. Considering the bottomleft green neighbor point in Fig. 3 , its corresponding position is depicted as a point in the bottom-left map of Fig. 8 . To generalize, this bottom-left map in Fig. 8 illustrates the offset position of all bottom-left neighbor points in the original 15 × 30 feature map. Similar information is conveyed in the projection maps of the other 7 directions. 4 validates the efficacy of our proposed sequential audio-visual fusion method, including an audio-visual spatial activation module (AV-SAM) and an audio-visual instance alignment module (AV-IAM). With the integration of AV-SAM, audio signals are seamlessly infused into the visual encoding space via element-wise summation and local convolution operation. It aggregates the audio and image features in a pixel-wise manner, furnishing the pixel decoder with robust localization cues of the sound-emitting object. Row 1 and 2 reveal a noticeable improvement in all metrics, underscoring the crucial role of the acoustic modality in salient object detection and the necessity of our AV-SAM. However, as the sound information of all objects is encompassed within a single audio feature, the model encounters challenges in distinguishing instance-level salient objects. Hence, further unmixing the audio signals is deemed necessary. Row 3 illustrates the performance boost conferred by the AV-IAM, with F 𝛽 increasing by +1.8%, S 𝛼 by +1.0%, and E 𝜖 by +1.3%. This improvement can be attributed to the model's capacity to perform an implicit separation of multiple sound sources, thereby enabling our spatio-temporal object decoder's object queries to possess distinct sound source semantic information. This results in a more efficient and precise frame-by-frame instance-level salient sound source localization. Moreover, we further discuss the cross-modal fusion at various stages. It can be seen that applying the cross-modal fusion at three stages will increase all metrics, showing our model can fuse and balance audio-visual features from multiple stages. audio-visual and visual-audio transformers. From the results, we can find that performing cross-modal fusion sequentially infers a finer attention map than doing in parallel. Audio prototypes have undergone regularization through contrastive learning before inputting into the visual-audio transformer. Each audio prototype is assigned to a visual object and exhibits heterogeneity with other audio prototypes. Therefore, it helps the visual-audio transformer to align pixels with audio and perform instance-aware fusion. To verify that dense audio has been disentangled, we upsample and visualize the attention matrices between audio prototypes and image features from the audio-visual transformer. As illustrated in Fig. 9 , when introducing contrastive learning, two audio prototypes can activate two different object instances, that is, the audio signal has been well unmixed and assigned to different sounding objects.Ours w. CL Ours wo. CL  
conclusion: 
 In this paper, we introduce two challenging tasks, i.e., instance-level PAV-SOD and PAV-SOR, and then propose a unified framework to solve the above tasks. Three key components of our model are designed: a distortion-aware pixel decoder mitigates panoramic distortions; a sequential cross-modal fusion method integrates audiovisual information in an instance-aware manner; a spatio-temporal object decoder generates segmentation mask and saliency rank for each salient object instance. Without bells and whistles, our model achieves satisfactory performances on PAVS10K benchmark. We hope the framework could serve as a preferred baseline for panoramic audio-visual saliency detection.  
: 
 complex scene. Modeling this ability is crucial for the understanding of how humans interpret videos, and facilitates a wide range of multi-modal applications, e.g., virtual reality, autonomous driving, robot navigation, robot-human interactions. Following this idea, in this paper, we introduce two novel tasks: instance-level panoramic audio-visual salient object detection and ranking, namely instancelevel PAV-SOD and PAV-SOR, as depicted in Fig. 1 (c-d ). The former segments individual salient instances, and the latter further predicts the relative saliency ranks of these instances. The goal of the proposed tasks is to perform more detailed parsing within detected salient regions and account for inter-observer variability by assigning confidence to different salient instances, making models mimic human perceptual mechanisms promising. For the above instance-level tasks, three key problems need to be considered: (i) How to solve panoramic distortions and perceive complicated geometries? 360°Panoramic video captures the entire surrounding environment and represents each pixel on a 3D sphere. In general, to facilitate storage and transportation, raw panoramic video is transformed into a regular 2D format via equirectangular projection (ERP). However, ERP exhibits severe geometrical distortions, especially in the polar area. Considering the ERP distortion, [36, 37] utilize the combination of equirectangular and cubemap projection as input to the framework, since the distortion can be removed by converting a single equirectangular image into several perspective ones. Nevertheless, this two-branch architecture has significantly increased the parameters that incurred high computational complexity. In addition, some works use deformable convolution [44] and MLP [11] to mitigate panoramic distortions. For this Problem, we present a distortion-aware transformer, which consists of the following steps: 1) sample a small set of neighbor points around each pixel on cubemap and equirectangular projection; 2) aggregate all neighbor features to output the pixel's de-distorted representation; 3) dynamically adjust neighbor positions with learnable position bias. Due to the combination of equirectangular and cubemap projections, our model can effectively avoid panoramic distortions while mitigating discontinuities on cube maps.(ii) How to parse audio signals and fuse audio-visual information? Audio exhibits high overlapping nature, as multiple objects can make sounds simultaneously. Even worse, these audio signals tend to be similar and indistinguishable when the sounding objects are homogeneous. Furthermore, background interference in panoramic videos is stronger than 2D videos with limited field-ofview, leading to audio-visual misalignment. Traditional approaches [45] directly combine entangled audio features with image embeddings by using a bi-linear layer or other fusion operations. However, they fail to effectively unmix audio and correctly locate the sources of sound in visual space. For this Problem, we first use the pretrained SoundNet [1] to extract audio features, and then adopt a sequential multi-modal fusion strategy to integrate visual and audio features. Specifically, an audio-visual spatial activation module embeds entangled audio information into image features and highlights all sounding regions via element-wise summation and convolution operation. To unmix audio signals, we present audio ProtoNet to map dense ambisonics into multiple audio prototypes by using an MLP and a transformer encoder. Then, an audio-visual instance alignment module, equipped with two cross-modal transformers, aligns audio prototypes with visual objects and performs instance-aware audio-visual fusion. Furthermore, we introduce a contrastive learning scheme in the training to ensure each visual object possesses a unique sound semantic corresponding to itself.(iii) How to identify individual object instances and predict their saliency ranks? Instance-level saliency detection is first proposed by [17] , and they propose a three-stage method, including salient region detection, object contour detection, and instance generation. This pipeline is cumbersome and its performance is sensitive to post-processing steps. Recently, CATR [18] adopts a DETR-like architecture for audio-visual segmentation (AVS) and treats audio as the prompt to query and segment all sounding object instances. Unlike the AVS task, PAV-SOD aims to predict subjects' fixations and segment the corresponding salient objects, that is, audio cannot be viewed as the necessary and sufficient condition for judging saliency. Moreover, saliency ranks are affected by various factors like object position, motion, sound, etc., making model prediction more difficult. For this Problem, a transformerbased object decoder is introduced to establish spatio-temporal relationships among instances and generate final segmentation masks. Note that audio is not the query, but has been embedded as auxiliary information into the visual features. We also show that saliency ranking can be addressed with the same network. After the object decoder, an MLP is added to predict the saliency score for each instance. Additionally, due to the lack of rank annotation, we provide the ground-truth saliency rank based on the attention shift of multiple observers for the PAVS10K dataset. In summary, our contributions can be summarized as:• To the best of our knowledge, this is the first work proposing a unified framework for the instance-level panoramic audiovisual saliency detection and ranking.  
 
 Appendix:  


Decision: An instance-level audio-visual framework has been proposed for multiple salient object segmentation and ranking simultaneously in panoramic videos. This framework comprises three components: a distortion-aware pixel decoder to mitigate the inherent geometric distortions in 360° panoramic videos, a sequential cross-modal fusion method to capture instance-aware audio-visual dependencies, and a spatio-temporal object decoder to generate segmentation masks and saliency ranks for each salient object instance. The proposed method appears novel and achieves state-of-the-art performance on the PAVS10K benchmark for both saliency detection and ranking tasks. The distortion-aware pixel decoder is lightweight and efficient, and it can serve as standard preprocessing for panoramic video analysis.

The limitations and problems raised by the two reviewers with BR are more related to technical details, equations, etc. Most of the reviewers' concerns have already been addressed in the rebuttal. As stated by the authors in their rebuttal, PAVS10K seems to be the only available dataset for this task. The authors should also incorporate these changes into the paper upon acceptance.

Reviews: review: paper_summary: paper_summary: The paper proposes an instance-level framework for segmenting and ranking multiple salient objects in panoramic videos. It introduces three key components: a distortion-aware pixel decoder to handle panoramic distortions, a sequential audio-visual fusion module to integrate audio-visual information, and a spatio-temporal object decoder to separate individual instances and predict their saliency scores. The framework achieves state-of-the-art performance on the PAVS10K benchmark for both saliency detection and ranking tasks .
strength_weakness: strength_weakness: 1. The framework introduced in this paper is the first of its kind to address instance-level saliency detection and ranking in panoramic videos.
2. The proposed model achieves satisfactory performances on PAVS10K benchmark.
limitations: limitations: 1. As shown in Fig. 1, I'm quite doubtful why the predicted masks of T1 and T2 are different.
2. As in Table 1, why are the performances of the four metrics not consistent? For example, the MAE of COSNet ranks first among all models, but the F-measure is quite limited.
3. In the experiments of saliency ranking, SA-SOR, FLOPS, and FPS are not compared.
4. The necessity of the new metric adaptive MAE is not explained; why not directly use pixel accuracy?
5. The source code is not provided, making some implementation details hard to understand.

If the author can adequately answer these questions, I will raise the score.
paper_summary: paper_summary: This paper primarily focuses on the field of panoramic audio-visual saliency detection and proposes an instance-level panoramic audio-visual saliency detection and ranking framework to detect and rank salient object instances in 360° panoramic videos with audio. The framework designs a distortion-aware pixel decoder to mitigate the inherent geometric distortions in 360° panoramic videos, a sequential cross-modal fusion method to capture instance-aware audio-visual dependencies, and a spatio-temporal object decoder to generate segmentation masks and saliency ranks for each salient object instance.
strength_weakness: strength_weakness: This study proposes an instance-level panoramic audio-visual saliency detection and ranking framework, capable of handling multi-instance saliency detection and ranking tasks simultaneously. The framework effectively mitigates geometric distortions in 360° panoramic videos, achieves efficient audio-visual information fusion, and can separate individual instances and predict their saliency scores. The paper is well-structured, but further validation is needed regarding its practical application prospects.
limitations: limitations: 1.Although extensive experiments have been conducted, the comparative experiments are primarily focused on the PAVS10K dataset, lacking comparisons with other types of datasets, which limits the generalizability and applicability of the results.

2.Although the method for generating labels based on attention shift data is introduced, there is a lack of specific construction steps description.

3.The results in Figure 6 are perplexing, with a noticeable misalignment between the predicted results of the third and fourth rows and the labels. For instance, the labels of the third row correspond to the predicted values of the fourth row, and the predictions of the fourth row align with the labels of the third row. The authors did not provide an explanation for this.

4.The explanation of Figure 8 is confusing. It is unclear whether the nine subplots in Figure 8 correspond one-to-one with the green dots in Figure 3. Additionally, there is a need to explain the significance of the points in the figure.
paper_summary: paper_summary: This paper proposes the first instance-level panoramic audio-visual saliency ranking task. A framework with distortion-aware pixel decoding, sequential audio-visual fusion and the spatio-temporal object decoding is proposed and is validated by comprehensive experiments.
strength_weakness: strength_weakness: 1. This work introduces the concept of panoramic audio-visual saliency ranking for the first time.
2. A novel method is proposed for panoramic audio-visual saliency detection and ranking in video.
3. The proposed method demonstrates good performance.
limitations: limitations: 1. In experiments for panoramic audio-visual saliency ranking, image-based methods are compared, but it is not specified how these methods are applied to videos.
2. In Table 1, the performance of the proposed method in the last column shows no advantage. Further analysis is needed to address this.
3. Although this paper is the first to address instance-level performance, no experiments are presented to demonstrate the method's effectiveness at this level.
paper_summary: paper_summary: This paper introduces Instance-level panoramic audio-visual salient object ranking (PAV-SOR), which is a new challenging task. To solve the difficulties in this task, three key modules are designed: a distortion-aware pixel decoder mitigates panoramic distortions, a sequential audio-visual fusion module to integrate audio-visual information; a spatio-temporal object decoder to detect instances and predict scores. Numerous experiments have confirmed the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. The new task proposed in this paper has practical value and provides ranking annotations for PAVS10K dataset, promoting the development of this topic.
2. The distortion-aware pixel decoder is lightweight and efficient, and can serve as the standard preprocessing for panoramic video analysis.
limitations: limitations: 1. The author needs to explain how the model annotates the ranking order between instances and how to train the model's ranking ability.
2. I am confused about the correspondence between the two queries f_n^Q and f_n^AV in Eq. (7). It seems that these two terms only correspond based on the index rather than positional embedding. Will this make it difficult for the training of the network?
3. Lines 577 and 563 seem contradictory. Does the symbol f_n^Q represent audio prototypes or query features?