ScanTD: 360° Scanpath Prediction based on Time-Series Diffusion

Authors: yujia  wang
neil a dodgson
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 14:09:26

Abstract: Scanpath generation in 360° images aims to model the realistic trajectories of gaze points that viewers follow when exploring panoramic environments. Existing methods for scanpath generation suffer from various limitations, including a lack of global attention to panoramic environments, insufficient diversity in generated scanpaths, and inadequate consideration of the temporal sequence of gaze points. To address these challenges, we propose a novel approach named ScanTD which employs a conditional Diffusion Model-based method to generate multiple scanpaths. Notably, a transformer-based time-series (TTS) module with a novel attention mechanism is integrated into ScanTD to capture the temporal dependency of gaze points effectively. Additionally, ScanTD utilizes a Vision Transformer-based method for image feature extraction, enabling better learning of scene semantic information. Experimental results demonstrate that our approach outperforms state-of-the-art methods across three datasets. We further demonstrate its generalizability by applying it to the 360° saliency detection task.

Main paper: 
 introduction: 
 In the rapidly evolving field of virtual reality (VR), understanding the visual attention of users is critical for creating immersive and interactive experiences. It offers meaningful insights into scene rendering, data transmission, and compression [11, 35] for developing better VR applications. However, recruiting actual participants and obtaining real experimental data using eye-tracking devices is time-consuming and complex. Therefore, it would significantly benefit this research field to develop a method capable of modeling human visual attention behavior and generating realistic visual attention patterns without the necessity for real-world experiments. 360 • images are experiencing a surge in popularity due to their ability to deliver highly immersive viewing experiences. Generally, the study of human visual attention on 360 • images has two research focuses: saliency maps and gaze point trajectories (scanpaths). Unlike saliency maps, which only present static results, gaze point movement in a scanpath is a dynamic process [5] . Recent studies have shown that the dynamics of gaze points can reflect various characteristics, such as the attractiveness and quality of areas viewed at different time stamps [38, 41, 43] .Scanpath generation aims to accurately model the trajectories of viewers' gaze points from two perspectives: the spatial positions of gaze points and their temporal order. The recent advances in deep learning techniques have enabled scanpath prediction in traditional 2D images [24, 25] . However, modeling scanpaths in 360 • images presents a more challenging task due to the significant differences from 2D planar images. In 360 • environments, observers engage not only in scanning the image with their gazes but also in changing their viewports by turning their heads or bodies [43] . This also necessitates better global scene feature extraction in the larger space offered by 360 • images. Additionally, most studies in scanpath modeling do not fully address the temporal dynamics of gaze points, which are crucial for accurately simulating realistic gaze point trajectories [5] . Capturing this temporal dependency among gaze points is particularly vital in interactive applications such as VR training simulation and interactive storytelling.Existing research and human attention datasets also indicate that, while observers may share similar gaze patterns for the same scene, there remains significant diversity in gaze behavior during free-viewing [32, 35, 42] . Therefore, generating multiple scanpaths is critical for real-world applications. Researchers [2, 27] have employed Generative Adversarial Networks (GANs) to generate diverse scanpaths (see Fig. 1 ). However, GAN-based methods suffer from instability issues, such as mode collapse. An alternative approach is Markov-based methods [12, 37] that use a Markov Chain to predict scanpaths (see Fig. 1 ), but they struggle with handling long-range dependencies due to their reliance on the immediate last state, leading to the loss of relevant information over time.In this research, we propose a novel approach, ScanTD, to address the above challenges in 360 • scanpath prediction. Based on recent research that confirmed the better capability of the Diffusion Model in learning to generate diverse sequential outputs [16, 21] , we incorporate the Diffusion Model into the scanpath prediction task. We employ a conditional Diffusion Model that uses the deep features of the given 360 • image as its condition and effectively learns the distribution of scanpath data via a gradual denoising procedure. Previous studies model the temporal domain information based on Recurrent Neural Networks (RNNs) [19, 30] or Markov Models [37] , which struggle to cope with increasing lengths of time series and suffer from low computational efficiency. Recently, Transformer-based time-series processing models [48] have demonstrated superior performance in capturing long-range temporal dependencies, achieving higher efficiency than RNN-based methods.In our conditional diffusion model, we adopt a Transformer-based time-series (TTS) module in the denoising procedure to capture the temporal information across time-ordered gaze points.When extracting 360 • image features for our diffusion model, we take into account the spherical nature of 360 • images to address the typical distortions [13] to enhance the scanpath learning process. Most existing scanpath modeling methods for 360 • images rely on normal 2D CNNs, which often struggle to capture distant spatial relationships between content elements [1, 27, 37] . Instead, we utilize Vision Transformer [17] , incorporated with a Spherical CNN [9] , to learn the global scene features in 360 • images. This use of global attention significantly enhances the accuracy and realism of scanpath modeling in 360 • images.Our main contributions include: (1) We propose a novel 360 • scanpath prediction method based on time-series diffusion, ScanTD, which boosts the performance of scanpath generation in 360 • images. (2) We develop a transformer-based time-series (TTS) module with a novel temporal attention mechanism to better capture the dynamic temporal information across time-ordered gaze points. This modification significantly improves the accuracy of both the spatial positions and the temporal order of the generated gaze points. (3) We integrate Spherical CNNs into Vision Transformer for better global scene feature extraction in 360 • images.2 Related Works 2.1 Scanpath Modeling on 2D/360 • Images Scanpath prediction and generation have been explored in traditional 2D images for many years and have achieved notable progress. Scanpaths differ from saliency maps, in that they are time-aware. These time-aware models [5] highlight the limitations of static saliency maps by underscoring the importance of incorporating temporal dynamics into the modeling of eye movement trajectories. In recent years, the application of Deep Learning and Probabilistic Generative Models [14, 22, 25] has proven the efficacy of neural networks in recognizing and understanding the complex patterns of human visual attention. However, the performance of these models can be sensitive to certain parameters, such as the number of Gaussian components, which limits their ability to handle complex scenes. Markov chain-based methods [12] for gaze behavior modeling provide insights into integrating various influences on gaze behavior, yet they fall short in efficiently capturing the temporal dependencies in gaze points and shifts in visual attention. Beyond scanpath modeling for free-viewing situations, several studies [8, 45] have also explored goal-directed human visual attention modeling.For 360 • images, several attempts have been made to model saliency maps [6, 28, 29, 35, 36] , but scanpath modeling has received less attention. Initially, several scanpath prediction models based on saliency maps were developed, but their performance largely depends on the accuracy of the saliency map predictions [1, 3, 50, 51] (see Fig. 1 ). While some methods have focused on improving saliency prediction models and sampling strategies, constructing effective sampling strategies for time-dependent visual behavior remains challenging, as evidenced by the unstable results from models like SaltiNet [1] . Assens et al. [2] , have attempted to adapt 2D models for 360 • images but have encountered difficulties in accurately replicating ground truth scanpaths behaviors. Approaches using generative models, such as GANs, all struggle with training stability [2, 16] . The recent ScanDMM model [37] , which simulates visual working memory through a Markov chain, shows promise in predicting realistic scanpaths. However, it struggles with the loss of state information over longer timescales, a limitation inherent to the Markov chain model itself. Furthermore, while ScanDMM [37] offers a probabilistic generative approach to capture gaze behavior diversity, it is constrained by the Markov assumption that future states depend solely on the current state, potentially limiting scanpath diversity due to this sequential dependence. This approach effectively models individual scanpaths but falls short in capturing the diversity of scanpaths across different viewers. In this context, our model, ScanTD, allows for more flexible modeling of temporal dependencies and panoramic scene semantic features, generating more diverse and realistic scanpaths. It is worth noting that all these studies use CNN-based methods for scene feature extraction in 360 • environments, which are limited by their local perception. Moreover, employing CNN-based methods without pre-trained models for scene feature extraction in 360 • images may result in poor generalization and challenges in capturing the complex spatial relationships inherent in panoramic content. In contrast, pre-trained models like the Vision Transformer (ViT) [17] offer robustness and enhanced generalization across diverse visual content, potentially improving scanpaths generation accuracy. 
diffusion model: 
 Diffusion models have shown success in various visual applications, including image production [33, 49] and audio synthesis [26] . These models consist of two processes: forward and reverse. The forward process adds Gaussian noise to the original data distribution 𝑥 0 ∼ 𝑞(𝑥 0 ), generating latent variables 𝑥 1 through 𝑥 𝑇 :EQUATIONThe reverse process aims to reconstruct the original data using a learned neural network:EQUATIONHere, 𝛽 𝑡 ∈ (0, 1) is the noise variance at time 𝑡, and 𝜇 𝜃 and Σ 𝜃 are learned parameters. Diffusion probabilistic models excel in matching complex data distributions and managing long-range dependencies by reversing a gradual, multi-step noising process [21] . Dhariwal et al. [16] highlight their advantages over GANs in synthesizing high-quality images, with a structured generative process providing greater stability and robustness. Additionally, diffusion models outperform GANs in covering distribution modes, suggesting superior capability in handling data variability [31] . For scanpath modeling in 360 • images, the unique demands align well with the strengths of diffusion models, enabling them to navigate complex spatial relationships and temporal sequences effectively. Similar to conditional-VAE and conditional-GAN, diffusion models can treat the encoded latent input, 𝑋 , as a condition [4, 47] . This approach extends to time-series prediction and audio waveform generation in models like CSDI [39] and WaveGard [7] . Traditionally, diffusion models for image-to-image processing use a U-net architecture [34] . However, for sequence-to-sequence tasks like text generation, Transformer-based methods have been employed as alternatives [20] . For instance, CSDI [39] uses a Transformer-based model with two-dimensional attention mechanisms for capturing temporal and feature dependencies in multivariate time-series data.Inspired by these advances, our proposal, ScanTD, leverages scene features from 360 • images as conditions to generate varied scanpaths. Specifically, we introduce a transformer-based time-series module with a novel multi-attention mechanism, replacing the Unet to learn the temporal order of gaze points more effectively. This innovation allows ScanTD to better capture the temporal and feature dependencies in multivariate time-series data, making it particularly suited for scanpath modeling. 
method: 
 Problem statement. In a 360 • environment, viewers' gaze point movements can be considered as a dynamic, temporally ordered process. A human scanpath can be defined as a time series of gaze points𝑋 1:𝑇 = (𝑋 1 , 𝑋 2 , . . . , 𝑋 𝑇 ) ∈ R 3×𝑇, where 𝑋 𝑡 is the 3D coordinate (𝑥 𝑡 , 𝑦 𝑡 , 𝑧 𝑡 ) of a gaze point. Specifically, given a 360 • image, the scanpath prediction method aims to generate realistic scanpaths X1:𝑇 . For better real-world application and to simulate different viewers, the method needs to efficiently generate multiple scanpaths X1 , X2 ..... X𝑚 for one given image, ensuring the diversity of the generation results [35] .As illustrated in Fig. Given a 360 • image, the output of this stage is an image feature embedding with a shape of 𝐵 × 20 × 196 (𝐵 is the batch size). Different from traditional CNN-based methods, ViT is a Transformerbased architecture, devoid of convolutions to process flattened image patches, enabling it to effectively capture long-range dependencies between different regions of an image. This convolutionfree approach of the ViT not only challenges the prevailing CNN paradigms but also sets new benchmarks in large-scale image classification tasks, achieving state-of-the-art performance on datasets like ImageNet [15] . The shape of vanilla ViT [17] is 𝐵 × 197 × 768, and we remove the last layer used for image classification and use the remaining ones with a shape of 𝐵 × 196 × 768 to output the scene features of the image. Traditional 2D CNNs can not handle the distortion introduced by the equirectangular projection-based representation of 360 • images. Therefore, we replace 2D CNNs with Spherical Convolutions [10] that have been shown to perform better for embedding equiretangular content in the ViT model. Spherical convolutions are a type of dilated convolution where the relationships between image elements are not established in 2D image space, but in a gnomonic, non-distorted spherical space. These spherical convolutions can represent kernels as patches tangent to a sphere where the 360 • is reprojected, and therefore allow the network to learn spatial relations of 360 • content, such as longitudinal continuities or spherical distortions. The core of Spherical Convolutions is adapting convolutional neural networks for 360 • image data by employing filter kernels that have a perception field covering the pixel positions aligned with the spherical nature of the images, effectively handling the distortions typical in 360 • data [10] . In a traditional ViT model, a 2D image 𝐼 is divided into 𝑁 2D image patches 𝑃 1 , 𝑃 2 , . . . , 𝑃 𝑁 . Each image patch 𝑃 𝑖 is transformed into a one-dimensional vector 𝑣 𝑖 through a patch embedding layer 𝐸: 𝑣 𝑖 = 𝐸 (𝑃 𝑖 ) . Then, the vectors 𝑣 1 , 𝑣 2 , . . . , 𝑣 𝑁 are fed into the Transformer encoder. In our modified approach, we replace the 2D patch embedding with spherical CNNs to extract features from the input 360 • image, preserving its spherical properties. These features are then partitioned into patches, each shaped into a feature vector. The resulting feature vector 𝐹 has a shape of 𝐵 × 196 × 768 and is subsequently fed into the ViT architecture.We use a conditional time-series data diffusion model for learning to generate scanpaths with better diversity and accuracy based on the extracted scene features. As shown in Fig. 2 , our model learns to denoise the gaze point sequence to make it progressively closer to the real data distribution through denoising steps. We use the scene features as the condition of the corresponding scanpaths of the same image by concatenating the features and the point sequence to form the sample space to model by the diffusion model. Notably, the noise is not added to the conditional scene features in the diffusion process. A Transformer-based time-series module is adopted to denoise the scanpath with the given condition, which enables a better exploration of the temporal dependencies among gaze points within the same scanpath.We denote the scene feature of an image as C with a shape of 𝐵 × 20 × 196 and denote the corresponding ground truth scanpaths as {S 𝑝 } = {S 1 , S 2 .....S 𝑛 } (𝑛 is the number of ground truth scanpaths) of that image. Each scanpath is a 20 × 3 matrix, representing a temporal sequence of 20 gaze points. We use the 3D coordinates of the point on the unit sphere 𝑥, 𝑦, 𝑧 to represent each point. Our conditional diffusion model learns to model the sample space where each sample is a combination of the image scene information C and one of the ground truth scanpaths S 𝑝 , expressed asz = C⊕S 𝑖 = [𝐶, 𝑆 𝑝 1 , 𝑆 𝑝 2 ...𝑆 𝑝 𝑚 ],where 𝑚 is the point number of a scanpath, normally set as 20. Note that if an image has 𝑛 ground truth scanpaths, there will be 𝑛 corresponding samples in the dataset used for learning. The forward diffusion process is expressed as:EQUATIONHere, 𝛽 𝑡 is a small constant that controls the noise level. 𝑡 represent the state of the variable z during the process. Different from conventional diffusion models that corrupt the whole z 𝑡 (both image scene information C and a ground truth scanpath S 𝑝 𝑡 ), we only add noises to the different states of the scanpath to obtain S 𝑝,1 , S 𝑝,2 , ..., S 𝑝,𝑇 , where S 𝑝,𝑡 represents the 𝑡-th state of the scanpath S 𝑝 during the partial noising process. The conditional part C remains unchanged during the forward process, making it a partial noising process. This modification allows for the generated scanpaths that not only mimic the variability found in human visual attention but also are deeply informed by the scene's content structure. In practical applications, this translates to the ability to create highly personalized user experiences in VR settings. By understanding the scene cues that guide visual attention, content creators can design experiences that are both informative and visually captivating, leading to enhanced engagement and satisfaction.We recover the original scanpath S 𝑝 in the reverse process using a conditional denoising module. Particularly, we utilize a Transformer-based time-series (TTS) learning module, denoted as 𝑓 𝜃 , to learn to reconstruct the original z 0 given the conditional part C that represents the scene's visual features.We adopt the TTS module to learn the transition from a later noised ground truth state S 𝑝,𝑡 back to an earlier noised state S 𝑝,𝑡 -1 , thereby effectively recovering the sequence of gaze points. Our TTS module comprises eight 2D multi-head self-attention blocks. As illustrated in Fig. 3 , each block integrates a spatial multi-head attention alongside a temporal multi-head attention, enabling the extraction of hidden relationships across both temporal and spatial dimensions. The spatial attention operates on the tensor of each feature to learn spatial dependencies, whereas the temporal attention processes the tensor of each timestamp of the gaze points to discern temporal dependencies. The input embedding with shape [𝐵, 𝐿, 𝐶] (batch size, length, channel number) is initially expanded in dimension to [𝐵, 𝐿, 𝐶, 𝐷] (batch size, length, channel number, depth). Within each 2D Multi-head Self-attention block, the expanded embedding undergoes a split by channel to facilitate feature extraction via spatial attention, followed by a sequence split for temporal feature extraction through temporal attention.Crucially, for the TTS module, we use a two-dimensional attention mechanism in each residual layer instead of convolution to capture spatial and temporal dependencies of multivariate time series. We employ a spatial Transformer layer and a temporal Transformer layer, both as 1-layer Transformer encoders. The spatial Transformer layer takes the features of each spatial position as inputs to learn spatial dependencies, whereas the temporal Transformer layer processes tensors at each time point to learn temporal dependencies. This capability is important for real-world applications, predicting where and when a viewer might look next can guide the adaptive streaming of high-definition content to just the right places at the right times.In the reverse process, we use the TTS module 𝑓 𝜃 to estimate the posterior distribution: where the 𝜇 𝜃 and 𝜎 𝜃 represent the parameters for the predicted mean and standard deviation of 𝑞 ( z 𝑡 -1|z 𝑡 , z 0 )) from the forward nosing process. The condition C is included in z and remains unchanged. After the conditional denoising process from 𝑡 = 𝑇 to 𝑡 = 0, we generate z 0:𝑇 =C ⊕ Ŝ𝑝,0:𝑇 . During the training process of our TTS-based denoising model, we optimize the parameters of the TTS module by the error between the generated scanpath states Ŝ1:𝑇 and the ground truth scanpaths S 1:𝑇 . The losses are computed following the original denoising diffusion model proposed in [21] :EQUATIONEQUATIONL 𝑉 𝐿𝐵 = E 𝑞 (z 1:𝑇 |z 0 ) log 𝑞(z 𝑇 |z 0 ) 𝑝 𝜃 (z 𝑇 ) 𝐿 𝑇 + (6) 𝑇 ∑︁ 𝑡 =2 log 𝑞(z 𝑡 -1 |z 0 , z 𝑡 ) 𝑝 𝜃 (z 𝑡 -1 |z 𝑡 ) 𝐿 𝑡 -1 -log 𝑝 𝜃 (z 0 |z 1 ) 𝐿 0The reverse process can be simplified by solving the following optimization problem:min 𝜃 L VLB = min 𝜃 𝑇 ∑︁ 𝑡 =1 ∥z 0 -𝑓 𝜃 (z 𝑡 , 𝑡)∥ 2 → min 𝜃 𝑇 ∑︁ 𝑡 =1 ∥ Ŝ𝑝,0 -f 𝜃 (z 𝑡 , 𝑡)∥ 2 (7)where the f 𝜃 (z 𝑡 , 𝑡) donates the fractions of reconstructed z 0 corresponding to Ŝ𝑝,0 .To predict scanpaths from a given 360 • image, FV-Net extract global scene features C, which serve as the condition for the following SD-Net. Subsequently, a standard noise N with the shape of (1, 20, 3), mirroring the shape of ground truth scanpaths, is generated based on the standard normal distribution. This noise is then concatenated with the condition C obtained from the FV-Net as: W = C ⊕ N. The concatenated tensor is subsequently fed to SD-Net. Within SD-Net, the mean and variance of the concatenated tensor W = C ⊕ N are used to denoise. This denoising process is iterated until a preset number of noising/denoising cycles are completed, resulting in the concatenation of the condition and the generated scanpaths: C⊕ Ŝ𝑝 . To ensure that the generated scanpaths exhibit the necessary diversity reflective of natural human gaze behavior when viewing images in a 360 • environment, we utilize different noise samples to generate scanpaths for the same image. 
datasets: 
 We use three 360 • image datasets: Sitzmann [35] , AOI [42] , and Salient360! [32] . Sitzmann [35] comprises 22 360 • images and 1,980 scanpaths collected from 169 different users. Each scanpath represents gaze information recorded over 30 seconds at a sampling rate of 120Hz. To simplify the data, we reduced the sampling frequency of these scanpaths to 1Hz, yielding 30 data points per scanpath. To address the limited size of the dataset, we augmented the data by rotating the 360 • images longitudinally and making corresponding adjustments to the scanpaths, thereby creating six distinct versions for each image. We used 19 of the 22 images for training, and the remaining three were included in the test set, effectively expanding our training dataset to 114 images. The AOI dataset [42] comprises 600 high-resolution 360 • images accompanied by 18,000 scanpaths, and we randomly select 100 360 • images for model evaluation. We use the training set in Salient360! [32] , which contains 85 images and 3,036 scanpaths, for evaluation as well.Evaluation Metrics. Many metrics have been proposed to quantitatively evaluate the similarity between two disparate scanpaths [18] . We used three metrics to quantitatively evaluate the performance of our ScanTD model: Levenshtein distance (LEV) , Dynamic Time Warping (DTW) , and Recurrence measure (REC) [18, 27] . LEV quantifies the similarity between two strings. In practical applications, LEV can also be employed to compare ordered sequences. In our experiment, the scanpath is defined as a series of 3D coordinate points 𝑋 1:𝑇 = (𝑋 1 , 𝑋 2 , . . . , 𝑋 𝑇 ) ∈ R 3×𝑇 . We convert these coordinate points into strings and then use LEV to assess their similarities. LEV is computed using the dynamic programming formula:EQUATION)where 𝛿 is the Kronecker delta function that equals 0 if 𝐴[𝐴] = 𝐵 [𝐵] and 1 otherwise. DTW is a metric designed to compare the similarity of different time-series sequences. The gaze points in a scanpath exhibit temporal order characteristics, and the output coordinate points typically contain (𝑥 𝑡 , 𝑦 𝑡 , 𝑧 𝑡 ), with each point representing a time step. Therefore, DTW can be used to measure the ability of each method in capturing the dynamic temporal order dependencies Room StudyDiverse Results of ScanTD. As illustrated in Fig. 4 , our approach ScanTD is capable of generating multiple scanpaths for the same scene, catering to the diversity required in real-world applications. Moreover, it ensures that the generated scanpaths are plausible and focus on relevant areas of interest within the scene. The road scene is from AOI dataset [42] , the room scene is from Salient360! dataset [32] , and the study scene is from Sitzmann dataset [35] . Selection of Methods for Comparison. We compare our approach ScanTD with three state-of-the-art 360 • scanpath prediction models: SaltiNet [1] , ScanGAN360 [27] , ScanDMM [37] and [46] and TAHA [44] . In addition, we noted a newly posted arXiv paper that also uses the Diffusion Model for scanpath prediction [23] . However, it did not share its source code and trained models. Furthermore, its evaluation metrics differ from those of the published related works ScanDMM [37] and ScanGAN360 [27] , and it did not provide sufficient details of their evaluation. Therefore, we chose not to compare with [23] .Our Comparison Methodology. ScanTD can produce any number of scanpaths as required to simulate different viewers. To perform a fair comparison, we use each method (SalitiNet [1] , ScanDMM [37] , ScanGAN360 [27] , HAT [46] , TAHA [44] and our ScanTD) to generate multiple scanpaths for each image and calculate both the average and the best results for each method using the evaluation metrics. Specifically, we compare each predicted scanpath against each ground truth scanpath, averaging the results for every image. Finally, the average result across each dataset is calculated. We also select the scanpath that best matches a corresponding ground truth scanpath for each image and then calculate the average of these results across the entire dataset. To minimize potential bias from randomly generated scanpaths, we tested each model 10 times per image. The results were then averaged to determine the final performance metrics as in [37] . For enhanced comparability and interpretability of the results, we calculate the human consistency [27] for each metric, serving as a realistic upper limit for model performance (refer to "Human" in Table 1 ). Additionally, we contrast our results with those of a chance model, which generates scanpaths by incorporating random Brownian motion into prior positions (refer to "Random walk" in Table 1 ) [37] .Comparison Results Analysis. Table 1 and Figure 5 show that our method, ScanTD, can generate results closer to the ground truth, with predicted gaze points showing reasonable displacements in the vertical direction. The beach scene is from the Salient360! dataset [32] , and the temporal order of our predicted gaze points across different parts of the scene more closely matches the ground truth compared to other methods. The square scene is from the AOI dataset [42] and our predicted gaze points concentrated on significant buildings. The office scene is from Sitzmann [35] , and the color distribution of gaze points demonstrates that ScanTD can better capture the temporal order of gaze points.We conducted ablation experiments to analyze the role of Spherical CNNs [9] in scene feature extraction and the ability of our designed TTS module to capture temporal information of gaze points. We compare ScanTD with three baselines: (1) 𝑆𝑐𝑎𝑛𝑇 𝐷 1 , using original ViT without spherical CNNs [9] for 360 TTS module. Moreover, to determine whether the improved performance is due to the employed TTS modules rather than the larger number of parameters, we employed ten spatial transformer blocks in 𝑆𝑐𝑎𝑛𝑇 𝐷 4 to ensure a similar number of parameters with the original 𝑆𝑐𝑎𝑛𝑇 𝐷 that has eight TTS blocks; both models have sizes of approximately 230 MB. Additionally, we compared another model, 𝑆𝑐𝑎𝑛𝑇 𝐷 5 , in which the number of TTS module blocks is reduced to four, to explore the impact of the number of parameters on model performance.GT ScanTD ScanDMM ScanGAN360 Gallery Room BuildingWe demonstrate the generalizability of our approach by applying it to saliency detection in 360 • images. Saliency detection in 360 • images aims to identify and predict areas within panoramic images that are most likely to capture and hold the viewer's visual attention. Intuitively, the series of gaze points generated by a scanpath prediction approach should be able to used to indicate the regions where the users concentrate within an image. We use the scanpaths predicted by ScanGAN360 [27] , ScanDMM [37] and our method to generate saliency maps. Following the method introduced in [37] to convolve fixation maps, we use an adapted Gaussian function [40] to generate a saliency map from multiple predicted scanpaths:EQUATIONwhere 𝜎 𝑥 = 𝜎 𝑦 cos(𝜃 ) . 𝜎 𝑦 = 15 • is a constant value, and 𝜃 ∈ -𝜋 2 , 𝜋 2 is the latitude of the gaze point. Fig. 6 shows a qualitative comparison where the saliency maps generated from ScanTD's prediction results are better than the others. The gallery scene is from the AOI dataset [42] , the room scene is from the Sailent360! dataset [32] , and the building scene is from Sitzmann [35] . 
conclusion and future work: 
 We propose a novel method, ScanTD, which offers a potential solution for the visual task of realistic scanpaths modeling on 360 • images. Through extensive experiments, we have confirmed that our method ScanTD achieves state-of-the-art performance across three test datasets. Our model, ScanTD, advances the theoretical understanding of scanpath generation in panoramic viewing and also holds significant potential for practical real-world applications. For instance, by accurately modeling gaze trajectories, our approach can aid content creators in designing VR and AR experiences that naturally guide the viewer's attention, enhancing narrative storytelling and informational clarity.There is still much to explore in scanpath modeling, especially when considering its potential applications in various sectors. For example, the task we are currently studying is free-viewing scanpaths. However, many practical applications of VR, such as navigational training for pilots or surgical procedure simulations for medical professionals, require task-driven human visual attention data. This distinction underscores the need for developing scanpath models that are tailored to specific tasks, offering a promising research direction that could enhance the effectiveness of training and education in VR environments. 
 
 Appendix:  


Decision: This paper presents ScanTD, a novel approach for predicting scanpaths in 360° images by means of diffusion models and features extracted from the image.  All the reviewers agree on the timeliness of the approach and the good shape of the paper. Some concerns were risen during the reviewing process but the authors were able to address them in the rebuttal. Given that the authors tackle all the reviewers' comments, I believe that the paper can be accepted for oral presentation.

Reviews: review: paper_summary: paper_summary: Overall the paper is well written with nice results. A nice study is conducted where the proposed method ScanTD shows very promising results in terms of Scanpath prediction in 360 degree images.

The main contributions of the paper are as follows:

1) It proposes a novel 360◦ scanpath prediction method based on time-series diffusion, ScanTD, which boosts the performance of scanpath generation in 360◦ images.
2) A transformer-based time-series (TTS) module with a novel temporal attention mechanism to better capture the dynamic temporal information across time-ordered gaze points. This modification significantly improves the accuracy of both the spatial positions and the temporal order of the generated gaze points.
3) Integrates Spherical CNNs into Vision Transformer for better global scene feature extraction in 360◦ images.
strength_weakness: strength_weakness: The strengths of the paper are as follows:

1) The use of Vision Transformer (ViT) along with the Spherical CNN (using patch embedding) indeed helps to extract the global features from the 360 degree images.
2) A conventional U-net is replaced with a Transformer-based Time-Series (TTS) network inside the Diffusion model to process the sequential data of the diverse scanpaths.
3) The overall pipeline shows novel contribution towards Scanpath prediction.
4) The ScanTD method outperforms other methods in terms of accuracy.
5) A nice ablation study is also performed to highlight the different modules and their respective contributions to the overall pipeline.
limitations: limitations: There are some few drawbacks as mentioned below:

1) More comparisons with other state-of-the-art methods could be performed.
2) It should be mentioned explicitly, the impact of the different modules inside the pipeline based on the ablation studies.
paper_summary: paper_summary: The paper proposes a novel approach called ScanTD for predicting scanpaths in 360° images. It uses a diffusion model conditioned on scene features extracted from the image by a Vision Transformer combined with Spherical CNNs. A transformer-based time-series module is incorporated into the diffusion model to capture the temporal dependencies in the scanpaths.
strength_weakness: strength_weakness: 1. The idea of using diffusion models for scanpath prediction is novel and potentially promising.
2. The use of Vision Transformers with Spherical CNNs for extracting global scene features from 360° images is a reasonable approach.
4. The paper provides a good overview of previous work on scanpath modeling and diffusion models.
limitations: limitations: 1. The paper lacks technical details and clarity in explaining the proposed method. The descriptions of the FV-Net and SD-Net components are vague, making it difficult to fully understand and reproduce the approach.
2. There is a lack of theoretical justification for the novel "temporal attention mechanism" mentioned, and no details are provided on how it works or how it improves over existing attention mechanisms.
3. The experimental results section is minimal, with only a brief statement about outperforming state-of-the-art methods on three datasets, without providing any insightful analysis.
4. More qualitative comparisons should be included. , ScanDMM looks better than the proposed method in some cases, such as the FIg. 5 (office) and Fig. 6 (Room)
5. The paper does not discuss the computational complexity or inference time of the proposed method.
paper_summary: paper_summary: This is a clearly written paper that introduces several novel methods to the domain of 360-degree panoramic images, achieving promising results in the task of Scanpath.
strength_weakness: strength_weakness: The main innovations of the paper are as follows:
1.	Replacing 2D CNN in ViT with Spherical CNN: This approach extracts global features from 360-degree images.
2.	Designing a Transformer-based Temporal Prediction Network: This network is used to generate gaze points.
3.	Utilizing a Conditional Diffusion Model: This model predicts points along the Scanpath.
limitations: limitations: 1.	Experimental Comparisons: The results differ from the original data in the ScanDMM and ScanGAN papers when using the same metrics. The original papers averaged results over 10 runs and reported the best, whereas this paper uses 100 runs. The reasoning behind this experimental setup needs further explanation.
2.	Ablation Study Details (Section 4.4): The setup and description in the ablation study are too simplistic and lack necessary analysis. For example, in ScanTD_2, TTS was not used, so it is important to consider whether parameters were reduced in the control group. It is also worth discussing if the reduced performance can be compensated by increasing network parameters. Simply showing decreased performance without analyzing the reasons is insufficient to demonstrate the effectiveness of the innovations.
3.	Generalization Evidence (Section 4.5): Using only three saliency detection visualization examples and evaluation formulas to claim superior generalization of the model is not robust for a scientific paper. It would be better to reduce Section 4.5 and enhance the experiments and analysis in Section 4.4.
4.	Model Architecture: In the TTS model, the transition from Spatial Multi-head Attention to Temporal Multi-head Attention only changes the input feature dimension. However, the model concatenates the two feature vectors instead of running them in parallel. Would changing the order of these layers yield the same results?
5.	Feature Vector Reduction: The experimental model significantly reduces the transformer input feature vector from B * 196 * 768 to B * 20 * 196. It is necessary to evaluate whether this reduction affects the model's performance.
6.	ViT Core Concept: One of ViT's main concepts is segmenting an image into small patches and learning the self-attention between them. This experiment replaces image segmentation with a whole image, requiring further experimental validation and analysis to verify that self-attention learning is still effective.