STAR-VP: Improving Long-term Viewport Prediction in 360° Videos via Space-aligned and Time-varying Fusion

Authors: baoqi  gao
daoxu  sheng
lei  zhang
qi  qi
zirui  zhuang
jingyu  wang
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 10:45:19

Abstract: Accurate long-term viewport prediction in tile-based 360° video adaptive streaming helps pre-download tiles for a further future, thus establishing a longer buffer to cope with network fluctuations. Long-term viewport motion is mainly influenced by Historical viewpoint Trajectory (HT) and Video Content information (VC). However, HT and VC are difficult to align in space due to their different modalities, and their relative importance in viewport prediction varies across prediction time steps. In this paper, we propose STAR-VP, a model that fuses HT and VC in a Space-aligned and Time-vARying manner for Viewport Prediction. Specifically, we first propose a novel saliency representation $salxyz$ and a Spatial Attention Module to solve the spatial alignment of HT and VC. Then, we propose a two-stage fusion approach based on Transformer and gating mechanisms to capture their time-varying importance. Visualization of attention scores intuitively demonstrates STAR-VP's capability in space-aligned and time-varying fusion. Evaluation on three public datasets shows that STAR-VP achieves state-of-the-art accuracy for long-term (2-5s) viewport prediction without sacrificing short-term ($<$1s) prediction performance.

Main paper: 
 introduction: 
 360-degree videos capture the full surrounding environment, enabling viewers to freely change perspectives during playback for a more immersive and interactive viewing experience. The data rate of 360-degree video needs to be approximately six times higher than that of regular video to achieve a comparable angular resolution [1] . The increased data rate poses significant challenges for the streaming of 360-degree video over the internet. Therefore, tilebased adaptive streaming [36] has been proposed to stream only the tiles within the user's Field of View (FoV) at higher bitrates, thus reducing bandwidth and computing resource overhead. Nevertheless, the effectiveness of tile-based adaptive streaming relies heavily on the accuracy of viewport prediction. Accurate long-term viewport prediction helps pre-download tiles for a further future, establishing a longer buffer to cope with network fluctuations. Long-term viewport motion is mainly influenced by Historical viewpoint Trajectory (HT) and Video Content information (VC), thus requiring the fusion of these two types of information for prediction.However, the fusion of HT and VC for viewport prediction faces two key challenges: 1) Spatial alignment. HT is represented as coordinate sequences, while commonly used VC information such as saliency map [26, 40, 43] and motion map [11, 12] is represented as image. This makes it difficult for the model to align the spatial positions represented by viewpoint coordinates in HT with the pixel positions in VC information. 2) Time-varying importance. The relative importance of HT and VC in viewpoint prediction varies across prediction time steps. Intuitively, in the short term, the viewpoint tends to continue its previous motion patterns due to inertia, thus HT is more important; whereas in the long term, the viewpoint tends to be attracted by saliency regions within the video content, thus VC is more important [31] . However, this timevarying pattern of relative importance can be very complex and cannot be described by simple rules.The example in Fig. 1 further illustrates the necessity of considering the above two challenges. In the spatial dimension, User 1 and User 2's viewpoints are attracted by different salient regions due to their different positions, thus the viewport prediction model needs to align the spatial position information expressed in HT and VC. In the temporal dimension, User 1's viewpoint maintains its previous motion pattern in the short term due to inertia, until it is mainly attracted by salient regions in the video content in the long term. Therefore, the viewport prediction model needs to capture the time-varying importance of HT and VC.In this paper, we propose STAR-VP, a model that fuses HT and VC in a Space-aligned and Time-vARying manner for Viewport Prediction. Specifically, to address the spatial alignment of HT and VC, we first propose a novel saliency representation 𝑠𝑎𝑙𝑥𝑦𝑧, which represents a 2D grayscale saliency map as a collection of (3D spatial coordinates + saliency value) tuples, explicitly injecting spatial coordinate information into saliency information. Then, we design a Spatial Attention Module with a Transformer-like architecture to solve the spatial alignment problem in conjunction with 𝑠𝑎𝑙𝑥𝑦𝑧. To address the time-varying importance of HT and VC, we propose a two-stage fusion approach. In the first fusion stage, we design a Temporal Attention Module with a Transformerlike architecture. This module enables the model to adaptively  
related work 2.1 viewport prediction in 360°videos: 
 To achieve more accurate prediction, most existing viewport prediction models consider not only the user's HT, but also auxiliary information such as saliency map [26, 32, 38, 41] , motion map [11, 12] , object trajectory [6, 20] , and other users' HT [2, 5, 25] . In this paper, we mainly explore the fusion of HT and the most commonly used VC information, saliency map.To address the spatial alignment of HT and VC, some works [26, 34] convert HT from coordinates into VC-like heatmap representations using Gaussian filtering. While this approach unifies the modalities of HT and VC, it increases computational and storage overheads. Moreover, the spatial position information expressed in 2D ERP images cannot well reflect the motion of the viewpoint on the 3D sphere.Regarding the time-varying importance of HT and VC, most existing viewport prediction models [11, 26, 37] concatenate HT and VC along the feature dimension. TRACK [31] uses three LSTM modules to separately process HT, VC, and their concatenated features, aiming to dynamically balance HT and VC across prediction time steps. However, due to the inherent inability of LSTMs to model complex and long-term dependencies, the TRACK model has limitations in capturing complex time-varying patterns of relative importance between HT and VC. HeMoG [30] is a white-box model built on physics of rotational motion and gravitation, achieving higher interpretability. However, its performance is sensitive to manually-tuned parameters. MFTR [42] uses three Transformer encoders [33] to separately process HT, VC, and their concatenated features. Compared to TRACK, MFTR more effectively learns the time-varying importance of HT and VC. However, MFTR still only adaptively selects HT and VC with time-varying patterns at the feature level, making it difficult to avoid sacrificing short-term prediction performance while improving long-term prediction performance.𝑃 !"#$ 𝑆 !"#$ 𝑃 !"%&' 𝑆 !"%&' 𝑃𝑆 𝐸 𝑊′′ 𝑊 ' 𝑆 𝑃 𝑆 !"# C ! C ( Q Q K V V K Q V K K V Q Linear LinearTransformer [33] was initially known for its excellent performance in natural language processing tasks such as machine translation. It aggregates information across the entire sequence with adaptive weights through attention mechanisms, capturing long-term dependencies in a parallel manner. In recent years, models based on the Transformer architecture have shown excellent performance not only in natural language processing tasks [9, 16, 28] but also in various visual tasks [3, 10, 17, 21, 22] and multimodal tasks [18, 19, 23] .Since the fusion of HT and VC involves time series and multimodal data, it is suitable to use Transformer-based models for processing. The application of gating mechanisms in neural networks can be traced back to the LSTM model [14] . LSTM uses three gating units to control the flow of information to better capture long-term dependencies. Since then, gating mechanisms have been widely used in various deep learning models [7, 27, 45] to better focus on important information and suppress irrelevant information. 
method 3.1 overview: 
 The overall architecture of STAR-VP is depicted in Fig. 2 . STAR-VP predicts the viewpoint positions for the future 𝑇 𝐻 time steps( 𝑃), given the viewpoint positions from the past 𝑇 𝑀 time steps ( 𝑃) and the video frames spanning both the past 𝑇 𝑀 and the future 𝑇 𝐻 time steps ( 𝑉 ). Each viewpoint position is represented as a 3D spatial coordinate on the unit sphere, and each video frame is represented as an image in Equi-Rectangular Projection (ERP) format.STAR-VP mainly consists of five modules: SalMap Processor, LSTM module, Spatial Attention Module, Temporal Attention Module, and Gating Fusion Module. First, the original video frames 𝑉 are converted into saliency maps 𝑆 by PAVER [39] , a model specifically designed for saliency detection in 360-degree videos. Then, the SalMap Processor converts 𝑆 into the 𝑠𝑎𝑙𝑥𝑦𝑧 representation 𝑆 𝑥 𝑦𝑧 .The LSTM module only uses the viewpoint positions from the past The following subsections will introduce the specific details of the five modules mentioned above and the internal Encoder/Decoder.The SalMap Processor converts the original saliency map into a compact saliency representation with pixel coordinate information. Given the saliency map 𝑆 ∈ R (𝑇 𝑀 +𝑇 𝐻 ) ×𝐻 ×𝑊 ×1 , the SalMap Processor represents each pixel's saliency information as (𝑥, 𝑦, 𝑧, 𝑠), where (𝑥, 𝑦, 𝑧) are the 3D coordinates of the pixel block center and 𝑠 is the corresponding saliency value. The 3D coordinates (𝑥, 𝑦, 𝑧) of the pixel block with index (𝑖, 𝑗) are calculated as follows:EQUATIONwhere 𝐻 and 𝑊 are the height and width of the saliency map respectively.To reduce data redundancy, the SalMap Processor retains only the top 𝑡𝑡 points with maximum saliency, which are then sampled at rate 𝑠𝑟 . Finally, the SalMap Processor outputs 𝑆 𝑥 𝑦𝑧 ∈ R (𝑇 𝑀 +𝑇 𝐻 ) ×𝐷 𝑃 ×4 , where 𝐷 𝑃 = ⌊𝑡𝑡 • 𝑠𝑟 ⌋.When watching 360-degree videos, users' viewpoint motion is relatively stable most of the time. Due to inertia, the future motion of the viewpoint, especially in the short term, is highly correlated with its historical trajectory. Therefore, we use LSTM, a lightweight and efficient temporal prediction model, to take only the viewpoint positions from the past 𝑇 𝑀 time steps as input and output predictions for the future 𝑇 𝐻 time steps in an autoregressive manner:EQUATIONwhere 𝑃 𝑡 -𝑇 𝑀 +1:𝑡 ∈ R 𝑇 𝑀 ×3 is the viewpoint positions from the past 𝑇 𝑀 time steps, and 𝑃 ′ 𝑡 +1:𝑡 +𝑇 𝐻 ∈ R 𝑇 𝐻 ×3 is the LSTM prediction results for the future 𝑇 𝐻 time steps. This prediction has excellent short-term performance. The specific details of the LSTM module are omitted here.The Spatial Attention Module is based on a Transformer-like architecture. Compared to the original Transformer model, we have modified its positional encoding part and the structure and query input of its Decoder.First, since both 𝑆 𝑥 𝑦𝑧 and the viewpoint data already contain 3D spatial coordinate information, we modify the positional encoding in the original Transformer to trainable parameter embedding, aiming to unify the representations of the two modalities. The Spatial Attention Module takes 𝑆 𝑠 -𝑖𝑛 and 𝑃 𝑠 -𝑖𝑛 as input, where𝑆 𝑠 -𝑖𝑛 = 𝑆 𝑥 𝑦𝑧 ∈ R (𝑇 𝑀 +𝑇 𝐻 ) ×𝐷 𝑃 ×4. The calculation formula of 𝑃 𝑠 -𝑖𝑛 is as follows:𝑃 𝑠 -𝑖𝑛 = C 𝑓 (C 𝑡 ( 𝑃 𝑡 -𝑇 𝑀 +1:𝑡 , 𝑃 ′ 𝑡 +1:𝑡 +𝑇 𝐻 ),𝑇 𝐸) ∈ R (𝑇 𝑀 +𝑇 𝐻 ) ×4 , (3)where C 𝑓 (•) and C 𝑡 (•) denote concatenating vectors along the feature dimension and the temporal dimension respectively, and 𝑇 𝐸 ∈ R (𝑇 𝑀 +𝑇 𝐻 ) ×1 is a vector expanded from a 1 × 1 trainable parameter along the temporal dimension, used to transform 𝑃 𝑠 -𝑖𝑛 into a form similar to (𝑥, 𝑦, 𝑧, 𝑠) of 𝑆 𝑠 -𝑖𝑛 , facilitating modality fusion and spatial alignment. Subsequently, two linear layers convert 𝑆 𝑠 -𝑖𝑛 and 𝑃 𝑠 -𝑖𝑛 into 𝑆 ∈ R (𝑇 𝑀 +𝑇 𝐻 ) ×𝐷 𝑃 ×𝐷 𝐶 and 𝑃 ∈ R (𝑇 𝑀 +𝑇 𝐻 ) ×𝐷 𝐶 , respectively.Next, we unify the Decoder to have the same architecture as the Encoder, and use viewpoint features as the query input to the Decoder, fully utilizing the cross-attention mechanism to decode viewpoint-specific saliency features. The Encoder encodes 𝑆 into 𝑆 ′ using self-attention, with 𝑆 serving as query, key, and value. The Decoder uses viewpoint position features 𝑃 as query and 𝑆 ′ as key and value, employing cross-attention to perceive the position information of viewpoint, giving higher attention to the saliency information near the viewpoint. Further details of the Encoder/Decoder are provided in Section 3.7.Similar to the Spatial Attention Module, the Temporal Attention Module is also based on a Transformer-like architecture. Compared to the original Transformer model, we have modified its positional encoding part and the structure and query input of its Decoder.First, we modify the positional encoding in the original Transformer to positional embedding and trainable modality-specific embedding. The Temporal Attention Module takes 𝑃 𝑠 -𝑜𝑢𝑡 = 𝑃 ∈ R (𝑇 𝑀 +𝑇 𝐻 ) ×𝐷 𝐶 and 𝑆 𝑠 -𝑜𝑢𝑡 ∈ R (𝑇 𝑀 +𝑇 𝐻 ) ×𝐷 𝐶 as input, and outputs the composite vector 𝑃𝑆 ∈ R 2(𝑇 𝑀 +𝑇 𝐻 ) ×𝐷 𝑃𝑆 as follows:EQUATIONwhere C 𝑓 (•) and C 𝑡 (•) denote concatenating vectors along the feature dimension and the temporal dimension respectively, 𝑃𝐸 ∈ R (𝑇 𝑀 +𝑇 𝐻 ) ×𝐷 𝑃 𝐸 is fixed 1D Fourier positional embeddings for a total of 𝑇 𝑀 + 𝑇 𝐻 time steps, 𝑇 𝐸 𝑃 ∈ R (𝑇 𝑀 +𝑇 𝐻 ) ×𝐷 𝑇 𝐸 and 𝑇 𝐸 𝑆 ∈ R (𝑇 𝑀 +𝑇 𝐻 ) ×𝐷 𝑇 𝐸 are trainable modality-specific embeddings, and𝐷 𝑃𝑆 = 𝐷 𝐶 + 𝐷 𝑃𝐸 + 𝐷 𝑇 𝐸 .Next, we use the vector concatenated by positional embeddings and modality-specific embeddings as the query input to the Decoder, fully utilizing the cross-attention mechanism to decode the viewpoint information at each prediction step. The query embeddings 𝐸 ∈ R 𝑇 𝐻 × (𝐷 𝑃 𝐸 +𝐷 𝑇 𝐸 ) are calculated as follows:EQUATIONThe Decoder in the Temporal Attention Module is also unified to have the same architecture as the Encoder. The Encoder encodes 𝑃𝑆 into 𝑃𝑆 ′ using self-attention, with 𝑃𝑆 serving as query, key, and value. The Decoder uses 𝐸 as query and 𝑃𝑆 ′ as key and value, employing cross-attention to perceive the specific temporal and modality information. This captures the time-varying importance of the viewpoint position features and saliency features. The Encoder/Decoder is introduced in detail in Section 3.7.As the first fusion stage, the Temporal Attention Module adaptively adjusts the focus on HT and VC features at each prediction time step, outputting predictions 𝑃 ′′ ∈ R 𝑇 𝐻 ×3 with excellent longterm performance.As the second fusion stage, the Gating Fusion Module generates vectors 𝑊 ′ ∈ R 𝑇 𝐻 ×1 and 𝑊 ′′ ∈ R 𝑇 𝐻 ×1 to control the weights of 𝑃 ′ and 𝑃 ′′ at each prediction time step, and outputs the final viewpoint prediction results 𝑃 ∈ R 𝑇 𝐻 ×3 :EQUATIONwhere ⊗ and ⊕ denote element-wise multiplication and addition respectively, C 𝑓 denotes concatenating vectors along the feature dimension, 𝜎 𝑟 is the ReLU activation function, 𝜎 𝑠 is the sigmoid activation function,𝑥 ∈ R 6𝑇 𝐻 ×1 , 𝑊 𝑟 ∈ R 𝐷 𝐺 ×6𝑇 𝐻 , 𝑏 𝑟 ∈ R 𝐷 𝐺 ×1 , 𝑊 𝑠 ∈ R 𝑇 𝐻 ×𝐷 𝐺 , 𝑏 𝑠 ∈ R 𝑇 𝐻 ×1 .Inspired by the universal multimodal fusion architecture Perceiver IO [15] , we unified the architecture of the Encoder and Decoder in the Spatial Attention Module and the Temporal Attention Module, and added some additional linear layers to adjust the feature dimensions of the vectors. The Encoder uses self-attention to extract deep features, while Decoder uses cross-attention to perceive current specific spatial/temporal information, achieving adaptive feature selection. The Encoder/Decoder consists of two components: the Multi-head Attention and the Multilayer Perceptron (MLP). The Multi-head Attention component performs the following calculation given 𝑄 𝑖𝑛 ∈ R 𝑛 𝑞 ×𝑑 𝑞 , 𝐾 𝑖𝑛 ∈ R 𝑛 𝑘 ×𝑑 𝑘 , and 𝑉 𝑖𝑛 ∈ R 𝑛 𝑣 ×𝑑 𝑣 :EQUATIONwhere L 𝑞,𝑘,𝑣 are linear layers mapping each input to a shared feature dimension 𝑑 𝑙 , and L 𝑜 is a linear layer projecting the output to a desired feature dimension. It is worth noting that all of these linear layers are applied convolutionally along the index dimension, and the batch and head dimensions have been excluded for clarity.The MLP component consists of two linear layers with a GELU [13] nonlinearity applied after the first layer:EQUATIONThe final output of the Encoder/Decoder is calculated as follows:Attention(𝑄, 𝐾, 𝑉 ) = A (𝑂 𝑎𝑡𝑡𝑛 , MLP(N (𝑂 𝑎𝑡𝑡𝑛 ))),EQUATIONwhere N denotes layer normalization, and A denotes addition in residual connection. 
experiments: 
 STAR-VP is evaluated on a unified evaluation framework [29] for viewport prediction methods in 360-degree videos. The framework unifies five datasets into a common format (sampling period: 0.2s; viewpoint representation: 3D coordinates on the unit sphere) and summarizes current combination approaches of HT and VC, evaluating their performance.4.1.1 Datasets. We evaluate STAR-VP on three datasets:(1) David_MMSys_18 [8] contains 19 20-second 360-degree videos in ERP format and head and eye tracking data from 57 participants watching the videos freely.(2) Nasrabadi_MMSys_19 [24] consists of 28 videos (60s) and viewport traces from 60 participants watching the videos. Each video is viewed by 30 participants.(3) Wu_MMSys_17 [35] includes head tracking data from 48 users watching 18 360-degree videos of 5 categories. The other two models use both HT and VC: (4) TRACK [31] uses three LSTM modules to separately process HT, VC, and their concatenated features, aiming to dynamically balance HT and VC across prediction time steps.(5) MFTR [42] is a complex multi-modal fusion Transformer-based model. However, its basic fusion framework for HT and VC is to use three Transformer-encoder-based modules to separately process HT, VC, and their concatenated features.It is worth noting that the above comparison models basically cover all the fusion methods of HT and VC in the current viewport prediction models we know.We use Orthodromic Distance (OD) [31] and Intersection over Union (IoU) [44] as the performance metrics.The Orthodromic Distance between points 𝑃1 and 𝑃2 is calculated as follows:EQUATION)where 𝑃 1 and 𝑃 2 are two 3D coordinates on the unit sphere, and • is the dot product operation. When we use "3D spatial coordinates on the unit sphere" to represent the viewpoint position, the orthodromic distance actually represents the shortest spherical distance between the two viewpoints.The Intersection over Union is an evaluation metric related to the specific tile partition method. We divide the ERP video frame into 9 × 16 tiles and set the user's field of view to 100°. If the angle between the center of the tile and the viewpoint is less than 50°, the tile is considered to be in the viewport, and its label is set to 1; otherwise, it is set to 0. Then, the Intersection over Union is computed on these labels as follows: 𝐼𝑜𝑈 = 𝑇 𝑃 𝑇𝑇 (11) where 𝑇 𝑃 represents True Positive, i.e., the intersection between prediction and ground-truth of tiles with label 1; 𝑇𝑇 represents True Total, i.e., the union of all tiles with label 1 in the prediction and in the ground-truth.4.1.4 Saliency Map Generation and Processing. We use PAVER [39] , a Vision Transformer-based model trained specifically for 360°video saliency detection, to compute the saliency map at each time step. The original 224 × 448 saliency maps are processed by the SalMap Processor module of STAR-VP to generate 128 × 4 compact saliency representation 𝑆 𝑥 𝑦𝑧 with a top threshold (𝑡𝑡) of 20480 and sample rate (𝑠𝑟 ) of 1/160. In addition, for model size and memory considerations, the competitors use downsampled 64 × 128 saliency maps as the actual VC input. Even so, the data volume of saliency information actually used by STAR-VP is only 1/16 of that of other competitors.The hyperparameters of all models are shown in Table 1 . All models predict the next 5s (25 steps) of viewpoint positions based on the past 3s (15 steps). Fig. 3 shows the prediction performance of STAR-VP and five competitors on three datasets. From the figure, we can see that HTonly models (linear-regression, deep-pos-only, VPT360) have better performance in short-term prediction (<1s) as they are not influenced by VC information. The two competitors (TRACK, MFTR) that consider both HT and VC for prediction have better longterm prediction (2-5s) performance, but worse short-term prediction performance than HT-only models. This indicates that their improvement in long-term prediction performance comes at the expense of short-term prediction performance. In contrast, STAR-VP achieves the best long-term prediction performance without sacrificing short-term prediction performance. This is mainly due to the space-aligned and time-varying fusion of HT and VC by using 𝑠𝑎𝑙𝑥𝑦𝑧 representation and the two-stage fusion method in STAR-VP. Table 2 further shows the average long-term prediction performance of each model on three datasets. We note that because TRACK and MFTR sacrifice short-term prediction performance to improve long-term prediction, their average performance over 2-5s is often worse than HT-only models such as deep-pos-only and VPT360. Across all datasets and metrics, STAR-VP achieves stateof-the-art performance in long-term prediction and comparable performance in short-term prediction. To analyze the time-varying attention pattern of STAR-VP, we multiply the attention matrices of its Temporal Attention Module to obtain the attention scores between each prediction step and the HT/VC inputs, and visualize them as the heatmap shown in Fig. 5 . It can be seen from the figure that as the prediction step increases, the attention scores of recent 2s HT gradually decrease, while the attention scores of VC and earlier HT gradually increase. This is consistent with the idea that HT reflects inertia for short-term viewpoint motion, while VC influences long-term motion through salient regions. We also observe that as the prediction step increases, STAR-VP gradually increases its attention to HT over the past 2s. This is because HT over the past 2s reflects certain user viewpoint motion preferences that are beneficial for long-term viewpoint prediction. In summary, the visualization results of the attention scores of the Temporal Attention Module indicate that STAR-VP performs feature selection on HT and VC with a time-varying pattern in the first fusion stage. It is worth noting that to facilitate a clearer observation of the time-varying attention pattern, we apply minmax normalization to the original attention scores matrix along the columns. To analyze the contribution of each module in STAR-VP to the prediction performance, we conducted an ablation study on the Spatial Attention Module (SAM), Temporal Attention Module (TAM), and Gating Fusion Module (GFM). For SAM, we replaced it with a linear layer. For TAM, we replaced it with an LSTM. For GFM, we removed it directly and used the output of TAM as the final prediction result. The results are depicted in Fig. 7 . It can be seen from the figure that removing any module will affect the prediction performance. Removing SAM and TAM mainly affects long-term prediction performance, while removing GFM mainly affects short-term performance. This result not only demonstrates the importance of each module but also indicates that the main role of SAM and TAM is to improve the long-term viewpoint prediction performance of STAR-VP, while the main role of GFM is to block the output of the long-term prediction module in short-term prediction to ensure the short-term viewpoint prediction performance of STAR-VP.4.4.2 Effects of 𝑠𝑎𝑙𝑥𝑦𝑧 Representation. We tested the long-term prediction performance of three models that consider both HT and VC with the original saliency map and the 𝑠𝑎𝑙𝑥𝑦𝑧 representation, respectively. The results are shown in Table 3 . From the test results, it can be seen that the performance of the TRACK and MFTR models decreases after using the 𝑠𝑎𝑙𝑥𝑦𝑧 representation, while only STAR-VP improves its performance after using the 𝑠𝑎𝑙𝑥𝑦𝑧 representation. This result demonstrates the improvement of the 𝑠𝑎𝑙𝑥𝑦𝑧 representation on long-term prediction performance and highlights the importance of the 𝑠𝑎𝑙𝑥𝑦𝑧 representation in conjunction with the Spatial Attention Module. 
conclusion: 
 In this paper, we propose a novel long-term viewpoint prediction model STAR-VP. It fuses viewpoint and saliency information in a space-aligned and time-varying manner, achieving the best longterm prediction performance without sacrificing short-term prediction performance. A viewpoint prediction model with excellent long-term performance helps 360°video streaming systems predownload tiles for a longer time ahead, thereby establishing a longer buffer to cope with network fluctuations. 
 
 Appendix:  


Decision: This paper proposes an approach for viewport prediction for 360-degree video streaming which takes viewport trajectories and video content into approach. The paper is well written, relevant and interesting. The reviewers in general like the paper and are inclined to its acceptance. However, after the rebuttal, more questions regarding the novelty contribution were risen, leading to some of reviewers to low their score to borderline accept. Thus, while I see the value of the paper, I would recommend it as a poster.

Reviews: review: paper_summary: paper_summary: The paper proposes a model called STAR-VP that improves long-term viewport prediction in 360° videos through space-aligned and time-varying fusion of historical viewpoint trajectory and video content information leveraging a transformer architecture.
strength_weakness: strength_weakness: -	The idea of aligning the spatial information and the saliency information for further processing is interesting and seems to work well in practice
-	The approach is well thought and improves compared to sota
-	The evaluation is performed using a common evaluation framework, meaning that the results should be easier to validate and use in future works as well
limitations: limitations: -	Even though interesting, not sure if the spatial representation is big enough to be claimed as a contribution to the paper
-	All of the proposed modules seem to contribute towards better performance for long term prediction compared to sota. What is unclear though is exactly how the relationship between short term prediction (dominated by the user trajectory) and long term one (dominated by the saliency map) plays out. A qualitative analysis could have been helpful to shed some light on this aspect
paper_summary: paper_summary: The paper proposes a novel viewport prediction algorithm for 360 degree video that historical trajectory and video content information through a Transformer architecture. In addition, a novel saliency representation salxyz is proposed. Its performance and generalizability are shown on three different, public datasets indicating its improvement for long-term prediction horizons.
strength_weakness: strength_weakness: - The paper is well-written, has a good structure and is fluent to follow. Its contributions are novel, relevant and timely. 
- The authors propose an interesting line of thought by linking the applicability of content-agnostic vs content-aware prediction to the envisioned prediction horizon.
- The performance of the model is proven on multiple, publicly accessible and well-known viewport prediction datasets.
limitations: limitations: - The stated problem regarding spatial alignment of movement trajectories with content information is not entirely clear to me. Additional explanation is required to this extent. 
- I don't fully understand the need for a novel saliency representation. How are other saliency methods represented? How does salxyz differs from these?
- It might be interesting to include static prediction (i.e. always take the last known orientation as the prediction) as a baseline, as it incorporates the most straightforward prediction possible. 
- The authors state that data is randomly divided in training, validation and test set at user level. Does this mean that trajectories from different users but a same, given video might appear in both training and validation/test set? How does this affect the results? How would the model perform on a previously unseen video?
paper_summary: paper_summary: This paper proposed a framework that combines saliency features with head movement history to predict long-term head movement.
strength_weakness: strength_weakness: The paper employs an attention structure to match two modalities: head movement history and saliency content. Attention is a well-studied building block with proven performance.
limitations: limitations: The paper used five different components to perform long-term viewport prediction. The structure is complex but lacks clarification. Specifically:
+Why did the authors consecutively stack encoders and decoders? Is there any empirical or mathematical evidence to prove the superiority of this approach?
+It is true that LSTM cannot handle long-term prediction. Then why incorporate LSTM into the structure as it would introduce biases and increase prediction errors? Isn't using the attention structure alone sufficient, as it can handle both long-term and short-term information?


The author states that the goal of the paper is to support 'real-time streaming of 360-degree video,' but this goal is difficult to achieve as the model is computationally expensive. It first uses SalMap Processor to extract saliency maps for every video frame, then uses multiple attention building blocks.
paper_summary: paper_summary: This paper proposes an approach to improve the accuracy of viewport prediction for 360-degree video streaming. The approach takes both historical viewpoint trajectories and video content as input, and predicts the viewpoint in the future 1-5 seconds. Existing works either perform well in the short term (< 1 s) or long term (2 - 5 s). The proposed method uses a transformer-like mechanism to conduct spatial and temporal fusions, achieving higher prediction accuracy in both the short and long term, compared to existing techniques.
strength_weakness: strength_weakness: The paper addresses an important problem in 360-degree video streaming, and provides a technically sound solution. It is well-known that both historical viewpoint trajectories and video content are useful for viewport prediction, but how to adaptively balance their contributions over time is a challenging issue. This paper presents a promising solution to this problem. The evaluation is comprehensive, conducted on three datasets and using two evaluation metrics. The visualizations and ablation study are clear and helpful.
limitations: limitations: While the proposed method outperforms existing techniques, especially in the longer term, the overall performance may not be significant enough to make it impactful in practice. For example, in the short term, the method's performance is similar to simpler approaches like linear regression. In the longer term (2-5 seconds), the IoU metric is around 0.3, which may still be insufficient for practical deployment.

The writing could be improved in a few places:
1) In Section 1, the second paragraph states that spatial alignment is hard, but it would be more logical to first explain why spatial alignment is necessary for the problem before discussing the challenges. The authors explain why in the third paragraph, which does not look right.
2) The dimensions of the Ss-out and Ps-out tensors are not explicitly mentioned in Section 3.5. Adding this information would help readers better understand the model architecture.