Lumos: Optimizing Live 360-degree Video Upstreaming via Spatial-Temporal Integrated Neural Enhancement

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 13:40:54

Abstract: As VR devices become increasingly prevalent, live 360-degree video has surged in popularity. However, current live 360-degree video systems heavily rely on uplink bandwidth to deliver high-quality live videos. Recent advancements in neural-enhanced streaming offer a promising solution to this limitation by leveraging server-side computation to conserve bandwidth. Nevertheless, these methods have primarily concentrated on neural enhancement within a single domain (either spatial or temporal), which may not adeptly adapt to diverse video scenarios and fluctuating bandwidth conditions. In this paper, we propose Lumos, a novel spatial-temporal integrated neural-enhanced live 360-degree video streaming system. To accommodate varied video scenarios, we devise a real-time Neural-enhanced Quality Prediction (NQP) model to predict the neural-enhanced quality for different video contents. To cope with varying bandwidth conditions, we design a Content-aware Bitrate Allocator, which dynamically allocates bitrates and selects an appropriate neural enhancement configuration based on the current bandwidth. Moreover, Lumos employs online learning to improve prediction performance and adjust resource utilization to optimize user quality of experience (QoE). Experimental results demonstrate that Lumos surpasses state-of-the-art neural-enhanced systems with an improvement of up to 0.022 in terms of SSIM, translating to an 8.2%-8.5% enhancement in QoE for live stream viewers.

Main paper: 
 introduction: 
 Live 360-degree video streaming has emerged as a significant portion of Internet traffic [41] , offering users an exceptionally immersive experience [9] . Viewers can adjust their viewing angles freely, enabling exploration of omni-directional video content during live events. Recent market reports [29] indicate a consistent growth in demand for live 360-degree video streaming. On the upstream side, upload clients typically employ a 360-degree camera equipped with multiple lenses to capture live video streams. These streams are subsequently sliced into tiles, encoded using various codecs [3, 5, 23, 31, 36] , and transmitted through protocols like HTTP, RTMP, as highlighted in [1, 25, 26] , to the media server.However, achieving high-quality live 360-degree video upstreaming encounters challenges due to constraints in uplink bandwidth [46, 47] and the computational capacity of the upload client. Limited uplink bandwidth necessitates the compression and transmission of video at lower quality, leading to poor video quality. Moreover, attaining high-quality live 360-degree video upstreaming mandates real-time 360-degree video encoding, posing challenges for mobile devices with limited computational capacity.Recent advancements in neural-enhanced video streaming have introduced a novel approach to enhance video quality by applying neural computation to video frames [15, 16, 21] . The fundamental concept is to leverage the computational power of the media server to offset the limitations of uplink bandwidth. Broadly, these approaches can be categorized into two primary categories: one employs super-resolution (SR) techniques for live video enhancement in the spatial domain [14, 35, 44] , while the other utilizes frameinterpolation (FI) techniques in the temporal domain [13, 24, 33] . However, downsampling in the spatial domain may compromise image details [19, 42] , especially in frames with intricate textures, whereas downsampling in the temporal domain may adversely affect motion information in high-action videos [6, 11] . Consequently, relying solely on neural enhancement in either the spatial or temporal domain has inherent drawbacks that may result in a subpar user experience.Nonetheless, the integration of spatial-temporal neural enhancement in live 360-degree video upstreaming introduces several new and non-trivial challenges. Firstly, the spatial-temporal combination necessitates the upload client to explore optimal neural enhancement configurations within an expanded solution space. However, due to the limited computational power of the upload client, rapidly searching for the optimal solution and performing neural enhancement is challenging. Secondly, video tiles in a live 360-degree stream exhibit different rate-distortions after encoding. Equally allocating bitrate among tiles would lead to inefficient bandwidth usage and degrade the quality of transmitted videos. Estimating the required bandwidth for each tile to optimize the rate-distortion poses a challenge. Thirdly, minimizing the degradation of user Quality of Experience (QoE) over time is challenging, as dynamic video scenes limit the effectiveness of historical data in improving system performance. Moreover, constrained bandwidth makes it difficult to refresh the dataset, intensifying the complexity of the problem.In this paper, we propose Lumos, a novel live 360-degree video streaming framework that integrates spatial and temporal domain neural enhancement techniques. By considering different ratios of downsampling in the spatial and temporal domains, Lumos generates a wide array of diverse neural enhancement configurations, each corresponding to varying levels of neural-enhanced quality. This enables Lumos to overcome the aforementioned limitations by exploring the optimal neural enhancement configuration in a twodimensional solution space, a capability not feasible with existing methods limited to a single dimension.To address the aforementioned challenges, Lumos incorporates several important innovative designs. Firstly, Lumos introduces a lightweight Neural-enhanced Quality Prediction (NQP) model trained using historical live 360-degree video data. This model takes the low-level spatial-temporal features of tile videos as input and generates upper and lower bounds of neural-enhanced quality predictions for various spatial-temporal neural enhancement configurations. This guidance aids in selecting optimal spatial-temporal configurations, reducing the computational burden on the upload client while enhancing the robustness of the configuration selection procedure. Secondly, based on our observation of a significant correlation between the spatial-temporal complexity of different tiles and their rate-distortions, Lumos adopts a dynamic bandwidth allocation strategy. This strategy assigns available bandwidth to each tile according to its specific spatial-temporal complexity. By doing so, Lumos ensures efficient bandwidth utilization, thereby facilitating the transmission of higher-quality live 360-degree videos. Finally, recognizing that even low-quality tiles can provide valuable labels for updating the dataset, Lumos utilizes the remaining bandwidth after transmitting 360-degree videos to send low-quality tiles for updating the NQP model. This proactive approach mitigates performance degradation and maintains user QoE.In summary, our paper makes the following contributions:• We conduct measurement experiments to explore the relationship between spatial-temporal complexity and tile ratedistortion, and find a strong correlation between them. Additionally, we observe that low-quality tiles can also effectively serve as training samples for updating neural models used in system components to maintain system performance. The remainder of this paper is organized as follows. In Section 2, we survey previous works related to our research. In Section 3, we conduct the measurement studies to understand the drawbacks of one-dimension neural enhancement and discover the key observation inspiring our design. In Section 4, we introduce the detailed design of Lumos. In Section 5, we demonstrate experiments results of Lumos compared with other baselines. Finally, we conclude our work in Section 6. 
related work 2.1 360-degree live video streaming: 
 Previous works on the optimisation of 360-degree video streaming systems have been approached from various perspectives. Yi et al. [45] conducted a detailed measurement study on the time consumption of a live 360-degree video streaming system and discovered the relationship between task time consumption and system latency. Feng et al. [10] developed a viewport prediction scheme for live 360-degree video streaming systems using motion tracking and user interest modelling. Ban et al. [2] proposed a multi-agent deep reinforcement learning based 360-degree video streaming system (MA360) to solve the multi-user streaming problem in edge caching networks. Sun et al. [32] proposed the use of "flocking" to improve viewport prediction and edge server caching for live 360-degree video streaming. Xie et al. [37] proposed a dynamic co-rendering solution for low-latency and high-quality mobile virtual reality. Eltobgy et al. [8] proposed a solution for live 360-degree video multicast to mobile users.Super-resolution Based Neural Enhanced Video Streaming System. Recent advances in deep learning, especially in superresolution, offer opportunities to reduce bandwidth consumption in video streaming applications. Dasari et al. [7] proposed a superresolution approach that significantly compresses the video on the server side and enhances it to higher quality on the client side using a neural network model. Kim et al. [14] introduced LiveNAS, a realtime video ingestion framework that enhances the quality of the original video stream by leveraging server-side computation. Live-NAS applies neural enhancement to the original video stream and uses online learning to maximize quality. Luo et al. [20] developed CrowdSR, a real-time video ingestion method that converts lowresolution video streams from weak devices into high-resolution streams through super-resolution. Wang et al. [35] proposed Vaser, a novel neural-enhanced 360-degree live video ingestion framework considering viewport information. While these works focus on the spatial domain, recovering original details and ensuring video quality for frames with complex textures can be challenging. Frame-interpolation Based Neural Enhanced Video Streaming System. Some existing works optimize video transmission by discarding packets or video frames. Yahia et al. [39] discard fixed bitrate video frames in live video streams using HTTP/2. Stewart et al. [30] modify the SCTP protocol to be partially reliable, allowing for discarding some packets without considering video frame characteristics. BETA [13] considers all B-frames as discardable, while VOXEL [24] modifies P-frames and B-frames for unreliable transmissions. Yang et al. [40] design an efficient end-to-end collaborative VR streaming system to increase the frame rate of VR video, but it is not easily applicable to real-time video streaming systems. Reparo [33] propose a new approach for real-time video streaming to improve QoE in low-rate networks, using a lightweight prediction model to determine whether to drop an even frame and recovering it through VFI at the media server. These works provide inspiration for compressing video in the time domain, but none of them handle video content with drastically changing scenes effectively. Inherent drawbacks of one-dimension neural enhancement. In a typical neural-enhanced live streaming video system, the video is usually downsampled either spatially or temporally to further compress its size, followed by neural enhancement using SR-DNN and FI-DNN. 
measurement study: 
 Figure 1a and Figure 1b illustrate the performance of different neural enhancement configurations on different types of tiles. We showcase the effects of neural enhancement using three different neural enhancement configurations: (a) temporal downsampling only (540P, 4fps), (b) spatial downsampling only (270P, 25fps), and (c) spatial-temporal downsampling (270P, 4fps). CRF is an FFmpeg encoding option used for bitrate control, where the larger the CRF, the lower the bitrate of the encoded video. As shown in Figure 1a , the DNN-based SR model falls short of effectively enhancing video tile with complex textures. Furthermore, simultaneous spatialtemporal downsampling yields comparable neural enhancement results with a smaller tile size. However, as shown in Figure 1b , temporal downsampling does harm to video quality in scenarios with strong dynamics, suggesting that neural enhancement in the temporal dimension is not applicable to these tiles.These results suggest that neural enhancement in either the spatial or temporal domain has its own drawbacks, and an integration of the two should be considered in practice to improve video quality. Correlation between rate-distortion and spatial-temporal complexity. In 360-degree video transmission systems, adaptive bandwidth allocation for each tile is essential. As illustrated in Figure 2a and 2b, the Tile 0 achieves remarkably high quality with less bandwidth requirement, while the Tile 12 requires a higher bandwidth allocation to achieve comparable quality, indicating the rate-distortion varies across different tiles. What's more, we observed a strong correlation between the tile's rate-distortion and its spatial-temporal complexity, as depicted in Figure 2a . To achieve satisfactory video quality, a tile with high spatial-temporal complexity requires a much higher bitrate than a tile with low  
framework design of lumos 4.1 system overview: 
 The core of Lumos is to utilize spatial-temporal integrated neural enhancement techniques to make full use of constrained upload bandwidth, to transmit higher-quality 360-degree videos. Figure 4 elucidates the architectural framework of Lumos, which consists of an upload client and a media server. Upload Client. At the upload client, Firstly, for each tile, the NQP model takes the low-level spatial-temporal feature as input and predicts the neural-enhanced quality of its corresponding neural enhancement configuration. Afterward, the available bitrates are dynamically allocated among tiles based on their spatial-temporal complexity, followed by selecting the appropriate neural enhancement configuration that satisfies the allocated bitrates. Then, the tile is encoded with the selected configuration and transmitted. Finally, leveraging the rest bandwidth after transmitting 360-degree videos, low-quality tiles, which are denoted as fragment sample in our design, are selected and sent to the media server for the NQP model updating. Media Server. At the media server, firstly, the received 360-degree tile videos are decoded, and applied the corresponding spatialtemporal neural enhancement techniques for recovery. The enhanced tiles are then stitched and transcoded into multiple versions waiting for viewer requests. Simultaneously, fragment training samples are decoded into tile frames. For each neural enhancement configuration, the tile frames are downsampled, encoded, and decoded using the highest and lowest bitrate settings. After that, the corresponding neural enhancement configuration is applied to enhance the tile frames to obtain the upper and lower bounds of enhanced quality, serving as labels for re-training the NQP model.Then, the NQP model is updated and sent back to the upload client periodically.The role of the Neural-enhanced Quality Predictor is to forecast the highest and lowest restore quality for each neural enhancement configuration.Extracting spatial-temporal features of tiles. As depicted in Figure 4 , firstly, the Neural-enhanced Quality Predictor extracts spatial-temporal features of each tile. Inspired by Reducto [17] , the Feature Extractor is designed to capture a range of low-level features to construct the spatial-temporal feature for each tile, such as pixel difference and area difference for the temporal domain and edge difference and variance of pixel values after applying the Laplacian operator [4] for the spatial domain. The spatial-temporal features of tiles are then utilized for predicting neural-enhanced quality, as well as for computing spatial-temporal complexity.Predicting neural-enhanced quality. Based on the aforementioned steps, the NQP model is trained to predict the neural-enhanced quality for its corresponding neural enhancement configuration. This prediction serves as a guidance for the selection of the neural enhancement configuration for each tile. At time chunk 𝑖, assume the spatial-temporal features of tile 𝑗 be denoted as 𝑓 𝑖 𝑗 and NQP 𝑘 denotes the NQP model of neural enhancement configuration 𝑘. For each configuration 𝑘, we predict the neural-enhanced quality for each tile, denoted as 𝑞 𝑘 𝑖 𝑗 . This procedure could be established as follows:EQUATIONTo ensure the effectiveness of our NQP model predictions, we propose two design strategies. First, for each neural enhancement configuration, rather than predicting the neural-enhanced quality under all CRF settings, we only forecast the upper and lower bounds of the enhanced quality. That is the neural-enhanced quality of tile videos which is encoded with corresponding neural enhancement configuration and the highest and lowest CRF. This approach not only reduces the number of prediction outputs to mitigate the risk of wrong selections but also affords greater flexibility in selecting spatial-temporal neural enhancement configurations, thereby enhancing the robustness of the proposed system. Furthermore, we employ lightweight Multilayer Perceptron (MLP) as the NQP model, which enables real-time inference under the constraint of limited computational resources.After predicting the neural-enhanced quality for each neural enhancement configuration, we proceed to allocate bitrates to each tile and select the appropriate configurations to downsample the tiles.Measuring spatial-temporal complexity. Firstly, it is crucial to measure the spatial-temporal complexity of each tile. We employ the spatial-temporal feature to calculate the spatial-temporal complexity of each tile, as we aggregate all the low-level features within the spatial-temporal feature of each tile to serve as its spatialtemporal complexity. Allocating bitrate among tiles. Then, bitrate allocation among tiles is conducted based on two considerations: first, tiles with lower spatial-temporal complexity should get lower bitrates as they can achieve satisfactory viewing quality with fewer bandwidth. Moreover, it is crucial not to allocate too much bandwidth to tiles with higher spatial-temporal complexity, as this may excessively degrade the quality of other tiles. Thus, a trade-off must be made between these two types of tiles to ensure full utilization of bandwidth and high-quality encoding of tiles.We design a weight-based heuristic bitrate allocation algorithm reflecting the proportion of total bitrates allocated to a specific tile 𝑗 in chunk 𝑖. Initially, each tile is assigned an initial weight 𝛼. Then, the weight of each tile is incremented by its corresponding spatialtemporal complexity 𝑧 𝑖 𝑗 , i.e., 𝑤 𝑖 𝑗 = 𝛼 + 𝑧 𝑖 𝑗 , where 𝑤 𝑖 𝑗 is the weight of tile 𝑗 in chunk 𝑖. After that, the weight function is normalized to ensure that the total weight across all tiles sums up to one. To prevent the bitrate allocation from excessively favoring certain tiles and hurting others' quality, the normalized weights of each tile are then truncated to predefined maximum and minimum thresholds. Finally, the truncated weights are renormalized to obtain the bitrate weight for each tile, which is then used to proportionally allocate bitrates to each tile according to the formula in Algorithm 1. Selecting spatial-temporal downsampling configurations. Given the predicted quality of each neural enhancement configuration and the allocated bitrates, we select the most suitable neural enhancement configuration for each tile. For each neural enhancement configuration, its upper and lower bounds of neural enhancement performance correspond to an encoding bitrate, respectively. For example, as shown in figure 5a , the upper and lower bounds of enhancement qualities for configuration y correspond to bitrates 60kbps and 40bps, respectively. Therefore, we follow these steps to select the neural enhancement configuration: (1) First, we sort the predicted enhancement qualities (including upper and lower bounds) of each configuration in descending order. (2) Second, we traverse through each configuration sequentially until we find one with a bitrate lower than the allocated bitrate for the tile (e.g. the configuration y in our examples). (3) If the allocated bitrate lies between the upper bound of configuration x and the upper (or lower) bound of configuration y (where y can be any configuration, including x), we select configuration x (e.g., the example in Figure 5a ); otherwise, if the allocated bitrate lies between the lower bound of configuration x and the upper (or lower) bound of configuration y (where y can be any configuration except x since the upper bound corresponds to a higher bitrate than the lower bound), we choose configuration y (e.g., the example in Figure 5b ).The above strategy is based on two considerations: (1) If the allocated bitrate falls between the upper bound of configuration x and the upper (or lower) bound of configuration y, it indicates that there exists a suitable CDF setting of configuration x that meets the requirements of the allocated bitrate, and selecting configuration x provides the opportunity for better neural-enhanced quality. (2) If the allocated bitrate falls between the lower bound of configuration x and the upper (or lower) bound of configuration y, it suggests that the allocated bitrate is not within the bitrate range of configuration x. In this case, we choose configuration y and select an appropriate CRF for tile encoding. Encoding tiles with selected neural enhancement configuration. For each tile, we downsampled it to the resolution and frame rate corresponding to its selected neural enhancement configuration. The remaining tile frames are then encoded using a CDF that meets the allocated bitrate and transmitted to the media server side.The purpose of the Fragment Sample Selector is to effectively utilize the remaining bandwidth for transmitting as many fragment samples as possible for the online training of the NQP model. Therefore, we base the selection procedure on the following principles:(1) Regionalization: Adjacent tiles in a 360-degree video exhibit similar characteristics. ( 2) Importance: Tiles should be sorted by spatial-temporal complexity. (3) Balance: Uniform sampling is necessary while considering Importance. Therefore, samples with lower spatial-temporal complexity should also be collected. We present the designed fragment sample selection procedure. Firstly, as illustrated in Figure 6 , the 360-degree video is divided into six regions, labeled 1 to 6. Subsequently, tiles within regions 1 to 4 are sorted in descending order of spatial-temporal complexity, while tiles in regions 5 to 6 are sorted in ascending order. It is worth noting that due to the wrapping-around property of 360-degree video ERP projection, there is no left or right distinction in the video. Finally, following the sequence of regions 1 to 6, tiles are selected from each region according to the sorted order and encoded in low quality to become a fragment sample. These samples are then transmitted to the server side utilizing the remaining bandwidth until it is depleted.The Neural-enhanced processor's role is to enhance the received downsampled 360-degree video tiles, restoring their original resolution and frame rate, and then stitching them together to form complete 360-degree video chunks. Enhancing received tiles with DNNs. For tiles with reduced spatial resolution, we use SR-DNN to recover the resolution, while for tiles with reduced frame rate only, we use FI-DNN to recover the frame rate. When both the resolution and frame rate of the tile are degraded, we first use SR-DNN to recover the resolution and then apply FI-DNN to recover the frame rate. With the above process, we achieve spatial-temporal integrated neural enhancement for all 360-degree video tiles.Subsequently, the neural-enhanced tiles are stitched together to form complete 360-degree video chunks and transcoded into versions of varying bitrates, awaiting download requests from viewers.As the similarity between historical and current live 360-degree videos diminishes over time, it becomes necessary to conduct online training for the NQP model, enabling it to adapt to upcoming video content.Extracting training dataset. Figure 4 illustrates the process of acquiring a dataset. For a fragment sample, we apply each neural enhancement configuration to it for spatial-temporal downsampling, and the downsampled tile frames are encoded using the maximum and minimum CRF settings. Then, the encoded tile is decoded and neural enhancement is applied accordingly to restore it to its original resolution and frame rate. Afterward, the restored tile is compared with the unprocessed tile to calculate the upper and lower bounds of the quality of the neural enhancement, which are used as training labels for this neural enhancement configuration. It is worth noting that the SSIM values computed on the fragment sample may be different from those computed on the original tile frame, but we are only concerned with the relative order of the upper and lower enhanced quality bounds of the different neural enhancement configurations, which the fragment sample can provide.The inputs corresponding to the extracted labels are the spatialtemporal features of the fragment sample tiles, which are merely floating-point numbers and directly transmitted from the upload client, occupying negligible bandwidth. Updating the NQP model. Employing the described approach, the media server continuously updates the training dataset and periodically retrains the NQP model using the extracted data. Subsequently, it forwards the updated model to the upload client, replacing the obsolete NQP model. 
evaluation: 
 In this section, we evaluate Lumos with a series of experiments and in-depth analysis to answer the following questions:(1) How much is the video quality enhancement of the proposed Lumos compared with other state-of-the-art systems that only consider a single dimension? (2) Does Lumos offer improved Quality of Experience (QoE) for end users? (3) Does the implementation of the Content-aware Bitrate Allocator and Fragment Sample Selector genuinely lead to performance improvement?Evaluation Setup. The Lumos framework is implemented using Python, and a testbed is established for the data generation and evaluation process. Our pipeline operates on Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz processors, complemented by NVIDIA GeForce RTX 3090 Super GPUs. For DNN training, both SR-DNN and FI-DNN are pre-trained on 360-degree videos from Gaze360 [38] that are not used for testing. We utilize the EDSR [18] model for spatial-domain neural enhancement and the RIFE [12] model for temporal-domain neural enhancement. These models are not updated online due to their acceptable generalization capabilities.In Evaluation Videos. We conducted experiments using nine 360degree videos from Gaze360 [38] and four 360-degree videos from SalientVR [34] . Finally, the buffer length is set to 10s for the experiments. Network Traces. To better evaluate the performance, we use real 4G network traces [28] , containing 135 traces. In our experiments, we filter out the traces whose average bandwidth is higher than 10Mbps to model a bandwidth-constrained environment just like previous works [14, 33] . As a result, the network bandwidth ranged from 4 to 10 Mbps. QoE Calculation. To assess the Quality of Experience (QoE), we utilize the linear QoE model introduced by Pensieve [22] :EQUATIONwhere 𝜇 is set to 4.3 like Pensieve, and the increase of SSIM is computed using its effective bitrate function, generated through linear interpolation similar to NAS [43] .Baselines. To demonstrate that Lumos can improve video quality and enhance users' QoE, we conduct comparisons between Lumos and two state-of-the-art neural-enhanced baseline methods:• LiveNAS-360 [14] , which primarily emphasizes spatial neural enhancement. In this approach, the upload client adjusts the resolution of tiles based on the available uplink bandwidth, and the media server restores the resolution of these tiles using SR-DNN, which is the same as the SR-DNN used by Lumos. It is worth noting that LiveNAS performs online training on SR-DNNs while we do not, which is orthogonal to our work. • Reparo-360 [33] , which emphasizes temporal neural enhancement. In this approach, the upload client utilizes the VFD model's prediction to discard even frames of tiles, and the media server restores the frame rate of these tiles using FI-DNN, which is the same as the FI-DNN used by Lumos. The VFD model is periodically updated on the server side and sent back to the client.Video Quality Improvement. Figures 7a and 7b present the average SSIM values of all video chunks from Gaze360 and SalientVR generated by Lumos and other baseline methods at four different bitrate levels. We have two main observations: Firstly, compared to LiveNAS-360 and Reparo-360, Lumos surpasses them in both datasets across all bitrate levels, with SSIM gains up to 0.012 and 0.005 on Gaze360 and up to 0.022 and 0.013 on SalientVR. Secondly, in our experiments, Reparo-360 performs poorly under low bandwidth conditions, while LiveNAS-360 exhibits subpar performance when bandwidth is relatively abundant. This suggests that relying solely on single-domain neural enhancement is not able to adequately adapt to a wide range of bandwidth variations. In contrast, our approach considers the combination of both temporal and spatial domains, thereby possessing greater bandwidth adaptability and outperforming both baselines at all bitrate levels. Figure 7c and Figure 7d show the SSIM CDF curves for all video blocks on Gaze360 and SalientVR. We see that the majority of the Lumos curves lie below the other curves, which demonstrates the superiority of spatial-temporal integrated neural enhancement approach over those that only consider either the temporal or spatial domain. To further elucidate the effectiveness of Lumos, the chunklevel ssim gain compared to LiveNAS-360 and Reparo-360 on 13 types videos from the two datasets is shown in Figure 8 , in which C1-C9 refers to 9 video types in Gaze360 and D1-D4 refers to 4 video types in SalientVR. Overall, Lumos has SSIM gain on the vast majority of video types, illustrating the robustness of our system. Quality of Experience. To assess the QoE, we conducted tests on four types of videos from SalientVR using real network traces. Figure 9a illustrates the comparison of QoE between Lumos and two other baselines. The findings demonstrate that Lumos surpasses the other baselines, achieving the highest QoE, with an improvement of 8.2% on LiveNAS-360 and 8.5% on Reparo-360. To better understand QoE gains obtained by Lumos, we analyze its performance on each term for the QoE objective (see Eq. 2), which includes bitrate utility, smoothness penalty, and rebuffer penalty. For a clearer visualization, we multiply the rebuffer penalty by a factor of 20 while maintaining the original values for other components. As depicted in Figure 9a , Lumos exhibits the highest bitrate utility, with an increase of 19.1% on LiveNAS-360 and 11.1% on Reparo-360. Additionally, considering that Reparo-360 can discard no more than half of the frames which limits its adaptability to dynamic bandwidth, its rebuffer penalty is 2.67 times higher than that of Lumos. Since Lumos explores a significantly larger spatial-temporal neural enhancement configuration space, it suffers a little more smoothness penalty. Figure 9b presents the CDF curves of QoE for all video chunks of our four 360-degree videos. Overall, the majority curve of Lumos which corresponds to higher QoE value is below the other two baselines, suggesting that the spatial-temporal integrated neural enhancement paradigm is effective in improving QoE. Besides, there are small portions of the curve corresponding to lower QoE values above the other two baselines. This aligns with the smoothness penalty results shown in Figure 9a and will be addressed as part of our future work. Ablation Study. To validate the effectiveness of the CBA and online training of the NQP model, we conducted an ablation study on two bitrate levels using four types of videos from SalientVR. Each video is sampled for 1 minute. As demonstrated in Table 1 , online training of the NQP model leads to improvements in video quality at both bitrate levels, indicating the effectiveness of employing fragment samples for online training. Furthermore, substituting the CBA with the uniform distribution of bitrates hurts video quality. This underscores the necessity of dynamically allocating bitrates based on the spatial-temporal complexity of tiles, further emphasizing the effectiveness of the CBA. Case study of Content-aware Bitrate Allocator. To further investigate the effectiveness of the CBA, we present the CDF curves of pertile SSIM with and without CBA in EQUATION 
conclusion: 
 In this paper, we introduce Lumos, a novel live 360-degree video streaming system equipped with spatial and temporal integrated neural enhancement techniques to improve live 360-degree video quality under limited upload bandwidth. Lumos applies the realtime NQP model to forecast the enhanced quality for each neural enhancement configurations, dealing with the computational power bottleneck of upload client. The Content-aware Bitrate Allocator assigns bitrates to each tile and selects the appropriate neural enhancement configuration to improve overall 360-degree video quality. Besides, fragment samples are utilized to improve the prediction performance, preventing system performance from decreasing over time. Compared to the baselines using either spatial or temporal neural enhancement, Lumos achieves up to 0.022 SSIM gain and produces significant (8.2%-8.5%) QoE improvement for live viewers.  
 
 Appendix:  


Decision: The paper integrates spatial and temporal neural enhancement techniques into 360-degree video upstreaming, addressing the limitations of systems that focus on a single domain. It introduces a lightweight, real-time NQP model that predicts enhanced quality efficiently and a CBA model that optimizes video quality without overburdening the upload client. Extensive evaluations using real-world network traces and diverse video datasets demonstrate significant improvements in SSIM and QoE over baseline methods. Some limitations remain, but several concerns have been addressed in the rebuttal. Regarding the quality of responses in the rebuttal and reviewers' feedback, I recommend accepting this paper as a poster.

Reviews: review: paper_summary: paper_summary: This paper presents a spatio-temporal neural-enhanced upstreaming framework for live 360-degree video. 
Meanwhile, a neural-enhanced quality prediction model is developed to predict the neural-enhanced quality for different video content. However, the rationality of the system design needs further verification. The results are based on simulation evaluation.
strength_weakness: strength_weakness: In this work, the relationship between spatio-temporal complexity and tile rate distortion is explored and analyzed in detail. It also considers both spatial and temporal quality enhancement of neural networks. Furthermore, it is motivated by measurement experiments.
limitations: limitations: 1) A single SR model is often difficult to adapt to various video contents. Therefore, in live streaming, it is often necessary to train SR models in real time. Proposed Neural-enhanced Quality Prediction (NQP) may be difficult to adapt to this situation.
2) Although the time cost of NQP per tile is 3ms, the bitrate allocation algorithm requires at least two inferences about each tile in each chunk, which is still very time consuming.
3) NQP only forecast the upper and lower bounds of the enhanced quality. It is not clear how to select the appropriate quality when there are many parameter configurations.
4) The bitrate allocation scheme based on the spatio-temporal complexity of the tiles may not be reasonable. For example, it may result in inconsistent quality between tiles in the same chunk?
5) The work controls the video quality by adjusting the CRF parameter, but the encoding bitrate of the tile can only be known after the it has been encoded, how can the bitrate assignment of tiles be done before encoding?
6) There are too many heuristics settings, such as weights pruning for bitrate assignment, considerdion of the highest and lowest CRF in NQP, and 10s buffer size. They may not work well for varying video content and real-world testing. Evaluations under simulation conditions have also shown limited user QoE gains, let alone in real-world systems.
paper_summary: paper_summary: This paper presents Lumos, a novel live streaming framework for 360-degree videos that integrates neural enhancement techniques in both the spatial and temporal domains. Lumos dynamically allocates bitrates using a Neural-enhanced Quality Prediction (NQP) model and a Content-aware Bitrate Allocator (CBA) to enhance video quality under constrained network bandwidth. Extensive experiments demonstrate that Lumos significantly improves the SSIM and QoE compared to the selected neural-enhanced baseline methods.
strength_weakness: strength_weakness: The main strength lies in the integration of spatial and temporal neural enhancement techniques into ive 360-degree video upstreaming, addressing the limitations of existing systems that focus primarily on only one domain.

The introduction of a lightweight, real-time NQP model enables the system to quickly predict the enhanced quality for various configurations, integrated with CBA that optimizing video quality efficiently without overburdening the upload client.

The authors provide an extensive evaluation using real-world network traces and diverse video datasets, demonstrating substantial improvements in SSIM and QoE over baseline methods.
limitations: limitations: The motivation behind the NQP model and online training is not well-explained. In measurement study section around figure 3, it is not clear how the relative orders of the upper and lower bounds of configurations relate to the feasibility of using low-quality training samples. Besides, it is unclear what advantages it offers compared to existing no-reference video quality estimation methods (e.g. VMAF or zero-inference method used in NeuralScaler). A comparative analysis in the experiments would be helpful.

The paper should better explain and exemplify (e.g., by visual samples) how the accuracy of the CBA and NQP affects output quality. Figure 10b shows that online training can improve prediction accuracy of the NQP but Table 1 shows that online update have a minor effects on quality, I'm not sure whether the effects to the visual experience really significant.

Although the system is designed for 360-degree video, it does not utilize much of the unique characteristics of 360-degree content. The paper should address why Lumos might outperform other advanced neural-enhanced live video methods, such as NeuralScaler, specifically for 360-degree video. Including this in the motivation and experimental comparison would strengthen the paper.

typo: In the "measurement study", there is a reference to "the lower bound (CRF33) of the configuration 540p, 12fps" in Figure 3, but Figure 3 does not contain a configuration for 540p, 12fps, CRF33. This inconsistency needs correction.
paper_summary: paper_summary: This paper introduced Lumos, a live 360-degree video streaming system equipped
with both spatial and temporal integrated neural enhancement. There is a
real-time neural-enhanced quality prediction model to forecast the enhanced
video quality, and a content-aware bitrate allocator to assign bitrates at the
tile level. Besides, fragment samples are utilized to improve the prediction performance.
strength_weakness: strength_weakness: * This paper is well written with very clear structure.

* The measurement study proved the drawback of one-dimension neural enhancement,
  which serves as a strong motivation for integrating both spatial and temporal
  enhencement.
limitations: limitations: * The QoE is not good. First, the QoE of Pensieve is proposed for 2-D video, not
  for 360-degree video. Second, the QoE metric does not reflect the latency,
  which is important in live streaming. 

* SSIM is mainly used for images rather than video. Hence, other metrics like
  PSNR, VMAF should be considered. 

* The evaluation results are not sufficient. As a live video solution, we
  are not only interested in the overall QoE improvement. Instead, the detailed
  improvement for every QoE component should be clearly illustrated. Moreover,
  it would be better to show the QoE improvement compared with spatial-only and
  temporal-only neural-enhancement.
paper_summary: paper_summary: The live-360 degree streaming has gained its popularity and has inbuilt faces the challenge to achieve right tradeoff between the QoE and uplink bandwidth. Neural enhancement with edge is best known solution that can save bandwidth in uplink and help client to avoid computational tasks, this method known to be used in single domain either spatial or temporal. The authors claim to introduce the combination of both for live 360-degree streaming. They use the prediction model to predict the neural enhanced quality for different video context  and design a context-aware bitrate allocator. The performance gain is significant with ~9% QoE improvement.
strength_weakness: strength_weakness: 1. Live 360-degree streaming is challenging.
2. Using the combination of both spatial and temporal information to reduce bitrate and use the same for assigning higher bitrate is simple and effective.
3. Gain is good compared to SOTA.
limitations: limitations: 1. I do not see any limitations