ResVR: Joint Rescaling and Viewport Rendering of Omnidirectional Images

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-02 18:50:37

Abstract: With the advent of virtual reality technology, omnidirectional image (ODI) rescaling techniques are increasingly embraced for reducing transmitted and stored file sizes while preserving high image quality. Despite this progress, current ODI rescaling methods predominantly focus on enhancing the quality of images in equirectangular projection (ERP) format, which overlooks the fact that the content viewed on head mounted displays (HMDs) is actually a rendered viewport instead of an ERP image. In this work, we emphasize that focusing solely on ERP quality results in inferior viewport visual experiences for users. Thus, we propose ResVR, which is the first comprehensive framework for the joint Rescaling and Viewport Rendering of ODIs. ResVR allows obtaining LR ERP images for transmission while rendering high-quality viewports for users to watch on HMDs. In our ResVR, a novel discrete pixel sampling strategy is developed to tackle the complex mapping between the viewport and ERP, enabling end-to-end training of ResVR pipeline. Furthermore, a spherical pixel shape representation technique is innovatively derived from spherical differentiation to significantly improve the visual quality of rendered viewports. Extensive experiments demonstrate that our ResVR outperforms existing methods in viewport rendering tasks across different fields of view, resolutions, and view directions while keeping a low transmission overhead.

Main paper: 
 introduction: 
 With the growing interest in virtual reality and augmented reality, omnidirectional images (ODIs), also referred to as 360°or panoramic images, attract great attention within the computer vision community for their immersive and interactive capabilities. Although ODIs can capture scenes across a comprehensive 360°×180°views, headmounted displays (HMDs) often present a limited field-of-view (FoV), necessitating resolutions as high as 4K×8K [2] to preserve details in a small viewport. High-resolution (HR) ODIs are typically stored on cloud servers by platforms of virtual reality media, requiring real-time download by users. This can degrade the visual experience of users, particularly under poor internet conditions.Image rescaling [12, 26, [46] [47] [48] 50] emerges as an effective method to reduce image file size for storage and transmission while preserving high quality in the reconstructed images at the user end. This technique first downscales HR images to low-resolution (LR) ones that keep the most important visual details, then upscales them back to their original HR versions. It not only minimizes the file size of LR images but also maintains the quality of reconstructed HR images [35] , thus becoming a straightforward approach for efficient transmission and storage for ODIs from cloud servers to user HMDs. However, the prevalent storage and transmission format for ODIs, i.e., the equirectangular projection (ERP), has directed state-of-theart ODI rescaling works [16] to focus on enhancing the quality of ERP images. As depicted in Fig. 1 (a), after receiving an LR ODI, the typical process on HMDs is two-step, involving (1) upscaling HR ODIs in ERP format from LR images ( ÎLR-ERP ↦ → ÎHR-ERP ), and (2) projecting them onto the viewport ( ÎHR-ERP ↦ → ÎHR-View ) using traditional interpolation methods such as bilinear. This pipeline does not fully account for the ultimate rendered image of the HMD viewport ÎHR-View , particularly lacking optimization for the final viewing experience. As Fig. 1 (c ) shows, the quality of images seen by the user on HMDs can be significantly lower than anticipated.In this paper, we point out that (1) the content viewed on HMDs is actually a rendered viewport, not an ERP image, and (2) focusing solely on the quality of ERP images will result in sub-optimal viewport visual experiences. To improve the viewing experience for users, there is a need to develop a comprehensive solution that is optimized for end-to-end ODI processing from the storage of ERP images to the display of the final viewport. To this end, we propose ResVR, a novel framework for joint Rescaling and Viewport Rendering of ODIs, marking an innovative step towards comprehensive end-to-end ODI processing. As shown in Fig. 1 (b) , ResVR aligns the optimization of network parameters with our primary goal of improving the quality of the final viewport. By utilizing such a new methodology, the viewports rendered through our ResVR framework exhibit enhanced details and fewer artifacts compared to those produced by conventional pipelines, as shown in Fig. 1 (c ).In our proposed ResVR, HR ODIs are firstly embedded into LR images to facilitate efficient transmission. Then, HR viewports are directly rendered from LR ERP images on HMDs. This process ( ÎLR-ERP ↦ → ÎHR-View ) does not need to produce HR ERP images. To deal with the irregular correspondence between the ERP area and the viewport, which hinders the joint optimization of downscaling and viewport rendering using traditional image-level loss training methods, we develop a discrete pixel sampling strategy. In each training iteration, this strategy is used to randomly sample paired sets of ground truth pixels and the reconstructed ones on viewports, thus making the end-to-end learning of our entire ResVR pipeline feasible in implementation. Furthermore, to enhance our ResVR's awareness of the positions of different viewport pixels on the spherical surface, we introduce a technique for spherical pixel shape representation. This technique employs spherical differentiation to calculate the geometric orientation and curvature of various viewport areas, offering positional information that contains more precise spherical attributes than existing 2D image representation methods [22] . This advanced representation effectively improves the quality of the final viewport, especially in regions of high latitude and longitude. Extensive experiments on various panoramic image datasets show that ResVR outperforms existing methods in multiple viewport rendering tasks across different FoVs, view directions, and resolutions. In summary, our contributions are:❑ We propose ResVR, a novel framework for the comprehensive processing of omnidirectional images, seamlessly integrates image Rescaling with Viewport Rendering. Our ResVR effectively balances the transmission efficiency and users' visual experience.❑ We develop a discrete pixel sampling strategy to tackle the complex correspondence between viewport and equirectangular projection (ERP) areas within our framework. This strategy makes the end-to-end training of our whole processing pipeline feasible.❑ We introduce a spherical pixel shape representation technique based on spherical differentiation to guide viewport rendering, which significantly enhances the visual quality of the final viewport.❑ Extensive empirical evaluations on various panoramic image datasets exhibit that ResVR consistently achieves new state-of-theart visual quality while maintaining a low transmission bitrate. 
related work 2.1 omnidirectional image super-resolution: 
 Image super-resolution (SR) seeks to construct HR images from LR ones. Since the advent of deep neural networks (DNNs) in SR-CNN [15] , subsequent studies [10, 13, 25, 27, 28, 42, 43, 60, 61] have significantly advanced SR performance beyond traditional methods. In the specific context of omnidirectional image superresolution (ODISR), DNN-based approaches have been tailored to account for the unique latitude-based characteristics of ODIs [1, 5-7, 14, 29, 31, 33, 39, 57, 58] . For example, LAU-Net [14] divides the entire ERP image into latitude-based patches for separate upscaling. SphereSR [57] introduces the spherical local implicit image function (SLIIF) alongside a novel feature extraction module to leverage information from arbitrary projection types. OSRT [58] employs a distortion-aware transformer targeting dimension-related distortions in ERP images. OPDN [39] introduces a dual-stage framework incorporating a position-aware deformable network. Despite these noteworthy advancements, the majority of these methodologies presuppose a fixed downscaling approach (e.g. bicubic [30] ) and overlook high-frequency components from HR inputs, thus limiting the quality of reconstructed details in SR ODIs.Different from SR, image rescaling focuses on downscaling HR images to create visually pleasing LR images that retain essential information for accurate HR reconstruction. Recently, invertible neural network (INN) [4, 9, 32] becomes a representative framework for image rescaling [12, 26, 34, 38, 46, 47, 51, 54] , offering a direct route to inversely map the downscaled images back to HR. For instance, IRN [46, 47] is the first attempt to model image downscaling and upscaling using invertible transmissions. Liang et al. [26] formulate high-frequency components in INNs as a conditional distribution on the LR image. HyberThumbnail [35] employs an asymmetric encoder-decoder architecture for real-time reconstruction of 6K images and also optimizes the JPEG compression process [41, 44, 49] . Very recently, DINN [16] makes the first attempt to apply image rescaling to ODI and highlights the significance of leveraging ERP's latitude characteristics by developing a latitudeaware conditional mechanism. However, existing ODI rescaling methods focus solely on improving the quality of ERP images. In contrast, our ResVR innovatively optimizes the quality of the final viewport, offering new improvements orthogonal to previous ODI rescaling methodologies.ODIs are designed to encapsulate a full spherical view, enabling an immersive viewing experience. However, when viewed through HMDs towards a particular direction, only a specific viewport is displayed [18, 19] . Achieving high resolution and quality in these viewports is crucial for immersive experiences, as highlighted by various ODI visual quality assessment techniques [52, 53, 55, 56] . As a widely adopted method for viewport rendering [8] , perspective projection employs a series of pixel mapping and resampling operations to effectively implement image warping. To reduce the interpolation-induced blurriness and artifacts, SRWarp [37] reinterprets image warping as an SR problem and introduces a differentiable warping module. LTEW [22] uses a continuous neural representation [11, 20, 23] for image warping by taking advantage of both Fourier features and spatially-varying Jacobian matrices. LeRF [24] assigns spatially varying steerable resampling functions to pixels, learning their orientations for continuous function prediction. Different from the above methods that focus on warping, our ResVR considers comprehensive ODI processing, serving as a new framework while enjoying high viewpoint rendering quality. 
methodology: 
 In this section, we begin with a concise review of the viewport rendering process (Sec. 3.1). Following this, we provide an overview of our ResVR framework (Sec. 3.2), which is illustrated in Fig. 2 . We then elaborate on the proposed discrete pixel sampling strategy (Sec. 3.3) and spherical pixel shape representation technique (Sec. 3.4). The training objectives are detailed in Sec. 3.5.ODIs inherently provide a full spherical view. However, when viewed through HMDs directed towards a specific direction, only the corresponding viewport is displayed. This viewport appears as a 2D image, derived through perspective projection [21] from a segment of the spherical image. To formalize this process, we consider the view direction in spherical coordinates (𝜃 𝑐 , 𝜙 𝑐 ), along with the horizontal and vertical fields of view (𝐹 ℎ , 𝐹 𝑣 ), and the height and width of the viewport (ℎ 𝑣 , 𝑤 𝑣 ). An invertible coordinate mapping 𝑓 : X ↦ → Y is established, where X := {x|x ∈ R 2 } denotes the coordinate space of ERP, and Y := {y|y ∈ R 2 } represents the coordinate space of the viewport to be rendered. In practice, the coordinates Y 𝑣𝑖𝑒𝑤 on the viewport are initially determined by (ℎ 𝑣 , 𝑤 𝑣 ), and then the corresponding coordinates on ERP are obtained through backward mapping X 𝑣𝑖𝑒𝑤 = 𝑓 -1 (Y 𝑣𝑖𝑒𝑤 ). The rendering of the viewport is achieved through resampling techniques, such as interpolation, ensuring that this process remains fully differentiable. Additional mathematical details of perspective projection and viewport rendering are elaborated in the supplementary material.An overview of proposed joint Rescaling and Viewport Rendering (ResVR) of ODIs is presented in Fig. 2 . The ODI processing of ResVR contains two steps: (1) ODI Downscaling & Compression: An HR ERP image is firstly downsampled and compressed to an LR ERP JPEG image for efficient transmission from cloud server to user HMDs, and (2) Decompression & Viewport Rendering: The LR ERP image is then decompressed and rendered to HR viewports on HMDs through our viewport rendering (VR) module.ODI Downscaling & Compression: Given an HR ERP image I HR-ERP ∈ R 3×𝐻 ×𝑊 , its LR representation is firstly generated through our downsampler, where 𝑠 is the rescaling factor. The downsampler is a U-Net [36] with dense blocks [17] . To further decrease the size of the transmitted image file, a JPEG encoder is employed for the LR representation to obtain a JPEG image. Concretely, we follow [35] to predict adaptive quantization tables for each image. Finally, the LR JPEG image is obtained, and adaptive quantization tables and quantized DCT coefficients are also encoded into the JPEG file. More details about the downsampler, the adaptive quantization table prediction module, and the training process of learned compression are provided in the supplementary material.Decompression & Viewport Rendering: After receiving theLR JPEG image, ÎLR-ERP ∈ R 3× 𝐻 𝑠 × 𝑊𝑠 is firstly reconstructed by the JPEG decoder through inverse discrete cosine transformation (IDCT). Then our goal is to render high-resolution viewport ÎHR-View ∈ R 3×ℎ 𝑣 ×𝑤 𝑣 directly from ÎLR-ERP . Inspired by recent implicit neural representation methods [11, 22, 23] , we develop a viewport rendering (VR) module to predict pixel values of the query coordinates Output:!! !" !# !$Coordinates: X 𝑣𝑖𝑒𝑤 from the latent space of input ÎLR-ERP instead of using traditional interpolation methods. As depicted in Fig. 3 , our VR module consists of an encoder E, a local texture estimator ℎ 𝜓 = {ℎ 𝑎 , ℎ 𝑓 , ℎ 𝑝 }, and an MLP decoder D. ℎ 𝜓 is a learnable dominant-frequency estimator, which is capable of characterizing image textures in 2D Fourier space [23] . Here, ℎ 𝑎 is an amplitude estimator (R 𝐶 ↦ → R 256 ), ℎ 𝑓 is a frequency estimator (R 𝐶 ↦ → R 2×128 ), and ℎ 𝑝 is a phase estimator (R 10 ↦ → R 128 ). Concretely, for a query point x ∈ X 𝑣𝑖𝑒𝑤 , the estimating function ℎ 𝜓 is defined as:EQUATIONwhere z = E ( ÎLR-ERP ). Denote 𝑗 as a pixel index of the ÎLR-ERP , x 𝑗 and z 𝑗 are the corresponding coordinates and latent variable of ÎLR-ERP , respectively. 𝛿 x is a local grid calculated by 𝛿 x = xx 𝑗 . < •, • > is an inner product, and ⊗ denotes element-wise multiplication. To be noted, s(x) is the spherical pixel shape representation of the query coordinate x, which is important for providing spatialvarying priors [22] for our VR module and will be elaborated on in Sec. EQUATIONwhere F 𝐵 is a bilinear interpolation operator to stabilize the convergence of network training and aid the VR module in learning high-frequency details. J is a neighborhood set of x, defined asJ = { 𝑗 | 𝑗 = x + [ 𝑚𝑠 𝑊 , 𝑛𝑠 𝐻 ],𝑚, 𝑛 ∈ {-1, 1}}, while 𝜔 𝑗 is a local ensemble coefficient. More details of our VR module are provided in the supplementary material. Different from existing ODI rescaling methods which first produce ÎHR-ERP and then obtain HR viewports through traditional interpolation methods, our ResVR directly renders the final HR viewports from ÎLR-ERP through our VR module.Motivation. In image rescaling and SR tasks, the ground truth HR images I HR-ERP are typically cropped into patches I HR-ERP-Patch to reduce GPU memory overhead while increasing the diversity of training data. The goals of our ResVR's end-to-end training process include: (1) optimizing the downscaling process (I HR-ERP-Patch ↦ → ÎLR-ERP-Patch ), and (2) optimizing the rendering process ( ÎLR-ERP-Patch ↦ → ÎHR-View ). In this framework, the former process requires the corresponding I LR-ERP-Patch as supervision, while the latter process needs the corresponding I HR-View as supervision. However, due to the natural geometric properties of ODIs' different projection types, there exists a challenge of shape mismatch between a viewport and its corresponding ERP area. As depicted in Situations #1 and #2 in Fig. 4 , one can not obtain I LR-ERP-Patch and I HR-View that are both with rectangular image shape at the same time. As a result, conventional image-level loss can not be directly used for end-toend training of ResVR. To address this, we innovatively propose a discrete pixel sampling strategy (DPS), as shown in "Ours" of Fig. 4 .Algorithm 1 Discrete pixel sampling strategy I HR-ERP , 𝑠: HR ERP image and the downscaling factor (𝑎, 𝑏), 𝑝: left top coordinate and the size of cropped patch (𝜃 𝑐 , 𝜙 𝑐 ), (𝐹 ℎ , 𝐹 𝑣 ): view direction and FoVs of the viewport (ℎ 𝑣 , 𝑤 𝑣 ), Y 𝑣𝑖𝑒𝑤 : shape and the coordinate space of the viewport function DisSamp(𝑎, 𝑏, 𝑝, 𝜃 𝑐 , 𝜙 𝑐 , 𝐹 ℎ , 𝐹 𝑣 , ℎ 𝑣 , 𝑤 𝑣 , Y 𝑣𝑖𝑒𝑤 , I HR-ERP ) I HR-ERP-Patch ← CropPatch(I HR-ERP , 𝑎, 𝑏, 𝑝) 𝑓 ← GetTransform(𝜃 𝑐 , 𝜙 𝑐 , 𝐹 ℎ , 𝐹 𝑣 , ℎ 𝑣 , 𝑤 𝑣 )X 𝑣𝑖𝑒𝑤 ← 𝑓 -1 (Y 𝑣𝑖𝑒𝑤 ) ⊲ Inverse mapping from viewport to ERP X ′ 𝑣𝑖𝑒𝑤 ← FilterWithBounds(X 𝑣𝑖𝑒𝑤 , 𝑎, 𝑏, 𝑝) ⊲ Eq. (3) if |X ′ 𝑣𝑖𝑒𝑤 | > 𝑁 then X ′ 𝑣𝑖𝑒𝑤 ← RandomSample(X ′ 𝑣𝑖𝑒𝑤 , 𝑁 ) end if X 𝑠𝑎𝑚𝑝 ← CoordSpaceTransform(X ′𝑣𝑖𝑒𝑤 , 𝑎, 𝑏, 𝑝) ⊲ Eq. ( 4) 𝑆 𝑝𝑖𝑥 ← BicubicSample(I HR-ERP-Patch , X 𝑠𝑎𝑚𝑝 ) I LR-ERP-Patch ← BicubicDownscale(I HR-ERP-Patch , s) return X 𝑠𝑎𝑚𝑝 , 𝑆 𝑝𝑖𝑥 , I LR-ERP-Patch end function Method. Denoting 𝑝 as both the height and width of training patches, our core idea is to keep the LR ERP patch with a rectangular image shape (I LR-ERP-Patch ∈ R 3× 𝑝 𝑠 × 𝑝 𝑠 ), while discretely sampling 𝑁 pixels 𝑆 𝑝𝑖𝑥 ∈ R 3×𝑁 with their coordinates X 𝑠𝑎𝑚𝑝 ∈ R 2×𝑁 in the irregular area. Thanks to the continuous representation ability of implicit neural representation methods [11] , our VR module is able to predict corresponding pixel values Ŝ𝑝𝑖𝑥 given coordinates X 𝑠𝑎𝑚𝑝 . Hence, we use the sampled 𝑆 𝑝𝑖𝑥 as supervision for the predicted Ŝ𝑝𝑖𝑥 . An overview of our strategy is presented in Algo. 1.Specifically, given the left top coordinate (𝑎, 𝑏) of the training patch, the view direction (𝜃 𝑐 , 𝜙 𝑐 ), FoVs (𝐹 ℎ and 𝐹 𝑣 ), shapes (ℎ 𝑣 , 𝑤 𝑣 ), and query coordinates (Y 𝑣𝑖𝑒𝑤 ) of the desired viewport, we first determine the coordinate mapping 𝑓 . Then the corresponding coordinates X 𝑣𝑖𝑒𝑤 in the ERP space can be obtained through inverse coordinate mapping as X 𝑣𝑖𝑒𝑤 = 𝑓 -1 (Y 𝑣𝑖𝑒𝑤 ). By setting appropriate view direction and FoVs, it is possible to ensure that some elements in X 𝑣𝑖𝑒𝑤 lie in the area of the cropped ERP patch. Recall that 𝑝 is the size of the cropped training patch, X 𝑣𝑖𝑒𝑤 is then filtered to preserve the subset overlapped with the patch as follows:EQUATIONwhere x 1 and x 2 represent the coordinates of a point x in the ERP space. This filter operation ensures that every point in X ′ 𝑣𝑖𝑒𝑤 can find its corresponding latent variable extracted from ÎLR-ERP-Patch . To balance the number of pixels sampled from different FoVs and view directions, if the filtered X ′ 𝑣𝑖𝑒𝑤 contains more than 𝑁 pixel coordinates, only 𝑁 elements will be randomly retained. Finally, to uniform the coordinate correspondence between LR and HR patches, X ′ 𝑣𝑖𝑒𝑤 ⊆ [0, 𝐻 ) × [0,𝑊 ) is converted from the whole HR ERP coordinate space to the patch coordinate space, and normalized to X 𝑠𝑎𝑚𝑝 ⊆ [-1, 1) × [-1, 1), as:X 𝑠𝑎𝑚𝑝 = {𝑇 (x)|x ∈ X ′ 𝑣𝑖𝑒𝑤 }, where 𝑇 (x) = 2 (((x 1 , x 2 ) -(𝑎, 𝑏)) /𝑝) -1. (4)As a result, 𝑆 𝑝𝑖𝑥 are sampled through bicubic interpolation as 𝑆 𝑝𝑖𝑥 = F 𝑏𝑖𝑐𝑢𝑏𝑖𝑐 (I HR-ERP-Patch , X 𝑠𝑎𝑚𝑝 ). By adopting Algo. 1, I LR-ERP-Patch ∈ R 3× 𝑝 𝑠 × 𝑝 𝑠 can be used as supervision of the predicted ÎLR-ERP-Patch , and 𝑆 𝑝𝑖𝑥 ∈ R 3×𝑁 can be used as supervision of the predicted Ŝ𝑝𝑖𝑥 .Motivation. Pixel shape representations described by grid orientation and curvature provide informative geometric spatial-varying priors for image warping tasks [22] . Existing shape representation methods are designed for 2D images. However, different from 2D images, ERP is an unfolding of the spherical surface along meridians, which leads to the fact that the adjacent pixels on the sphere can be far apart in the ERP image. As a result, the previous 2D shape representation methods fall in the high-latitude/-longitude areas due to the nature properties of (1) latitude-related distortion and (2) wraparound consistency of ERP images (Fig. 7 in Sec. 4.3). Therefore, we develop a spherical pixel shape representation (SSR) technique to solve this challenge. As Fig. 5 shows, SSR leverages the information of transformed coordinates on the sphere as a more effective shape representation to guide the viewport rendering process.Method. In image warping tasks, the first-order partial derivatives (i.e. Jacobian matrix) and the second-order partial derivatives (i.e. Hessian matrix), describe the orientation and curvature of pixels resulting from the transformation, respectively [22] . These two matrices provide informative geometric spatial-varying priors to guide the process of viewport rendering. Inspired by this, for a point x on the original ERP plane, we represent its pixel shape s(x) with the gradient of the inverse coordinate transformation 𝑓 -1 . Specifically, we start with the point y = 𝑓 (x) on the viewport plane, and the corresponding Jacobian matrix J 𝑓 -1 (y) and Hessian matrix H 𝑓 -1 (y) are analytically computed as:EQUATIONwhere x = 𝑓 -1 (y) = (x 1 , x 2 ) represents the coordinates in the original ERP plane. We propose a spherical pixel shape representation (SSR) technique to numerically estimate J 𝑓 -1 (y) and H 𝑓 -1 (y) based on spherical differentiation. Specifically, as depicted in Fig. 5 , the inverse coordinate transformation 𝑓 -1 is firstly applied to y and its eight nearest points(y + [ 𝑚 𝑤 𝑣 , 𝑛 ℎ 𝑣 ] with 𝑚, 𝑛 ∈ {-1, 0, 1})to get x and its neighborhood on the ERP plane. Different from existing 2D shape representation methods [22] which estimate the shape of x directly on the ERP image plane, we further transform x and its neighborhood to the sphere denoted by the set {p 𝑖 |p 𝑖 = (𝜃 𝑖 , 𝜙 𝑖 ), 𝑖 = 1, 2, . . . , 9}, and innovatively calculate numerical derivatives to estimate J 𝑓 -1 (y) and H 𝑓 -1 (y) by the proposed spherical central difference method, overriding Eq. ( 5) as: 6) and (7) .J 𝑓 -1 (where 𝐷 (p 𝑖 , p 𝑗 ) calculates the distance between p 𝑖 and p 𝑗 in the spherical coordinate system as:EQUATIONwhere | • | represents the absolute value operation. The min(•) operator ensures the computation of minor arc distances, and the average latitude is utilized to adjust the longitudinal distance to account for the convergence of meridians. We follow [22] to use six elements in H 𝑓 -1 (y), and finally s(x) ∈ R 10 is obtained by concatenating and flattening J 𝑓 -1 (y) and H 𝑓 -1 (y). As a result, s(x) serves as an auxiliary input, together with the LR latent representation z = E ( ÎLR-ERP ) to predict final viewports through the VR module.Thanks to the discrete pixel sampling strategy (Sec. 3.3) and the fully differentiable pipeline (Fig. 2 ), the processes of downscaling, compression, and viewport rendering can be jointly optimized endto-end, which aligns the learning of network parameters with our goals of obtaining high-quality viewport while reducing the transmission overhead. The total loss is a weighted sum of a pixel-level reconstruction loss, an LR guidance loss, and a bitrate loss as:EQUATIONwhere 𝜆 1 and 𝜆 2 are two trade-off parameters.Pixel-level reconstruction and guidance loss. In the training phase, we employ Algo. 1 to get 𝑆 𝑝𝑖𝑥 as the ground truth and use the VR module to predict the corresponding values Ŝ𝑝𝑖𝑥 given 𝑋 𝑠𝑎𝑚𝑝 .EQUATIONwhere 𝑁 is the number of sampled pixels. Additionally, following [35, 46, 47] , an 𝐿 2 guidance loss on the LR ERP patch is defined as:EQUATION)Bitrate loss. To optimize the size of the transmitted JPEG image, we firstly follow [3] to estimate the rate 𝑅 of the quantized coefficients 𝐶 with differentiable fully factorized entropy models as:𝑅 = E 𝑥∼𝑝 𝑥 [-log 𝑝 𝐿 ( 𝐶 𝑦 ) -log 𝑝 𝐶 ( 𝐶 𝐶𝑏 ) -log 𝑝 𝐶 ( 𝐶 𝐶𝑟 )],where 𝑝 𝐿 and 𝑝 𝐶 are two fully factorized entropy models for luma and chroma coefficient maps, respectively. Then the bpp of the transmitted LR ERP JPEG image is calculated as:EQUATION 
experiments 4.1 experimental setup: 
 Implementation details. We follow the previous work [35] to put more computation into the downsampler to keep the VR module lightweight. The VR module is composed of a lightweight feature extractor and a tiny MLP. Please refer to our supplementary materials for more details of our network architecture.Training details. The downscaling factor 𝑠 is fixed to 4, and patch size 𝑝 of random cropped I HR-ERP-Patch is set to 256. To ensure the overlapping of I HR-ERP-Patch and discretely sampled points in Algo. 1, we calculate the viewport center (𝜃 𝑐 , 𝜙 𝑐 ) according to the center of cropped patch. To enable ResVR to handle different resolutions and FoVs, we randomly sample FoVs and resolutions during training. Specifically, the FoVs (𝐹 ℎ , 𝐹 𝑣 ) are randomly sampled from {80 • , 90 • , 100 • , 110 • , 120 • } and the resolutions of viewport (ℎ 𝑣 , 𝑤 𝑣 ) are randomly sampled from {512, 576, 640, 768, 832, 960, 1024}. The number of sampled pixels 𝑁 is set to 25600, and the batch size is set to 16. All experiments are conducted on one V100 GPU. The network is trained for 5 × 10 5 iterations with learning rate 2 × 10 -4 . 𝜆 1 and 𝜆 2 are set to 0.6 and 0.01 for all experiments, respectively.Datasets and evaluation metrics. ODI-SR dataset [14] and SUN360 Panorama dataset [45] are used in our experiment. We follow the data split setting in [14] and train on the ODI-SR training set. For evaluation of transmission efficiency, we follow [35] to use the real file size of JPEG for evaluating the bitrate: bpp = E 𝑥∼𝑝 𝑥 [filesize/(𝐻 ×𝑊 )]. For evaluation of the quality of rendered viewports, we choose ten different view directions, get the ground truth image using bicubic interpolation, and calculate the average PSNR, SSIM, and LPIPS [59] . More details are provided in the supplementary materials. For those competing methods that need to explicitly produce HR ERP images to render final viewports, we evaluate the WS-PSNR [40] of their predicted ÎHR-ERP .We compare three categories of methods: (1) a baseline method which downscales with Bicubic interpolation, compresses with standard JPEG codec, and renders viewports with Bicubic interpolation; (2) SR pipeline which downscales with Bicubic, compresses with standard JPEG codec, upscales with state-of-the-art methods [58] , and uses bicubic interpolation for viewport rendering; (3) Rescaling pipeline which firstly uses state-of-the-art asymmetric rescaling methods [35] and renders viewports with Bicubic interpolation. For a fair comparison, we retrain SR methods [58] and rescaling methods [35] with our training sets and constrain their bpp around 0.3 by adjusting the quality factor of JPEG compression in all baselines.Tab. 1 presents the quantitative comparisons of the quality of rendered viewports with (𝐹 ℎ , 𝐹 𝑣 ) = (120 • , 90 • ) and (𝑤 𝑣 , ℎ 𝑣 ) = (2048, 1536) among different methods. Taking advantage of directly optimizing the final viewport through end-to-end training, ResVR outperforms previous methods by reconstruction accuracy (about 0.4dB gain on PSNR) and with better realness (about 0.02 gain on LPIPS). Notably, even though existing rescaling methods can achieve PSNR of about 28dB, they only obtain sub-optimal results due to the lack of awareness and optimization of the viewport rendering process, which demonstrates the fact that focusing solely on the quality of ERP images results in sub-optimal visual experience of viewports. ResVR directly optimizes the rendered viewports without the need to predict HR ERP and achieves SOTA performance. Fig. 6 further provides the visual comparison of rendered viewports.It can be seen that our ResVR reconstructs enhanced details with fewer artifacts (e.g. the text "labatt's" in the top image and the lines in the bottom image) compared to other methods. Additionally, ResVR shows better wraparound continuity at the location of the seams (e.g., pipe in the zoomed-in area) in the bottom case. We attribute this to the awareness of arbitrary view directions during training and our MLP's continuous representation of images.In this section, we analyze the effect of the proposed discrete pixel sampling strategy and spherical pixel shape representation. We conduct experiments on three ResVR variants: (1) Case #1: We train the rescaling process and VR module separately with 2D shape representation; (2) Case #2: We train the whole pipeline in an endto-end manner by employing discrete pixel sampling strategy with 2D shape representation; (3) The complete ResVR with our sphere pixel shape representation. Quantitative results are shown in Tab. 2.Effect of discrete pixel sampling strategy. Comparison #1 vs. #2 exhibits the effectiveness of the proposed discrete pixel sampling strategy which enables end-to-end training of the whole pipeline, thus providing substantial PSNR improvement of 4.12-4.44dB. To further analyze the sampled coordinates and pixels, we visualize the sampled areas and pixels in the supplementary materials.Effect of spherical pixel shape representation. Comparison #2 vs. ResVR reveals that the spherical shape representation improves PSNR by 0.40-0.55dB. We observe that conventional 2D shape representation methods [22] fall in high-longitude and highlatitude areas, as depicted in Fig. 7 . We attribute this to the following reasons: adjacent pixels on the viewport should be very close to each other on the sphere. However, (1) for high-latitude areas, due to the natural distortion of ERP, the distance between corresponding points is very close to the original sphere but is very far on the ERP image. (2) For high-longitude areas, due to the wraparound consistency of the left and right ends of ERP, the corresponding points are originally adjacent pixels on the sphere but are at both ends of the ERP image. The traditional shape representation method based on 2D ERP ignores the spherical characteristics, so in the above situations, it provides wrong pixel shape priors for the VR module, thus leading to abnormal visual results. The proposed SSR is performed on the native spherical surface, thus ensuring stable and high-quality rendering quality in various view directions.LR JPEG image. The file size and visual quality of LR JPEG images are also important for efficient transmission and user preview.ResVR effectively balances transmission efficiency with the final viewer experience through our end-to-end training. Visualizations of LR images are provided in the supplementary materials. Arbitrary-resolution viewport rendering. Due to the continuous representation ability [11, 22, 23] of the MLP in our VR module, ResVR is able to render viewports of different resolutions using only one set of parameters. Tab. 3 compares ResVR with SOTA methods on different resolution settings. It can be seen that ResVR achieves better performance at different resolutions. 
conclusion: 
 The rise of virtual reality and augmented reality applications has popularized ODI rescaling to shrink the file size of images while maintaining their quality. However, existing methods focus on improving the ERP image quality, ignoring that HMDs use rendered viewports for display, not ERP images. This focus on ERP quality alone will lead to compromised user experiences. To address this, in this paper, we propose ResVR, a novel framework for joint rescaling and viewport rendering of ODIs. In ResVR, we develop a discrete pixel sampling strategy to tackle the irregular correspondence between the ERP area and the viewport, enabling end-to-ending training of the whole pipeline. A spherical pixel shape representation technique is introduced to serve as an effective guidance for the rendering process, further enhancing the visual quality of viewports. Experiments demonstrate that our ResVR achieves state-of-the-art performance across different settings of FoVs, view directions, and resolutions while keeping a low transmission overhead.Limitations and future work. Existing ODI datasets are of relatively low resolution and with compression artifacts, which limits the further performance improvements of ResVR. Besides, although ResVR provides a promising pipeline for comprehensive ODI processing and achieves SOTA performance, it is crucial to extend ResVR to 360 • video tasks. We leave this for future work.  
abstract: 
 With the advent of virtual reality technology, omnidirectional image (ODI) rescaling techniques are increasingly embraced for reducing transmitted and stored file sizes while preserving high image quality. Despite this progress, current ODI rescaling methods predominantly focus on enhancing the quality of images in equirectangular projection (ERP) format, which overlooks the fact that the content viewed on head mounted displays (HMDs) is actually a rendered viewport instead of an ERP image. In this work, we emphasize that focusing solely on ERP quality results in inferior viewport visual experiences for users. Thus, we propose ResVR, which is the first comprehensive framework for the joint Rescaling and Viewport Rendering of ODIs. ResVR allows obtaining LR ERP images for transmission while rendering high-quality viewports for users to watch on HMDs. In our ResVR, a novel discrete pixel sampling strategy is developed to tackle the complex mapping between the viewport and ERP, enabling end-to-end training of ResVR pipeline. Furthermore, a spherical pixel shape representation technique is innovatively derived from spherical differentiation to significantly improve the visual quality of rendered viewports. Extensive experiments demonstrate that our ResVR outperforms existing methods in viewport rendering tasks across different fields of view, resolutions, and view directions while keeping a low transmission overhead 1 . 
 
 Appendix:  


Decision: The paper is clearly written. 
It provides significant original contributions to the field, as appreciated by all four reviewers.
The less strong critical points / limitations have been addressed in the rebuttal, where possible in revision.

The positive and few critical points mentioned by the reviewers (summarized / partly cited from original reviews) are:

(wHUF)
-  well-written
-   solid contributions
-   technically sound
-   sufficient analysis

"...Moreover, the proposed method seems technically sound, while the performance evaluation and ablation studies are adequate."

(F9kL)
- technically sound, 
- providing a clear description of the methods and algorithms used. 
- spherical pixel shapes derived from spherical differentiation to handle pixel distortions effectively is a significant technical contribution that improves the rendering quality of viewports; 
- diagrams and comparisons used effectively to illustrate improvements ResVR brings and to explain the operational mechanisms of the framework
- paper clearly outlines implications of improved ODI processing for VR and AR applications.

(note: negative points have been addressed in rebuttal with additions to paper)

(JV9Q)
- authors achieved better performance for ODI SR task through 1) a discrete pixel sampling strategy (DPS) and 2) representation considering spherical coordinates.
- coordinate-grid based SR method, LTE is well-known but this paper effectively introduced it to the ERP rescaling task.
- DPS considering geometric properties demonstrates potential to be widely applied across ERP image task. Specifically, DPS was designed for effective learning within ERP datasets, taking viewport rendering into account. 
- method has shown significant performance improvements in experimental results. 
- Seems valuable to introduce this model to the low-level vision task for ODI. 
- paper also demonstrates that SSR can effectively remove the artifacts from ERP images.

(kv6y)
- rescaling and viewport rendering in single pipeline ensures optimization is directly aligned with final visual output on head-mounted displays (HMDs)
- pixel sampling addresses the irregular correspondence issues inherent in spherical images, enabling more accurate and high-quality reconstructions.
- spherical pixel shape representation technique gives better geometric orientation and curvature information, particularly improving visual quality in regions with high latitude and longitude variations, which are typically problematic in conventional methods
- proposed method achieves state-of-the-art performance in multiple viewport rendering tasks. Validation across different fields of view, resolutions, and view directions highlights robustness and versatility of proposed framework.

As a result, I propose to accept this paper as oral presentation. It could even be a candidate for best paper, with the final score of 5.75 after rebuttal.
In case of final acceptance of the paper, authors are strongly encouraged to make sure that all points mentioned in the rebuttal and further reviewer comments are considered for the camera-ready version of the paper.

Reviews: review: paper_summary: paper_summary: In this paper, a learning-based framework for joint rescaling and viewport rendering of omnidirectional images is proposed. The paper assumes as a conventional pipeline that omnidirectional images are downscaled (LR version) and upscaled (HR version) in the ERP domain, followed by viewport extraction to render an HR view. The proposed approach instead relies on a learning-based module that extracts the viewport directly from the downscaled omnidirectional image (LR) to render an HR view. The main contributions of the paper include the discrete pixel sampling strategy and the spherical pixel shape representation technique.
strength_weakness: strength_weakness: + well-written
+ solid contributions
+ technically sound
+ sufficient analysis

I find the paper quite good. The manuscript is well-written and well-structured, with clear motivations and contributions. Moreover, the proposed method seems technically sound, while the performance evaluation and ablation studies are adequate.
limitations: limitations: - missing complexity aspects

The main aspect I am missing is related to the complexity of the proposed approach during the inference stage. Specifically, there is no information on whether the proposed method can be used for real-time consumption of an omnidirectional image e.g., a user inspects the omnidirectional image through a head-mounted display and can choose the viewport in real time through head movements - is this feasible using the proposed method? What is the maximum/minimum update frame rate, assuming different viewports between successive frames? How does this compare to the conventional pipeline?

Cosmetic changes:
- "LR" is not defined as an abbreviation in the Abstract
- In section 4.2, it is written:
"(3) Rescaling pipeline which firstly uses state-of-the-art asymmetric rescaling methods [35] and renders viewports with Bicubic interpolation."
However, it seems that Nearest and Bilinear methods are additionally used in Table 1. Please clarify.
- It is not clear in Table 1 whether the average performance indexes are reported across all contents of a dataset, or something different. Moreover, it would be good to mention the number of contents/stimuli included in each selected dataset and potentially other characteristics.
paper_summary: paper_summary: This paper presents ResVR, a framework designed to optimize omnidirectional image processing for head-mounted displays by focusing on joint rescaling and viewport rendering. ResVR targets the actual viewports seen by users, improving both transmission efficiency and viewport image quality. It introduces a novel discrete pixel sampling strategy and a spherical pixel shape representation technique, enhancing the quality of rendered viewports.
strength_weakness: strength_weakness: The paper is technically sound, providing a clear description of the methods and algorithms used. The introduction of spherical pixel shapes derived from spherical differentiation to handle pixel distortions effectively is a significant technical contribution that improves the rendering quality of viewports; 
Diagrams and comparisons are used effectively to illustrate the improvements ResVR brings and to explain the operational mechanisms of the framework; 
The paper clearly outlines the implications of improved ODI processing for VR and AR applications.
limitations: limitations: Generally I think it's a very good paper.
Since the success of VR/AR applications heavily depends on user acceptance and satisfaction, the paper might not sufficiently explore the challenges of deploying the framework in actual virtual or augmented reality systems, such as integration issues with existing systems, user acceptance, and quality of experience in practical use.
What is the details in the downscaling stage? Maybe you did not consider the distortion in "polar" areas. Please check OSRT(https://arxiv.org/pdf/2302.03453) again for its Deep Feature Extraction which consider distortion in downsampling stage too besides only as a SR method in your paper. Thank you.
paper_summary: paper_summary: This paper proposes a rescaling with viewport rendering SR model for omnidirectional images (ODI). These rescaling models can significantly reduce the burden of processing by compressing low-resolution images with a conventional JPEG codec [1]. Overall, this paper clearly describes the proposed method and the performance improvement is also substantial. In addition, the comparison of experimental results is solid, and additional visualization materials also explain the proposed techniques well.
strength_weakness: strength_weakness: In this paper, the authors achieved better performance for ODI SR task through 1) a discrete pixel sampling strategy (DPS) and 2) representation considering spherical coordinates. 

A coordinate-grid based SR method, LTE is well-known but this paper effectively introduced it to the ERP rescaling task.

The DPS considering geometric properties demonstrates the potential to be widely applied across the ERP image task. Specifically, DPS was designed for effective learning within ERP datasets, taking viewport rendering into account. This method has shown significant performance improvements in experimental results. It seems valuable to introduce this model to the low-level vision task for ODI. The paper also demonstrates that SSR can effectively remove the artifacts from ERP images.
limitations: limitations: This paper did not compare with SOTA ERP SR model, DINN360 [2]. It seems that it was impossible to reproduce.

It would be good to describe the overall runtime of the proposed model. The comparison model, HyperThumbnail [1], explicitly states the runtime benefits through the JPEG codec. I think this paper is also expected to be similar.

[1] Qi, Chenyang, et al. "Real-time 6K Image Rescaling with Rate-distortion Optimization." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[2] Guo, Yichen, et al. "Dinn360: Deformable invertible neural network for latitude-aware 360deg image rescaling." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
paper_summary: paper_summary: The paper addresses the challenges related to bandwidth requirement for an omnidirectional contents (ODI images). Since many applications downscale the images during transmission to save bandwidth and then upscale it at user's end, the quality of content gets degraded.
The proposed paper presents a method, which instead of upscaling the whole ODI, only focuses on the viewport rendering part, means the planar patch of image which is currently being viewed by the user. The pipeline works by first down sampling and compression of the ODI image (in ERP format). This image is then decompressed, and then up sampled for the target viewport by using a learning based up sampling module.
In the comparison the method achieves significant performance in terms of visual quality when compared to the other mentioned methods.
strength_weakness: strength_weakness: **1**. The rescaling and viewport rendering in single pipeline ensures optimization is directly aligned with the final visual output on head-mounted displays (HMDs).

**2**. The pixel sampling addresses the irregular correspondence issues inherent in spherical images, enabling more accurate and high-quality reconstructions. 

**3**. Spherical pixel shape representation technique gives better geometric orientation and curvature information, particularly improving visual quality in regions with high latitude and longitude variations, which are typically problematic in conventional methods 

**4**. The proposed method achieves state-of-the-art performance in multiple viewport rendering tasks. This validation across different fields of view, resolutions, and view directions highlights the robustness and versatility of the proposed framework.
limitations: limitations: **1.** The paper focuses on the visual quality of the rendered FoV, however given the final application viewing a 360 image, where the user might move the head randomly at different latitudes and longitudes. The runtime of model becomes significant. The inference time should have been mentioned in the table 1,2 and 3. 

**2.** The paper does not mention about the scalability of the ResVR framework in real-time applications. Given the high computational demands, there may be issues related to latency and performance, especially under varying network conditions and hardware capabilities that are typical in practical VR scenarios.

**3.** For comparison the paper uses **[58]Yu, F., Wang, X., Cao, M., Li, G., Shan, Y. and Dong, C., 2023. Osrt: Omnidirectional image super-resolution with distortion-aware transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 13283-13292)**. However considering the planar nature of final viewport image, addition of a planar image SR method will have added more weights to the results **Liang, J., Cao, J., Sun, G., Zhang, K., Van Gool, L. and Timofte, R., 2021. Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 1833-1844).**

 

**Minor Notes:**
In section 4.1 (experimental setup), the line 676, it mentions about the number of iterations the model was trained for. Since these iterations will change according to data points and batch size. It will be better to mention about number of epochs (even though authors have mentioned about these 2 values)