Reviewerly/exHarmony huggingface dataset:
Optimal Tile Size and Streaming Field of View for VR Streaming

Virtual reality (VR) video services require a high bitrate, and hence, viewport-adaptive streaming techniques like motion-constrained-tile-set (MCTS) have been found important to reduce streaming-rate and storage demands. The tiling scheme and streaming field of view (FOV) are among the key elements in designing an optimal VR viewport-adaptive streaming solution, in terms of rate-distortion (R-D) performance. The aim of this study is to propose an optimal configuration for the tile grid and streaming FOV, considering different VR viewing situations such as head motion speed, system delay, and head-mounted display FOV. To achieve this, a wide range of tiling schemes and streaming FOVs are examined to study the storage and streaming R-D performance of the MCTS-based technique in both viewport and non-viewport areas using a quality metric called Zonal-cubic PSNR. The findings demonstrate that for VR applications focused on preserving high viewport quality, fine tile grids lead to higher performance. In scenarios featuring small and large HMD FOV, the optimal configuration involves a small and medium streaming FOV, respectively.
The Subjective Quality of Stereoscopic 3D Video Following Display Stream Compression

Stereoscopic displays present similar but different images/videos separately to the left and right eyes. Here we report the results of a subjective assessment of computer-generated high dynamic range (HDR) stereoscopic video compression targeting lightweight, low-latency visually lossless performance. The ISO/IEC 29170–2 non-flicker paradigm (Annex A) was adapted for stereoscopic videos and used to evaluate the VESA DSC 1.2a and VDC-M 1.2.2 codecs operating at and beyond their recommended maximum compression levels. Both codecs exhibited visually lossless performance at their target compressions levels. There were differences in compression artifact detection rates with some image sequences between stereoscopic and 2D presentation although the direction of the difference was inconsistent. The results confirm the effectiveness of these codecs for Stereoscopic 3D (S3D) videos and suggest that observers are relatively insensitive to compression artifacts in moving stereoscopic imagery.
HiStyle: Reinventing Historic Portrait for 3D Colorization and Zero-Shot Stylization

Restoring and reinventing historical portraits has long been a challenging task in computer vision. In order to faithfully explore the portrait images, it is necessary to restore the color and reconstruct the 3D geometry. Furthermore, the ability to stylize historic portraits is crucial for extending their use to different forms of media. Existing methods for each specific task make huge progress. However, they struggle with conflicts of multi-tasks, which hinders their ability to meet the requirement in a unified model. To achieve this goal, we propose HiStyle, a generative model for generally reinventing historic portraits, which simultaneously realizes 2D-to-3D reconstruction, gray-to-RGB restoration, and photo-to-style image translation. We introduce a GAN inversion technique to transfer a gray historical portrait into the latent space of a 3D generator, restoring the lost color information and meanwhile lifting the 2D image to 3D representation. Besides, we incorporate the power of CLIP model with 3D-aware GANs to achieve zero-shot text-driven style transfer. The results demonstrate the superior of HiStyle in the quality and diversity of synthesized images and also highlight the potential of 3D-aware GANs for preserving cultural heritage.
Transform Domain Temporal Prediction for Dynamic Point Cloud Compression

With the advent of immersive multimedia technologies, interest in dynamic point cloud compression standards, such as video-based point cloud compression (V-PCC), has been steadily growing. At the core of V-PCC, conventional video codecs such as High Efficiency Video Coding are employed for encoding the geometry video, attribute video and the occupancy map. Standard video codecs perform temporal prediction using motion compensation by simple pixel-copying from the reference frame. However, conventional pixel-domain prediction does not exploit spatial correlation within the block. This is highly sub-optimal especially due to large homogeneous regions in the geometry video. In this paper, spatial decorrelation is first achieved via the Discrete Cosine Transform. Prediction is then performed in the transform domain, where the observed variation in temporal correlation across frequencies is exploited for a more accurate prediction. Design of the correlation filters required for this procedure poses challenges. The inherent instability in closed-loop design is tackled using the Asymptotic Closed Loop paradigm which uses an iterative open-loop algorithm that asymptotically converges to closed-loop operation. Experimental results on geometry videos generated from common test conditions show significant bitrate savings of up to 5.8% (about 3% on average) and highlight the efficacy of our approach.
Edge-Assisted Joint Rate Adaptation and Quality Enhancement for 360-Degree Video Streaming

360-degree video offers users an immersive viewing experience by enabling them to look around during playback. However, this type of video consumes a significant amount of data, necessitating high bandwidth and minimal streaming delays. Super-Resolution (SR) is an emerging technique that can enhance video quality while reducing data transmission requirements. It achieves this by sending low-quality video and utilizing an SR model on the client side to enhance its quality. Nonetheless, SR is computationally demanding and typically requires high-performance hardware, making it impractical for many weak clients, particularly mobile devices. To overcome this limitation, we propose EdgeSR, an edge-assisted approach that combines rate adaptation and quality enhancement for 360-degree video streaming. In EdgeSR, the client can request low-quality video chunks from the cloud and perform SR at the edge to enhance video quality while minimizing transmission delays from the cloud to the edge. Simultaneously, the edge node can prefetch a video chunk from the cloud while the previous one is being transmitted from the edge to the client, thereby reducing delays. Experimental results under various settings demonstrate that EdgeSR outperforms the most competitive baseline methods, improving the overall Quality of Experience (QoE) by 27%.
Cloud Game Video Coding Based On Human Eye Fixation Point

Cloud Gaming enables users to run high-quality games on thin clients with limited graphics processing and data computing capabilities. Under the running mode of cloud games, all games are run on the server side, and the rendered game picture is compressed and sent to the user through the network. On the client side, the user's gaming device doesn't need any high-end processors or graphics cards, only basic video extraction capabilities. However, cloud gaming requires a high bandwidth connection to present a good-quality game picture to the user. At present, the main problem is limited bandwidth during transmission, which leads to poor quality of the game image received by users and poor user experience, which has become an important problem hindering the popularity of cloud games. To solve this problem, we observed that when playing a game, the user's eyes are not always focused on the entire picture, and due to the characteristics of visual perception, the user pays more attention to the area around the eye fixation point. In this paper, for the first time, we add information about user interactions with devices to the network to more accurately predict user fixation points. Using visual perception features, more bit rates are assigned to ROI regions near the user's fixation point. Our results show that our network is able to predict more accurate user fixation points, and that our approach can significantly improve the subjective quality of the area near the user's focus of the game frame at the same bitrate, with the VMAF scores is 2 to 3 points higher on average compared to H.264, the most commonly used standard encoder in cloud games today.
M-DVC: Moderate Deep Video Compression Embracing the Metaverse New Era

In the Metaverse new era, video compression methods are required to provide low latency, low complexity, and high coding performance. In the low-latency scenarios, utilizing multiple reference frames in learned video compression methods has shown their capabilities of improving coding performance. However, these sophisticated methods generally yield to high complexity, mismatching the real-world Metaverse applications. In this paper, we propose a moderate deep video compression (M-DVC) method to balance the coding performance and complexity. Specifically, we first calculate the motion vector (MV) between the current frame and the previous frame to capture the motion information. Then, we introduce a multi-reference frame motion compensation (MFMC) module to exploit the high correlation among multiple frames and obtain more accurate prediction frames, thereby enhancing coding performance. In our M-DVC, we strive to utilize lightweight network structures to achieve a balance between coding performance improvement and reduced time complexity. Experimental results demonstrate that our M-DVC achieves a favorable trade-off between time complexity and coding performance. Specifically, on the HEVC Class E dataset, our M-DVC achieves a bitrate saving of 43.8% and 55.2% compared to H.264 when evaluated based on PSNR and MS-SSIM, respectively. In contrast, DVC achieves a bitrate saving of only 34.9% and 28.5% compared to H.264 based on PSNR and MS-SSIM evaluations, respectively. Additionally, our M-DVC introduces a minimal increase of only 5.9% in terms of time complexity compared to DVC.
Image-Feature Parallel Compression for Indoor Surveillance Video

Intelligent applications such as metaverse built on IoT technology are usually inseparable from the collection of surveillance video data. Yet, how to effectively compress indoor surveillance video has always been a huge challenge for data transmission. In this paper, we propose an image-feature parallel compression framework for indoor surveillance video. In other words, the video is divided into images and features which are compressed separately. Specifically, according to the Mean-shift algorithm, the background image is generated, encoded and shared among frames. Based on a series of detection and matching algorithms, a human feature (HF) extractor is designed to extract a frontage body image, identity features, boundary features and structural features which are used for the generation of foreground body images. At the same time, body masks are generated in order to segment body areas from body images. Finally, with the guidance of body mask and boundary features, the body area covers the corresponding area of generated background image to reconstruct the video, and its quality is improved by a quality enhancement network. Experimental results show that the proposed compression scheme achieves coding gain up to 66.91% and 51.84% averagely compared with HEVC and VVC respectively.
SPNet: An RGB-D Sequence Progressive Network for Road Semantic Segmentation

Road semantic segmentation is an essential component of autonomous driving and blind navigation. Although many excellent RGB-based road semantic segmentation algorithms have been proposed, these methods may not detect correctly due to the lack of geometric information. Recently, RGB-D road semantic segmentation methods attract more research attention. However, the existing RGB-D methods ignore the impact of unknown noise in sensors. To solve this problem, we propose an RGB-D Sequence Progressive Network (SPNet) for road semantic segmentation. Specifically, we first propose a sequence-based RGB-D feature extractor to alleviate the effect of noise. Then, We propose a multi-modal feature fusion (MMFF) module to enhance the feature representation of multi-modal data by further alleviating the effect of noise. Finally, we propose a semantic flow prediction (SFP) module that aims to align the multi-modal features in the decoder. Extensive experiments are conducted on several challenging datasets, including KITTI and GMRP. Our method achieves an F-score of 97.21% on the KITTI official leaderboard and ranked third in the official leaderboard.
Unveiling the Limitations of Novel Image Quality Metrics

Subjective image quality measurement plays a crucial role in the advancement of image-processing technologies. The primary goal of a visual quality metric is to reflect subjective evaluation results. Despite the rapid development of these metrics, their potential limitations have not been sufficiently explored. This research paper aims to address this gap by demonstrating how image preprocessing before compression can artificially inflate the quality scores of widely used metrics such as DISTS, LPIPS, HaarPSI, VIF, STLPIPS, ADISTS, MR-Perceptual, AHIQ, IQT, and CONTRIQUE. We present several CNN-based preprocessing models that significantly increase these metrics when images are JPEG-compressed. However, a subjective assessment (with 1027 participants) of preprocessed images reveals that the visual quality either decreases or remains unchanged, thereby challenging the universal applicability of these metrics. The detection of metric attacks integrated into image processing systems has emerged as a significant issue. Attack detection can be achieved by comparing subjective evaluation results with metric results. If these results are anticorrelated, it suggests that the metric has been attacked. However, the time-consuming nature of subjective evaluations and the need for numerous participants make them impractical for routine attack detection. To address this, we propose using other metrics to determine whether the target metric has been attacked. Our results show that attacking one metric affects the output of other metrics, offering a potential method for attack detection.
Region of Interest Enabled Learned Image Coding for Machines

Image and video coding for machines has been recently gaining more and more interest from both the industry and the research community. One successful approach is based on end-to-end (E2E) learned compression and has shown significant gains over the state-of-the-art conventional image coding methods. However, one of the remaining challenges for such E2E-learned image codecs for machines is to adaptively allocate the bits over different regions of the image, while retaining the machine vision performance. In this paper, we propose a method that leverages Regions-Of-Interest (ROIs) for bitrate allocation within a Learned Image Codec (LIC) for machines. In particular, the proposed method reduces the bits allocated for the background regions of the image by reducing the variance of the elements corresponding to the background regions in the latent representation. This results in more heavily quantized background areas, while keeping the quality of the ROI areas suitable for machine tasks. The proposed method achieves significant gains, -15.80% and -22.43% Pareto BD-rate reduction, over the baseline LIC on object detection and instance segmentation tasks, respectively. To the best of our knowledge, this is the first research paper proposing an ROI-based inference-time technology for Learned Image Coding for machines.
Channel Affinity Knowledge Distillation for Semantic Segmentation

In recent years, convolutional neural networks have achieved significant success in computer vision tasks. However, the deployment of these algorithms remains challenging. Knowledge distillation (KD) as a type of important method enables a tiny model to extract helpful information from a large model. Most existing KD methods based semantic segmentation aim to align predicted maps in the spatial domain, but channel distillation also may help to improve segmentation performance. Additionally, pairwise pixel affinity provides efficiently structured reasoning for semantic segmentation. Motivated by these considerations, we propose a novel Channel Affinity KD (CAKD) framework for semantic segmentation that focuses on channel and cross-channel affinity relationship distillation to better align the distribution of the student and teacher models. Extensive experiments demonstrate that our proposed approach outperforms state-of-the-art KD methods on Cityscapes, Pascal VOC, and ADE20k datasets.
Fast LoG SIFT Keypoint Detector

Scale-invariant feature transform (SIFT) is a classical computer vision technique for scale-invariant keypoint detection and feature extraction. SIFT exhibits invariance to various transformations such as scale, rotation, noise, and illumination, making it applicable in a wide range of applications like object recognition, image matching and stitching, environment mapping, navigation, robotics, camera calibration, and more. A key contribution of SIFT is its utilization of the Difference-of-Gaussian (DoG) feature pyramid, which approximates the scale-space response of the Laplacian-of-Gaussian (LoG) filter. The DoG feature pyramid is computed by taking the separable Gaussian filtering and stacking the difference of Gaussian blurred images. In this paper, we propose a novel approach called “Fast LoG” filtering, which offers direct computation of the LoG filter to model the scale-space response solution. The “Fast LoG” filter is achieved by decomposing the LoG filter into two separable filters via SVD, and the scale-space response is computed by a direct polynomial fitting and differentiation, which is analytically more accurate. The polynomial fitting and differentiation only happen after the LoG peak strength thresholding, therefore the overall complexity is low compared with the DoG-based SIFT. The experimental results show that the keypoint generated by the Fast LoG method matches the SIFT keypoints, and per-pixel filtering complexity is lower.
A Simple Relationship Between SSIM and PSNR for DCT-Based Compressed Images and Video: SSIM as Content-Aware PSNR

The aim of this paper is to show how the structural similarity metric SSIM for image quality assessment can be seen in many cases, such as DCT-based compressed images and video, as a content-aware version of the peak signal-to-noise ratio (PSNR). In fact, under some assumptions described in the paper, the first can be derived directly from the latter based on a single content-dependent parameter, i.e. the variance of the image / video frame, not requiring joint access to original and compressed image/frames. Results show the validity of the assumptions and high accuracy for the derivation / estimation, in particular at quality levels of interest in real use cases.
μOpt: An Efficient Optimal Autoscaler for Microservice Applications

Microservices are a popular architecture for cloudbased applications subject to stringent performance requirements. To effectively serve variable workloads, autoscaling allocates computational resources ideally at the lowest possible cost. Although several autoscaling techniques have already been proposed in the literature, they suffer from high computational complexity. Here we propose μOpt as a computationally efficient model-based autoscaler for microservices. By solving a nonlinear optimization problem that embeds a layered queueing network (LQN) model, μOpt computes optimal configurations maximizing performance while minimizing allocated resources. We validate μOpt on a benchmark microservice application, reporting fast solution times (~10 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">-1</sup> s) that enable prompt reactions to highly variable workloads. Compared to a state-of-the-art autoscaler based on LQN and genetic algorithms, μOpt achieves higher performance (~6%-8%) with significantly fewer allocated resources (~15%-35%) in the presence of both synthetic and real-world workloads.
DASA: Domain Adaptation via Saliency Augmentation

This paper aims for supervised domain adaptation of image classifiers via saliency augmentation. The idea is to utilize domain-independent saliency extraction to enrich source and target domains and bring them closer. We then align their lower-order statistics to solve the problem. Because saliency augmentation suppresses uncommon background features across the domains, only the foreground features get aligned, as one would desire in the domain adaptation of image classifiers. Exploring this new direction of saliency augmentation for domain adaptation makes our work novel and promising. Despite providing far fewer labeled data in the target domain than in the source domain, our extensive experiments comprehensively demonstrate our method's commendable effectiveness and accuracy.
A Framework for Autonomic Computing for In Situ Imageomics

In situ imageomics is a new approach to study ecological, biological and evolutionary systems wherein large image and video data sets are captured in the wild and machine learning methods are used to infer biological traits of individual organisms, animal social groups, species, and even whole ecosystems. Monitoring biological traits over large spaces and long periods of time could enable new, data-driven approaches to wildlife conservation, biodiversity, and sustainable ecosystem management. However, to accurately infer biological traits, machine learning methods for images require voluminous and high quality data. Adaptive, data-driven approaches are hamstrung by the speed at which data can be captured and processed. Camera traps and unmanned aerial vehicles (UAVs) produce voluminous data, but lose track of individuals over large areas, fail to capture social dynamics, and waste time and storage on images with poor lighting and view angles. In this vision paper, we make the case for a research agenda for in situ imageomics that depends on significant advances in autonomic and self-aware computing. Precisely, we seek autonomous data collection that manages camera angles, aircraft positioning, conflicting actions for multiple traits of interest, energy availability, and cost factors. Given the tools to detect object and identify individuals, we propose a research challenge: Which optimization model should the data collection system employ to accurately identify, characterize, and draw inferences from biological traits while respecting a budget? Using zebra and giraffe behavioral data collected over three weeks at the Mpala Research Centre in Laikipia County, Kenya, we quantify the volume and quality of data collected using existing approaches. Our proposed autonomic navigation policy for in situ imageomics collection has an F1 score of 82% compared to an expert pilot, and provides greater safety and consistency, suggesting great potential for state-of-the-art autonomic approaches if they can be scaled up to fully address the problem.
Prolego: Time-Series Analysis for Predicting Failures in Complex Systems

Failures in large, complex systems can be difficult to diagnose and expensive for both the system maintainers and users. We present techniques for predicting failures when there is sparse ground truth in sensor logs and usage modes of the system evolve rapidly. We demonstrate our methods using real-world logs from three different systems. Our method achieves over 80% prediction accuracy for various amounts of lead time.