Reviewerly/exHarmony huggingface dataset:
Attentive Fine-Grained Structured Sparsity for Image Restoration

Image restoration tasks have witnessed great performance improvement in recent years by developing large deep models. Despite the outstanding performance, the heavy computation demanded by the deep models has restricted the application of image restoration. To lift the restriction, it is required to reduce the size of the networks while maintaining accuracy. Recently, N:M structured pruning has appeared as one of the effective and practical pruning approaches for making the model efficient with the accuracy constraint. However, it fails to account for different computational complexities and performance requirements for different layers of an image restoration network. To further optimize the trade-off between the efficiency and the restoration accuracy, we propose a novel pruning method that determines the pruning ratio for N:M structured sparsity at each layer. Extensive experimental results on super-resolution and deblurring tasks demonstrate the efficacy of our method which outperforms previous pruning methods significantly. PyTorch implementation for the proposed methods will be publicly available at https://github.com/JungHunOh/SLS_CVPR2022
LC-FDNet: Learned Lossless Image Compression with Frequency Decomposition Network

Recent learning-based lossless image compression methods encode an image in the unit of subimages and achieve comparable performances to conventional non-learning algorithms. However, these methods do not consider the performance drop in the high-frequency region, giving equal consideration to the low and high-frequency areas. In this paper, we propose a new lossless image compression method that proceeds the encoding in a coarse-to-fine manner to separate and process low and high-frequency regions differently. We initially compress the low-frequency components and then use them as additional input for encoding the remaining high-frequency region. The low-frequency components act as a strong prior in this case, which leads to improved estimation in the high-frequency area. In addition, we design the frequency decomposition process to be adptive to color channel, spatial location, and image characteristics. As a result, our method derives an image-specific optimal ratio of low/high-frequency components. Experiments show that the proposed method achieves state-of-the-art performance for benchmark high-resolution datasets.
Learned Queries for Efficient Local Attention

Vision Transformers (ViT) serve as powerful vision models. Unlike convolutional neural networks, which dominated vision research in previous years, vision transformers enjoy the ability to capture long-range dependencies in the data. Nonetheless, an integral part of any transformer architecture, the self-attention mechanism, suffers from high latency and inefficient memory utilization, making it less suitable for high-resolution input images. To alleviate these shortcomings, hierarchical vision models locally employ self-attention on non-interleaving windows. This relaxation reduces the complexity to be linear in the input size; however, it limits the cross-window interaction, hurting the model performance. In this paper, we propose a new shift-invariant local attention layer, called query and attend (QnA), that aggregates the input locally in an overlapping manner, much like convolutions. The key idea behind QnA is to introduce learned queries, which allow fast and efficient implementation. We verify the effectiveness of our layer by incorporating it into a hierarchical vision transformer model. We show improvements in speed and memory complexity while achieving comparable accuracy with state-of-the-art models. Finally, our layer scales especially well with window size, requiring up to x10 less memory while being up to x5 faster than existing methods. The code is publicly available at https://github.com/moabarar/qna.
LiDAL: Inter-frame Uncertainty Based Active Learning for 3D LiDAR Semantic Segmentation

We propose LiDAL, a novel active learning method for 3D LiDAR semantic segmentation by exploiting inter-frame uncertainty among LiDAR frames. Our core idea is that a well-trained model should generate robust results irrespective of viewpoints for scene scanning and thus the inconsistencies in model predictions across frames provide a very reliable measure of uncertainty for active sample selection. To implement this uncertainty measure, we introduce new inter-frame divergence and entropy formulations, which serve as the metrics for active selection. Moreover, we demonstrate additional performance gains by predicting and incorporating pseudo-labels, which are also selected using the proposed inter-frame uncertainty measure. Experimental results validate the effectiveness of LiDAL: we achieve 95% of the performance of fully supervised learning with less than 5% of annotations on the SemanticKITTI and nuScenes datasets, outperforming state-of-the-art active learning methods. Code release: https://github.com/hzykent/LiDAL .
Unsupervised Domain Adaptive 3-D Detection With Data Adaption From LiDAR Point Cloud

Existing unsupervised domain adaptive (UDA) 3D detection methods only address the domain gap caused by the prior size of 3D bounding boxes between different datasets, which ignore the difference in the distribution of point clouds. To address this challenge, we propose an unsupervised domain adaptive 3D detection by data adaption, which trains the model by transferring the source domain instances into the target domain scenes by adaptive point distribution. First, an instance transferring method is proposed for selecting and transferring suitable instances from the source domain into the target domain scene; Second, we propose an adaptive downsampling method to adjust the point cloud distribution of the transferred instances to approximate the points distribution of the target domain. Finally, our method trains the randomly initialized detector with the pseudo-instances in the target domain. To the best of our knowledge, we first address the UDA problem of the 3D detectors from the perspective of data. Extensive experiments on several popular datasets show that the proposed method outperforms the existing state-of-the-art methods by a large margin. Further experiments also show our approach is detector-agnostic and achieves consistent and significant gains on all types of 3D detectors.
Adaptive Co-teaching for Unsupervised Monocular Depth Estimation

Unsupervised depth estimation using photometric losses suffers from local minimum and training instability. We address this issue by proposing an adaptive co-teaching framework to distill the learned knowledge from unsupervised teacher networks to a student network. We design an ensemble architecture for our teacher networks, integrating a depth basis decoder with multiple depth coefficient decoders. Depth prediction can then be formulated as a combination of the predicted depth bases weighted by coefficients. By further constraining their correlations, multiple coefficient decoders can yield a diversity of depth predictions, serving as the ensemble teachers. During the co-teaching step, our method allows different supervision sources from not only ensemble teachers but also photometric losses to constantly compete with each other, and adaptively select the optimal ones to teach the student, which effectively improves the ability of the student to jump out of the local minimum. Our method is shown to significantly benefit unsupervised depth estimation and sets new state of the art on both KITTI and Nuscenes datasets.
Automatic Group-Based Structured Pruning for Deep Convolutional Networks

Structured pruning methods have been used in several convolutional neural networks (CNNs). However, group-based structured pruning is a challenging task. In previous methods, the number of groups is manually determined for all layers, which is suboptimal. Moreover, which kernels should be appropriately removed? Model accuracy may be significantly reduced when the number of kernels is removed. To address these challenges, we propose an automatic group-based structured pruning method with reinforcement learning, named AGSPRL, which can generate pruned models with different compression rates automatically. We first develop a reinforcement learning (RL) framework to learn the pruning rate for group-based channel pruning layer by layer. Then, based on the learned kernel pruning rate, we propose an efficient group configuration algorithm to adaptively determine the number of groups for each convolution layer. Finally, we introduce a channel pruning method with an attention mechanism as a tiny auxiliary filter selector for each group to dynamically determine which part of the kernels should be selected into the group convolution and which part of the kernels should be removed. To demonstrate the efficiency of our method, we apply it to a variety of CNNs in classification and detection datasets. The experimental results show that the AGSPRL not only adaptively but also accurately configures the number of groups. The accuracy is reduced by less than 1% and improved by 1%. Moreover, compared to other state-of-the-art methods, AGSPRL is more effective and has less accuracy loss.
Arrow, Bézier Curve, or Halos? – Comparing 3D Out-of-View Object Visualization Techniques for Handheld Augmented Reality

Handheld augmented reality (AR) applications allow users to interact with their virtually augmented environment on the screen of their tablet or smartphone by simply pointing its camera at nearby objects or “points of interest” (POIs). However, this often requires users to carefully scan their surroundings in search of POIs that are out of view. Proposed 2D guides for out-of-view POIs can, unfortunately, be ambiguous due to the projection of a 3D position to 2D screen space. We address this by using 3D visualizations that directly encode the POI’s 3D direction and distance. Based on related work, we implemented three such visualization techniques: (1) 3D Arrow, (2) 3D Bézier Curve, and (3) 3D Halos. We confirmed the applicability of these three techniques in a case study and then compared them in a user study, evaluating performance, workload, and user experience. Participants performed best using 3D Arrow, while surprisingly, 3D Halos led to poor results. We discuss the design implications of these results that can inform future 3D out-of-view object visualization techniques.
A Comprehensive Survey of Security Issues of Smart Home System: “Spear” and “Shields,” Theory and Practice

The rapid development and wide application of the Internet of Things (IoT) and sensor technologies have produced good opportunities for the development of IoT-based smart home systems (SHSs).However, during the rapid market expansion of SHSs, security challenges associated with SHSs have become a primary concern of people because they are so closely related to people's daily lives.These security problems may damage information assets and pose a serious threat to people's health and life.This study investigates security issues in SHSs and provides a comprehensive overview of research to date.In this review, after analyzing the existing definition and concept of SHSs proposed by authoritative encyclopedias and academic literature, we propose a more accurate, elaborated definition of SHSs, analyze their architecture, extract six natural and contextual features, and summarize spears (cyber-attack means) vs. shields (countermeasures) in detail in the context of SHSs.Then, the security frameworks and evaluation technologies in SHSs are discussed.Different scenarios for technology integration and the practical research results in SHSs, such as blockchain, cloud computing, Internet of Vehicles, and AI are presented meticulously.After that, two special issues related to security are discussed.We believe that future research on SHS security should focus on four aspects: the unification of architecture, resource limitation, fragmentation, and code and firmware security.In addition, research on SHS security should be given priority over its commercialization process.
Illumination-Guided Transformer-Based Network for Multispectral Pedestrian Detection

Multi-modal information (e.g., visible and thermal) can generate reliable and robust pedestrian detection results in various computer vision applications. Despite its broad applications, it remains a crucial problem that how to fuse the two modalities effectively. The self-attention operator of transformer can obtain long-range dependencies and integrate information across the entire input, which has been widely used for cross-modal fusion. However, there is still a lack of further analysis and design for transformer to use in multispectral pedestrian detection task. To benefit from both RGB and thermal modalities, we propose a novel illumination-guided transformer-based network (ITNet) for multispectral pedestrian detection in this paper. Firstly, different from the previous methods that apply the original transformer structure directly, we designed two different transformer-based fusion modules to make the RGB and thermal modalities complement each other. Secondly, an illumination-guided module is used to adaptively re-weight and fuse the multi-modal features according to the illumination conditions. Extensive evaluations on two benchmarks demonstrate the effectiveness of our proposed approach for multispectral pedestrian detection.
$$\text {Face2Face}^\rho $$: Real-Time High-Resolution One-Shot Face Reenactment

Existing one-shot face reenactment methods either present obvious artifacts in large pose transformations, or cannot well-preserve the identity information in the source images, or fail to meet the requirements of real-time applications due to the intensive amount of computation involved. In this paper, we introduce $$\text {Face2Face}^\rho $$ , the first Real-time High-resolution and One-shot (RHO, $$\rho $$ ) face reenactment framework. To achieve this goal, we designed a new 3DMM-assisted warping-based face reenactment architecture which consists of two fast and efficient sub-networks, i.e., a u-shaped rendering network to reenact faces driven by head poses and facial motion fields, and a hierarchical coarse-to-fine motion network to predict facial motion fields guided by different scales of landmark images. Compared with existing state-of-the-art works, $$\text {Face2Face}^\rho $$ can produce results of equal or better visual quality, yet with significantly less time and memory overhead. We also demonstrate that $$\text {Face2Face}^\rho $$ can achieve real-time performance for face images of $$1440\times 1440$$ resolution with a desktop GPU and $$256\times 256$$ resolution with a mobile CPU.
RepNet: Efficient On-Device Learning via Feature Reprogramming

Transfer learning, where the goal is to transfer the well-trained deep learning models from a primary source task to a new task, is a crucial learning scheme for on-device machine learning, due to the fact that IoT/edge devices collect and then process massive data in our daily life. However, due to the tiny memory constraint in IoT/edge devices, such on-device learning requires ultra-small training memory footprint, bringing new challenges for memory-efficient learning. Many existing works solve this problem by reducing the number of trainable parameters. However, this doesn't directly translate to memory saving since the major bottleneck is the activations, not parameters. To develop memory-efficient on-device transfer learning, in this work, we are the first to approach the concept of transfer learning from a new perspective of intermediate feature re-programming of a pre-trained model (i.e., backbone). To perform this lightweight and memory-efficient reprogramming, we propose to train a tiny Reprogramming Network (Rep-Net) directly from the new task input data, while freezing the backbone model. The proposed Rep-Net model interchanges the features with the backbone model using an activation connector at regular intervals to mutually benefit both the backbone model and Rep-Net model features. Through extensive experiments, we validate each design specs of the proposed Rep-Net model in achieving highly memory-efficient on-device reprogramming. Our experiments establish the superior performance (i.e., low training memory and high accuracy) of Rep-Net compared to SOTA on-device transfer learning schemes across multiple benchmarks. Code is available at https://github.com/ASU-ESIC-FAN-Lab/RepNet.
AMMF: Attention-Based Multi-Phase Multi-Task Fusion for Small Contour Object 3D Detection

Recently significant progress has been made in 3D detection. However, it is still challenging to detect small contour objects under complex scenes. This paper proposes a novel Attention-based Multi-phase Multi-task Fusion (AMMF) that uses point-level, RoI-level, and multi-task fusions to complement the disadvantages of LiDAR and camera, to solve this challenge. First, at the feature extraction phase, AMMF uses the Low and High-level Fusion with Matching Attention (LHF-MA) and efficient FPN (eFPN) to perform point-level fusion for cross sensors and single sensor, respectively. Instead of merging each level and using expensive 3D CNN like other methods, LHF-MA fuses low-level spatial location and high-level contextual feature of 2D CNN customized feature extractors and ignores the fusion of middle levels, reducing the computational cost. Then, at the proposal generation phase, Progressive Proposal Fusion (PPF) with learned attention map is used to perform coarse-to-fine RoI-level fusion, instead of only combining coarse-grained features at high-level of network. PPF using progressively increasing IoU thresholds could avoid overfitting and improve the performance. Note that the matching attentions and learned attention maps are utilized to weigh the priority of different sensors. Moreover, to solve the sparseness of point-wise fusion between LiDAR BEV and RGB image, AMMF uses multi-task fusion that generates pseudo-LiDAR from camera by depth estimation task, to guide this point-wise fusion. Finally, AMMF performs excellently for detecting small contour objects like pedestrians, cyclists, and distant cars. On the KITTI, AMMF finishes 3.62% improvements in the moderate instance for pedestrians. It achieves a 2.21% improvement in the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$&gt;$</tex-math> </inline-formula> 50 instance of LEVEL <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$_{-}$</tex-math> </inline-formula> 2 level for vehicle on the Waymo Open Dataset. And AMMF is further verified on our customized dataset consisting of challenging scenarios like strong illumination and heavy shadow cases.
An effective method for text line segmentation in historical document images

In this paper, we present a text-line segmentation method for historical documents.Historical documents are challenging given their characteristics of highly degradation, writing style variation and diacritics.From these observations, we proposed an effective approach for text line segmentation by analysing the properties of document layouts.We combine the idea of seam carving method with the novel cost functions to accurately split text lines.Experiments were conducted on two challenging datasets of historical documents, namely the DIVA-HisDB dataset and our ChamDoc dataset.Our methods provided good results on the DIVA-HisDB dataset with 99.36% of Line IU and 98.86% of Pixel IU.On the ChamDoc dataset, the proposed method outperformed the two baseline approaches i.e. seam carving-based and A* path planning by a large margin.
StyleLight: HDR Panorama Generation for Lighting Estimation and Editing

We present a new lighting estimation and editing framework to generate high-dynamic-range (HDR) indoor panorama lighting from a single limited field-of-view (LFOV) image captured by low-dynamic-range (LDR) cameras. Existing lighting estimation methods either directly regress lighting representation parameters or decompose this problem into LFOV-to-panorama and LDR-to-HDR lighting generation sub-tasks. However, due to the partial observation, the high-dynamic-range lighting, and the intrinsic ambiguity of a scene, lighting estimation remains a challenging task. To tackle this problem, we propose a coupled dual-StyleGAN panorama synthesis network (StyleLight) that integrates LDR and HDR panorama synthesis into a unified framework. The LDR and HDR panorama synthesis share a similar generator but have separate discriminators. During inference, given an LDR LFOV image, we propose a focal-masked GAN inversion method to find its latent code by the LDR panorama synthesis branch and then synthesize the HDR panorama by the HDR panorama synthesis branch. StyleLight takes LFOV-to-panorama and LDR-to-HDR lighting generation into a unified framework and thus greatly improves lighting estimation. Extensive experiments demonstrate that our framework achieves superior performance over state-of-the-art methods on indoor lighting estimation. Notably, StyleLight also enables intuitive lighting editing on indoor HDR panoramas, which is suitable for real-world applications. Code is available at https://style-light.github.io/ .
SecSkyline: Fast Privacy-Preserving Skyline Queries Over Encrypted Cloud Databases

The well-known benefits of cloud computing have spurred the popularity of database service outsourcing, where one can resort to the cloud to conveniently store and query databases. Coming with such popular trend is the threat to data privacy, as the cloud gains access to the databases and queries which may contain sensitive information, like medical or financial data. A large body of work has been presented for querying encrypted databases, which has been mostly focused on secure keyword search. In this paper, we instead focus on the support for secure skyline query processing over encrypted outsourced databases, where little work has been done. Skyline query is an advanced kind of database query which is important for multi-criteria decision-making systems and applications. We propose SecSkyline, a new system framework building on lightweight cryptography for fast privacy-preserving skyline queries. SecSkyline ambitiously provides strong protection for not only the content confidentiality of the outsourced database, the query, and the result, but also for data patterns that may incur indirect data leakages, such as dominance relationships among data points and search access patterns. Extensive experiments demonstrate that SecSkyline is substantially superior to the state-of-the-art in query latency, with up to <inline-formula><tex-math notation="LaTeX">$813\times$</tex-math></inline-formula> improvement.