Reviewerly/exHarmony huggingface dataset:
Promoting Monocular Depth Estimation by Multi-Scale Residual Laplacian Pyramid Fusion

Deep learning approach has achieved great success in monocular depth estimation. However, the learned deep network may produce a depth map with fewer details and incorrect global depth layout, especially when the learned network is applied to a high-resolution image. In order to generate a high-quality depth map with better global structure and richer details, we propose a multi-scale residual Laplacian pyramid fusion net (MS-RLap-FNet), to fuse the multi-scale depth maps estimated by the existing depth estimation models, for depth refinement. Our approach relies on a proposed multi-scale residual Laplacian pyramid decomposition of the multi-scale depth maps, and the fusion network modules to gradually refine the depth maps based on the decomposition from low to high resolution. Comprehensive experiments show that our method, by refining the depth maps based on three popular monocular depth estimation models (DPT, MiDas, SGR), outperforms the existing state-of-the-art methods both in quantity and quality on three public datasets with different image resolutions. The depth map refined by our method has better global depth layout with richer fine details.
Micro-Doppler Extraction for Cone-Shaped Target Using Constrained Nonnegative Matrix Factorization

Micro-Doppler (MD) extraction is the precondition for target classification based on micro-motion. This letter proposes a method for MD extraction based on constrained nonnegative matrix factorization (NMF). The proposed algorithm is applied to the time-frequency spectrogram of the cone-shaped precession target. Taking the sparsity constraint, temporal continuity constraint and approximate orthogonality constraint into consideration, the basic NMF is improved to match the MD separation problem, and thus the time-frequency spectrogram of each scattering center is reconstructed by deriving the basis matrix and coefficient matrix. Morphological processing, valid frequency extraction and MD curve fitting are conducted on the reconstructed spectrograms to extract the MD accurately. Experiment results show that the proposed approach outperforms other algorithms with better accuracy and robustness.
Bringing Online Egocentric Action Recognition Into the Wild

To enable a safe and effective human-robot cooperation, it is crucial to develop models for the identification of human activities. Egocentric vision seems to be a viable solution to solve this problem, and therefore many works provide deep learning solutions to infer human actions from first person videos. However, although very promising, most of these do not consider the major challenges that comes with a realistic deployment, such as the portability of the model, the need for real-time inference, and the robustness with respect to the novel domains (i.e., new spaces, users, tasks). With this paper, we set the boundaries that egocentric vision models should consider for realistic applications, defining a novel setting of egocentric action recognition in the wild, which encourages researchers to develop novel, applications-aware solutions. We also present a new model-agnostic technique that enables the rapid repurposing of existing architectures in this new context, demonstrating the feasibility to deploy a model on a tiny device (Jetson Nano) and to perform the task directly on the edge with very low energy consumption (2.4W on average at 50 fps). The code is publicly available at: https://github.com/EgocentricVision/EgoWild.
Deep Learning Based Cognitive Radio Modulation Parameter Estimation

Automatic Modulation Classification (AMC) is a critical issue in electromagnetic spatial perception.Currently traditional recognition techniques are difficult to adapt to complex signal situations.Most existing modulation classification algorithms ignore the complementarity between different features and the importance of feature fusion.Based on this, we proposed a method for image feature fusion for AMC that fully uses the complementarity between different image features.The original signal is converted into an image by the Gramian Angular Field (GAF) method, and the GAF image is used as the input to the network, meanwhile the received signal is converted from the Inphase-Quadrature (I-Q) domain to the r-θ domain using the Accumulated Polar Feature conversion technique, and the original signal is feature coded from the r-θ domain and then converted into an image.The fused features of the two images are used as input to the neural network for model training to achieve automatic modulation classification of multiple types of signals.In the evaluation phase, the differences in the recognition effectiveness of the proposed method by different neural networks are discussed.Experiments show that the best performance is achieved using the Swin-Transformer network model, with a more than 90% recognition rate for the modulation method at signal-to-noise ratios greater than 4dB.
Insulator Detection for High-Resolution Satellite Images Based on Deep Learning

The detection of electrical insulators in Unmanned Aerial Vehicle (UAV) images using deep learning has made great progress in recent years, but little research has been conducted in the same field in remote sensing (RS) images. In this paper, a novel method was proposed to detect insulators on 500kV transmission towers in RS images. The proposed method consists of three components including 1) a super-resolution (SR) network to improve image resolution, 2) an object detection model to detect 110kV, 220kV and 500kV electrical power towers along transmission pipelines and 3) a semantic segmentation network to identify insulators on the detected 500kV towers. In addition, the online hard example mining (OHEM) method and class weight calculation method were utilized to handle the imbalanced data among different classes during training. The proposed model was evaluated on SuperView-1 and WorldView-3 satellite images collected in 4 regions. Experiment results show that the proposed method can effectively detect insulators in high-resolution satellite images and achieved the highest F1 score of 0.7952. The codes are available at https://github.com/hardworking-jws/insulator-detection-remote-sensing.
Oblivious Statistic Collection With Local Differential Privacy in Mutual Distrust

Location data is valuable for various applications such as epidemiology, natural disasters, and urban planning but causes exposure of sensitive information, e.g., home or work place, from collected data in a datastore. Local Differential Privacy (LDP)-based data collection is a promising technology to protect sensitive information. A mobile device modify data to make each piece of data indistinguishable from others but keep its intrinsic value for statistical characteristics in data. Although LDP fundamentally protects the privacy exposure from a data store, a datastore suffer a shortcomings on it; as a datastore can never validate the modified data due to concealed raw data, that allows anyone to tamper with one’s data or inject any amount of data, and thus manipulate the statistics of the whole data in a datastore, called data poisoning attack. As a device does not disclose raw data and a datastore cannot collaborate to validate data with a device who may be an adversary on this mutual distrust relationship, data collection needs an ability to avoid the effect of data poisoning.. The cause of data poisoning is the direct relationship between data volume and statistic; the more data a device sends gives more statistical changes on merged data in a datastore. In this paper, we propose to decouple statistical characteristics from data volumes on LDP-based data collection process to minimize the effect of poisoned data on a datastore. We utilize Oblivious Transfer (OT) protocol to retrieve only statistic characteristics of receiving data at a datastore. As OT protocol inevitably strengthen privacy protection on LDP-based data collection and accordingly drops statistic characteristics of data, We adjust LDP processing to collaboratively work with OT protocol. The proposed adjustment method adapts the protection strength of LDP to OT protocol behavior so that a data store receives data containing sufficient statistical characteristics. We conduct qualitative and experimental overhead analysis and show that our method decouples the relationship between statistical characteristics from data volume. Our experimental result also prove that the overhead can be acceptable on devices such as smartphones and IoT.
Superconducting-Oscillatory Neural Network With Pixel Error Detection for Image Recognition

Brain-inspired oscillatory neural networks (ONNs) utilize coupled oscillators to emulate biological neuronal dynamics. ONNs are naturally suitable for image/pattern recognition. Many hardware implementations of ONNs have been explored recently by using analog or digital CMOS systems, which can be limited by Moore's law. In this work, we used one of the advanced superconducting technologies for coupled oscillator networks to further improve the energy efficiency and processing speed. Inductively coupled ring oscillators using rapid single flux quantum (RSFQ) technology were designed and simulated. These Josephson junction (JJ) based oscillators can operate as fast as tens of GHz but only consume a energy of several aJ per operation. Excluding cryo-cooling factors, the estimated power consumption of our RSFQ oscillator is hundreds of nW at a frequency of 12.5 GHz, which is multiple times smaller than a typical CMOS ring oscillator. Furthermore, a system for pattern recognition with error detection was designed based on these oscillators. Synchronization dynamics in a pair of coupled oscillators were used to identify whether a test pixel matches a reference pixel. A network comprised of 16 pairs of oscillators were wired together, along with synchronization detectors and a fluxon integrator to demonstrate pattern recognition function. This work was performed using standard JJ models and demonstrated through WRspice and JoSIM software simulations. The circuits and systems proposed in this work aim to provide insights into the use of superconducting technology for implementing ONNs and may serve as basic building blocks for the design of more complex oscillatory computational networks in image processing and beyond. Single oscillator modeling and systematic network function exploration will be performed in future work.
Data Secure De-Duplication and Recovery Based on Public Key Encryption With Keyword Search

In the current era of information explosion, users' demand for data storage is increasing, and data on the cloud has become the first choice of users and enterprises.Cloud storage facilitates users to backup and share data, effectively reducing users' storage expenses.As the duplicate data of different users are stored multiple times, leading to a sudden decrease in storage utilization of cloud servers.Data stored in plaintext form can directly remove duplicate data, while cloud servers are semi-trusted and usually need to store data after encryption to protect user privacy.In this paper, we focus on how to achieve secure de-duplication and recover data in ciphertext for different users, and determine whether the indexes of public key searchable encryption and the matching relationship of trapdoor are equal in ciphertext to achieve secure de-duplication.For the duplicate file, the data user's re-encryption key about the file is appended to the ciphertext chain table of the stored copy.The cloud server uses the re-encryption key to generate the specified transformed ciphertext, and the data user decrypts the transformed ciphertext by its private key to recover the file.The proposed scheme is secure and efficient through security analysis and experimental simulation analysis.
Towards Understanding Variation-Constrained Deep Neural Networks

Multi-layer feedforward networks have been used to approximate a wide range of nonlinear functions. A fundamental problem is understanding the generalizability of a neural network model through its statistical risk, or the expected test error. In particular, it is important to understand the phenomenon that overparameterized deep neural networks may not suffer from overfitting when the number of neurons and learning parameters rapidly grow with <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$n$</tex-math></inline-formula> or even surpass <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$n$</tex-math></inline-formula> . In this paper, we show that a class of variation-constrained regression neural networks, with arbitrary width, can achieve a near-parametric rate <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$n^{-1/2+\delta }$</tex-math></inline-formula> for an arbitrarily small positive constant <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\delta$</tex-math></inline-formula> . It is equivalent to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$n^{-1 +2\delta }$</tex-math></inline-formula> under the mean squared error. This rate is also observed from numerical experiments. The result provides an insight into the benign overparameterization phenomenon. It indicates that the number of trainable parameters may not be a suitable complexity measure as often perceived for classical regression models. We also discuss the convergence rate regarding other network parameters, including the input dimension, network layer, and coefficient norm.
Adversarial Attacks and Batch Normalization: A Batch Statistics Perspective

Batch Normalization (BatchNorm) is an effective architectural component in deep learning models that helps to improve model performance and speed up training. However, it has also been found to increase the vulnerability of models to adversarial attacks. In this study, we investigate the mechanism behind this vulnerability and took first steps towards a solution called RobustNorm. We observed that adversarial inputs tend to shift the distributions of the output of the BatchNorm layer, leading to inaccurate train-time statistics and increased vulnerability. Through a series of experiments on various architectures and datasets, we confirm our hypothesis. We also demonstrate the effectiveness of RobustNorm in improving the robustness of models under adversarial perturbation while maintaining the benefits of BatchNorm.
Priors Guided Extreme Underwater Image Compression for Machine Vision and Human Vision

A low bit-rate compression is required for underwater images due to the limited bandwidth in underwater acoustic communication, which limits performances of both machine analysis and human perception in most underwater applications. Few existing compression methods consider the unique characteristics of underwater images such as color shift and haze effect to better fulfill requirements of various applications under low bit-rates. To address this problem, we propose a novel extreme underwater image compression framework, which can provide scalability to support machine vision and human vision with the assistance of underwater priors. Specifically, the base layer is composed of a feature extractor and a generator, where global structural edges and high-level features of regions with a significant impact on machine analysis are extracted and used for reconstructing a feature-matching image for analysis purpose. Considering the negative influence of underwater imaging processes on machine vision, in this article, a feature degradation removal module guided by underwater priors is proposed to alleviate feature-level degradation via taking analysis-friendly enhanced images as auxiliary information. As for the enhancement layer aiming for human vision, the residual between the original image and the reconstruction from base layer is compressed. A feature attention block and a background light recovery block are designed utilizing features extracted from enhanced images and the underwater before further recovering the original scene with a good perception quality under low bit-rates. Experimental results demonstrate the superiority of our framework in both machine vision tasks and perception quality compared with traditional compression methods and learned-based methods.
An Efficient Data Structure for Dynamic Graph on GPUs

There is a growing interest to offload dynamic graph computation to GPU and resort to its high parallel processing ability and larger memory bandwidths compared with CPUs. The existing GPU graph systems usually use compressed sparse row (CSR) as the de-facto structure. However, CSR has a critical weakness for dynamic change due to the large overhead of re-balance process after update. GPMA+ is a state-of-art dynamic PMA-based structure that uses PMA structure and segment-oriented parallel update procedure to address the dynamic weakness of CSR, but it still has a bottleneck on the array expansion. In this paper, we propose an leveled structure (called LPMA) instead of continue array to retain low time complexity and high parallel update and lift the expansion bottleneck of GPMA+. More specifically, we propose a series of optimization techniques, including bottom-up update, top-down update and on-demand hybrid update strategies as well as consistence-guaranteed parallel processing for update-query mixed workloads. We theoretically analyze the benefits of LPMA compared in terms of re-balance cost during updates. Extensive experiments on four large real-life graphs prove the superiority of LPMA compared with the-state-of-arts.
In-Sensor Artificial Intelligence and Fusion With Electronic Medical Records for At-Home Monitoring

This work presents an artificial intelligence (AI) framework for real-time, personalized sepsis prediction four hours before onset through fusion of electrocardiogram (ECG) and patient electronic medical record. An on-chip classifier combines analog reservoir-computer and artificial neural network to perform prediction without front-end data converter or feature extraction which reduces energy by 13× compared to digital baseline at normalized power efficiency of 528 TOPS/W, and reduces energy by 159× compared to RF transmission of all digitized ECG samples. The proposed AI framework predicts sepsis onset with 89.9% and 92.9% accuracy on patient data from Emory University Hospital and MIMIC-III respectively. The proposed framework is non-invasive and does not require lab tests which makes it suitable for at-home monitoring.
Optimizing GPU-Based Graph Sampling and Random Walk for Efficiency and Scalability

Graph sampling and random walk algorithms are playing increasingly important roles today because they can significantly reduce graph size while preserving structural information, thus enabling computationally intensive tasks on large-scale graphs. Current frameworks designed for graph sampling and random walk tasks are generally not efficient in terms of memory requirement and throughput. Not to mention that some of them result in biased results. To solve the above problems, we introduce Skywalker+, a high-performance graph sampling and random walk framework on multiple GPUs supporting multiple algorithms. Skywalker+ makes four key contributions: First, it realizes highly paralleled alias method on GPUs. Second, it applies finely adjusted workload-balancing techniques and locality-aware execution modes to present a highly efficient execution engine. Third, it optimizes the GPU memory usage with efficient buffering and data compression schemes. Last, it scales to multi-GPU to further enhance the system throughput. Abundant experiments show that Skywalker+ exhibits significant advantage over the baselines both in performance and utility.
Illumination-Guided RGBT Object Detection With Inter- and Intra-Modality Fusion

Robust object detection is hindered by various illumination conditions in real-world applications. Common practice introduces thermal modality to augment the detection capability of RGB images in poor illumination conditions. However, a major challenge for such work is how to leverage the complementary information of RGB and thermal images effectively. In this paper, we tackle this by combining the strengths of illumination-guided inter- and intra-modality information in both modalities. Specifically, we propose an effective object detection network dubbed TINet to adaptively fuse the complementary features extracted from RGB and thermal images. Firstly, we design an illumination-guided feature weighting module to guide the network in leaning towards a reliable modality. Then, an inter-modality attention module is developed to amplify and complement differential features between these two modalities. We further design an intra-modality attention module using object heatmap prediction to enhance the foreground features of each modality. Finally, we fuse the inter- and intra-modality features for object detection with illumination-guided feature weights. Experimental results demonstrate the effectiveness of the proposed method on the FLIR-aligned dataset and the KAIST multispectral pedestrian detection dataset.
Eye gaze for menu display selection on smart wheelchair using k-nearest neighbors method

Everyone wants to have high mobility in their own life. For patients with disabilities who especially experience paralysis on their hands, they will experience less mobility in daily activities. This causes the patients the need for nurses for their treatment. However, nurses cannot keep them monitored for 24 hours continuously. They need more attention in the treatment period. In this research, the authors developed a system to help patients with disabilities on their hands in their daily activities. The author proposes menu selection using the face on smart wheelchair to provide an alternative in menu selection by utilizing the eye gaze. The right eye position is used to select menus, while the blinking of the left eye is used for menu execution. The menus will be used to execute the features of the smart wheel cursor on the device. There are four optional menus in this research. Eye detection use facial landmark methods and classification use k-nearest neighbors. Disabilities, especially those experiencing paralysis on the hands, can be helped in their daily lives by implementing the system. The higher accuracy result of the method is 90,42% with k=9, with 0,441 seconds of computation time with k=9.
Learnings From Discontinued Central Bank Digital Currencies (CBDC)

The financial and monetary system is transitioning into contactless payment options and digital currencies. About 90 percent of the world's central banks are engaged in CBDC research, proof of concept, pilot, development, and launch, while the remaining are overlooking CBDC offerings. Notwithstanding, many central banks have cancelled their CBDC following their launch. While learning from the past failure is extoled as a virtue, learning from failed CBDC is seldom in the CBDC ecosystem, contrary to dominant literature on CBDC development and scalability. This study explores reasons why central banks discontinued already launched CBDC's. The authors adopted criterion sampling to select failed CBDC. This multiple case content analysis shows that low levels of trust, cybersecurity concerns, inadequate digital identification infrastructure, and obsolete and uncompetitive technology are the leading triggers of failed CBDC. Through the lens of actor-network theory (ANT), actors contributing to CBDC failure was identified