Reviewerly/exHarmony huggingface dataset:
Image Deblurring With Image Blurring

Deep learning (DL) based methods for motion deblurring, taking advantage of large-scale datasets and sophisticated network structures, have reported promising results. However, two challenges still remain: existing methods usually perform well on synthetic datasets but cannot deal with complex real-world blur, and in addition, over- and under-estimation of the blur will result in restored images that remain blurred and even introduce unwanted distortion. We propose a motion deblurring framework that includes a Blur Space Disentangled Network (BSDNet) and a Hierarchical Scale-recurrent Deblurring Network (HSDNet) to address these issues. Specifically, we train an image blurring model to facilitate learning a better image deblurring model. Firstly, BSDNet learns how to separate the blur features from blurry images, which is adaptable for blur transferring, dataset augmentation, and ultimately directing the deblurring model. Secondly, to gradually recover sharp information in a coarse-to-fine manner, HSDNet makes full use of the blur features acquired by BSDNet as a priori and breaks down the non-uniform deblurring task into various subtasks. Moreover, the motion blur dataset created by BSDNet also bridges the gap between training images and actual blur. Extensive experiments on real-world blur datasets demonstrate that our method works effectively on complex scenarios, resulting in the best performance that significantly outperforms many state-of-the-art approaches.
Frequency-Domain Inference Acceleration for Convolutional Neural Networks Using ReRAMs

Convolutional neural networks (CNNs) (including 2D and 3D convolutions) are popular in video analysis tasks such as action recognition and activity understanding. Fast algorithms such as fast Fourier transforms (FFTs) are promising in significantly reducing computation complexity by transforming convolution into frequency domain. In frequency space, conventional spatial convolutions are replaced with simpler element-wise complex multiplications. Conventional application-specific-integrated-circuit (ASIC) based frequency-domain accelerators can achieve effective performance boost but come at the cost of significant energy consumption, owing to the hierarchical memory organization. We propose a frequency-domain resistive random access memory (ReRAM) based inference accelerator called FDA that can process element-wise complex multiplication in memory for both 2D and 3D CNNs. Each ReRAM-based frequency-domain process element (PE) with two ReRAM cells can perform an element-wise complex multiplication in two continuous execution cycles. We then provide a flexible dataflow to alleviate the redundant data movements by frequency-domain data reuse and inherent symmetrical characteristic for both 2D and 3D convolutions. Evaluation results based on representative both 2D and 3D CNN benchmarks demonstrate that FDA outperforms state-of-the-art baselines with better performance and energy efficiency.
Composite Object Relation Modeling for Few-Shot Scene Recognition

The goal of few-shot image recognition is to classify different categories with only one or a few training samples. Previous works of few-shot learning mainly focus on simple images, such as object or character images. Those works usually use a convolutional neural network (CNN) to learn the global image representations from training tasks, which are then adapted to novel tasks. However, there are many more abstract and complex images in real world, such as scene images, consisting of many object entities with flexible spatial relations among them. In such cases, global features can hardly obtain satisfactory generalization ability due to the large diversity of object relations in the scenes, which may hinder the adaptability to novel scenes. This paper proposes a composite object relation modeling method for few-shot scene recognition, capturing the spatial structural characteristic of scene images to enhance adaptability on novel scenes, considering that objects commonly co- occurred in different scenes. In different few-shot scene recognition tasks, the objects in the same images usually play different roles. Thus we propose a task-aware region selection module (TRSM) to further select the detected regions in different few-shot tasks. In addition to detecting object regions, we mainly focus on exploiting the relations between objects, which are more consistent to the scenes and can be used to cleave apart different scenes. Objects and relations are used to construct a graph in each image, which is then modeled with graph convolutional neural network. The graph modeling is jointly optimized with few-shot recognition, where the loss of few-shot learning is also capable of adjusting graph based representations. Typically, the proposed graph based representations can be plugged in different types of few-shot architectures, such as metric-based and meta-learning methods. Experimental results of few-shot scene recognition show the effectiveness of the proposed method.
Prototypical Matching Networks for Video Object Segmentation

Semi-supervised video object segmentation is the task of segmenting the target in sequential frames given the ground truth mask in the first frame. The modern approaches usually utilize such a mask as pixel-level supervision and typically exploit pixel-to-pixel matching between the reference frame and current frame. However, the matching at pixel level, which overlooks the high-level information beyond local areas, often suffers from confusion caused by similar local appearances. In this paper, we present Prototypical Matching Networks (PMNet) - a novel architecture that integrates prototypes into matching-based video objection segmentation frameworks as high-level supervision. Specifically, PMNet first divides the foreground and background areas into several parts according to the similarity to the global prototypes. The part-level prototypes and instance-level prototypes are generated by encapsulating the semantic information of identical parts and identical instances, respectively. To model the correlation between prototypes, the prototype representations are propagated to each other by reasoning on a graph structure. Then, PMNet stores both the pixel-level features and prototypes in the memory bank as the target cues. Three affinities, i.e., pixel-to-pixel affinity, prototype-to-pixel affinity, and prototype-to-prototype affinity, are derived to measure the similarity between the query frame and the features in the memory bank. The features aggregated from the memory bank using these affinities provide powerful discrimination from both the pixel-level and prototype-level perspectives. Extensive experiments conducted on four benchmarks demonstrate superior results than the state-of-the-art video object segmentation techniques.
Keyframe Selection Via Deep Reinforcement Learning for Skeleton-Based Gesture Recognition

Skeleton-based gesture recognition has attracted extensive attention and has made great progress. However, mainstream methods generally treat all frames as equally important, which may limit performance, especially when dealing with high inter-class variance in gesture. To tackle this issue, we propose an approach that models a Markov decision process to identify keyframes while discarding irrelevant ones. This article proposes a deep reinforcement learning double-feature double-motion network comprising two main components: a baseline gesture recognition model and a frame selection network. These two components mutually influence each other, resulting in enhanced overall performance. Following the evaluation of the SHREC-17 and F-PHAB datasets, our proposed method demonstrates superior performance.
DIC measurement method based on binocular stereo vision for image 3D displacement detection

The deformation detection of large machinery is usually achieved using three-dimensional displacement measurement. Binocular stereo vision measurement technology, as a commonly used digital image correlation method, has received widespread attention in the academic community. Binocular stereo vision achieves the goal of three-dimensional displacement measurement by simulating the working mode of the human eyes, but the measurement is easily affected by light refraction. Based on this, the study introduces particle swarm optimization algorithm for target displacement measurement on Canon imaging dataset, and introduces backpropagation neural network for mutation processing of particles in particle swarm algorithm to generate fusion algorithm. It combines the four coordinate systems of world, pixel, physics, and camera to establish connections. Taking into account environmental factors and lens errors, the camera parameters and deformation coefficients were revised by shooting a black and white checkerboard. Finally, the study first conducted error analysis on binocular stereo vision technology in three dimensions, and the relative error remained stable at 1 % within about 60 seconds. At the same time, three algorithms, including the spotted hyena algorithm, were introduced to conduct performance comparison experiments using particle swarm optimization and backpropagation network algorithms. The experiment shows that the three-dimensional error of the fusion algorithm gradually stabilizes within the range of [–0.5 %, 0.5 %] over time, while the two-dimensional error generally hovers around 0 value. Its performance is significantly superior to other algorithms, so the binocular stereo vision of this fusion algorithm can achieve good measurement results.
An Automatic Detection System for Fake Japanese Shopping Sites Using fastText and LightGBM

In recent years, the number of fake shopping sites that scam people out of their money or steal their personal information has skyrocketed. To address this problem, Japanese law enforcement agencies such as the police have been detecting fake shopping sites through information provided by a third party and by conducting manual investigations. However, this current approach is quite inefficient. Despite a number of recent studies that use machine learning to detect fake sites, there is still no system for automatically detecting fake shopping sites. Therefore, in this study, we developed an automatic detection system for fake shopping sites to solve the problem of detection inefficiency faced by law enforcement agencies in Japan. The proposed system successfully identified an average of 118,000 target URLs per day from the list of newly registered domains and collected an average of 51,000 sets of HTML data. Also, it was able to determine with 98.5% accuracy using machine learning whether the collected data were fake shopping sites or not. Since this system was able to meet the time requirements for actual operation, we developed an automatic detection system for fake Japanese shopping sites.
Vision-Based Uncertainty-Aware Motion Planning Based on Probabilistic Semantic Segmentation

For safe operation, a robot must be able to avoid collisions in uncertain environments. Existing approaches for motion planning under uncertainties often assume parametric obstacle representations and Gaussian uncertainty, which can be inaccurate. While visual perception can deliver a more accurate representation of the environment, its use for safe motion planning is limited by the inherent miscalibration of neural networks and the challenge of obtaining adequate datasets. To address these limitations, we propose to employ ensembles of deep semantic segmentation networks trained with massively augmented datasets to ensure reliable probabilistic occupancy information. To avoid conservatism during motion planning, we directly employ the probabilistic perception in a scenario-based path planning approach. A velocity scheduling scheme is applied to the path to ensure a safe motion despite tracking inaccuracies. We demonstrate the effectiveness of the massive data augmentation in combination with deep ensembles and the proposed scenario-based planning approach in comparisons to state-of-the-art methods and validate our framework in an experiment with a human hand as an obstacle.
Cluster-N-Engage: A New Framework for Measuring User Engagement of Website With User Navigational Behavior

User engagement is a part of user experience that measures how attracted the users are in using a certain product or services. The level of user attention while navigating a website is a key factor in determining its effectiveness. Web usage mining can be performed on the weblog of a website to extract user navigational behavior and gain valuable insights into the useful activities of the users. The user navigational behavior can then be further evaluated to measure the user engagement of the website. This paper presents a new framework that uses user navigational behavior extracted from web usage mining to measure the engagement level of the users on the website. In this proposed framework, web session clustering is first performed on the pre-processed weblog to group similar user access patterns from web sessions. After that, four proposed engagement metrics, which are the hourly activeness, the hourly traffic, the daily activeness, and the daily traffic are calculated for each cluster to determine the engagement level of the users on the website. According to the engagement level determined by the user engagement metrics, an attention score is formulated to show how attracted the users in the session cluster are to the website. In this paper, two weblogs from different websites are used to measure their user engagement with the Cluster-N-Engage framework. The framework shows that attention score can assess the website’s effectiveness in achieving its objectives.
Dynamic and Static Enhanced BIRCH for Functional Data Clustering

Accurate and efficient clustering of large-scale functional data is of utmost importance in the era of big data. However, the current research falls short in fully considering the differentiability inherent in functional data. To tackle this significant challenge, we propose a novel method, namely Dynamic and Static Enhanced-BIRCH (DSE-BIRCH), which incorporates both the constant and derívate features to simultaneously measure the static and dynamic distances between functional samples. To this end, a novel matrix factorization-based approach is introduced to transform constant features, extracted through principal component analysis, into derivative features. Subsequently, these two sets of features are fused to form global clustering features with different weighting coefficients are assigned to each of them, reflecting their respective importance. Finally, an enhanced BIRCH algorithm is employed to handle both static and dynamic constraints, enabling hierarchical clustering from a more comprehensive perspective. The mathematical definition of the algorithm is rigorously provided. The superior empirical performance of our method on publicly available datasets and simulated datasets fully demonstrates its effective capture of dynamic information and its capability to achieve accurate clustering on real-world data. Further experiments involving noise and complexity attest to the algorithm’s robustness and efficiency, highlighting its broad potential for applications in various complex scenarios involving large-scale functional data.
Graph Learning in Robotics: A Survey

Deep neural networks for graphs have emerged as a powerful tool for learning on complex non-euclidean data, which is becoming increasingly common for a variety of different applications. Yet, although their potential has been widely recognised in the machine learning community, graph learning is largely unexplored for downstream tasks such as robotics applications. To fully unlock their potential, hence, we propose a review of graph neural architectures from a robotics perspective. The paper covers the fundamentals of graph-based models, including their architecture, training procedures, and applications. It also discusses recent advancements and challenges that arise in applied settings, related for example to the integration of perception, decision-making, and control. Finally, the paper provides an extensive review of various robotic applications that benefit from learning on graph structures, such as bodies and contacts modelling, robotic manipulation, action recognition, fleet motion planning, and many more. This survey aims to provide readers with a thorough understanding of the capabilities and limitations of graph neural architectures in robotics, and to highlight potential avenues for future research.
Global-View and Speaker-Aware Emotion Cause Extraction in Conversations

Emotion cause extraction in conversations, the task of recognizing and extracting the causes behind the emotions in a conversation, is a new and under-explored task. It was previously treated as an utterance-level task, that can only extract cause of one emotion from one utterance at a time and is difficult to model the correlation between different emotions and causes in the conversation. The role of speakers was also not fully utilized in the previous methods. In this paper, we introduce a global-view and speaker-aware conversational emotion cause extraction framework. It can fully model the interaction between utterances and emotions in the conversation and simultaneously extract all the causes corresponding to all emotions or one given emotion in a conversation, and can be applied to both real-time and non-real-time task settings. We further propose a Speaker-aware Couple-Decoder Module and a Speaker-Emotion Graph Attention Network, to better model the role of speakers in the conversation. The experimental results prove our approach's advantages in both emotion cause extraction performance and computational efficiency.
Chiral‐Quantum‐Optics‐Based Supervised Learning

Abstract Based on the excellent control of single photons realized by atom‐photon‐chiral couplings,a novel quantum‐optics scheme for supervised learning is proposed. The single‐photon rotating and phase‐shift operations, which can be controlled by another single photon, are realized by proper atom‐photon‐chiral couplings. Then, an algorithm to perform the supervised learning tasks, composed by integrating the realized gates, and implemented by the tunable gate parameters, is realized.
Differential Privacy on Edge Computing

This paper presents an overview of privacy protection, with a focus on differential privacy (DP), from the perspective of edge computing. It explores the application of DP in various associative analysis techniques, including heavy hitter mining, frequent itemset mining, and association rules mining, within the context of edge computing. The paper also highlights the current challenges and future research directions in this area, including differentially private hybrid models and federated learning. By examining the intersection of privacy protection and edge computing, this paper provides insights into the application of DP and its potential for preserving privacy in associative analysis tasks within edge computing environments.
Orientation Detector for Ship Targets in SAR Images Based on Semantic Flow Feature Alignment and Gaussian Label Matching

To address the challenges in synthetic aperture radar (SAR) ship target detection, this paper proposes a SAR ship small target orientation detector named FADet based on semantic flow feature alignment and Gaussian label matching. First, to solve the feature misalignment problem caused by feature extraction downsampling and residual connections, we introduce the FAM module into FPN, which automatically aligns deep and shallow fine-grained semantics information through semantic flow alignment. Second, due to the scattering characteristics of SAR imaging, the boundary information of SAR targets is not obvious, we combining attention mechanisms design an adaptive boundary enhancement module to enhance the target boundary information. Finally, to solve the problem that small targets have difficulty matching positive samples under IOU rules, we design a label matching strategy based on Gaussian distribution. This matching strategy can still learn regression information when two boxes do not intersect. Based on the SSDD+ and RSDD-SAR datasets, the effectiveness of each module in FADet is verified by ablation experiments. Additionally, through comparison experiments with the latest orientation detection methods, FADet achieves a good compromise between accuracy and inference speed. The AP50 and AP75 on the SSDD+ and RSDD-SAR is 91.03, 59.94 and 90.78, 59.91 respectively, and the FPS is 19.83.
Infrared Thermal Image Instance Segmentation Method for Power Substation Equipment Based on Visual Feature Reasoning

Accurate infrared thermal image instance segmentation of substation equipment is a prerequisite for intelligent analysis of its temperature status. To address the issues of low accuracy and false detection in existing substation instance segmentation methods, we propose a visual feature reasoning-based substation infrared thermal image instance segmentation method to compensate for the limitations of deep learning methods and improve the instance segmentation accuracy. We propose utilizing distinctive visual features as a priori knowledge for three types of substation equipment and construct a two-branch instance segmentation model (FR-SOLOv2) based on power domain expertise and visual feature reasoning. FR-SOLOv2 comprises distinctive visual feature extraction and reasoning network (FR) as well as a substation equipment image segmentation network (SOLOv2). FR combines power domain knowledge and visual feature reasoning to provide accurate localization and classification information for each substation device. SOLOv2 accomplishes segmenting substation devices in infrared images based on residual networks and feature fusion pyramids. The test results well demonstrate the superiority of our model for instance segmentation of the infrared image of substation equipment. Additionally, FR-SOLOv2 demonstrates an average accuracy of 83.18% on the substation equipment infrared image dataset, a significant improvement of 13.5% compared to the baseline model. The method relying on prior knowledge for visual feature reasoning on deep learning methods also presents a new approach to substation equipment image segmentation.
Co-occurrence Object Detection of the Transmission Lines Based on the Cross-Domain Interactive Feature Enhancement

The inspection of the transmission lines is of great significance for the safe and stable transportation of electric power. However, the small-sized and easily confused objects of transmission lines are always difficult to detect accurately. In this article, a new network based on the combination of the cross-domain interactive (CDI) enhanced pyramid network and the knowledge graph relational reasoning (KGRR) network is proposed to improve the detection accuracy. First, the self-attention is applied to the feature layer fusion process, and the object features located on different feature layers are enhanced by calculating the similarity of the features of the adjacent layers. Secondly, constructing the knowledge graph of the fitting and using Graph Convolution Network (GCN) to adjust the output of the raw object detector. The experimental results show that the method proposed in this article has greatly improved the detection accuracy of the small-sized fittings of the transmission line, the mean Average Precision (mAP) of this proposed method is improved by 1.9% compared with the baseline algorithm.