Reviewerly/exHarmony huggingface dataset:
ARFace: Attention-Aware and Regularization for Face Recognition With Reinforcement Learning

Different face regions have different contributions to recognition. Especially in the wild environment, the difference of contributions will be further amplified due to a lot of interference. Based on this, this paper proposes an attention-aware face recognition method based on a deep convolutional neural network and reinforcement learning. The proposed method composes of an Attention-Net and a Feature-net. The Attention-Net is used to select patches in the input face image according to the facial landmarks and trained with reinforcement learning to maximize the recognition accuracy. The Feature-net is used for extracting discriminative embedding features. In addition, a regularization method has also been introduced. The mask of the input layer is also applied to the intermediate feature maps, which is an approximation to train a series of models for different face patches and provide a combined model. Our method achieves satisfactory recognition performance on its application to the public prevailing face verification database.
Deep anomaly detection with self-supervised learning and adversarial training

Deep anomaly detection, which utilizes neural networks to discover anomalies, is a vital research topic in pattern recognition. With the burgeoning of inference mechanism, inference-based methods show the promising performance. However, inference-based methods have two limitations: (1) they use an adversarial training way to learn data features. Such training way fails to learn task-specific features which can be conducive to capture the difference between normal and anomaly data. (2) The structure of detection network cannot capture the marginal distributions of normal data and corresponding features, which influences on the performance of anomaly detection. To overcome these limitations, this paper proposes a deep adversarial anomaly detection (DAAD) method. Specifically, an auxiliary task with self-supervised learning is first designed to learn task-specific features. Then a deep adversarial training (DAT) model is constructed to capture marginal distributions of normal data in different spaces. In addition, a majority voting strategy is applied to obtain reliable detection results. Experimental results on image and sequence datasets show that proposed method performs significantly better than many strong baselines.
Wearable sensor-based pattern mining for human activity recognition: deep learning approach

Purpose This paper aims to deal with the human activity recognition using human gait pattern. The paper has considered the experiment results of seven different activities: normal walk, jogging, walking on toe, walking on heel, upstairs, downstairs and sit-ups. Design/methodology/approach In this current research, the data is collected for different activities using tri-axial inertial measurement unit (IMU) sensor enabled with three-axis accelerometer to capture the spatial data, three-axis gyroscopes to capture the orientation around axis and 3° magnetometer. It was wirelessly connected to the receiver. The IMU sensor is placed at the centre of mass position of each subject. The data is collected for 30 subjects including 11 females and 19 males of different age groups between 10 and 45 years. The captured data is pre-processed using different filters and cubic spline techniques. After processing, the data are labelled into seven activities. For data acquisition, a Python-based GUI has been designed to analyse and display the processed data. The data is further classified using four different deep learning model: deep neural network, bidirectional-long short-term memory (BLSTM), convolution neural network (CNN) and CNN-LSTM. The model classification accuracy of different classifiers is reported to be 58%, 84%, 86% and 90%. Findings The activities recognition using gait was obtained in an open environment. All data is collected using an IMU sensor enabled with gyroscope, accelerometer and magnetometer in both offline and real-time activity recognition using gait. Both sensors showed their usefulness in empirical capability to capture a precised data during all seven activities. The inverse kinematics algorithm is solved to calculate the joint angle from spatial data for all six joints hip, knee, ankle of left and right leg. Practical implications This work helps to recognize the walking activity using gait pattern analysis. Further, it helps to understand the different joint angle patterns during different activities. A system is designed for real-time analysis of human walking activity using gait. A standalone real-time system has been designed and realized for analysis of these seven different activities. Originality/value The data is collected through IMU sensors for seven activities with equal timestamp without noise and data loss using wirelessly. The setup is useful for the data collection in an open environment outside the laboratory environment for activity recognition. The paper also presents the analysis of all seven different activity trajectories patterns.
Blockchain state-of-the-art: architecture, use cases, consensus, challenges and opportunities

Blockchain is a chain of blocks where each block contains a set of transactions that are digitally signed by its verifier and stored across the distributed network so that all the legitimate stakeholders can access/verify them. Due to the attributes of Blockchain such as decentralization, immutability, auditability, transparency, and cryptographic security, it offers various benefits to different domains such as cryptocurrency, financial sectors, private/public segments, insurance, healthcare, supply chain management, Internet of Things, etc. However, the technology is in its early stage and still, there is a range of concerns that are yet to be addressed before its wide adoption. Through this paper, we intend to cover extensive study on the Blockchain that includes taxonomy, application/use-cases, consensus mechanisms, prospective research, future directions, and related technologies. This paper also aims to discuss the opportunities, benefits, and challenges of Blockchain technology and to assist the research community in understanding the same.
FACIAL: Synthesizing Dynamic Talking Face with Implicit Attribute Learning

In this paper, we propose a talking face generation method that takes an audio signal as input and a short target video clip as reference, and synthesizes a photo-realistic video of the target face with natural lip motions, head poses, and eye blinks that are in-sync with the input audio signal. We note that the synthetic face attributes include not only explicit ones such as lip motions that have high correlations with speech, but also implicit ones such as head poses and eye blinks that have only weak correlation with the input audio. To model such complicated relationships among different face attributes with input audio, we propose a FACe Implicit Attribute Learning Generative Adversarial Network (FACIAL-GAN), which integrates the phonetics-aware, context-aware, and identity-aware information to synthesize the 3D face animation with realistic motions of lips, head poses, and eye blinks. Then, our Rendering-to-Video network takes the rendered face images and the attention map of eye blinks as input to generate the photorealistic output video frames. Experimental results and user studies show our method can generate realistic talking face videos with not only synchronized lip motions, but also natural head movements and eye blinks, with better qualities than the results of state-of-the-art methods.
An evidence of cognitive benefits from immersive design review: Comparing three-dimensional perception and presence between immersive and non-immersive virtual environments

The adoption of VR solutions in design review has grown rapidly in the construction sector worldwide. However, the extent to which these enhance professionals' cognitive capabilities in the task – an essential step in determining their validity and effectiveness – is still unclear. Through a controlled experiment, this study sought to quantitatively verify the ability of an immersive VR system to provide users with an enhanced 3D perception of the virtual model and greater levels of presence compared to a non-immersive one. Controlling for individual factors and order effects, findings indicate that the immersive system improves the 3D perception of the model and provides more immersive experiences, which is expected to benefit collaborative design review and increase productivity.
Spatiotemporal Multimodal Learning With 3D CNNs for Video Action Recognition

Extracting effective spatial-temporal information is significantly important for video-based action recognition. Recently 3D convolutional neural networks (3D CNNs) that could simultaneously encode spatial and temporal dynamics in videos have made considerable progress in action recognition. However, almost all existing 3D CNN-based methods recognize human actions only using RGB videos. The single modality may limit the performance capacity of 3D networks. In this paper, we extend 3D CNN to depth and pose data besides RGB data to evaluate its capacity for spatiotemporal multimodal learning for video action recognition. We propose a novel multimodal two-stream 3D network framework, which can exploit complementary multimodal information to improve the recognition performance. Specifically, we first construct two discriminative video representations under depth and pose data modalities respectively, referred as depth residual dynamic image sequence (DRDIS) and pose estimation map sequence (PEMS). DRDIS captures spatial-temporal evolution of actions in depth videos by progressively aggregating the local motion information. PEMS eliminates the interference of cluttered backgrounds and describes the spatial configuration of body parts intuitively. The multimodal two-stream 3D CNN deals with two separate data streams to learn spatiotemporal features from DRDIS and PEMS representations. Finally, the classification scores from two streams are fused for action recognition. We conduct extensive experiments on four challenging action recognition datasets. The experimental results verify the effectiveness and superiority of our proposed method.
Adversarial Attacks and Defenses for Deep-Learning-Based Unmanned Aerial Vehicles

The introduction of deep learning (DL) technology can improve the performance of cyber–physical systems (CPSs) in many ways. However, this also brings new security issues. To tackle these challenges, this article explores the vulnerabilities of DL-based unmanned aerial vehicles (UAVs), which are typical CPSs. Although many research works have been reported previously on adversarial attacks of DL models, only few of them are concerned about safety-critical CPSs, especially regression models in such systems. In this article, we analyze the problem of adversarial attacks against DL-based UAVs and propose two adversarial attack methods against regression models in UAVs. The experiments demonstrate that the proposed nontargeted and targeted attack methods both can craft imperceptible adversarial images and pose a considerable threat to the navigation and control of UAVs. To address this problem, adversarial training and defensive distillation methods are further investigated and evaluated, increasing the robustness of DL models in UAVs. To our knowledge, this is the first study on adversarial attacks and defenses against DL-based UAVs, which calls for more attention to the security and safety of such safety-critical applications.
A critical review of text-based research in construction: Data source, analysis method, and implications

The advancement of natural language processing and text mining techniques facilitate automatic non-trivial pattern extraction and knowledge discovery from text data. However, text-based research has received less attention compared to image- and sensor-based research in the construction industry. Hence, this paper performs a comprehensive review to understand the current state and future insights of text analytics focusing on the data source and analysis method. This study identifies various kinds of text data sources from project documents as well as open data in the websites. In addition, the review finds that the ontology- and rule-based approach has been dominant, at the same time, recent research has attempted to apply the state-of-the-art machine learning methods. It is envisioned that there are potential advancements in construction engineering and management based on the latest text analysis methods along with the enriched data by the digital transformation.
From General to Specific: Informative Scene Graph Generation via Balance Adjustment

The scene graph generation (SGG) task aims to detect visual relationship triplets, i.e., subject, predicate, object, in an image, providing a structural vision layout for scene understanding. However, current models are stuck in common predicates, e.g., "on" and "at", rather than informative ones, e.g., "standing on" and "looking at", resulting in the loss of precise information and overall performance. If a model only uses "stone on road" rather than "blocking" to describe an image, it is easy to misunderstand the scene. We argue that this phenomenon is caused by two key imbalances between informative predicates and common ones, i.e., semantic space level imbalance and training sample level imbalance. To tackle this problem, we propose BA-SGG, a simple yet effective SGG framework based on balance adjustment but not the conventional distribution fitting. It integrates two components: Semantic Adjustment (SA) and Balanced Predicate Learning (BPL), respectively for adjusting these imbalances. Benefited from the model-agnostic process, our method is easily applied to the state-of-the-art SGG models and significantly improves the SGG performance. Our method achieves 14.3%, 8.0%, and 6.1% higher Mean Recall (mR) than that of the Transformer model at three scene graph generation sub-tasks on Visual Genome, respectively. Codes are publicly available <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
Pseudo-mask Matters in Weakly-supervised Semantic Segmentation

Most weakly supervised semantic segmentation (WSSS) methods follow the pipeline that generates pseudo-masks initially and trains the segmentation model with the pseudo-masks in fully supervised manner after. However, we find some matters related to the pseudo-masks, including high quality pseudo-masks generation from class activation maps (CAMs), and training with noisy pseudo-mask supervision. For these matters, we propose the following designs to push the performance to new state-of-art: (i) Coefficient of Variation Smoothing to smooth the CAMs adaptively; (ii) Proportional Pseudo-mask Generation to project the expanded CAMs to pseudo-mask based on a new metric indicating the importance of each class on each location, instead of the scores trained from binary classifiers. (iii) Pretended Under-Fitting strategy to suppress the influence of noise in pseudo-mask; (iv) Cyclic Pseudo-mask to boost the pseudo-masks during training of fully supervised semantic segmentation (FSSS). Experiments based on our methods achieve new state-of-art results on two changeling weakly supervised semantic segmentation datasets, pushing the mIoU to 70.0% and 40.2% on PAS-CAL VOC 2012 and MS COCO 2014 respectively. Codes including segmentation framework are released at https://github.com/Eli-YiLi/PMM
Toward Proactive Human–Robot Collaborative Assembly: A Multimodal Transfer-Learning-Enabled Action Prediction Approach

Human–robot collaborative assembly (HRCA) is vital for achieving high-level flexible automation for mass personalization in today’s smart factories. However, existing works in both industry and academia mainly focus on the adaptive robot planning, while seldom consider human operator’s intentions in advance. Hence, it hinders the HRCA transition toward a proactive manner. To overcome the bottleneck, this article proposes a multimodal transfer-learning-enabled action prediction approach, serving as the prerequisite to ensure the proactive HRCA. First, a multimodal intelligence-based action recognition approach is proposed to predict ongoing human actions by leveraging the visual stream and skeleton stream with short-time input frames. Second, a transfer-learning-enabled model is adapted to transfer learnt knowledge from daily activities to industrial assembly operations rapidly for online operator intention analysis. Third, a dynamic decision-making mechanism, including robotic decision and motion control, is described to allow mobile robots to assist operators in a proactive manner. Finally, an aircraft bracket assembly task is demonstrated in the laboratory environment, and the comparative study result shows that the proposed approach outperforms other state-of-the-art ones for efficient action prediction.
Joint Contrast Enhancement and Exposure Fusion for Real-World Image Dehazing

Due to the complexity of real environment and potential defects of current simulation datasets, either prior-based or deep learning-based single image dehazing methods may not work well in certain scenarios. In this work, we propose an efficient joint contrast enhancement and exposure fusion (CEEF) framework to formulate image dehazing task as a problem of enhancing local visibility and global contrast. In the contrast enhancement stage, several intermediate images are generated through two pre-processing steps. Specifically, gamma correction (GC) is used to adjust local visibility of an input hazy image. To address the issue of applying adaptive histogram equalization (AHE) to each color channel independently, we introduce color-preserving AHE (CP-AHE) to improve global contrast of the input hazy image. In the fusion stage, we develop a fast structural patch decomposition-based fusion strategy with an adaptive kernel size to fuse the inputs obtained by GC and CP-AHE. Extensive experiments on the real-world datasets demonstrate superiority of the proposed method to state-of-the-art methods in terms of visual and quantitative evaluation. Particularly for nighttime hazy scenes, our approach is shown to retain fine details and reduce color artifacts against three latest nighttime defogging methods. Moreover, we discuss potential applications of our CP-AHE in low-light enhancement and image editing.
HSWOA: An ensemble of hunger games search and whale optimization algorithm for global optimization

The search for food stimulated by hunger is a common phenomenon in the animal world. Mimicking the concept, recently, an optimization algorithm Hunger Games Search (HGS) has been proposed for global optimization. On the other side, the Whale Optimization Algorithm (WOA) is a commonly utilized nature-inspired algorithm portrayed by a straightforward construction with easy parameters imitating the hunting behavior of humpback whales. However, due to minimum exploration of the search space, WOA has a high chance of trapping into local solutions, and more exploitation leads it towards premature convergence. The concept of hunger from HGS is merged with the food searching techniques of the whale to lessen the inherent drawbacks of WOA. Two weights of HGS are adaptively designed for every whale using the respective hunger level for balancing search strategies. Performance verification of the proposed hunger search-based whale optimization algorithm (HSWOA) is done by comparing it with 10 state-of-the-art algorithms, including three very recently developed algorithms on 30 classical benchmark functions. Comparison with some basic algorithms, recently modified algorithms, and WOA variants is performed using IEEE CEC 2019 function set. Statistical performance of the proposed algorithm is verified with Friedman's test, boxplot analysis, and Nemenyi multiple comparison test. The operating speed of the algorithm is determined and tested with complexity analysis and convergence analysis. Finally, seven real-world engineering problems are solved and compared with a list of metaheuristic algorithms. Numerical and statistical performance comparison with state-of-the-art algorithms confirms the efficacy of the newly designed algorithm.
Current Time Series Anomaly Detection Benchmarks are Flawed and are Creating the Illusion of Progress

Time series anomaly detection has been a perennially important topic in data science, with papers dating back to the 1950s. However, in recent years there has been an explosion of interest in this topic, much of it driven by the success of deep learning in other domains and for other time series tasks. Most of these papers test on one or more of a handful of popular benchmark datasets, created by Yahoo, Numenta, NASA, etc. In this work we make a surprising claim. The majority of the individual exemplars in these datasets suffer from one or more of four flaws. Because of these four flaws, we believe that many published comparisons of anomaly detection algorithms may be unreliable, and more importantly, much of the apparent progress in recent years may be illusionary. In addition to demonstrating these claims, with this paper we introduce the UCR Time Series Anomaly Archive. We believe that this resource will perform a similar role as the UCR Time Series Classification Archive, by providing the community with a benchmark that allows meaningful comparisons between approaches and a meaningful gauge of overall progress.
Hierarchical Memory Matching Network for Video Object Segmentation

We present Hierarchical Memory Matching Network (HMMN) for semi-supervised video object segmentation. Based on a recent memory-based method [33], we propose two advanced memory read modules that enable us to perform memory reading in multiple scales while exploiting temporal smoothness. We first propose a kernel guided memory matching module that replaces the non-local dense memory read, commonly adopted in previous memory-based methods. The module imposes the temporal smoothness constraint in the memory read, leading to accurate memory retrieval. More importantly, we introduce a hierarchical memory matching scheme and propose a top-k guided memory matching module in which memory read on a fine-scale is guided by that on a coarse-scale. With the module, we perform memory read in multiple scales efficiently and leverage both high-level semantic and low-level fine-grained memory features to predict detailed object masks. Our network achieves state-of-the-art performance on the validation sets of DAVIS 2016/2017 (90.8% and 84.7%) and YouTube-VOS 2018/2019 (82.6% and 82.5%), and test-dev set of DAVIS 2017 (78.6%). The source code and model are available online: https://github.com/Hongje/HMMN.
Progressively Normalized Self-Attention Network for Video Polyp Segmentation

Existing video polyp segmentation(VPS) models typically employ convolutional neural networks (CNNs) to extract features. However, due to their limited receptive fields, CNNs cannot fully exploit the global temporal and spatial information in successive video frames, resulting in false positive segmentation results. In this paper, we propose the novel PNS-Net (Progressively Normalized Self-attention Network), which can efficiently learn representations from polyp videos with real-time speed (\(\sim \)140fps) on a single RTX 2080 GPU and no post-processing. Our PNS-Net is based solely on a basic normalized self-attention block, equipping with recurrence and CNNs entirely. Experiments on challenging VPS datasets demonstrate that the proposed PNS-Net achieves state-of-the-art performance. We also conduct extensive experiments to study the effectiveness of the channel split, soft-attention, and progressive learning strategy. We find that our PNS-Net works well under different settings, making it a promising solution to the VPS task.