Reviewerly/exHarmony huggingface dataset:
Dynamic GCN: Context-enriched Topology Learning for Skeleton-based Action Recognition

raph Convolutional Networks (GCNs) have attracted increasing interests for the task of skeleton-based action recognition. The key lies in the design of the graph structure, which encodes skeleton topology information. In this paper, we propose Dynamic GCN, in which a novel convolutional neural network named Context-encoding Network (CeN) is introduced to learn skeleton topology automatically. In particular, when learning the dependency between two joints, contextual features from the rest joints are incorporated in a global manner. CeN is extremely lightweight yet effective, and can be embedded into a graph convolutional layer. By stacking multiple CeN-enabled graph convolutional layers, we build Dynamic GCN. Notably, as a merit of CeN, dynamic graph topologies are constructed for different input samples as well as graph convolutional layers of various depths. Besides, three alternative context modeling architectures are well explored, which may serve as a guideline for future research on graph topology learning. CeN brings only ~7% extra FLOPs for the baseline model, and Dynamic GCN achieves better performance with 2x ~4x fewer FLOPs than existing methods. By further combining static physical body connections and motion modalities, we achieve state-of-the-art performance on three large-scale benchmarks, namely NTU-RGB+D, NTU-RGB+D 120 and Skeleton-Kinetics.
Multi-scale Positive Sample Refinement for Few-Shot Object Detection

Few-shot object detection (FSOD) helps detectors adapt to unseen classes with few training instances, and is useful when manual annotation is time-consuming or data acquisition is limited. Unlike previous attempts that exploit few-shot classification techniques to facilitate FSOD, this work highlights the necessity of handling the problem of scale variations, which is challenging due to the unique sample distribution. To this end, we propose a Multi-scale Positive Sample Refinement (MPSR) approach to enrich object scales in FSOD. It generates multi-scale positive samples as object pyramids and refines the prediction at various scales. We demonstrate its advantage by integrating it as an auxiliary branch to the popular architecture of Faster R-CNN with FPN, delivering a strong FSOD solution. Several experiments are conducted on PASCAL VOC and MS COCO, and the proposed approach achieves state of the art results and significantly outperforms other counterparts, which shows its effectiveness. Code is available at https://github.com/jiaxi-wu/MPSR.
Effective template update mechanism in visual tracking with background clutter

Today, artificial intelligence is everywhere in people’s daily lives. Visual tracking, which is used to identify and continuously track specific targets, is an important research domain in the study of artificial intelligence. However, current visual tracking methods are not accurate enough for object tracking with background clutter, which can easily lead to tracking failures. Therefore, in this paper, in order to solve the problem of tracking failure in clutter background, we propose a template update mechanism to improve the accuracy of visual tracking. First, an original template is saved when the background clutter is detected. During background clutter, we use both the original template and the current template at the location estimated by the optical flow and choose better one. Next, the original template is reused after the background clutter is ended. Finally, the proposed mechanism is used both in the KCF and BACF algorithm to verify the effectiveness of the mechanism. With experiments on the OTB2015 dataset, results show that the proposed mechanism has improved accuracy and success rate of the two baseline algorithms. Meanwhile, in state-of-the-art algorithms, the algorithm using the proposed mechanism also has excellent tracking performance. In addition, this method also has strong tracking robustness and adaptation capability to sequential learning for video data.
Adversarial Spatio-Temporal Learning for Video Deblurring

Camera shake or target movement often leads to undesired blur effects in videos captured by a hand-held camera. Despite significant efforts having been devoted to video-deblur research, two major challenges remain: 1) how to model the spatio-temporal characteristics across both the spatial domain (i.e., image plane) and the temporal domain (i.e., neighboring frames) and 2) how to restore sharp image details with respect to the conventionally adopted metric of pixel-wise errors. In this paper, to address the first challenge, we propose a deblurring network (DBLRNet) for spatial-temporal learning by applying a 3D convolution to both the spatial and temporal domains. Our DBLRNet is able to capture jointly spatial and temporal information encoded in neighboring frames, which directly contributes to the improved video deblur performance. To tackle the second challenge, we leverage the developed DBLRNet as a generator in the generative adversarial network (GAN) architecture and employ a content loss in addition to an adversarial loss for efficient adversarial training. The developed network, which we name as deblurring GAN, is tested on two standard benchmarks and achieves the state-of-the-art performance.
Blockchain-Enabled Secure Data Sharing Scheme in Mobile-Edge Computing: An Asynchronous Advantage Actor–Critic Learning Approach

Mobile-edge computing (MEC) plays a significant role in enabling diverse service applications by implementing efficient data sharing. However, the unique characteristics of MEC also bring data privacy and security problem, which impedes the development of MEC. Blockchain is viewed as a promising technology to guarantee the security and traceability of data sharing. Nonetheless, how to integrate blockchain into MEC system is quite challenging because of dynamic characteristics of channel conditions and network loads. To this end, we propose a secure data sharing scheme in the blockchain-enabled MEC system using an asynchronous learning approach in this article. First, a blockchain-enabled secure data sharing framework in the MEC system is presented. Then, we present an adaptive privacy-preserving mechanism according to available system resources and privacy demands of users. Next, an optimization problem of secure data sharing is formulated in the blockchain-enabled MEC system with the aim to maximize the system performance with respect to the decreased energy consumption of MEC system and the increased throughput of blockchain system. Especially, an asynchronous learning approach is employed to solve the formulated problem. The numerical results demonstrate the superiority of our proposed secure data sharing scheme when compared with some popular benchmark algorithms in terms of average throughput, average energy consumption, and reward.
FFB6D: A Full Flow Bidirectional Fusion Network for 6D Pose Estimation

In this work, we present FFB6D, a Full Flow Bidirectional fusion network designed for 6D pose estimation from a single RGBD image. Our key insight is that appearance information in the RGB image and geometry information from the depth image are two complementary data sources, and it still remains unknown how to fully leverage them. Towards this end, we propose FFB6D, which learns to combine appearance and geometry information for representation learning as well as output representation selection. Specifically, at the representation learning stage, we build bidirectional fusion modules in the full flow of the two networks, where fusion is applied to each encoding and decoding layer. In this way, the two networks can leverage local and global complementary in-formation from the other one to obtain better representations. Moreover, at the output representation stage, we designed a simple but effective 3D keypoints selection algorithm considering the texture and geometry information of objects, which simplifies keypoint localization for precise pose estimation. Experimental results show that our method outperforms the state-of-the-art by large margins on several benchmarks. Code and video are available at https://github.com/ethnhe/FFB6D.git.
Recent Advances on Federated Learning for Cybersecurity and Cybersecurity for Federated Learning for Internet of Things

Decentralized paradigm in the field of cybersecurity and machine learning (ML) for the emerging Internet of Things (IoT) has gained a lot of attention from the government, academia, and industries in recent years. Federated cybersecurity (FC) is regarded as a revolutionary concept to make the IoT safer and more efficient in the future. This emerging concept has the potential of detecting security threats, taking countermeasures, and limiting the spreading of threats over the IoT network system efficiently. An objective of cybersecurity is achieved by forming the federation of the learned and shared model on top of various participants. Federated learning (FL), which is regarded as a privacy-aware ML model, is particularly useful to secure the vulnerable IoT environment. In this article, we start with background and comparison of centralized learning, distributed on-site learning, and FL, which is then followed by a survey of the application of FL to cybersecurity for IoT. This survey primarily focuses on the security aspect but it also discusses several approaches that address the performance issues (e.g., accuracy, latency, resource constraint, and others) associated with FL, which may impact the security and overall performance of the IoT. To anticipate the future evolution of this new paradigm, we discuss the main ongoing research efforts, challenges, and research trends in this area. With this article, readers can have a more thorough understanding of FL for cybersecurity as well as cybersecurity for FL, different security attacks, and countermeasures.
Baking Neural Radiance Fields for Real-Time View Synthesis

Neural volumetric representations such as Neural Radiance Fields (NeRF) have emerged as a compelling technique for learning to represent 3D scenes from images with the goal of rendering photorealistic images of the scene from unobserved viewpoints. However, NeRF’s computational requirements are prohibitive for real-time applications: rendering views from a trained NeRF requires querying a multilayer perceptron (MLP) hundreds of times per ray. We present a method to train a NeRF, then precompute and store (i.e. "bake") it as a novel representation called a Sparse Neural Radiance Grid (SNeRG) that enables real-time rendering on commodity hardware. To achieve this, we introduce 1) a reformulation of NeRF’s architecture, and 2) a sparse voxel grid representation with learned feature vectors. The resulting scene representation retains NeRF’s ability to render fine geometric details and view-dependent appearance, is compact (averaging less than 90 MB per scene), and can be rendered in real-time (higher than 30 frames per second on a laptop GPU). Actual screen captures are shown in our video.
Action Recognition With Spatio–Temporal Visual Attention on Skeleton Image Sequences

Action recognition with 3D skeleton sequences became popular due to its speed and robustness. The recently proposed convolutional neural networks (CNNs)-based methods show a good performance in learning spatio-temporal representations for skeleton sequences. Despite the good recognition accuracy achieved by previous CNN-based methods, there existed two problems that potentially limit the performance. First, previous skeleton representations were generated by chaining joints with a fixed order. The corresponding semantic meaning was unclear and the structural information among the joints was lost. Second, previous models did not have an ability to focus on informative joints. The attention mechanism was important for skeleton-based action recognition because different joints contributed unequally toward the correct recognition. To solve these two problems, we proposed a novel CNN-based method for skeleton-based action recognition. We first redesigned the skeleton representations with a depth-first tree traversal order, which enhanced the semantic meaning of skeleton images and better preserved the associated structural information. We then proposed the general two-branch attention architecture that automatically focused on spatio-temporal key stages and filtered out unreliable joint predictions. Based on the proposed general architecture, we designed a global long-sequence attention network with refined branch structures. Furthermore, in order to adjust the kernel's spatio-temporal aspect ratios and better capture long-term dependencies, we proposed a sub-sequence attention network (SSAN) that took sub-image sequences as inputs. We showed that the two-branch attention architecture could be combined with the SSAN to further improve the performance. Our experiment results on the NTU RGB+D data set and the SBU kinetic interaction data set outperformed the state of the art. The model was further validated on noisy estimated poses from the subsets of the UCF101 data set and the kinetics data set.
Provably Secure Fine-Grained Data Access Control Over Multiple Cloud Servers in Mobile Cloud Computing Based Healthcare Applications

Mobile cloud computing (MCC) allows mobile users to have on-demand access to cloud services. A mobile cloud model helps in analyzing the information regarding the patients' records and also in extracting recommendations in healthcare applications. In MCC, a fine-grained level access control of multiserver cloud data is a prerequisite for successful execution of end-users applications. In this paper, we propose a new scheme that provides a combined approach of fine-grained access control over cloud-based multiserver data along with a provably secure mobile user authentication mechanism for the Healthcare Industry 4.0. To the best of our knowledge, the proposed scheme is the first to pursue fine-grained data access control over multiple cloud servers in a MCC environment. The proposed scheme has been validated extensively in different heterogeneous environment where its performance was found good in comparison to other existing schemes.
Tumor Detection in Automated Breast Ultrasound Using 3-D CNN and Prioritized Candidate Aggregation

Automated whole breast ultrasound (ABUS) has been widely used as a screening modality for examination of breast abnormalities. Reviewing hundreds of slices produced by ABUS, however, is time consuming. Therefore, in this paper, a fast and effective computer-aided detection system based on 3-D convolutional neural networks (CNNs) and prioritized candidate aggregation is proposed to accelerate this reviewing. First, an efficient sliding window method is used to extract volumes of interest (VOIs). Then, each VOI is estimated the tumor probability with a 3-D CNN, and VOIs with higher estimated probability are selected as tumor candidates. Since the candidates may overlap each other, a novel scheme is designed to aggregate the overlapped candidates. During the aggregation, candidates are prioritized based on estimated tumor probability to alleviate over-aggregation issue. The relationship between the sizes of VOI and target tumor is optimally exploited to effectively perform each stage of our detection algorithm. On evaluation with a test set of 171 tumors, our method achieved sensitivities of 95% (162/171), 90% (154/171), 85% (145/171), and 80% (137/171) with 14.03, 6.92, 4.91, and 3.62 false positives per patient (with six passes), respectively. In summary, our method is more general and much faster than preliminary works and demonstrates promising results.
Deep Video Dehazing With Semantic Segmentation

Recent research have shown the potential of using convolutional neural networks (CNNs) to accomplish single image dehazing. In this work, we take one step further to explore the possibility of exploiting a network to perform haze removal for videos. Unlike single image dehazing, video based approaches can take advantage of the abundant information that exists across neighboring frames. In this work, assuming that a scene point yields highly correlated transmission values between adjacent video frames, we develop a deep learning solution for video dehazing, where a CNN is trained end-to-end to learn how to accumulate information across frames for transmission estimation. The estimated transmission map is subsequently used to recover a haze-free frame via atmospheric scattering model. In addition, as the semantic information of a scene provides a strong prior for image restoration, we propose to incorporate global semantic priors as input to regularize the transmission maps so that the estimated maps can be smooth in the regions of the same object and only discontinuous across the boundaries of different objects. To train this network, we generate a dataset consisted of synthetic hazy and haze-free videos for supervision based on the NYU depth dataset. We show that the features learned from this dataset are capable of removing haze that arises in outdoor scenes in a wide range of videos. Extensive experiments demonstrate that the proposed algorithm performs favorably against the state-of-the-art methods on both synthetic and real-world videos.
Low-Resolution Face Recognition in the Wild via Selective Knowledge Distillation

Typically, the deployment of face recognition models in the wild needs to identify low-resolution faces with extremely low computational cost. To address this problem, a feasible solution is compressing a complex face model to achieve higher speed and lower memory at the cost of minimal performance drop. Inspired by that, this paper proposes a learning approach to recognize low-resolution faces via selective knowledge distillation. In this approach, a two-stream convolutional neural network (CNN) is first initialized to recognize high-resolution faces and resolution-degraded faces with a teacher stream and a student stream, respectively. The teacher stream is represented by a complex CNN for high-accuracy recognition, and the student stream is represented by a much simpler CNN for low-complexity recognition. To avoid significant performance drop at the student stream, we then selectively distil the most informative facial features from the teacher stream by solving a sparse graph optimization problem, which are then used to regularize the fine-tuning process of the student stream. In this way, the student stream is actually trained by simultaneously handling two tasks with limited computational resources: approximating the most informative facial cues via feature regression, and recovering the missing facial cues via low-resolution face classification. Experimental results show that the student stream performs impressively in recognizing low-resolution faces and costs only 0.15-MB memory and runs at 418 faces per second on CPU and 9433 faces per second on GPU.
Two-Stream Convolutional Networks for Blind Image Quality Assessment

Traditional image quality assessment (IQA) methods do not perform robustly due to the shallow hand-designed features. It has been demonstrated that deep neural network can learn more effective features than ever. In this paper, we describe a new deep neural network to predict the image quality accurately without relying on the reference image. To learn more effective feature representations for non-reference IQA, we propose a two-stream convolution network that includes two subcomponents for image and gradient image. The motivation for this design is using a two-stream scheme to capture different-level information of inputs and easing the difficulty of extracting features from one steam. The gradient stream focuses on extracting structure features in details, and the image stream pays more attention to the information in intensity. In addition, to consider the locally non-uniform distribution of distortion in images, we add a region-based fully convolutional layer for using the information around the center of the input image patch. The final score of the overall image is calculated by averaging of the patch scores. The proposed network performs in an end-to-end manner in both the training and testing phases. The experimental results on a series of benchmark datasets, e.g., LIVE, CISQ, IVC, TID2013, and Waterloo Exploration Database, show that the proposed algorithm outperforms the state-of-the-art methods, which verifies the effectiveness of our network architecture.
Privacy-Oriented Blockchain-Based Distributed Key Management Architecture for Hierarchical Access Control in the IoT Scenario

The rapid development of the Internet of Things (IoT) and the explosive growth of valuable data produced by user equipment have led to strong demand for access control, especially hierarchical access control, which is performed from a group communication perspective.However, the key management strategies for such a future Internet are based mostly on a trusted third party that requires full trust of the key generation center (KGC) or central authority (CA).Recent studies indicate that centralized cloud centers will be unlikely to deliver satisfactory services to customers because we place too much trust in third parties; therefore, these centers do not apply to user privacy-oriented scenarios.This paper addresses these issues by proposing a novel blockchain-based distributed key management architecture (BDKMA) with fog computing to reduce latency and multiblockchains operated in the cloud to achieve cross-domain access.The proposed scheme utilizes blockchain technology to satisfy the decentralization, fine-grained auditability, high scalability, and extensibility requirements, as well as the privacy-preserving principles for hierarchical access control in IoT.We designed system operations methods and introduced different authorization assignment modes and group access patterns to reinforce the extensibility.We evaluated the performance of our proposed architecture and compared it with existing models using various performance measures.The simulation results show that the multiblockchain structure substantially improves system performance, and the scalability is excellent as the network size increases.Furthermore, dynamic transaction collection time adjustment enables the performance and system capacity to be optimized for various environments.
Generalizable Person Re-Identification by Domain-Invariant Mapping Network

We aim to learn a domain generalizable person re-identification (ReID) model. When such a model is trained on a set of source domains (ReID datasets collected from different camera networks), it can be directly applied to any new unseen dataset for effective ReID without any model updating. Despite its practical value in real-world deployments, generalizable ReID has seldom been studied. In this work, a novel deep ReID model termed Domain-Invariant Mapping Network (DIMN) is proposed. DIMN is designed to learn a mapping between a person image and its identity classifier, i.e., it produces a classifier using a single shot. To make the model domain-invariant, we follow a meta-learning pipeline and sample a subset of source domain training tasks during each training episode. However, the model is significantly different from conventional meta-learning methods in that: (1) no model updating is required for the target domain, (2) different training tasks share a memory bank for maintaining both scalability and discrimination ability, and (3) it can be used to match an arbitrary number of identities in a target domain. Extensive experiments on a newly proposed large-scale ReID domain generalization benchmark show that our DIMN significantly outperforms alternative domain generalization or meta-learning methods.