Reviewerly/exHarmony huggingface dataset:
Video Matting via Consistency-Regularized Graph Neural Networks

Learning temporally consistent foreground opacity from videos, i.e., video matting, has drawn great attention due to the blossoming of video conferencing. Previous approaches are built on top of image matting models, which fail in maintaining the temporal coherence when being adapted to videos. They either utilize the optical flow to smooth frame-wise prediction, where the performance is dependent on the selected optical flow model; or naively combine feature maps from multiple frames, which does not model well the correspondence of pixels in adjacent frames. In this paper, we propose to enhance the temporal coherence by Consistency-Regularized Graph Neural Networks (CRGNN) with the aid of a synthesized video matting dataset. CRGNN utilizes Graph Neural Networks (GNN) to relate adjacent frames such that pixels or regions that are incorrectly predicted in one frame can be corrected by leveraging information from its neighboring frames. To generalize our model from synthesized videos to real-world videos, we propose a consistency regularization technique to enforce the consistency on the alpha and foreground when blending them with different backgrounds. To evaluate the efficacy of CRGNN, we further collect a real-world dataset with annotated alpha mattes. Compared with state-of-the-art methods that require hand-crafted trimaps or backgrounds for modeling training, CRGNN generates favorably results with the help of unlabeled real training dataset. The source code and datasets are available at https://github.com/TiantianWang/VideoMattingCRGNN.git.
Parallel Governance for Decentralized Autonomous Organizations enabled by Blockchain and Smart Contracts

Decentralized autonomous organizations(DAOs) enabled by blockchain and smart contracts is regarded as an effective tool to solve corporate governance problems. It can minimize the contract risks, principal-agent dilemmas, and other issues in corporate governance. However, DAOs also face a variety of governance issues, such as internal governance issues and external governance issues caused by the technical attributes of blockchain and smart contracts. DAO is a typical complex system with the characteristics of both social and engineering complexity. It is difficult for existing governance theories to solve governance problems in DAOs. The parallel intelligence theory based on the artificial systems + computational experiments + parallel execution (ACP) method is a suitable research and practical approach to solve management and control problems for complex systems. Therefore, we propose a parallel governance framework for DAO based on the parallel intelligence theory and discuss its key research issues. Finally, we construct a parallel governance system for GnosisDAO, and conduct computational experiments on its voting mechanism. The results show the defects of GnosisDAO governance mechanism and we thus propose further research directions. We consider that parallel governance represents the future trend of DAO research and practice. This paper is aimed at stimulating further effort and providing helpful guidance and reference for future research works.
Going through the emotions of regret and fear: Revisiting protection motivation for identity theft protection

We explore why and how individuals adaptively and maladaptively respond to the threat of identity theft. We use protection motivation theory and regret theory, shedding light on how the individual's reflection of a future negative event, which they did nothing to prevent, would influence their current behavior. Fear appeal is experimentally manipulated to test different models of high and low threat. By comparing the impact of anticipated regret and fear on individuals' protection motivation, we find that discrete emotions of fear and anticipated regret behave differently in increasing adaptive and reducing maladaptive responses to identity theft. Specifically, whereas fear is only effective when threat is high, anticipated regret is effective in both high and low threat conditions. Also, we find that anticipated regret has the most potent effect on increasing adaptive coping responses in a low threat model. This means that anticipated regret could be used in situations where the threat is low rather than fear. This research provides empirical evidence of conditions under which fear and regret motivate personal security protection measures, enabling practitioners to promote identity theft protection more efficiently.
Temporal Cue Guided Video Highlight Detection with Low-Rank Audio-Visual Fusion

Video highlight detection plays an increasingly important role in social media content filtering, however, it remains highly challenging to develop automated video highlight detection methods because of the lack of temporal annotations (i.e., where the highlight moments are in long videos) for supervised learning. In this paper, we propose a novel weakly supervised method that can learn to detect highlights by mining video characteristics with video level annotations (topic tags) only. Particularly, we exploit audio-visual features to enhance video representation and take temporal cues into account for improving detection performance. Our contributions are threefold: 1) we propose an audio-visual tensor fusion mechanism that efficiently models the complex association between two modalities while reducing the gap of the heterogeneity between the two modalities; 2) we introduce a novel hierarchical temporal context encoder to embed local temporal clues in between neighboring segments; 3) finally, we alleviate the gradient vanishing problem theoretically during model optimization with attention-gated instance aggregation. Extensive experiments on two benchmark datasets (YouTube Highlights and TVSum) have demonstrated our method outperforms other state-of-the-art methods with remarkable improvements.
Efficient Implementation of Lightweight Hash Functions on GPU and Quantum Computers for IoT Applications

Secure communication is important for Internet of Things (IoT) applications, to avoid cybersecurity attacks.One of the key security aspects is data integrity, which can be protected by employing cryptographic hash functions.Recently, US National Institute of Standards and Technology (NIST) announced a competition to standardize lightweight hash functions, which can be used in IoT applications.IoT communication involves various hardware platforms, from low-end microcontrollers to high-end cloud servers with GPU accelerators.Since many sensor nodes are connected to the gateway devices and cloud servers, performing high throughput integrity check is important to secure IoT applications.However, this is a time consuming task even for high-end servers, which may affect the response time in IoT systems.Moreover, no prior work had evaluated the performance of NIST candidates on contemporary processors like GPU and quantum computers.In this study, we showed that with carefully crafted implementation techniques, all the finalist hash function candidates in the NIST standardization competition can achieve high throughput (up-to 1,000 Gbps) on a RTX 3080 GPU.This research output can be used by IoT gateway devices and cloud servers to perform data integrity checks at high speed, thus ensuring a timely response.In addition, this is also the first study that showcase the implementation of NIST lightweight hash functions on a quantum computer (ProjectQ).Besides securing the communication in IoT, these efficient implementations on a GPU and quantum computer can be used to evaluate the strength of respective hash functions against brute-force attack.
BorderPointsMask: One-stage instance segmentation with boundary points representation

The mechanism of human vision can easily detect and segment objects based on boundary information. Even though the deep learning instance segmentation model based on boundary information can mimic this human vision mechanism, its prospect is rarely explored. In this work, we propose a one-stage and anchor-free instance segmentation framework based on boundary points representation, named BorderPointsMask. The proposed BorderPointsMask doesn’t need to rely on the object detector. It formulates instance segmentation as instance classification, boundary points location prediction, and boundary points attribute prediction. Furthermore, we design two effective approaches to improve the performance of this proposed framework. Specifically, we utilize the proposed BorderPoints Center-ness to suppress the predicted low-quality masks. And the Deformation Before-and-After Stacking Module (DBASM) is formulated to promote instance classification and boundary points learning. BorderPointsMask obtains 35.0% in mask Average Precision (AP) with single-model (ResNet-101-FPN) and single-scale training/testing on the COCO benchmark, which demonstrates the superior performance among one-stage frameworks. The code will be available.
Active Learning for Lane Detection: A Knowledge Distillation Approach

Lane detection is a key task for autonomous driving vehicles. Currently, lane detection relies on a huge amount of annotated images, which is a heavy burden. Active learning has been proposed to reduce annotation in many computer vision tasks, but no effort has been made for lane detection. Through experiments, we find that existing active learning methods perform poorly for lane detection, and the reasons are twofold. On one hand, most methods evaluate data uncertainties based on entropy, which is undesirable in lane detection because it encourages to select images with very few lanes or even no lane at all. On the other hand, existing methods are not aware of the noise of lane annotations, which is caused by heavy occlusion and unclear lane marks. In this paper, we build a novel knowledge distillation framework and evaluate the uncertainty of images based on the knowledge learnt by the student model. We show that the proposed uncertainty metric overcomes the above two problems. To reduce data redundancy, we explore the influence sets of image samples, and propose a new diversity metric for data selection. Finally we incorporate the uncertainty and diversity metrics, and develop a greedy algorithm for data selection. The experiments show that our method achieves new state-of-the-art on the lane detection benchmarks. In addition, we extend this method to common 2D object detection and the results show that it is also effective.
On the KLM Properties of a Fuzzy DL with Typicality

The paper investigates the properties of a fuzzy logic of typicality. The extension of fuzzy logic with a typicality operator was proposed in recent work to define a fuzzy multipreference semantics for Multilayer Perceptrons, by regarding the deep neural network as a conditional knowledge base. In this paper, we study its properties. First, a monotonic extension of a fuzzy \(\mathcal {ALC}\) with typicality is considered (called \(\mathcal {ALC}^{\mathbf{F}}\mathbf{T}\)) and a reformulation the KLM properties of a preferential consequence relation for this logic is devised. Most of the properties are satisfied, depending on the reformulation and on the fuzzy combination functions considered. We then strengthen \(\mathcal {ALC}^{\mathbf{F}}\mathbf{T}\) with a closure construction by introducing a notion of faithful model of a weighted knowledge base, which generalizes the notion of coherent model of a conditional knowledge base previously introduced, and we study its properties.
Heterogeneous graph embedding by aggregating meta-path and meta-structure through attention mechanism

Heterogeneous information networks embedding, which is a promising technique to learn low-dimensional representations for nodes with different types, has obtained very high interests recently. Plenty of graph neural networks models have been proposed for heterogeneous graph embedding. However, there are two limitations in existing models: (1) despite the complex structures of heterogeneous nodes, almost all these models are mainly based on meta-paths; (2) the different importance of different structures are neglected in most models. To address these problems, we propose a meta-path and meta-structure integrated heterogeneous graph neural network through attention mechanisms (PSHGAN). PSHGAN first maps features of heterogeneous nodes into the same space. Then, PSHGAN learns the weights of two nodes at each end of the meta-path or meta-structure by a local attention mechanism. Finally, PSHGAN learns weights of meta-paths and meta-structures by a global attention mechanism and aggregates the nodes representations. Extensive experiments are conducted on real-world benchmark datasets and show that our proposed model outperforms the state-of-the-art models in the node classification and link prediction tasks. Moreover, we make comprehensive analysis on the impacts of meta-structures on the performance of classification training with meta-paths.
Robust Kalman filter for systems subject to parametric uncertainties

State estimation plays a fundamental role in control systems that rely on the knowledge of the underlying system state, especially when it is not readily available. The Kalman filter is among the most popular techniques in this matter. Nonetheless, one of its well-known shortcomings is the assumption that an exact system model is available. The difficulty in fulfilling this premise justifies the demand for estimation strategies that limit the effect of model uncertainties. In this paper, we propose a robust Kalman filter for uncertain linear discrete-time systems. We assume a general setting in which all matrices of both the system and measurement models are subject to norm-bounded parametric uncertainties. By adopting a purely deterministic viewpoint and applying the penalty function method, we formulate a robust regularized least-squares estimation problem. We further analyze how the penalty parameter influences the estimation performance. The resulting filter is presented both in a symmetric matrix arrangement and as explicit algebraic expressions in a Kalman-like structure that is suitable for online applications. Under reasonable conditions, we show that the steady-state filter is stable and, for quadratically stable systems, guarantees a bounded error variance. With a numerical example, we evaluate the proposed filter and compare its performance with other robust filtering approaches.
Attention-based LSTM-FCN for earthquake detection and location

SUMMARY Earthquake detection and localization are challenging since the seismic signal usually is noisy and the microearthquakes are hidden in the seismic noise. Traditional detection and localization methods often rely on manually picked phases or computationally intensive algorithms. Inspired by the successful application of a deep learning model, ConvNetQuake, in detecting and locating the seismic events, we train an attention-based long short-term memory fully convolutional network (LSTM-FCN) model to improve the detection and location accuracy on the same data set. We use a parallel structure of FCN and LSTM to extract different features separately and merge them as a vector for better classification. In particular, FCN is used to extract high-level features and, similarly, LSTM is employed to model the temporal dependences. Besides, an attention mechanism is added to the LSTM to select a significant input segment along with a squeeze-and-excitation block in FCN to enhance useful feature maps for classification. We show that the trained model has a classification accuracy of 89.1 per cent, which represents 14.5 per cent improvement compared to the ConvNetQuake model. Moreover, the ConvNetQuake model only considered classifying seismic events roughly into one of the six geographic regions. But our model can locate the seismic events with a higher resolution by classifying them into as a greater number of regions as to 15 while a relatively high accuracy is maintained. We also demonstrate that the incorporated attention mechanism can effectively improve the classification performance by automatically and selectively enhancing the significant feature maps and inputs.
Inferring substitutable and complementary products with Knowledge-Aware Path Reasoning based on dynamic policy network

Inferring the substitutable and complementary products for a given product is an essential and fundamental concern for the recommender system. To achieve this, existing approaches take advantage of the knowledge graphs to learn more evidences for inference, whereas they often suffer from invalid reasoning for lack of elegant decision making strategies. Therefore, we propose a novel Knowledge-Aware Path Reasoning (KAPR) model which leverages the dynamic policy network to make explicit reasoning over knowledge graphs, for inferring the substitutable and complementary relationships. Our contributions can be highlighted as three aspects. Firstly, we model this inference scenario as a Markov Decision Process in order to accomplish a knowledge-aware path reasoning over knowledge graphs. Secondly, we integrate both structured and unstructured knowledge to provide adequate evidences for making accurate decision-making. Thirdly, we evaluate our model on a series of real-world datasets, achieving competitive performance compared with state-of-the-art approaches. Our code is released on https://gitee.com/yangzijing_flower/kapr/tree/master.
Adaptive Informative Path Planning Using Deep Reinforcement Learning for UAV-based Active Sensing

Aerial robots are increasingly being utilized for environmental monitoring and exploration. However, a key challenge is efficiently planning paths to maximize the information value of acquired data as an initially unknown environment is explored. To address this, we propose a new approach for informative path planning based on deep reinforcement learning (RL). Combining recent advances in RL and robotic applications, our method combines tree search with an offline-learned neural network predicting informative sensing actions. We introduce several components making our approach applicable for robotic tasks with high-dimensional state and large action spaces. By deploying the trained network during a mission, our method enables sample-efficient online replanning on platforms with limited computational resources. Simulations show that our approach performs on par with existing methods while reducing runtime by 8-10×. We validate its performance using real-world surface temperature data.
Temporal consistency two-stream CNN for human motion prediction

Fusion is critical for a two-stream network. In this paper, we propose a novel temporal fusion (TF) module to fuse the two-stream joints’ information to predict human motion, including a temporal concatenation and a reinforcement trajectory spatial-temporal (TST) block that specifically designed to keep prediction temporal consistency. In particular, the temporal concatenation keeps the temporal consistency of preliminary predictions from two streams. Meanwhile, the TST block improves the spatial-temporal feature coupling. However, the TF module can increase the temporal continuities between the first predicted pose and the given poses and between each predicted pose. The fusion is based on a two-stream network that consists of a dynamic velocity stream (V-Stream) and a static position stream (P-Stream) because we found that the joints’ velocity information improves our short-term prediction, while the joints’ position information is better at long-term prediction, and they are complementary in motion prediction. Finally, our approach achieves impressive results on three benchmark datasets, including H3.6M, CMU-Mocap, and 3DPW in both short-term and long-term predictions, confirming its effectiveness and efficiency.
Exploring graph capsual network for graph classification

Graph Neural Network (GNN) has received tremendous attention due to their power in learning graph representations by modeling the topological structure and aggregating feature information. However, the scalar node representations learned from GNN may not be sufficient to effectively preserve the attributes of the node/graph features, resulting in sub-optimal graph representation. Repeated averaging gathers too much noise, which makes the features of nodes in different classes over-mixed and leads to the problem of over smoothing. Inspired by the concept of capsule network proposed by Hinton, we propose a new framework for graph classification, named CapsualGNN, which takes full advantage of Graph Neural Network and Capsule Network. Specifically, we firstly represent nodes as groups of capsules, in which each capsule extracts distinctive features of its corresponding node. Then, we exploit routing mechanism to capture important information and properties at the graph level by the generated multiple embeddings for each graph, and utilize attention mechanism to focus on important features. Finally, to solve the problem of over smoothing, we introduce class residual connection for GCN. In addition, we also introduce parameter for distinguishing self-connected nodes and other nodes. We evaluate the framework by using six graph datasets on biological information and social networks, and demonstrate that CapsualGNN outperforms other SOTA techniques on the task of graph classification.
Multimodal Representation Learning via Maximization of Local Mutual Information

We propose and demonstrate a representation learning approach by maximizing the mutual information between local features of images and text. The goal of this approach is to learn useful image representations by taking advantage of the rich information contained in the free text that describes the findings in the image. Our method trains image and text encoders by encouraging the resulting representations to exhibit high local mutual information. We make use of recent advances in mutual information estimation with neural network discriminators. We argue that the sum of local mutual information is typically a lower bound on the global mutual information. Our experimental results in the downstream image classification tasks demonstrate the advantages of using local features for image-text representation learning. Our code is available at: https://github.com/RayRuizhiLiao/mutual_info_img_txt.
Memory-Safety Challenge Considered Solved? An In-Depth Study with All Rust CVEs

Rust is an emerging programming language that aims at preventing memory-safety bugs without sacrificing much efficiency. The claimed property is very attractive to developers, and many projects start using the language. However, can Rust achieve the memory-safety promise? This article studies the question by surveying 186 real-world bug reports collected from several origins, which contain all existing Rust common vulnerability and exposures (CVEs) of memory-safety issues by 2020-12-31. We manually analyze each bug and extract their culprit patterns. Our analysis result shows that Rust can keep its promise that all memory-safety bugs require unsafe code, and many memory-safety bugs in our dataset are mild soundness issues that only leave a possibility to write memory-safety bugs without unsafe code. Furthermore, we summarize three typical categories of memory-safety bugs, including automatic memory reclaim, unsound function, and unsound generic or trait. While automatic memory claim bugs are related to the side effect of Rust newly-adopted ownership-based resource management scheme, unsound function reveals the essential challenge of Rust development for avoiding unsound code, and unsound generic or trait intensifies the risk of introducing unsoundness. Based on these findings, we propose two promising directions toward improving the security of Rust development, including several best practices of using specific APIs and methods to detect particular bugs involving unsafe code. Our work intends to raise more discussions regarding the memory-safety issues of Rust and facilitate the maturity of the language.