Reviewerly/exHarmony huggingface dataset:
AGSS-VOS: Attention Guided Single-Shot Video Object Segmentation

Most video object segmentation approaches process objects separately. This incurs high computational cost when multiple objects exist. In this paper, we propose AGSS-VOS to segment multiple objects in one feed-forward path via instance-agnostic and instance-specific modules. Information from the two modules is fused via an attention-guided decoder to simultaneously segment all object instances in one path. The whole framework is end-to-end trainable with instance IoU loss. Experimental results on Youtube- VOS and DAVIS-2017 dataset demonstrate that AGSS-VOS achieves competitive results in terms of both accuracy and efficiency.
Task-Driven Modular Networks for Zero-Shot Compositional Learning

One of the hallmarks of human intelligence is the ability to compose learned knowledge into novel concepts which can be recognized without a single training example. In contrast, current state-of-the-art methods require hundreds of training examples for each possible category to build reliable and accurate classifiers. To alleviate this striking difference in efficiency, we propose a task-driven modular architecture for compositional reasoning and sample efficient learning. Our architecture consists of a set of neural network modules, which are small fully connected layers operating in semantic concept space. These modules are configured through a gating function conditioned on the task to produce features representing the compatibility between the input image and the concept under consideration. This enables us to express tasks as a combination of sub-tasks and to generalize to unseen categories by reweighting a set of small modules. Furthermore, the network can be trained efficiently as it is fully differentiable and its modules operate on small sub-spaces. We focus our study on the problem of compositional zero-shot classification of object-attribute categories. We show in our experiments that current evaluation metrics are flawed as they only consider unseen object-attribute pairs. When extending the evaluation to the generalized setting which accounts also for pairs seen during training, we discover that naive baseline methods perform similarly or better than current approaches. However, our modular network is able to outperform all existing approaches on two widely-used benchmark datasets.
SO-HandNet: Self-Organizing Network for 3D Hand Pose Estimation With Semi-Supervised Learning

3D hand pose estimation has made significant progress recently, where Convolutional Neural Networks (CNNs) play a critical role. However, most of the existing CNN-based hand pose estimation methods depend much on the training set, while labeling 3D hand pose on training data is laborious and time-consuming. Inspired by the point cloud autoencoder presented in self-organizing network (SO-Net), our proposed SO-HandNet aims at making use of the unannotated data to obtain accurate 3D hand pose estimation in a semi-supervised manner. We exploit hand feature encoder (HFE) to extract multi-level features from hand point cloud and then fuse them to regress 3D hand pose by a hand pose estimator (HPE). We design a hand feature decoder (HFD) to recover the input point cloud from the encoded feature. Since the HFE and the HFD can be trained without 3D hand pose annotation, the proposed method is able to make the best of unannotated data during the training phase. Experiments on four challenging benchmark datasets validate that our proposed SO-HandNet can achieve superior performance for 3D hand pose estimation via semi-supervised learning.
Retraining Strategy-Based Domain Adaption Network for Intelligent Fault Diagnosis

Industrial Internet of Things (IIoT) obtains big data from industrial facilities. Based on these data, health conditions for facilities can be predicted using machine learning methods, which in turn improves the trustworthiness of IIoT. Intelligent fault diagnosis is developing with this process. Since machine damages always happen under different circumstances, the designed intelligent fault diagnosis methods should have domain adaption ability. In this article, a data-driven fault diagnosis method called domain adaption network (DAN) is present, in which the difference between training and testing data can be minimized while the maximum training accuracy can be maintained. First, DAN uses a combined loss for consistent feature learning and optimum training classification performance. Then, a DAN retraining (DAN-R) strategy is employed based on weighted pseudo-labeled testing dataset. Finally, three experiments are performed on two fault diagnosis datasets. The results reveal that DAN-R is applicable and has outstanding domain adaption ability.
Fashion Retrieval via Graph Reasoning Networks on a Similarity Pyramid

Matching clothing images from customers and online shopping stores has rich applications in E-commerce. Existing algorithms encoded an image as a global feature vector and performed retrieval with the global representation. However, discriminative local information on clothes are submerged in this global representation, resulting in sub-optimal performance. To address this issue, we propose a novel Graph Reasoning Network (GRNet) on a Similarity Pyramid, which learns similarities between a query and a gallery cloth by using both global and local representations in multiple scales. The similarity pyramid is represented by a Graph of similarity, where nodes represent similarities between clothing components at different scales, and the final matching score is obtained by message passing along edges. In GRNet, graph reasoning is solved by training a graph convolutional network, enabling to align salient clothing components to improve clothing retrieval. To facilitate future researches, we introduce a new benchmark FindFashion, containing rich annotations of bounding boxes, views, occlusions, and cropping. Extensive experiments show that GRNet obtains new state-of-the-art results on two challenging benchmarks, e.g. pushing the top-1, top-20, and top-50 accuracies on DeepFashion to 26%, 64%, and 75% (i.e. 4%, 10%, and 10% absolute improvements), outperforming competitors with large margins. On FindFashion, GRNet achieves considerable improvements on all empirical settings.
Prefigurative Design as a Method for Research Justice

While there is growing concern around justice and equity, they mean different things in different socio-political and cultural contexts. Additionally, it can be difficult to make sense of how to incorporate the more abstract concept of justice into our research practices. This paper discusses prefigurative design as a framework for more just research practices to challenge inequity, particularly in community-based collaborations. I draw from past fieldwork with activist organizations and radical organizing literature to explore opportunities for how to engage with justice as academics, identifying three main opportunities for intervention through research: social relationships, resource distribution, and counter-institutions. I offer these contributions in the spirit of generative critique, in hopes that other researchers with similar concerns will iterate on these practices to commit to more just and equitable scholarly impacts.
Action Assessment by Joint Relation Graphs

We present a new model to assess the performance of actions from videos, through graph-based joint relation modelling. Previous works mainly focused on the whole scene including the performer's body and background, yet they ignored the detailed joint interactions. This is insufficient for fine-grained, accurate action assessment, because the action quality of each joint is dependent of its neighbouring joints. Therefore, we propose to learn the detailed joint motion based on the joint relations. We build trainable Joint Relation Graphs, and analyze joint motion on them. We propose two novel modules, the Joint Commonality Module and the Joint Difference Module, for joint motion learning. The Joint Commonality Module models the general motion for certain body parts, and the Joint Difference Module models the motion differences within body parts. We evaluate our method on six public Olympic actions for performance assessment. Our method outperforms previous approaches (+0.0912) and the whole-scene analysis (+0.0623) in the Spearman's Rank Correlation. We also demonstrate our model's ability to interpret the action assessment process.
ProtoGAN: Towards Few Shot Learning for Action Recognition

Few-shot learning (FSL) for action recognition is a challenging task of recognizing novel action categories which are represented by few instances in the training data. In a more generalized FSL setting (G-FSL), both seen as well as novel action categories need to be recognized. Conventional classifiers suffer due to inadequate data in FSL setting and inherent bias towards seen action categories in G-FSL setting. In this paper, we address this problem by proposing a novel ProtoGAN framework which synthesizes additional examples for novel categories by conditioning a conditional generative adversarial network with class prototype vectors. These class prototype vectors are learnt using a Class Prototype Transfer Network (CPTN) from examples of seen categories. Our synthesized examples for a novel class are semantically similar to real examples belonging to that class and is used to train a model exhibiting better generalization towards novel classes. We support our claim by performing extensive experiments on three datasets: UCF101, HMDB51 and Olympic-Sports. To the best of our knowledge, we are the first to report the results for G-FSL and provide a strong benchmark for future research. We also outperform the state-of-the-art method in FSL for all the aforementioned datasets.
A Survey of Restoration and Enhancement for Underwater Images

Images captured underwater usually suffer from color distortion, detail blurring, low contrast, and a bluish or greenish tone due to light scattering and absorption in the underwater medium, which in turn the visibility is adversely affected by these factors seriously. Over the last decades, various image restoration and enhancement methods have been developed by many researchers to improve the quality (visibility and highlight richer details) of underwater images. This paper introduces the overview of state-of-the-art underwater image restoration and enhancement techniques and classifies the approaches in two categories: image restoration (physical-based model) and image enhancement (nonphysical-based model). Furthermore, the classification of these two methods is elaborated. Then, the typical underwater image restoration and enhancement methods are discussed in detail, as well as a comprehensive study and fair evaluation of the methods is carried out from both qualitative and quantitative perspectives. Finally, the research process of underwater image restoration and enhancement is summarized and the suggestions for future research are prospected.
FuseMODNet: Real-Time Camera and LiDAR Based Moving Object Detection for Robust Low-Light Autonomous Driving

Moving object detection is a critical task for autonomous vehicles. As dynamic objects represent higher collision risk than static ones, our own ego-trajectories have to be planned attending to the future states of the moving elements of the scene. Motion can be perceived using temporal information such as optical flow. Conventional optical flow computation is based on camera sensors only, which makes it prone to failure in conditions with low illumination. On the other hand, LiDAR sensors are independent of illumination, as they measure the time-of-flight of their own emitted lasers. In this work we propose a robust and real-time CNN architecture for Moving Object Detection (MOD) under low-light conditions by capturing motion information from both camera and LiDAR sensors. We demonstrate the impact of our algorithm on KITTI dataset where we simulate a low-light environment creating a novel dataset "Dark-KITTI". We obtain a 10.1 % relative improvement on Dark-KITTI, and a 4.25 % improvement on standard KITTI relative to our baselines. The proposed algorithm runs at 29 fps on a standard desktop GPU using 256x1224 resolution images.
Image caption generation with dual attention mechanism

As a crossing domain of computer vision and natural language processing, the image caption generation has been an active research topic in recent years, which contributes to the multimodal social media translation from unstructured image data to structured text data. The conventional research works have proposed a series of image captioning methods, such as template-based, retrieval-based, encode-decode. Among these methods, the one with encode-decode framework is widely used in the image caption generation, in which the encoder extracts the image features by Convolutional Neural Network (CNN), and the decoder adopts Recurrent Neural Network (RNN) to generate the image description. The Neural Image Caption (NIC) model has achieved good performance in image captioning, and however, there still remains some challenges to be addressed. To tackle the challenges of the lack of image information and the deviation from the core content of the image, our proposed model explores visual attention to deepen the understanding of the image, incorporating the image labels generated by Fully Convolutional Network (FCN) into the generation of image caption. Furthermore, our proposed model exploits textual attention to increase the integrity of the information. Finally, the label generation, attached to the textual attention mechanism, and the image caption generation, have been merged to form an end-to-end trainable framework. In this paper, extensive experiments have been carried out on the AIC-ICC image caption benchmark dataset, and the experimental results show that our proposed model is effective and feasible in the image caption generation.
Biomedical named entity recognition using deep neural networks with contextual information

In biomedical text mining, named entity recognition (NER) is an important task used to extract information from biomedical articles. Previously proposed methods for NER are dictionary- or rule-based methods and machine learning approaches. However, these traditional approaches are heavily reliant on large-scale dictionaries, target-specific rules, or well-constructed corpora. These methods to NER have been superseded by the deep learning-based approach that is independent of hand-crafted features. However, although such methods of NER employ additional conditional random fields (CRF) to capture important correlations between neighboring labels, they often do not incorporate all the contextual information from text into the deep learning layers.We propose herein an NER system for biomedical entities by incorporating n-grams with bi-directional long short-term memory (BiLSTM) and CRF; this system is referred to as a contextual long short-term memory networks with CRF (CLSTM). We assess the CLSTM model on three corpora: the disease corpus of the National Center for Biotechnology Information (NCBI), the BioCreative II Gene Mention corpus (GM), and the BioCreative V Chemical Disease Relation corpus (CDR). Our framework was compared with several deep learning approaches, such as BiLSTM, BiLSTM with CRF, GRAM-CNN, and BERT. On the NCBI corpus, our model recorded an F-score of 85.68% for the NER of diseases, showing an improvement of 1.50% over previous methods. Moreover, although BERT used transfer learning by incorporating more than 2.5 billion words, our system showed similar performance with BERT with an F-scores of 81.44% for gene NER on the GM corpus and a outperformed F-score of 86.44% for the NER of chemicals and diseases on the CDR corpus. We conclude that our method significantly improves performance on biomedical NER tasks.The proposed approach is robust in recognizing biological entities in text.
Effective Scheme against 51% Attack on Proof-of-Work Blockchain with History Weighted Information

Proof-of-Work (PoW) is a popular protocol used in Blockchain systems to resolve double-spending problems. However, if an attacker has access to calculation hash power greater than half of the total hash power, this attacker can create a double-spending attack or 51% attack. The cost of creating a 51% attack is surprisingly low if hash power is abundantly available. That posts a great threat to lots of PoW blockchains. We propose a technique to combine history weighted information of miners with the total calculation difficulty to alleviate the 51% attack problem. Analysis indicates that with the new technique, the cost of a traditional attack is increased by two orders of magnitude.
Blockchain leveraged decentralized IoT eHealth framework

Blockchain technologies recently emerging for eHealth, can facilitate a secure, decentralized and patient-driven, record management system. However, Blockchain technologies cannot accommodate the storage of data generated from IoT devices in remote patient management (RPM) settings as this application requires a fast consensus mechanism, careful management of keys and enhanced protocols for privacy. In this paper, we propose a Blockchain leveraged decentralized eHealth architecture which comprises three layers: (1) The Sensing layer – Body Area Sensor Networks include medical sensors typically on or in a patient body transmitting data to a smartphone. (2) The NEAR processing layer – Edge Networks consist of devices at one hop from data sensing IoT devices. (3) The FAR processing layer – Core Networks comprise Cloud or other high computing servers). A Patient Agent (PA) software replicated on the three layers processes medical data to ensure reliable, secure and private communication. The PA executes a lightweight Blockchain consensus mechanism and utilizes a Blockchain leveraged task-offloading algorithm to ensure patient’s privacy while outsourcing tasks. Performance analysis of the decentralized eHealth architecture has been conducted to demonstrate the feasibility of the system in the processing and storage of RPM data.
Unsupervised Low-Light Image Enhancement Using Bright Channel Prior

Recent approaches for low-light image enhancement achieve excellent performance through supervised learning based on convolutional neural networks. However, it is still challenging to collect a large amount of low-/normal-light image pairs in real environments for training the networks. In this letter, we propose an unsupervised learning approach for single low-light image enhancement using the bright channel prior (BCP) that the brightest pixel in a small patch is likely to be close to 1. An unsupervised loss function is defined with the pseudo ground-truth generated using the BCP. An enhancement network, consisting of a simple encoder-decoder, is then trained using the unsupervised loss function. To the best of our knowledge, this is the first attempt that enhances a low-light image through unsupervised learning. Furthermore, we introduce saturation loss and self-attention map for preserving image details and naturalness in the enhanced result. The performance of the proposed method is validated on various public datasets. Experimental results demonstrate that the proposed unsupervised approach achieves competitive performance over state-of-the-art methods based on supervised learning.
User authentication on mobile devices: Approaches, threats and trends

Mobile devices have brought a great convenience to us these years, which allow the users to enjoy the anytime and anywhere various applications such as the online shopping, Internet banking, navigation and mobile media. While the users enjoy the convenience and flexibility of the ”Go Mobile” trend, their sensitive private information (e.g., name and credit card number) on the mobile devices could be disclosed. An adversary could access the sensitive private information stored on the mobile device by unlocking the mobile devices. Moreover, the user’s mobile services and applications are all exposed to security threats. For example, the adversary could utilize the user’s mobile device to conduct non-permitted actions (e.g., making online transactions and installing malwares). The authentication on mobile devices plays a significant role to protect the user’s sensitive information on mobile devices and prevent any non-permitted access to the mobile devices. This paper surveys the existing authentication methods on mobile devices. In particular, based on the basic authentication metrics (i.e., knowledge, ownership and biometrics) used in existing mobile authentication methods, we categorize them into four categories, including the knowledge-based authentication (e.g., passwords and lock patterns), physiological biometric-based authentication (e.g., fingerprint and iris), behavioral biometrics-based authentication (e.g., gait and hand gesture), and two/multi-factor authentication. We compare the usability and security level of the existing authentication approaches among these categories. Moreover, we review the existing attacks to these authentication approaches to reveal their vulnerabilities. The paper points out that the trend of the authentication on mobile devices would be the multi-factor authentication, which determines the user’s identity using the integration (not the simple combination) of more than one authentication metrics. For example, the user’s behavior biometrics (e.g., keystroke dynamics) could be extracted simultaneously when he/she inputs the knowledge-based secrets (e.g., PIN), which can provide the enhanced authentication as well as sparing the user’s trouble to conduct multiple inputs for different authentication metrics.
Boosting Tree-Assisted Multitask Deep Learning for Small Scientific Datasets

Machine learning approaches have had tremendous success in various disciplines. However, such success highly depends on the size and quality of datasets. Scientific datasets are often small and difficult to collect. Currently, improving machine learning performance for small scientific datasets remains a major challenge in many academic fields, such as bioinformatics or medical science. Gradient boosting decision tree (GBDT) is typically optimal for small datasets, while deep learning often performs better for large datasets. This work reports a boosting tree-assisted multitask deep learning (BTAMDL) architecture that integrates GBDT and multitask deep learning (MDL) to achieve near-optimal predictions for small datasets when there exists a large dataset that is well correlated to the small datasets. Two BTAMDL models are constructed, one utilizing purely MDL output as GBDT input while the other admitting additional features in GBDT input. The proposed BTAMDL models are validated on four categories of datasets, including toxicity, partition coefficient, solubility, and solvation. It is found that the proposed BTAMDL models outperform the current state-of-the-art methods in various applications involving small datasets.