Reviewerly/exHarmony huggingface dataset:
Continual learning of context-dependent processing in neural networks

Deep neural networks (DNNs) are powerful tools in learning sophisticated but fixed mapping rules between inputs and outputs, thereby limiting their application in more complex and dynamic situations in which the mapping rules are not kept the same but changing according to different contexts. To lift such limits, we developed a novel approach involving a learning algorithm, called orthogonal weights modification (OWM), with the addition of a context-dependent processing (CDP) module. We demonstrated that with OWM to overcome the problem of catastrophic forgetting, and the CDP module to learn how to reuse a feature representation and a classifier for different contexts, a single network can acquire numerous context-dependent mapping rules in an online and continual manner, with as few as $\sim$10 samples to learn each. This should enable highly compact systems to gradually learn myriad regularities of the real world and eventually behave appropriately within it.
All You Need Is Low (Rank)

Recent studies have demonstrated that machine learning approaches like deep learning methods are easily fooled by adversarial attacks. Recently, a highly-influential study examined the impact of adversarial attacks on graph data and demonstrated that graph embedding techniques are also vulnerable to adversarial attacks. Fake users on social media and fake product reviews are examples of perturbations in graph data that are realistic counterparts of the adversarial models proposed. Graphs are widely used in a variety of domains and it is highly important to develop graph analysis techniques that are robust to adversarial attacks. One of the recent studies on generating adversarial attacks for graph data is Nettack. The Nettack model has shown to be very successful in deceiving the Graph Convolutional Network (GCN) model. Nettack is also transferable to other node classification approaches e.g. node embeddings. In this paper, we explore the properties of Nettack perturbations, in search for effective defenses against them. Our first finding is that Nettack demonstrates a very specific behavior in the spectrum of the graph: only high-rank (low-valued) singular components of the graph are affected. Following that insight, we show that a low-rank approximation of the graph, that uses only the top singular components for its reconstruction, can greatly reduce the effects of Nettack and boost the performance of GCN when facing adversarial attacks. Indicatively, on the CiteSeer dataset, our proposed defense mechanism is able to reduce the success rate of Nettack from 98% to 36%. Furthermore, we show that tensor-based node embeddings, which by default project the graph into a low-rank subspace, are robust against Nettack perturbations. Lastly, we propose LowBlow, a low-rank adversarial attack which is able to affect the classification performance of both GCN and tensor-based node embeddings and we show that the low-rank attack is noticeable and making it unnoticeable results in a high-rank attack.
Robots or frontline employees? Exploring customers’ attributions of responsibility and stability after service failure or success

Purpose Service robots are taking over the organizational frontline. Despite a recent surge in studies on this topic, extant works are predominantly conceptual in nature. The purpose of this paper is to provide valuable empirical insights by building on the attribution theory. Design/methodology/approach Two vignette-based experimental studies were employed. Data were collected from US respondents who were randomly assigned to scenarios focusing on a hotel’s reception service and restaurant’s waiter service. Findings Results indicate that respondents make stronger attributions of responsibility for the service performance toward humans than toward robots, especially when a service failure occurs. Customers thus attribute responsibility to the firm rather than the frontline robot. Interestingly, the perceived stability of the performance is greater when the service is conducted by a robot than by an employee. This implies that customers expect employees to shape up after a poor service encounter but expect little improvement in robots’ performance over time. Practical implications Robots are perceived to be more representative of a firm than employees. To avoid harmful customer attributions, service providers should clearly communicate to customers that frontline robots pack sophisticated analytical, rather than simple mechanical, artificial intelligence technology that explicitly learns from service failures. Originality/value Customer responses to frontline robots have remained largely unexplored. This paper is the first to explore the attributions that customers make when they experience robots in the frontline.
Blockchain-Envisioned Secure Data Delivery and Collection Scheme for 5G-Based IoT-Enabled Internet of Drones Environment

The Internet of Drones (IoD) is widely used in a wide range of applications from military to civilian applications from the past years. However, during communication either with the control room/ground station server(s) or moving access points in the sky, security and privacy is one the crucial issues which needs to be tackled efficiently. In this direction, blokchain technology can be one of the viable solutions due to the immutability and traceability of various transactions and decentralized nature. In this paper, we provide in-depth challenges and issues of applicability of blokchain in 5G-based Internet of Things (IoT)-enabled IoD environment. We propose and analyze a new blokchain based secure framework for data management among IoD communication entities. The proposed scheme has ability to resist several potential attacks that are essential in IoT-enabled IoD environment. A detailed comparative analysis exhibits that the proposed scheme offers better security and functionality requirements, and also provides less communication and computation overheads as compared to other related schemes.
Learning Dense Representations for Entity Retrieval

We show that it is feasible to perform entity linking by training a dual encoder (two-tower) model that encodes mentions and entities in the same dense vector space, where candidate entities are retrieved by approximate nearest neighbor search.Unlike prior work, this setup does not rely on an alias table followed by a re-ranker, and is thus the first fully learned entity retrieval model.We show that our dual encoder, trained using only anchor-text links in Wikipedia, outperforms discrete alias table and BM25 baselines, and is competitive with the best comparable results on the standard TACKBP-2010 dataset.In addition, it can retrieve candidates extremely fast, and generalizes well to a new dataset derived from Wikinews.On the modeling side, we demonstrate the dramatic value of an unsupervised negative mining algorithm for this task.
Dual Graph Convolutional Networks for Aspect-based Sentiment Analysis

Aspect-based sentiment analysis is a finegrained sentiment classification task.Recently, graph neural networks over dependency trees have been explored to explicitly model connections between aspects and opinion words.However, the improvement is limited due to the inaccuracy of the dependency parsing results and the informal expressions and complexity of online reviews.To overcome these challenges, in this paper, we propose a dual graph convolutional networks (Du-alGCN) model that considers the complementarity of syntax structures and semantic correlations simultaneously.Particularly, to alleviate dependency parsing errors, we design a SynGCN module with rich syntactic knowledge.To capture semantic correlations, we design a SemGCN module with self-attention mechanism.Furthermore, we propose orthogonal and differential regularizers to capture semantic correlations between words precisely by constraining attention scores in the SemGCN module.The orthogonal regularizer encourages the SemGCN to learn semantically correlated words with less overlap for each word.The differential regularizer encourages the SemGCN to learn semantic features that the SynGCN fails to capture.Experimental results on three public datasets show that our DualGCN model outperforms state-of-theart methods and verify the effectiveness of our model.
GlobalTrack: A Simple and Strong Baseline for Long-Term Tracking

A key capability of a long-term tracker is to search for targets in very large areas (typically the entire image) to handle possible target absences or tracking failures. However, currently there is a lack of such a strong baseline for global instance search. In this work, we aim to bridge this gap. Specifically, we propose GlobalTrack, a pure global instance search based tracker that makes no assumption on the temporal consistency of the target's positions and scales. GlobalTrack is developed based on two-stage object detectors, and it is able to perform full-image and multi-scale search of arbitrary instances with only a single query as the guide. We further propose a cross-query loss to improve the robustness of our approach against distractors. With no online learning, no punishment on position or scale changes, no scale smoothing and no trajectory refinement, our pure global instance search based tracker achieves comparable, sometimes much better performance on four large-scale tracking benchmarks (i.e., 52.1% AUC on LaSOT, 63.8% success rate on TLP, 60.3% MaxGM on OxUvA and 75.4% normalized precision on TrackingNet), compared to state-of-the-art approaches that typically require complex post-processing. More importantly, our tracker runs without cumulative errors, i.e., any type of temporary tracking failures will not affect its performance on future frames, making it ideal for long-term tracking. We hope this work will be a strong baseline for long-term tracking and will stimulate future works in this area.
Towards Fair and Privacy-Preserving Federated Deep Models

The current standalone deep learning framework tends to result in overfitting and low utility. This problem can be addressed by either a centralized framework that deploys a central server to train a global model on the joint data from all parties, or a distributed framework that leverages a parameter server to aggregate local model updates. Server-based solutions are prone to the problem of a single-point-of-failure. In this respect, collaborative learning frameworks, such as federated learning (FL), are more robust. Existing federated learning frameworks overlook an important aspect of participation: fairness. All parties are given the same final model without regard to their contributions. To address these issues, we propose a decentralized Fair and Privacy-Preserving Deep Learning (FPPDL) framework to incorporate fairness into federated deep learning models. In particular, we design a local credibility mutual evaluation mechanism to guarantee fairness, and a three-layer onion-style encryption scheme to guarantee both accuracy and privacy. Different from existing FL paradigm, under FPPDL, each participant receives a different version of the FL model with performance commensurate with his contributions. Experiments on benchmark datasets demonstrate that FPPDL balances fairness, privacy and accuracy. It enables federated learning ecosystems to detect and isolate low-contribution parties, thereby promoting responsible participation.
Directions in abusive language training data, a systematic review: Garbage in, garbage out

Data-driven and machine learning based approaches for detecting, categorising and measuring abusive content such as hate speech and harassment have gained traction due to their scalability, robustness and increasingly high performance. Making effective detection systems for abusive content relies on having the right training datasets, reflecting a widely accepted mantra in computer science: Garbage In, Garbage Out. However, creating training datasets which are large, varied, theoretically-informed and that minimize biases is difficult, laborious and requires deep expertise. This paper systematically reviews 63 publicly available training datasets which have been created to train abusive language classifiers. It also reports on creation of a dedicated website for cataloguing abusive language data hatespeechdata.com. We discuss the challenges and opportunities of open science in this field, and argue that although more dataset sharing would bring many benefits it also poses social and ethical risks which need careful consideration. Finally, we provide evidence-based recommendations for practitioners creating new abusive content training datasets.
Adaptively Sparse Transformers

Gonçalo M. Correia, Vlad Niculae, André F. T. Martins. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Evolving Unsupervised Deep Neural Networks for Learning Meaningful Representations

Deep Learning (DL) aims at learning the \emph{meaningful representations}. A meaningful representation refers to the one that gives rise to significant performance improvement of associated Machine Learning (ML) tasks by replacing the raw data as the input. However, optimal architecture design and model parameter estimation in DL algorithms are widely considered to be intractable. Evolutionary algorithms are much preferable for complex and non-convex problems due to its inherent characteristics of gradient-free and insensitivity to local optimum. In this paper, we propose a computationally economical algorithm for evolving \emph{unsupervised deep neural networks} to efficiently learn \emph{meaningful representations}, which is very suitable in the current Big Data era where sufficient labeled data for training is often expensive to acquire. In the proposed algorithm, finding an appropriate architecture and the initialized parameter values for a ML task at hand is modeled by one computational efficient gene encoding approach, which is employed to effectively model the task with a large number of parameters. In addition, a local search strategy is incorporated to facilitate the exploitation search for further improving the performance. Furthermore, a small proportion labeled data is utilized during evolution search to guarantee the learnt representations to be meaningful. The performance of the proposed algorithm has been thoroughly investigated over classification tasks. Specifically, error classification rate on MNIST with $1.15\%$ is reached by the proposed algorithm consistently, which is a very promising result against state-of-the-art unsupervised DL algorithms.
A comprehensive survey and analysis of generative models in machine learning

Generative models have been in existence for many decades. In the field of machine learning, we come across many scenarios when directly learning a target is intractable through discriminative models, and in such cases the joint distribution of the target and the training data is approximated and generated. These generative models help us better represent or model a set of data by generating data in the form of Markov chains or simply employing a generative iterative process to do the same. With the recent innovation of Generative Adversarial Networks (GANs), it is now possible to make use of AI to generate pieces of art, music, etc. with a high extent of realism. In this paper, we review and analyse critically all the generative models, namely Gaussian Mixture Models (GMM), Hidden Markov Models (HMM), Latent Dirichlet Allocation (LDA), Restricted Boltzmann Machines (RBM), Deep Belief Networks (DBN), Deep Boltzmann Machines (DBM), and GANs. We study their algorithms and implement each of the models to provide the reader some insights on which generative model to pick from while dealing with a problem. We also provide some noteworthy contributions done in the past to these models from the literature.
Reducing Transformer Depth on Demand with Structured Dropout

Overparametrized transformer networks have obtained state of the art results in various natural language processing tasks, such as machine translation, language modeling, and question answering. These models contain hundreds of millions of parameters, necessitating a large amount of computation and making them prone to overfitting. In this work, we explore LayerDrop, a form of structured dropout, which has a regularization effect during training and allows for efficient pruning at inference time. In particular, we show that it is possible to select sub-networks of any depth from one large network without having to finetune them and with limited impact on performance. We demonstrate the effectiveness of our approach by improving the state of the art on machine translation, language modeling, summarization, question answering, and language understanding benchmarks. Moreover, we show that our approach leads to small BERT-like models of higher quality than when training from scratch or using distillation.
SKEP: Sentiment Knowledge Enhanced Pre-training for Sentiment Analysis

Recently, sentiment analysis has seen remarkable advance with the help of pre-training approaches. However, sentiment knowledge, such as sentiment words and aspect-sentiment pairs, is ignored in the process of pre-training, despite the fact that they are widely used in traditional sentiment analysis approaches. In this paper, we introduce Sentiment Knowledge Enhanced Pre-training (SKEP) in order to learn a unified sentiment representation for multiple sentiment analysis tasks. With the help of automatically-mined knowledge, SKEP conducts sentiment masking and constructs three sentiment knowledge prediction objectives, so as to embed sentiment information at the word, polarity and aspect level into pre-trained sentiment representation. In particular, the prediction of aspect-sentiment pairs is converted into multi-label classification, aiming to capture the dependency between words in a pair. Experiments on three kinds of sentiment tasks show that SKEP significantly outperforms strong pre-training baseline, and achieves new state-of-the-art results on most of the test datasets. We release our code at https://github.com/baidu/Senta.
The role of blockchain technology in telehealth and telemedicine

Objective. Telehealth and telemedicine systems aim to deliver remote healthcare services to mitigate the spread of COVID-9. Also, they can help to manage scarce healthcare resources to control the massive burden of COVID-19 patients in hospitals. However, a large portion of today's telehealth and telemedicine systems are centralized and fall short of providing necessary information security and privacy, operational transparency, health records immutability, and traceability to detect frauds related to patients’ insurance claims and physician credentials. Methods. The current study has explored the potential opportunities and adaptability challenges for blockchain technology in telehealth and telemedicine sector. It has explored the key role that blockchain technology can play to provide necessary information security and privacy, operational transparency, health records immutability, and traceability to detect frauds related to patients’ insurance claims and physician credentials. Results. Blockchain technology can improve telehealth and telemedicine services by offering remote healthcare services in a manner that is decentralized, tamper-proof, transparent, traceable, reliable, trustful, and secure. It enables health professionals to accurately identify frauds related to physician educational credentials and medical testing kits commonly used for home-based diagnosis. Conclusions. Wide deployment of blockchain in telehealth and telemedicine technology is still in its infancy. Several challenges and research problems need to be resolved to enable the widespread adoption of blockchain technology in telehealth and telemedicine systems.
ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context

Convolutional neural networks (CNN) have shown promising results for end-to-end speech recognition, albeit still behind other state-of-the-art methods in performance. In this paper, we study how to bridge this gap and go beyond with a novel CNN-RNN-transducer architecture, which we call ContextNet. ContextNet features a fully convolutional encoder that incorporates global context information into convolution layers by adding squeeze-and-excitation modules. In addition, we propose a simple scaling method that scales the widths of ContextNet that achieves good trade-off between computation and accuracy. We demonstrate that on the widely used LibriSpeech benchmark, ContextNet achieves a word error rate (WER) of 2.1%/4.6% without external language model (LM), 1.9%/4.1% with LM and 2.9%/7.0% with only 10M parameters on the clean/noisy LibriSpeech test sets. This compares to the previous best published system of 2.0%/4.6% with LM and 3.9%/11.3% with 20M parameters. The superiority of the proposed ContextNet model is also verified on a much larger internal dataset.
Unsupervised Data Augmentation

Despite its success, deep learning still needs large labeled datasets tosucceed. Data augmentation has shown much promise in alleviating the need formore labeled data, but it so far has mostly been applied in supervised settingsand achieved limited gains. In this work, we propose to apply data augmentationto unlabeled data in a semi-supervised learning setting. Our method, namedUnsupervised Data Augmentation or UDA, encourages the model predictions to beconsistent between an unlabeled example and an augmented unlabeled example.Unlike previous methods that use random noise such as Gaussian noise or dropoutnoise, UDA has a small twist in that it makes use of harder and more realisticnoise generated by state-of-the-art data augmentation methods. This small twistleads to substantial improvements on six language tasks and three vision taskseven when the labeled set is extremely small. For example, on the IMDb textclassification dataset, with only 20 labeled examples, UDA outperforms thestate-of-the-art model trained on 25,000 labeled examples. On standardsemi-supervised learning benchmarks, CIFAR-10 with 4,000 examples and SVHN with1,000 examples, UDA outperforms all previous approaches and reduces more than$30\%$ of the error rates of state-of-the-art methods: going from 7.66% to5.27% and from 3.53% to 2.46% respectively. UDA also works well on datasetsthat have a lot of labeled data. For example, on ImageNet, with 1.3M extraunlabeled data, UDA improves the top-1/top-5 accuracy from 78.28/94.36% to79.04/94.45% when compared to AutoAugment.
Self-labelling via simultaneous clustering and representation learning

Combining clustering and representation learning is one of the most promising approaches for unsupervised learning of deep neural networks. However, doing so naively leads to ill posed learning problems with degenerate solutions. In this paper, we propose a novel and principled learning formulation that addresses these issues. The method is obtained by maximizing the information between labels and input data indices. We show that this criterion extends standard cross-entropy minimization to an optimal transport problem, which we solve efficiently for millions of input images and thousands of labels using a fast variant of the Sinkhorn-Knopp algorithm. The resulting method is able to self-label visual data so as to train highly competitive image representations without manual labels. Compared to the best previous method in this class, namely DeepCluster, our formulation minimizes a single objective function for both representation learning and clustering; it also significantly outperforms DeepCluster in standard benchmarks.