Reviewerly/exHarmony huggingface dataset:
Mapping Natural Language Instructions to Mobile UI Action Sequences

We present a new problem: grounding natural language instructions to mobile user interface actions, and create three new datasets for it.For full task evaluation, we create PIX-ELHELP, a corpus that pairs English instructions with actions performed by people on a mobile UI emulator.To scale training, we decouple the language and action data by (a) annotating action phrase spans in HowTo instructions and (b) synthesizing grounded descriptions of actions for mobile user interfaces.We use a Transformer to extract action phrase tuples from long-range natural language instructions.A grounding Transformer then contextually represents UI objects using both their content and screen position and connects them to object descriptions.Given a starting screen and instruction, our model achieves 70.59% accuracy on predicting complete ground-truth action sequences in PIXELHELP.
Unsupervised Real-World Super Resolution with Cycle Generative Adversarial Network and Domain Discriminator

This paper proposes an unsupervised single-image Super-Resolution(SR) model using cycleGAN and domain discriminator to solve the problem of SR with unknown degradation using unpaired dataset. In previous approaches, paired dataset is required for training with assumed levels of image degradation. In real world SR applications, however, training sets are typically not of low and high resolution image pairs, but only low resolution images with unknown degradation are provided as inputs. To address the problem, we introduce a cycle-in-cycle GAN based unsupervised learning model using an unpaired dataset. In addition, we combine several losses attributed to image contents, such as pixel-wise loss, VGG feature loss and SSIM loss, for stable learning and performance improvement. We also propose a domain discriminator, which consists of noise discriminator, texture discriminator and color discriminator, to guide generated images to follow target domain distribution rather than source domain. We validate effectiveness of our model in quantitative and qualitative experiments using NTIRE2020 real-world SR challenge dataset.
Learning Task-aware Local Representations for Few-shot Learning

Few-shot learning for visual recognition aims to adapt to novel unseen classes with only a few images. Recent work, especially the work based on low-level information, has achieved great progress. In these work, local representations (LRs) are typically employed, because LRs are more consistent among the seen and unseen classes. However, most of them are limited to an individual image-to-image or image-to-class measure manner, which cannot fully exploit the capabilities of LRs, especially in the context of a certain task. This paper proposes an Adaptive Task-aware Local Representations Network (ATL-Net) to address this limitation by introducing episodic attention, which can adaptively select the important local patches among the entire task, as the process of human recognition. We achieve much superior results on multiple benchmarks. On the miniImagenet, ATL-Net gains 0.93% and 0.88% improvements over the compared methods under the 5-way 1-shot and 5-shot settings. Moreover, ATL-Net can naturally tackle the problem that how to adaptively identify and weight the importance of different key local parts, which is the major concern of fine-grained recognition. Specifically, on the fine-grained dataset Stanford Dogs, ATL-Net outperforms the second best method with 5.39% and 9.69% gains under the 5-way 1-shot and 5-shot settings.
A Call for More Rigor in Unsupervised Cross-lingual Learning

We review motivations, definition, approaches, and methodology for unsupervised crosslingual learning and call for a more rigorous position in each of them.An existing rationale for such research is based on the lack of parallel data for many of the world's languages.However, we argue that a scenario without any parallel data and abundant monolingual data is unrealistic in practice.We also discuss different training signals that have been used in previous work, which depart from the pure unsupervised setting.We then describe common methodological issues in tuning and evaluation of unsupervised cross-lingual models and present best practices.Finally, we provide a unified outlook for different types of research in this area (i.e., cross-lingual word embeddings, deep multilingual pretraining, and unsupervised machine translation) and argue for comparable evaluation of these models.
Concept2Robot: Learning Manipulation Concepts from Instructions and Human Demonstrations

We aim to endow a robot with the ability to learn manipulation concepts that link natural language instructions to motor skills.Our goal is to learn a single multi-task policy that takes as input a natural language instruction and an image of the initial scene and outputs a robot motion trajectory to achieve the specified task.This policy has to generalize over different instructions and environments.Our insight is that we can approach this problem through Learning from Demonstration by leveraging large-scale video datasets of humans performing manipulation actions.Thereby, we avoid more time-consuming processes such as teleoperation or kinesthetic teaching.We also avoid having to manually design task-specific rewards.We propose a two-stage learning process where we first learn single-task policies through reinforcement learning.The reward is provided by scoring how well the robot visually appears to perform the task.This score is given by a video-based action classifier trained on a large-scale human activity dataset.In the second stage, we train a multi-task policy through imitation learning to imitate all the single-task policies.In extensive simulation experiments, we show that the multi-task policy learns to perform a large percentage of the 78 different manipulation tasks on which it was trained.The tasks are of greater variety and complexity than previously considered robot manipulation tasks.We show that the policy generalizes over variations of the environment.We also show examples of successful generalization over novel but similar instructions.
Mātauranga Māori in geomorphology: existing frameworks, case studies, and recommendations for incorporating Indigenous knowledge in Earth science

Abstract. Mixed-method bicultural research in Aotearoa New Zealand, including the weaving of Indigenous and other knowledge, is emerging within many academic disciplines. However, mātauranga Māori (the knowledge, culture, values, and world view of the Indigenous peoples of Aotearoa New Zealand) and Te Ao Māori (the Māori world) is poorly represented within geomorphological investigations. Here, we review international efforts to include Indigenous knowledge in geologic and geomorphic studies and provide an overview of the current state of mātauranga Māori within research endeavours in Aotearoa New Zealand. We review three theoretical frameworks (i.e. methodologies) for including mātauranga Māori in research projects and three models (i.e. methods) for including Māori values within research. We identify direct benefits to geomorphology and discuss how these frameworks and models can be adapted for use with Indigenous knowledge systems outside of Aotearoa New Zealand. The aim of this review is to encourage geomorphologists around the world to engage with local Indigenous peoples to develop new approaches to geomorphic research. In Aotearoa New Zealand, we hope to inspire geomorphologists to embark on research journeys in genuine partnership with Māori that promote toitū te mātauranga – the enduring protection, promotion and respect of mātauranga Māori.
Malicious UAV Detection Using Integrated Audio and Visual Features for Public Safety Applications

Unmanned aerial vehicles (UAVs) have become popular in surveillance, security, and remote monitoring. However, they also pose serious security threats to public privacy. The timely detection of a malicious drone is currently an open research issue for security provisioning companies. Recently, the problem has been addressed by a plethora of schemes. However, each plan has a limitation, such as extreme weather conditions and huge dataset requirements. In this paper, we propose a novel framework consisting of the hybrid handcrafted and deep feature to detect and localize malicious drones from their sound and image information. The respective datasets include sounds and occluded images of birds, airplanes, and thunderstorms, with variations in resolution and illumination. Various kernels of the support vector machine (SVM) are applied to classify the features. Experimental results validate the improved performance of the proposed scheme compared to other related methods.
Bridging the gap between constant step size stochastic gradient descent and Markov chains

We consider the minimization of a strongly convex objective function given access to unbiased estimates of its gradient through stochastic gradient descent (SGD) with constant step size.While the detailed analysis was only performed for quadratic functions, we provide an explicit asymptotic expansion of the moments of the averaged SGD iterates that outlines the dependence on initial conditions, the effect of noise and the step size, as well as the lack of convergence in the general (nonquadratic) case.For this analysis we bring tools from Markov chain theory into the analysis of stochastic gradient.We then show that Richardson-Romberg extrapolation may be used to get closer to the global optimum, and we show empirical improvements of the new extrapolation scheme.
Extensively Matching for Few-shot Learning Event Detection

Current event detection models under supervised learning settings fail to transfer to new event types.Few-shot learning has not been explored in event detection even though it allows a model to perform well with high generalization on new event types.In this work, we formulate event detection as a few-shot learning problem to enable to extend event detection to new event types.We propose two novel loss factors that matching examples in the support set to provide more training signals to the model.Moreover, these training signals can be applied in many metric-based few-shot learning models.Our extensive experiments on the ACE-2005 dataset (under a few-shot learning setting) show that the proposed method can improve the performance of few-shot learning.
Double-Blockchain Assisted Secure and Anonymous Data Aggregation for Fog-Enabled Smart Grid

As a future energy system, the smart grid is designed to improve the efficiency of traditional power systems while providing more stable and reliable services. However, this efficient and reliable service relies on collecting and analyzing users’ electricity consumption data frequently, which induces various security and privacy threats. To address these challenges, we propose a double-blockchain assisted secure and anonymous data aggregation scheme for fog-enabled smart grid named DA-SADA. Specifically, we design a three-tier architecture-based data aggregation framework by integrating fog computing and the blockchain, which provides strong support for achieving efficient and secure data collection in smart grids. Subsequently, we develop a secure and anonymous data aggregation mechanism with low computational overhead by jointly leveraging the Paillier encryption, batch aggregation signature and anonymous authentication. In particular, the system achieves fine-grained data aggregation and provides effective support for power dispatching and price adjustment by the designed double-blockchain and two-level data aggregation. Finally, the superiority of the proposed scheme is illustrated by a series of security and computation cost analyses.
A novel multi‐image visually meaningful encryption algorithm based on compressive sensing and Schur decomposition

Abstract Image encryption algorithm is an important telecommunication technology for protecting personal security. In this article, we propose a novel multi‐image visual encryption algorithm based on compressive sensing (CS) and Schur decomposition. First, multiple images are sparsed by a discrete wavelet transform, and the compressed images are obtained by zigzag confusion and CS. Second, these compressed images are combined according to random sequences and scrambled again to obtain the secret image. Third, the carrier image is subjected to a two‐level lifting wavelet transform where the low‐frequency and high‐frequency subbands are selected to perform a discrete cosine transform (DCT). Fourth, Schur decomposition is employed on the secret image to calculate an upper triangular matrix and a unitary matrix (orthogonal matrix) to be embedded into the DCT coefficients of the previous low‐frequency and high‐frequency subbands, respectively. With an inverse transformation, the final visually meaningful encrypted image is achieved. Here, the initial position of the zigzag confusion is related to the plain image, which can effectively resist known‐plaintext and chosen‐plaintext attacks. The simulation results demonstrate that the proposed algorithm has high security and good imperceptibility.
Fraud Transactions Detection via Behavior Tree with Local Intention Calibration

Fraud transactions obtain the rights and interests of e-commerce platforms by illegal ways, and have been the emerging threats to the healthy development of these platforms. Recently, user behavioral data is extensively exploited to detect fraud transactions, and it is usually processed as a sequence consisting of individual actions. However, such sequence-like user behaviors have logical patterns associated with user intentions, which motivates a fine-grained management strategy that binds and cuts off these actions into intention-related segments. In this paper, we devise a tree-like structure named behavior tree to reorganize the user behavioral data, in which a group of successive sequential actions denoting a specific user intention are represented as a branch on the tree. We then propose a novel neural method coined LIC Tree-LSTM(Local Intention Calibrated Tree-LSTM) to utilize the behavior tree for fraud transactions detection. In our LIC Tree-LSTM, the global user intention is captured by an attentional method applied on different branches. Then, we calibrate the entire tree by attentions within tree branches to pinpoint the balance between global and local user intentions. We investigate the effectiveness of LIC Tree-LSTM on a real-world dataset of Alibaba platform, and the experimental results show that our proposed algorithm outperforms state-of-the-art methods in both offline and online modes. Furthermore, our model provides good interpretability which helps us better understand user behaviors.
Layer-Specific Optimization for Mixed Data Flow With Mixed Precision in FPGA Design for CNN-Based Object Detectors

Convolutional neural networks (CNNs) require both intensive computation and frequent memory access, which lead to a low processing speed and large power dissipation. Although the characteristics of the different layers in a CNN are frequently quite different, previous hardware designs have employed common optimization schemes for them. This paper proposes a layer-specific design that employs different organizations that are optimized for the different layers. The proposed design employs two layer-specific optimizations: layer-specific mixed data flow and layer-specific mixed precision. The mixed data flow aims to minimize the off-chip access while demanding a minimal on-chip memory (BRAM) resource of an FPGA device. The mixed precision quantization is to achieve both a lossless accuracy and an aggressive model compression, thereby further reducing the off-chip access. A Bayesian optimization approach is used to select the best sparsity for each layer, achieving the best trade-off between the accuracy and compression. This mixing scheme allows the entire network model to be stored in BRAMs of the FPGA to aggressively reduce the off-chip access, and thereby achieves a significant performance enhancement. The model size is reduced by 22.66-28.93 times compared to that in a full-precision network with a negligible degradation of accuracy on VOC, COCO, and ImageNet datasets. Furthermore, the combination of mixed dataflow and mixed precision significantly outperforms the previous works in terms of both throughput, off-chip access, and on-chip memory requirement.
robosuite: A Modular Simulation Framework and Benchmark for Robot Learning

robosuite is a simulation framework for robot learning powered by the MuJoCo physics engine. It offers a modular design for creating robotic tasks as well as a suite of benchmark environments for reproducible research. This paper discusses the key system modules and the benchmark environments of our new release robosuite v1.0.
Resource Optimized Federated Learning-Enabled Cognitive Internet of Things for Smart Industries

Leveraging the cognitive Internet of things (C-IoT), emerging computing technologies, and machine learning schemes for industries can assist in streamlining manufacturing processes, revolutionizing operational analytics, and maintaining factory efficiency. However, further adoption of centralized machine learning in industries seems to be restricted due to data privacy issues. Federated learning has the potential to bring about predictive features in industrial systems without leaking private information. However, its implementation involves key challenges including resource optimization, robustness, and security. In this article, we propose a novel dispersed federated learning (DFL) framework to provide resource optimization, whereby distributed fashion of learning offers robustness. We formulate an integer linear optimization problem to minimize the overall federated learning cost for the DFL framework. To solve the formulated problem, first, we decompose it into two sub-problems: association and resource allocation problem. Second, we relax the association and resource allocation sub-problems to make them convex optimization problems. Later, we use the rounding technique to obtain binary association and resource allocation variables. Our proposed algorithm works in an iterative manner by fixing one problem variable (for example, association) and compute the other (for example, resource allocation). The iterative algorithm continues until convergence of the formulated cost optimization problem. Furthermore, we compare the proposed DFL with two schemes; namely, random resource allocation and random association. Numerical results show the superiority of the proposed DFL scheme.
Evolutionary Extreme Learning Machine with novel activation function for credit scoring

The term credit scoring is extensively used in credit industries for decision making and measuring the risk associated with an applicant. It uses applicants' historical data for credit risk evaluation by applying machine learning or statistical techniques. Credit risk evaluation has become progressively important field in financial risk management for credit industries. In this study, Extreme Learning Machine (ELM) is utilized as a classification tool for credit risk evaluation model. ELM requires more number of hidden neurons and random determination of the input weights and hidden biases. Moreover, ELM performance depends on activation function, weights and biases assigned to hidden neurons. An appropriate approach for selection of activation function, weights and biases may improve the performance of ELM. Hence, we have proposed a novel activation function and an evolutionary approach to get optimized weights and biases by utilizing Bat optimization algorithm. Further, the simulations are performed on four bench-marked credit scoring datasets with various activation functions. Simulation results demonstrate that proposed Evolutionary ELM (EELM) is more suitable for credit risk evaluation.
Deep anomaly detection through visual attention in surveillance videos

Abstract This paper describes a method for learning anomaly behavior in the video by finding an attention region from spatiotemporal information, in contrast to the full-frame learning. In our proposed method, a robust background subtraction (BG) for extracting motion, indicating the location of attention regions is employed. The resulting regions are finally fed into a three-dimensional Convolutional Neural Network (3D CNN). Specifically, by taking advantage of C3D (Convolution 3-dimensional), to completely exploit spatiotemporal relation, a deep convolution network is developed to distinguish normal and anomalous events. Our system is trained and tested against a large-scale UCF-Crime anomaly dataset for validating its effectiveness. This dataset contains 1900 long and untrimmed real-world surveillance videos and splits into 950 anomaly events and 950 normal events, respectively. In total, there are approximately ~ 13 million frames are learned during the training and testing phase. As shown in the experiments section, in terms of accuracy, the proposed visual attention model can obtain 99.25 accuracies. From the industrial application point of view, the extraction of this attention region can assist the security officer on focusing on the corresponding anomaly region, instead of a wider, full-framed inspection.
Digital contact tracing and exposure notification: ethical guidance for trustworthy pandemic management

Abstract There is growing interest in contact tracing apps (CT apps) for pandemic management. It is crucial to consider ethical requirements before, while, and after implementing such apps. In this paper, we illustrate the complexity and multiplicity of the ethical considerations by presenting an ethical framework for a responsible design and implementation of CT apps. Using this framework as a starting point, we briefly highlight the interconnection of social and political contexts, available measures of pandemic management, and a multi-layer assessment of CT apps. We will discuss some trade-offs that arise from this perspective. We then suggest that public trust is of major importance for population uptake of contact tracing apps. Hasty, ill-prepared or badly communicated implementations of CT apps will likely undermine public trust, and as such, risk impeding general effectiveness.
TuiGAN: Learning Versatile Image-to-Image Translation with Two Unpaired Images

An unsupervised image-to-image translation (UI2I) task deals with learning a mapping between two domains without paired images. While existing UI2I methods usually require numerous unpaired images from different domains for training, there are many scenarios where training data is quite limited. In this paper, we argue that even if each domain contains a single image, UI2I can still be achieved. To this end, we propose TuiGAN, a generative model that is trained on only two unpaired images and amounts to one-shot unsupervised learning. With TuiGAN, an image is translated in a coarse-to-fine manner where the generated image is gradually refined from global structures to local details. We conduct extensive experiments to verify that our versatile method can outperform strong baselines on a wide variety of UI2I tasks. Moreover, TuiGAN is capable of achieving comparable performance with the state-of-the-art UI2I models trained with sufficient data.