Reviewerly/exHarmony huggingface dataset:
Structured Self-AttentionWeights Encode Semantics in Sentiment Analysis

Neural attention, especially the self-attention made popular by the Transformer, has become the workhorse of state-of-the-art natural language processing (NLP) models.Very recent work suggests that the self-attention in the Transformer encodes syntactic information; Here, we show that self-attention scores encode semantics by considering sentiment analysis tasks.In contrast to gradient-based feature attribution methods, we propose a simple and effective Layer-wise Attention Tracing (LAT) method to analyze structured attention weights.We apply our method to Transformer models trained on two tasks that have surface dissimilarities, but share common semanticssentiment analysis of movie reviews and timeseries valence prediction in life story narratives.Across both tasks, words with high aggregated attention weights were rich in emotional semantics, as quantitatively validated by an emotion lexicon labeled by human annotators.Our results show that structured attention weights encode rich semantics in sentiment analysis, and match human interpretations of semantics.
Enhanced factorization machine via neural pairwise ranking and attention networks

The factorization machine models attract significant attention nowadays since they improve recommendation performance by incorporating context information into recommendation modeling. However, traditional factorization machine models often adopt the point-wise learning method for model parameter learning, as well as only model the linear interactions between features. They substantially fail to capture the complex interactions among features, which degrades the performance of factorization machine models. In this research, we propose a neural pairwise ranking factorization machine for item recommendation, namely NPRFM, which integrates the multi-layer perceptual neural networks into the pairwise ranking factorization machine model. Specifically, to capture the high-order and nonlinear interactions among features, we stack a multi-layer perceptual neural network over the bi-interaction layer, which encodes the second-order interactions between features. Moreover, instead of the prediction of the absolute scores, the pair-wise ranking model is adopted to learn the relative preferences of users. Since NPRFM does not take into account the importance of feature interactions, we propose a new variant of NPRFM, which learns the importance of feature interactions by introducing the attention mechanism. The empirical results on real-world datasets indicate that the proposed neural pairwise ranking factorization machine outperforms the traditional factorization machine models.
The Non-native Speaker Aspect: Indian English in Social Media

As the largest institutionalized second language variety of English, Indian English has received a sustained focus from linguists for decades.However, to the best of our knowledge, no prior study has contrasted webexpressions of Indian English in noisy social media with English generated by a social media user base that are predominantly native speakers.In this paper, we address this gap in the literature through conducting a comprehensive analysis considering multiple structural and semantic aspects.In addition, we propose a novel application of language models to perform automatic linguistic quality assessment.Error type
Enhanced Sentence Alignment Network for Efficient Short Text Matching

Cross-sentence attention has been widely applied in text matching, in which model learns the aligned information between two intermediate sequence representations to capture their semantic relationship.However, commonly the intermediate representations are generated solely based on the preceding layers and the models may suffer from error propagation and unstable matching, especially when multiple attention layers are used.In this paper, we propose an enhanced sentence alignment network with simple gated feature augmentation, where the model is able to flexibly integrate both original word and contextual features to improve the cross-sentence attention.Moreover, our model is less complex with fewer parameters compared to many state-of-the-art structures.Experiments on three benchmark datasets validate our model capacity for text matching.
Scene Restoring for Narrative Machine Reading Comprehension

This paper focuses on machine reading comprehension for narrative passages.Narrative passages usually describe a chain of events.When reading this kind of passage, humans tend to restore a scene according to the text with their prior knowledge, which helps them understand the passage comprehensively.Inspired by this behavior of humans, we propose a method to let the machine imagine a scene during reading narrative for better comprehension.Specifically, we build a scene graph by utilizing Atomic as the external knowledge and propose a novel Graph Dimensional-Iteration Network (GDIN) to encode the graph.We conduct experiments on the ROCStories, a dataset of Story Cloze Test (SCT), and Cos-mosQA, a dataset of multiple choice.Our method achieves state-of-the-art.
Sentiment Analysis of Tweets using Heterogeneous Multi-layer Network Representation and Embedding

Sentiment classification on tweets often needs to deal with the problems of under-specificity, noise, and multilingual content.This study proposes a heterogeneous multi-layer networkbased representation of tweets to generate multiple representations of a tweet and address the above issues.The generated representations are further ensembled and classified using a neural-based early fusion approach.Further, we propose a centrality aware random-walk for node embedding and tweet representations suitable for the multi-layer network.From various experimental analysis, it is evident that the proposed method can address the problem of under-specificity, noisy text, and multilingual content present in a tweet and provides better classification performance than the textbased counterparts.Further, the proposed centrality aware based random walk provides better representations than unbiased and other biased counterparts.
Multi-parameter estimation with multi-mode Ramsey interferometry

Abstract Estimating multiple parameters simultaneously is of great importance to measurement science and application. For a single parameter, atomic Ramsey interferometry (or equivalently optical Mach–Zehnder interferometry) is capable of providing the precision at the standard quantum limit (SQL) using unentangled probe states as input. In such an interferometer, the first beam splitter represented by unitary transformation U generates a quantum phase sensing superposition state, while the second beam splitter U −1 recombines the phase encoded paths to realize interferometric sensing in terms of population measurements. We prove that such an interferometric scheme can be directly generalized to estimation of multiple parameters (associated with commuting generators) to the SQL precision using multi-mode unentangled states, if (but not iff) U is orthogonal, i.e. a unitary transformation with only real matrix elements. We show that such a U can always be constructed experimentally in a simple and scalable manner. The effects of particle number fluctuation and detection noise on such multi-mode interferometry are considered. Our findings offer a simple solution for estimating multiple parameters corresponding to mutually commuting generators.
Abstractive Multi-Document Summarization via Joint Learning with Single-Document Summarization

Single-document and multi-document summarizations are very closely related in both task definition and solution method.In this work, we propose to improve neural abstractive multi-document summarization by jointly learning an abstractive single-document summarizer.We build a unified model for single-document and multi-document summarizations by fully sharing the encoder and decoder and utilizing a decoding controller to aggregate the decoder's outputs for multiple input documents.We evaluate our model on two multi-document summarization datasets: Multi-News and DUC-04.Experimental results show the efficacy of our approach, and it can substantially outperform several strong baselines.We also verify the helpfulness of single-document summarization to abstractive multi-document summarization task.
Uncertainty-Aware Semantic Augmentation for Neural Machine Translation

As a sequence-to-sequence generation task, neural machine translation (NMT) naturally contains intrinsic uncertainty, where a single sentence in one language has multiple valid counterparts in the other.However, the dominant methods for NMT only observe one of them from the parallel corpora for the model training but have to deal with adequate variations under the same meaning at inference.This leads to a discrepancy of the data distribution between the training and the inference phases.To address this problem, we propose uncertainty-aware semantic augmentation, which explicitly captures the universal semantic information among multiple semantically-equivalent source sentences and enhances the hidden representations with this information for better translations.Extensive experiments on various translation tasks reveal that our approach significantly outperforms the strong baselines and the existing methods.
Machine learning-based energy consumption modeling and comparison of H.264/AVC and google VP8 encoders

Advancement of the prediction models used in a variety of fields is a result of the contribution of machine learning approaches. Utilizing such modeling in feature engineering is exceptionally imperative and required. In this research, we show how to utilize machine learning to save time in research experiments, where we save more than five thousand hours of measuring the energy consumption of encoding recordings. Since measuring the energy consumption has got to be done by humans and since we require more than eleven thousand experiments to cover all the combinations of video sequences, video bit_rate, and video encoding settings, we utilize machine learning to model the energy consumption utilizing linear regression. VP8 codec has been offered by Google as an open video encoder in an effort to replace the popular MPEG-4 Part 10, known as H.264/AVC video encoder standard. This research model energy consumption and describes the major differences between H.264/AVC and VP8 encoders in terms of energy consumption and performance through experiments that are based on machine learning modeling. Twenty-nine raw video sequences are used, offering a wide range of resolutions and contents, with the frame sizes ranging from QCIF(176x144) to 2160p(3840x2160). For fairness in comparison analysis, we use seven settings in VP8 encoder and fifteen types of tuning in H.264/AVC. The settings cover various video qualities. The performance metrics include video qualities, encoding time, and encoding energy consumption.
Towards Context-Aware Code Comment Generation

Code comments are vital for software maintenance and comprehension, but many software projects suffer from the lack of meaningful and up-to-date comments in practice.This paper presents a novel approach to automatically generate code comments at a function level by targeting object-oriented programming languages.Unlike prior work that only uses information locally available within the target function, our approach leverages broader contextual information by considering all other functions of the same class.To propagate and integrate information beyond the scope of the target function, we design a novel learning framework based on the bidirectional gated recurrent unit and a graph attention network with a pointer mechanism.We apply our approach to produce code comments for Java methods and compare it against four strong baseline methods.Experimental results show that our approach outperforms most methods by a large margin and achieves a comparable result with the state-of-the-art method.
Control, Generate, Augment: A Scalable Framework for Multi-Attribute Text Generation

We introduce CGA, a conditional VAE architecture, to control, generate, and augment text.CGA is able to generate natural English sentences controlling multiple semantic and syntactic attributes by combining adversarial learning with a context-aware loss and a cyclical word dropout routine.We demonstrate the value of the individual model components in an ablation study.The scalability of our approach is ensured through a single discriminator, independently of the number of attributes.We show high quality, diversity and attribute control in the generated sentences through a series of automatic and human assessments.As the main application of our work, we test the potential of this new NLG model in a data augmentation scenario.In a downstream NLP task, the sentences generated by our CGA model show significant improvements over a strong baseline, and a classification performance often comparable to adding same amount of additional real data.
The Performance Analysis of Signal Recognition Using Attention Based CNN Method

Modulation recognition has always been an important task in the development of the cognitive radio. At present, there are two main application methods for signal data, namely, directly using the signal sequence and using some conversions such as constellation diagram. In this paper, the converted contour stella images are adopted as data source for research. The deep learning method has been proposed, which is called Image-based CNN with Attention Model (ICAM). ICAM is based on Residual Neural Network (ResNet). To evaluate the performance of ICAM, we generate a dataset which contains contour stella images involving of 8 kinds of modulation types with signal to noise ratio (SNR) from -6dB to 20dB. Compared with other state-of-the-art image-based methods, which are including those using constellation diagram and contour stellar images, ICAM makes more satisfactory performance. Besides, the Grad-Cam technology is applied to visualize and prove the effectiveness of ICAM.
Real-time Neural Networks Implementation Proposal for Microcontrollers

The adoption of intelligent systems with Artificial Neural Networks (ANNs) embedded in hardware for real-time applications currently faces a growing demand in fields like the Internet of Things (IoT) and Machine to Machine (M2M). However, the application of ANNs in this type of system poses a significant challenge due to the high computational power required to process its basic operations. This paper aims to show an implementation strategy of a Multilayer Perceptron (MLP) type neural network, in a microcontroller (a low-cost, low-power platform). A modular matrix-based MLP with the full classification process was implemented, and also the backpropagation training in the microcontroller. The testing and validation were performed through Hardware in the Loop (HIL) of the Mean Squared Error (MSE) of the training process, classification result, and the processing time of each implementation module. The results revealed a linear relationship between the values of the hyperparameters and the processing time required for classification, also the processing time concurs with the required time for many applications on the fields mentioned above. These findings show that this implementation strategy and this platform can be applied successfully on real-time applications that require the capabilities of ANNs.
Defogging Technology Based on Dual-Channel Sensor Information Fusion of Near-Infrared and Visible Light

Since the method to remove fog from images is complicated and detail loss and color distortion could occur to the defogged images, a defogging method based on near-infrared and visible image fusion is put forward in this paper. The algorithm in this paper uses the near-infrared image with rich details as a new data source and adopts the image fusion method to obtain a defog image with rich details and high color recovery. First, the colorful visible image is converted into HSI color space to obtain an intensity channel image, color channel image, and saturation channel image. The intensity channel image is fused with a near-infrared image and defogged, and then it is decomposed by Nonsubsampled Shearlet Transform. The obtained high-frequency coefficient is filtered by preserving the edge with a double exponential edge smoothing filter, while low-frequency antisharpening masking treatment is conducted on the low-frequency coefficient. The new intensity channel image could be obtained based on the fusion rule and by reciprocal transformation. Then, in color treatment of the visible image, the degradation model of the saturation image is established, which estimates the parameters based on the principle of dark primary color to obtain the estimated saturation image. Finally, the new intensity channel image, the estimated saturation image, and the primary color image are reflected to RGB space to obtain the fusion image, which is enhanced by color and sharpness correction. In order to prove the effectiveness of the algorithm, the dense fog image and the thin fog image are compared with the popular single image defogging and multiple image defogging algorithms and the visible light-near infrared fusion defogging algorithm based on deep learning. The experimental results show that the proposed algorithm is better in improving the edge contrast and the visual sharpness of the image than the existing high-efficiency defogging method.
Learning discrete distributions: user vs item-level privacy

Much of the literature on differential privacy focuses on item-level privacy, where loosely speaking, the goal is to provide privacy per item or training example. However, recently many practical applications such as federated learning require preserving privacy for all items of a single user, which is much harder to achieve. Therefore understanding the theoretical limit of user-level privacy becomes crucial. We study the fundamental problem of learning discrete distributions over $k$ symbols with user-level differential privacy. If each user has $m$ samples, we show that straightforward applications of Laplace or Gaussian mechanisms require the number of users to be $\mathcal{O}(k/(m\alpha^2) + k/\epsilon\alpha)$ to achieve an $\ell_1$ distance of $\alpha$ between the true and estimated distributions, with the privacy-induced penalty $k/\epsilon\alpha$ independent of the number of samples per user $m$. Moreover, we show that any mechanism that only operates on the final aggregate counts should require a user complexity of the same order. We then propose a mechanism such that the number of users scales as $\tilde{\mathcal{O}}(k/(m\alpha^2) + k/\sqrt{m}\epsilon\alpha)$ and hence the privacy penalty is $\tilde{\Theta}(\sqrt{m})$ times smaller compared to the standard mechanisms in certain settings of interest. We further show that the proposed mechanism is nearly-optimal under certain regimes. We also propose general techniques for obtaining lower bounds on restricted differentially private estimators and a lower bound on the total variation between binomial distributions, both of which might be of independent interest.
Variational Policy Gradient Method for Reinforcement Learning with General Utilities

In recent years, reinforcement learning (RL) systems with general goals beyond a cumulative sum of rewards have gained traction, such as in constrained problems, exploration, and acting upon prior experiences. In this paper, we consider policy optimization in Markov Decision Problems, where the objective is a general concave utility function of the state-action occupancy measure, which subsumes several of the aforementioned examples as special cases. Such generality invalidates the Bellman equation. As this means that dynamic programming no longer works, we focus on direct policy search. Analogously to the Policy Gradient Theorem \cite{sutton2000policy} available for RL with cumulative rewards, we derive a new Variational Policy Gradient Theorem for RL with general utilities, which establishes that the parametrized policy gradient may be obtained as the solution of a stochastic saddle point problem involving the Fenchel dual of the utility function. We develop a variational Monte Carlo gradient estimation algorithm to compute the policy gradient based on sample paths. We prove that the variational policy gradient scheme converges globally to the optimal policy for the general objective, though the optimization problem is nonconvex. We also establish its rate of convergence of the order $O(1/t)$ by exploiting the hidden convexity of the problem, and proves that it converges exponentially when the problem admits hidden strong convexity. Our analysis applies to the standard RL problem with cumulative rewards as a special case, in which case our result improves the available convergence rate.
Solar PV Power Forecasting Using Modified SVR with Gauss-Newton Method

Globally among 28% of renewable energy in total generation mix, solar photovoltaic (PV) contributes more than 23% and it is growing rapidly in recent years. Solar PV power forecasting plays an important role in grid integration of solar PV plants, optimum solar PV power utilization, energy market trading, ancillary services and maintaining power system stability and control. In last decade, machine learning methods, especially Support Vector Regression (SVR) are widely used in solar PV output power forecasting. SVR requires less amount of data for training, has less computational complexity and better forecasting accuracy. However, performance of SVR based forecasting models depends on the initial guess values provided in the training phase. Finding appropriate initial values using optimization methods increase computational time and complexity of these models. This paper proposes a simple, quick learning and robust method to modify the SVR model using Gauss-Newton method. The proposed model is tested to forecast power on sunny, partly cloudy and mostly cloudy day. It is compared with forecasted power by SVR and actual power output of a real solar PV plant. The modified model of SVR with Gauss-Newton method successfully converts nonlinear data of solar insolation and PV power output into linear data and achieves better forecasting accuracy than traditional SVR model in all three weather conditions.
Adversarial Augmentation Policy Search for Domain and Cross-Lingual Generalization in Reading Comprehension

Reading comprehension models often overfit to nuances of training datasets and fail at adversarial evaluation.Training with adversarially augmented dataset improves robustness against those adversarial attacks but hurts generalization of the models.In this work, we present several effective adversaries and automated data augmentation policy search methods with the goal of making reading comprehension models more robust to adversarial evaluation, but also improving generalization to the source domain as well as new domains and languages.We first propose three new methods for generating QA adversaries, that introduce multiple points of confusion within the context, show dependence on insertion location of the distractor, and reveal the compounding effect of mixing adversarial strategies with syntactic and semantic paraphrasing methods.Next, we find that augmenting the training datasets with uniformly sampled adversaries improves robustness to the adversarial attacks but leads to decline in performance on the original unaugmented dataset.We address this issue via RL and more efficient Bayesian policy search methods for automatically learning the best augmentation policy combinations of the transformation probability for each adversary in a large search space.Using these learned policies, we show that adversarial training can lead to significant improvements in in-domain, out-of-domain, and crosslingual (German, Russian, Turkish) generalization.