Reviewerly/exHarmony huggingface dataset:
STARC: Structured Annotations for Reading Comprehension

We present STARC (Structured Annotations for Reading Comprehension), a new annotation framework for assessing reading comprehension with multiple choice questions.Our framework introduces a principled structure for the answer choices and ties them to textual span annotations.The framework is implemented in OneStopQA, a new high-quality dataset for evaluation and analysis of reading comprehension in English.We use this dataset to demonstrate that STARC can be leveraged for a key new application for the development of SAT-like reading comprehension materials: automatic annotation quality probing via span ablation experiments.We further show that it enables in-depth analyses and comparisons between machine and human reading comprehension behavior, including error distributions and guessing ability.Our experiments also reveal that the standard multiple choice dataset in NLP, RACE (Lai et al., 2017), is limited in its ability to measure reading comprehension.47% of its questions can be guessed by machines without accessing the passage, and 18% are unanimously judged by humans as not having a unique correct answer.OneStopQA provides an alternative test set for reading comprehension which alleviates these shortcomings and has a substantially higher human ceiling performance. 1
Contextual Neural Machine Translation Improves Translation of Cataphoric Pronouns

The advent of context-aware NMT has resulted in promising improvements in the overall translation quality and specifically in the translation of discourse phenomena such as pronouns.Previous works have mainly focused on the use of past sentences as context with a focus on anaphora translation.In this work, we investigate the effect of future sentences as context by comparing the performance of a contextual NMT model trained with the future context to the one trained with the past context.Our experiments and evaluation, using generic and pronoun-focused automatic metrics, show that the use of future context not only achieves significant improvements over the context-agnostic Transformer, but also demonstrates comparable and in some cases improved performance over its counterpart trained on past context.We also perform an evaluation on a targeted cataphora test suite and report significant gains over the contextagnostic Transformer in terms of BLEU.
Learning with Noise: Improving Distantly-Supervised Fine-grained Entity Typing via Automatic Relabeling

Fine-grained entity typing (FET) is a fundamental task for various entity-leveraging applications. Although great success has been made, existing systems still have challenges in handling noisy samples in training data introduced by distant supervision methods. To address these noise, previous studies either focus on processing the clean samples (i,e., have only one label) and noisy samples (i,e., have multiple labels) with different strategies or filtering the noisy labels based on the assumption that the distantly-supervised label set certainly contains the correct type label. In this paper, we propose a probabilistic automatic relabeling method which treats all training samples uniformly. Our method aims to estimate the pseudo-truth label distribution of each sample, and the pseudo-truth distribution will be treated as part of trainable parameters which are jointly updated during the training process. The proposed approach does not rely on any prerequisite or extra supervision, making it effective on real applications. Experiments on several benchmarks show that our method outperforms previous approaches and alleviates the noisy labeling problem.
Adaptive Compression of Word Embeddings

Distributed representations of words have been an indispensable component for natural language processing (NLP) tasks.However, the large memory footprint of word embeddings makes it challenging to deploy NLP models to memory-constrained devices (e.g., selfdriving cars, mobile devices).In this paper, we propose a novel method to adaptively compress word embeddings.We fundamentally follow a code-book approach that represents words as discrete codes such as (8, 5, 2, 4).However, unlike prior works that assign the same length of codes to all words, we adaptively assign different lengths of codes to each word by learning downstream tasks.The proposed method works in two steps.First, each word directly learns to select its code length in an end-to-end manner by applying the Gumbel-softmax tricks.After selecting the code length, each word learns discrete codes through a neural network with a binary constraint.To showcase the general applicability of the proposed method, we evaluate the performance on four different downstream tasks.Comprehensive evaluation results clearly show that our method is effective and makes the highly compressed word embeddings without hurting the task accuracy.Moreover, we show that our model assigns word to each code-book by considering the significance of tasks.
THE EFFECT OF CURRICULUM 2013 ON ECONOMICS LEARNING ACHIEVEMENT: MOTIVATION AS A MEDIATING VARIABLE

The National Standard of Education (NSE) in Indonesia is a legitimate instrument to reach the best achievement. Yet, there are still the other positive factors, especially on economics subjects. This research aimed to find out the direct effect of the national standard of education (NSE) involving in a standard of content (SC), standard for educator and education staff (SEE), standard of facilities (SF), standard of assessment (SA), standard of the process (SP) and competence graduate standard (CGS) on achievement motivation (AM) and economics learning achievement (ELA), and to find out the indirect influence of SC, SEE, SF, SA, SP, and CGS on ELA through AM. This descriptive quantitative research used a survey to collect the data. The population is Indonesian senior high school students who have learned the economy for, at least. A total of 1065 students were selected through using proportional stratified random sampling with the Slovin formula (error=1%). The data in this study were collected through The data were subsequently analyized using Structural Equation Modeling (Partial Least Square approach). The result of this research showed that there was a direct effect of SC, SEE, SF, SA, SP, and CGS and AM on ELA. Then, there was an indirect effect of SC, SEE, SF, SA, and CGS on ELA through AM as a mediating variable.
Pattern recognition and artificial intelligence techniques for cultural heritage

This paper is the editorial of the virtual special issue (VSI) “Pattern recognition and artificial intelligence techniques for cultural heritage”, of which the authors of this paper have been the guest editors. It aims to bring together the work of experts from the fields of pattern recognition and artificial intelligence and that of cultural heritage. This multidisciplinary subject covers a wide spectrum spanning from the study of the cultural heritage to the development of tools based on PR/AI techniques for cultural heritage analysis, reconstruction and understanding. The papers included in this special issue allowed us to highlight the advances on this subject from a wide-angle perspective, as well as to stimulate new theoretical and applied researches for better characterizing the state of the art in this domain.
Deep Reinforcement Learning amidst Lifelong Non-Stationarity

As humans, our goals and our environment are persistently changing throughout our lifetime based on our experiences, actions, and internal and external drives. In contrast, typical reinforcement learning problem set-ups consider decision processes that are stationary across episodes. Can we develop reinforcement learning algorithms that can cope with the persistent change in the former, more realistic problem settings? While on-policy algorithms such as policy gradients in principle can be extended to non-stationary settings, the same cannot be said for more efficient off-policy algorithms that replay past experiences when learning. In this work, we formalize this problem setting, and draw upon ideas from the online learning and probabilistic inference literature to derive an off-policy RL algorithm that can reason about and tackle such lifelong non-stationarity. Our method leverages latent variable models to learn a representation of the environment from current and past experiences, and performs off-policy RL with this representation. We further introduce several simulation environments that exhibit lifelong non-stationarity, and empirically find that our approach substantially outperforms approaches that do not reason about environment shift.
Federated Learning Meets Multi-objective Optimization

Federated learning has emerged as a promising, massively distributed way to train a joint deep model over large amounts of edge devices while keeping private user data strictly on device. In this work, motivated from ensuring fairness among users and robustness against malicious adversaries, we formulate federated learning as multi-objective optimization and propose a new algorithm FedMGDA+ that is guaranteed to converge to Pareto stationary solutions. FedMGDA+ is simple to implement, has fewer hyperparameters to tune, and refrains from sacrificing the performance of any participating user. We establish the convergence properties of FedMGDA+ and point out its connections to existing approaches. Extensive experiments on a variety of datasets confirm that FedMGDA+ compares favorably against state-of-the-art.
Gestatten: Estimation of User's Attention in Mobile MOOCs From Eye Gaze and Gaze Gesture Tracking

The rapid proliferation of Massive Open Online Courses (MOOC) has resulted in many-fold increase in sharing the global classrooms through customized online platforms, where a student can participate in the classes through her personal devices, such as personal computers, smartphones, tablets, etc. However, in the absence of direct interactions with the students during the delivery of the lectures, it becomes difficult to judge their involvements in the classroom. In academics, the degree of student's attention can indicate whether a course is efficacious in terms of clarity and information. An automated feedback can hence be generated to enhance the utility of the course. The precision of discernment in the context of human attention is a subject of surveillance. However, visual patterns indicating the magnitude of concentration can be deciphered by analyzing the visual emphasis and the way an individual visually gesticulates, while contemplating the object of interest. In this paper, we develop a methodology called Gestsatten which captures the learner's attentiveness from his visual gesture patterns. In this approach, the learner's visual gestures are tracked along with the region of focus. We consider two aspects in this approach -- first, we do not transfer learner's video outside her device, so we apply in-device computing to protect her privacy; second, considering the fact that a majority of the learners use handheld devices like smartphones to observe the MOOC videos, we develop a lightweight approach for in-device computation. A three level estimation of learner's attention is performed based on these information. We have implemented and tested Gestatten over 48 participants from different age groups, and we observe that the proposed technique can capture the attention level of a learner with high accuracy (average absolute error rate is 8.68%), which meets her ability to learn a topic as measured through a set of cognitive tests.
Medical Image Enhancement Using Super Resolution Methods

Deep Learning image processing methods are gradually gaining popularity in a number of areas including medical imaging. Classification, segmentation, and denoising of images are some of the most demanded tasks. In this study, we aim at enhancing optic nerve head images obtained by Optical Coherence Tomography (OCT). However, instead of directly applying noise reduction techniques, we use multiple state-of-the-art image Super-Resolution (SR) methods. In SR, the low-resolution (LR) image is upsampled to match the size of the high-resolution (HR) image. With respect to image enhancement, the upsampled LR image can be considered as low quality, noisy image, and the HR image would be the desired enhanced version of it. We experimented with several image SR architectures, such as super-resolution Convolutional Neural Network (SRCNN), very deep Convolutional Network (VDSR), deeply recursive Convolutional Network (DRCN), and enhanced super-resolution Generative Adversarial Network (ESRGAN). Quantitatively, in terms of peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM), the SRCNN, VDSR, and DRCN significantly improved the test images. Although the ERSGAN showed the worst PSNR and SSIM, qualitatively, it was the best one.
Blockchain-Enabled Corporate Governance and Regulation

There is considerable hype about blockchain in almost every industry, including finance, with significant investments globally. We conduct a systematic review of 851 records and construct a final article sample of 183 for the sample period 2012 to 2020 to identify relevant factors for blockchain adoption in corporate governance. We conduct textual and empirical analysis to develop a decentralized autonomous governance framework and link traditional corporate governance theories to blockchain adoption. Furthermore, we explore present and future use cases and implications of blockchains in corporate governance. Using our systematic review and textual analysis, we further identify gaps and common trends between prior academic and industry literature. Moreover, for our empirical analysis, we compile a unique database of blockchain investments to forecast future investments. In addition, we explore blockchain potential in corporate governance during and post COVID-19. We find prior academic articles to mostly focus on regulation (49 studies) and Initial Coin Offerings (ICOs) (46 studies), while industry articles tend to concentrate on exchanges (10 studies) and cryptocurrencies (9 articles). A significant growth in literature is observed for 2017 and 2018. Finally, we provide behavioural, regulatory, ethical and managerial perspectives of blockchain adoption in corporate governance.
DeepAlign: Alignment-Based Process Anomaly Correction Using Recurrent Neural Networks

In this paper, we propose DeepAlign, a novel approach to multi-perspective process anomaly correction, based on recurrent neural networks and bidirectional beam search. At the core of the DeepAlign algorithm are two recurrent neural networks trained to predict the next event. One is reading sequences of process executions from left to right, while the other is reading the sequences from right to left. By combining the predictive capabilities of both neural networks, we show that it is possible to calculate sequence alignments, which are used to detect and correct anomalies. DeepAlign utilizes the case-level and event-level attributes to closely model the decisions within a process. We evaluate the performance of our approach on an elaborate data corpus of 252 realistic synthetic event logs and compare it to three state-of-the-art conformance checking methods. DeepAlign produces better corrections than the rest of the field reaching an overall $$F_1$$ score of 0.9572 across all datasets, whereas the best comparable state-of-the-art method reaches 0.6411.
uBLEU: Uncertainty-Aware Automatic Evaluation Method for Open-Domain Dialogue Systems

Because open-domain dialogues allow diverse responses, basic reference-based metrics such as BLEU do not work well unless we prepare a massive reference set of high-quality responses for input utterances.To reduce this burden, a human-aided, uncertainty-aware metric, ∆BLEU, has been proposed; it embeds human judgment on the quality of reference outputs into the computation of multiplereference BLEU.In this study, we instead propose a fully automatic, uncertainty-aware evaluation method for open-domain dialogue systems, υBLEU.This method first collects diverse reference responses from massive dialogue data and then annotates their quality judgments by using a neural network trained on automatically collected training data.Experimental results on massive Twitter data confirmed that υBLEU is comparable to ∆BLEU in terms of its correlation with human judgment and that the state of the art automatic evaluation method, RUBER, is improved by integrating υBLEU.
A Broader Study of Cross-Domain Few-Shot Learning

Recent progress on few-shot learning largely relies on annotated data for meta-learning: base classes sampled from the same domain as the novel classes. However, in many applications, collecting data for meta-learning is infeasible or impossible. This leads to the cross-domain few-shot learning problem, where there is a large shift between base and novel class domains. While investigations of the cross-domain few-shot scenario exist, these works are limited to natural images that still contain a high degree of visual similarity. No work yet exists that examines few-shot learning across different imaging methods seen in real world scenarios, such as aerial and medical imaging. In this paper, we propose the Broader Study of Cross-Domain Few-Shot Learning (BSCD-FSL) benchmark, consisting of image data from a diverse assortment of image acquisition methods. This includes natural images, such as crop disease images, but additionally those that present with an increasing dissimilarity to natural images, such as satellite images, dermatology images, and radiology images. Extensive experiments on the proposed benchmark are performed to evaluate state-of-art meta-learning approaches, transfer learning approaches, and newer methods for cross-domain few-shot learning. The results demonstrate that state-of-art meta-learning methods are surprisingly outperformed by earlier meta-learning approaches, and all meta-learning methods underperform in relation to simple fine-tuning by 12.8% average accuracy. Performance gains previously observed with methods specialized for cross-domain few-shot learning vanish in this more challenging benchmark. Finally, accuracy of all methods tend to correlate with dataset similarity to natural images, verifying the value of the benchmark to better represent the diversity of data seen in practice and guiding future research.
Fooling Automatic Short Answer Grading Systems

With the rising success of adversarial attacks on many NLP tasks, systems which actually operate in an adversarial scenario need to be reevaluated. For this purpose, we pose the following research question: How difficult is it to fool automatic short answer grading systems? In particular, we investigate the robustness of the state of the art automatic short answer grading system proposed by Sung et al. towards cheating in the form of universal adversarial trigger employment. These are short token sequences that can be prepended to students’ answers in an exam to artificially improve their automatically assigned grade. Such triggers are especially critical as they can easily be used by anyone once they are found. In our experiments, we discovered triggers which allow students to pass exams with passing thresholds of $$50\%$$ without answering a single question correctly. Furthermore, we show that such triggers generalize across models and datasets in this scenario, nullifying the defense strategy of keeping grading models or data secret.
A Short-Term Power Output Forecasting Model Based on Correlation Analysis and ELM-LSTM for Distributed PV System

Accurate short-term power output forecasting results are conducive to reducing the scheduling difficulty of grid-connected operation of distributed photovoltaic (PV) systems, thus improving the safety and stability of power grid operation. In this paper, a one-day-ahead short-term power output forecasting model based on correlation analysis and combination algorithms for distributed PV system is proposed to solve the problems within the current methods. Firstly, the basic information of distributed PV system is introduced, and the main influence factors affecting the power output of distributed PV system are determined. Secondly, the influence factors with higher correlation with PV output are selected by Spearman rank-order correlation coefficient (SROCC) analysis in multiple timescales. Then, based on the multimodel univariate extreme learning machine (ELM) submodel and the single-model multivariate long short-term memory (LSTM) submodel, the ELM-LSTM model is established. The case study analysis based on the actual data indicates that the ELM-LSTM forecasting model proposed in this paper has higher forecasting accuracy than the traditional forecasting methods.
Boltzmann machine learning with a variational quantum algorithm

A Boltzmann machine is a powerful tool for modeling probability distributions that govern the training data. A thermal equilibrium state is typically used for the Boltzmann machine learning to obtain a suitable probability distribution. The Boltzmann machine learning consists of calculating the gradient of the loss function given in terms of the thermal average, which is the most time-consuming procedure. Here, we propose a method to implement the Boltzmann machine learning by using noisy intermediate-scale quantum devices. We prepare an initial pure state that contains all possible computational basis states with the same amplitude, and we apply a variational imaginary time simulation. Readout of the state after the evolution in the computational basis approximates the probability distribution of the thermal equilibrium state that is used for the Boltzmann machine learning. We perform the numerical simulations of our scheme and confirm that the Boltzmann machine learning works well. Our scheme leads to a significant step toward an efficient machine learning using quantum hardware.
Conditional Set Generation with Transformers

A set is an unordered collection of unique elements--and yet many machine learning models that generate sets impose an implicit or explicit ordering. Since model performance can depend on the choice of order, any particular ordering can lead to sub-optimal results. An alternative solution is to use a permutation-equivariant set generator, which does not specify an order-ing. An example of such a generator is the DeepSet Prediction Network (DSPN). We introduce the Transformer Set Prediction Network (TSPN), a flexible permutation-equivariant model for set prediction based on the transformer, that builds upon and outperforms DSPN in the quality of predicted set elements and in the accuracy of their predicted sizes. We test our model on MNIST-as-point-clouds (SET-MNIST) for point-cloud generation and on CLEVR for object detection.
Using autoencoders for session-based job recommendations

Abstract In this work, we address the problem of providing job recommendations in an online session setting, in which we do not have full user histories. We propose a recommendation approach, which uses different autoencoder architectures to encode sessions from the job domain. The inferred latent session representations are then used in a k-nearest neighbor manner to recommend jobs within a session. We evaluate our approach on three datasets, (1) a proprietary dataset we gathered from the Austrian student job portal Studo Jobs, (2) a dataset released by XING after the RecSys 2017 Challenge and (3) anonymized job applications released by CareerBuilder in 2012. Our results show that autoencoders provide relevant job recommendations as well as maintain a high coverage and, at the same time, can outperform state-of-the-art session-based recommendation techniques in terms of system-based and session-based novelty.