Reviewerly/exHarmony huggingface dataset:
Relative Importance in Sentence Processing

Determining the relative importance of the elements in a sentence is a key factor for effortless natural language understanding.For human language processing, we can approximate patterns of relative importance by measuring reading fixations using eye-tracking technology.In neural language models, gradientbased saliency methods indicate the relative importance of a token for the target objective.In this work, we compare patterns of relative importance in English language processing by humans and models and analyze the underlying linguistic patterns.We find that human processing patterns in English correlate strongly with saliency-based importance in language models and not with attention-based importance.Our results indicate that saliency could be a cognitively more plausible metric for interpreting neural language models.The code is available on github: https://github.com/beinborn/relative_importance.
Cross-Domain Review Generation for Aspect-Based Sentiment Analysis

Supervised learning methods have proven to be effective for Aspect-Based Sentiment Analysis (ABSA).However, the lack of finegrained labeled data hinders their effectiveness in many domains.To address this issue, unsupervised domain adaptation methods are desired to transfer knowledge from a labeled source domain to any unlabeled target domain.In this paper, we propose a new domain adaptation paradigm called cross-domain review generation (CDRG), which aims to generate target-domain reviews with fine-grained annotation based on the source-domain labeled reviews.To achieve this goal, we propose a two-step approach as a concrete realization of CDRG.It first converts a sourcedomain review to a domain-independent review by masking its source-specific attributes, and then converts the domain-independent review to a target-domain review with a masked language model pre-trained in the target domain.We further propose two ways to leverage the generated target-domain reviews for two cross-domain ABSA tasks.Extensive experiments demonstrate the superiority of our CDRG-based approaches over the state-of-theart domain adaptation methods.
Augmented Partial Mutual Learning with Frame Masking for Video Captioning

Recent video captioning work improves greatly due to the invention of various elaborate model architectures. If multiple captioning models are combined into a unified framework not only by simple more ensemble, and each model can benefit from each other, the final captioning might be boosted further. Jointly training of multiple model have not been explored in previous works. In this paper, we propose a novel Augmented Partial Mutual Learning (APML) training method where multiple decoders are trained jointly with mimicry losses between different decoders and different input variations. Another problem of training captioning model is the "one-to-many" mapping problem which means that one identical video input is mapped to multiple caption annotations. To address this problem, we propose an annotation-wise frame masking approach to convert the "one-to-many" mapping to "one-to-one" mapping. The experiments performed on MSR-VTT and MSVD datasets demonstrate our proposed algorithm achieves the state-of-the-art performance.
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search

Despite transformers' impressive accuracy, their computational cost is often prohibitive to use with limited computational resources.Most previous approaches to improve inference efficiency require a separate model for each possible computational budget.In this paper, we extend PoWER-BERT (Goyal et al., 2020) and propose Length-Adaptive Transformer that can be used for various inference scenarios after one-shot training.We train a transformer with LengthDrop, a structural variant of dropout, which stochastically determines a sequence length at each layer.We then conduct a multi-objective evolutionary search to find a length configuration that maximizes the accuracy and minimizes the efficiency metric under any given computational budget.Additionally, we significantly extend the applicability of PoWER-BERT beyond sequence-level classification into tokenlevel classification with Drop-and-Restore process that drops word-vectors temporarily in intermediate layers and restores at the last layer if necessary.We empirically verify the utility of the proposed approach by demonstrating the superior accuracy-efficiency trade-off under various setups, including span-based question answering and text classification.Code is available at https://github.com/clovaai/lengthadaptive-transformer.
Free-Form Deformation Digital Image Correlation (FFD-DIC): A non-invasive spline regularization for arbitrary finite element measurements

Regularizing Finite-Element Digital Image Correlation (FE-DIC) problems is essential when using fine meshes, for instance to get a direct bridge with simulation tools or in view of performing data assimilation. Shape optimization brings comparable concerns so it is proposed to draw inspiration from this field of research. In this respect, a new general spline-based non-invasive regularization scheme is proposed, based on the Free-Form Deformation (FFD) concept that offers the opportunity to decouple the design space from that of the actual geometry to be updated. The main idea is to embed the input FE mesh, with any element type or size, into a simple cuboid B-spline box and to relate the nodal FE dof of the measured field to another more regular field discretized by the box. The regularization length is thus directly driven by the box refinement. A specific care is taken to handle possible conditioning problems caused by non-influential control points. From a practical point of view, the method can be interpreted as a projection on a reduced basis; therefore, it results in a regularized but conventional FE measured field which can be simply computed from a standard FE-DIC code. The developed scheme is applied for FE displacement measurement using DIC and for mesh-based measurement in stereo-DIC. For the latter, a novel procedure based on additional projections over local directions complements the FFD method for full efficiency. The performance of the strategy is assessed on real images coming from various experimental tests and comparisons are made with other published techniques to prove its effectiveness.
On the use of data from multiple mobile network operators in Europe to fight COVID-19

Abstract The rapid spread of COVID-19 infections on a global level has highlighted the need for accurate, transparent and timely information regarding collective mobility patterns to inform de-escalation strategies as well as to provide forecasting capacity for re-escalation policies aiming at addressing further waves of the virus. Such information can be extracted using aggregate anonymized data from innovative sources such as mobile positioning data. This paper presents lessons learnt and results of a unique Business-to-Government initiative between several mobile network operators in Europe and the European Commission. Mobile positioning data have supported policy-makers and practitioners with evidence and data-driven knowledge to understand and predict the spread of the disease, the effectiveness of the containment measures, their socio-economic impacts while feeding scenarios at European Union scale and in a comparable way across countries. The challenges of these data sharing initiative are not limited to data quality, harmonization, and comparability across countries, however important they are. Equally essential aspects that need to be addressed from the onset are related to data privacy, security, fundamental rights, and commercial sensitivity.
A Bottom-Up DAG Structure Extraction Model for Math Word Problems

Research on automatically solving mathematical word problems (MWP) has a long history. Most recent works adopt Seq2Seq approach to predict the result equations as a sequence of quantities and operators. Although result equations can be written as a sequence, it is essentially a structure. More precisely, it is a Direct Acyclic Graph (DAG) whose leaf nodes are the quantities, and internal and root nodes are arithmetic or comparison operators. In this paper, we propose a novel Seq2DAG approach to extract the equation set directly as a DAG structure. It is extracted in a bottom-up fashion by aggregating quantities and sub-expressions layer by layer iteratively. The advantages of our approach approach are three-fold: it is intrinsically suitable to solve multivariate problems, it always outputs valid structure, and its computation satisfies commutative law for +, x and =. Experimental results on Math23K and DRAW1K demonstrate that our model outperforms state-of-the-art deep learning methods. We also conduct detailed analysis on the results to show the strengths and limitations of our approach.
Semantic-guided Reinforced Region Embedding for Generalized Zero-Shot Learning

Generalized zero-shot Learning (GZSL) aims to recognize images from either seen or unseen domain, mainly by learning a joint embedding space to associate image features with the corresponding category descriptions. Recent methods have proved that localizing important object regions can effectively bridge the semantic-visual gap. However, these are all based on one-off visual localizers, lacking of interpretability and flexibility. In this paper, we propose a novel Semantic-guided Reinforced Region Embedding (SR2E) network that can localize important objects in the long-term interests to construct semantic-visual embedding space. SR2E consists of Reinforced Region Module (R2M) and Semantic Alignment Module (SAM). First, without the annotated bounding box as supervision, R2M encodes the semantic category guidance into the reward and punishment criteria to teach the localizer serialized region searching. Besides, R2M explores different action spaces during the serialized searching path to avoid local optimal localization, which thereby generates discriminative visual features with less redundancy. Second, SAM preserves the semantic relationship into visual features via semantic-visual alignment and designs a domain detector to alleviate the domain confusion. Experiments on four public benchmarks demonstrate that the proposed SR2E is an effective GZSL method with reinforced embedding space, which obtains averaged 6.1% improvements.
Meta-Learning Adversarial Domain Adaptation Network for Few-Shot Text Classification

Meta-learning has emerged as a trending technique to tackle few-shot text classification and achieved state-of-the-art performance.However, existing solutions heavily rely on the exploitation of lexical features and their distributional signatures on training data, while neglecting to strengthen the model's ability to adapt to new tasks.In this paper, we propose a novel meta-learning framework integrated with an adversarial domain adaptation network, aiming to improve the adaptive ability of the model and generate high-quality text embedding for new classes.Extensive experiments are conducted on four benchmark datasets and our method demonstrates clear superiority over the state-of-the-art models in all the datasets.In particular, the accuracy of 1shot and 5-shot classification on the dataset of 20 Newsgroups is boosted from 52.1% to 59.6%, and from 68.3% to 77.8%, respectively 1 .
Boosting Semi-supervised Image Segmentation with Global and Local Mutual Information Regularization

The scarcity of labeled data often impedes the application of deep learning to the segmentation of medical images. Semi-supervised learning seeks to overcome this limitation by exploiting unlabeled examples in the learning process. In this paper, we present a novel semi-supervised segmentation method that leverages mutual information (MI) on categorical distributions to achieve both global representation invariance and local smoothness. In this method, we maximize the MI for intermediate feature embeddings that are taken from both the encoder and decoder of a segmentation network. We first propose a global MI loss constraining the encoder to learn an image representation that is invariant to geometric transformations. Instead of resorting to computationally-expensive techniques for estimating the MI on continuous feature embeddings, we use projection heads to map them to a discrete cluster assignment where MI can be computed efficiently. Our method also includes a local MI loss to promote spatial consistency in the feature maps of the decoder and provide a smoother segmentation. Since mutual information does not require a strict ordering of clusters in two different assignments, we incorporate a final consistency regularization loss on the output which helps align the cluster labels throughout the network. We evaluate the method on four challenging publicly-available datasets for medical image segmentation. Experimental results show our method to outperform recently-proposed approaches for semi-supervised segmentation and provide an accuracy near to full supervision while training with very few annotated images.
MIDV-2020: a comprehensive benchmark dataset for identity document analysis

Identity documents recognition is an important sub-field of document analysis, which deals with tasks of robust document detection, type identification, text fields recognition, as well as identity fraud prevention and document authenticity validation given photos, scans, or video frames of an identity document capture. Significant amount of research has been published on this topic in recent years, however a chief difficulty for such research is scarcity of datasets, due to the subject matter being protected by security requirements. A few datasets of identity documents which are available lack diversity of document types, capturing conditions, or variability of document field values. In this paper, we present a dataset MIDV-2020 which consists of 1000 video clips, 2000 scanned images, and 1000 photos of 1000 unique mock identity documents, each with unique text field values and unique artificially generated faces, with rich annotation. The dataset contains 72409 annotated images in total, making it the largest publicly available identity document dataset to the date of publication. We describe the structure of the dataset, its content and annotations, and present baseline experimental results to serve as a basis for future research. For the task of document location and identification content-independent, feature-based, and semantic segmentation-based methods were evaluated. For the task of document text field recognition, the Tesseract system was evaluated on field and character levels with grouping by field alphabets and document types. For the task of face detection, the performance of Multi Task Cascaded Convolutional Neural Networks-based method was evaluated separately for different types of image input modes. The baseline evaluations show that the existing methods of identity document analysis have a lot of room for improvement given modern challenges. We believe that the proposed dataset will prove invaluable for advancement of the field of document analysis and recognition.
Improved Effort and Cost Estimation Model Using Artificial Neural Networks and Taguchi Method with Different Activation Functions

Software estimation involves meeting a huge number of different requirements, such as resource allocation, cost estimation, effort estimation, time estimation, and the changing demands of software product customers. Numerous estimation models try to solve these problems. In our experiment, a clustering method of input values to mitigate the heterogeneous nature of selected projects was used. Additionally, homogeneity of the data was achieved with the fuzzification method, and we proposed two different activation functions inside a hidden layer, during the construction of artificial neural networks (ANNs). In this research, we present an experiment that uses two different architectures of ANNs, based on Taguchi’s orthogonal vector plans, to satisfy the set conditions, with additional methods and criteria for validation of the proposed model, in this approach. The aim of this paper is the comparative analysis of the obtained results of mean magnitude relative error (MMRE) values. At the same time, our goal is also to find a relatively simple architecture that minimizes the error value while covering a wide range of different software projects. For this purpose, six different datasets are divided into four chosen clusters. The obtained results show that the estimation of diverse projects by dividing them into clusters can contribute to an efficient, reliable, and accurate software product assessment. The contribution of this paper is in the discovered solution that enables the execution of a small number of iterations, which reduces the execution time and achieves the minimum error.
Nested Named Entity Recognition with Partially-Observed TreeCRFs

Named entity recognition (NER) is a well-studied task in natural language processing. However, the widely-used sequence labeling framework is difficult to detect entities with nested structures. In this work, we view nested NER as constituency parsing with partially-observed trees and model it with partially-observed TreeCRFs. Specifically, we view all labeled entity spans as observed nodes in a constituency tree, and other spans as latent nodes. With the TreeCRF we achieve a uniform way to jointly model the observed and the latent nodes. To compute the probability of partial trees with partial marginalization, we propose a variant of the Inside algorithm, the Masked Inside algorithm, that supports different inference operations for different nodes (evaluation for the observed, marginalization for the latent, and rejection for nodes incompatible with the observed) with efficient parallelized implementation, thus significantly speeding up training and inference. Experiments show that our approach achieves the state-of-the-art (SOTA) F1 scores on the ACE2004, ACE2005 dataset, and shows comparable performance to SOTA models on the GENIA dataset. We release the code at https://github.com/FranxYao/Partially-Observed-TreeCRFs.
Twitter Event Summarization by Exploiting Semantic Terms and Graph Network

Twitter is a fast communication channel for gathering and spreading breaking news, and it generates a large volume of tweets for most events. Automatically creating a summary for an event is necessary and important. In this study, we explored two extractive approaches for summarizing events on Twitter. The first one exploits the semantic types of event related terms, and ranks the tweets based on the score computed from these semantic terms. The second one utilizes a graph convolutional network built from a tweet relation graph to generate tweet hidden features for tweet salience estimation. And the most salient tweets are selected as the summary of the event. Our experiments show that these two approaches outperform the compared methods.
Defending against Model Stealing via Verifying Embedded External Features

Obtaining a well-trained model involves expensive data collection and training procedures, therefore the model is a valuable intellectual property. Recent studies revealed that adversaries can `steal' deployed models even when they have no training samples and can not get access to the model parameters or structures. Currently, there were some defense methods to alleviate this threat, mostly by increasing the cost of model stealing. In this paper, we explore the defense from another angle by verifying whether a suspicious model contains the knowledge of defender-specified external features. Specifically, we embed the external features by tempering a few training samples with style transfer. We then train a meta-classifier to determine whether a model is stolen from the victim. This approach is inspired by the understanding that the stolen models should contain the knowledge of features learned by the victim model. We examine our method on both CIFAR-10 and ImageNet datasets. Experimental results demonstrate that our method is effective in detecting different types of model stealing simultaneously, even if the stolen model is obtained via a multi-stage stealing process. The codes for reproducing main results are available at Github (https://github.com/zlh-thu/StealingVerification).
The Prevalence of Code Smells in Machine Learning projects

Artificial Intelligence (AI) and Machine Learning (ML) are pervasive in the current computer science landscape. Yet, there still exists a lack of software engineering experience and best practices in this field. One such best practice, static code analysis, can be used to find code smells, i.e., (potential) defects in the source code, refactoring opportunities, and violations of common coding standards. Our research set out to discover the most prevalent code smells in ML projects. We gathered a dataset of 74 open-source ML projects, installed their dependencies and ran Pylint on them. This resulted in a top 20 of all detected code smells, per category. Manual analysis of these smells mainly showed that code duplication is widespread and that the PEP8 convention for identifier naming style may not always be applicable to ML code due to its resemblance with mathematical notation. More interestingly, however, we found several major obstructions to the maintainability and reproducibility of ML projects, primarily related to the dependency management of Python projects. We also found that Pylint cannot reliably check for correct usage of imported dependencies, including prominent ML libraries such as PyTorch.
SEAL-Embedded: A Homomorphic Encryption Library for the Internet of Things

The growth of the Internet of Things (IoT) has led to concerns over the lack of security and privacy guarantees afforded by IoT systems. Homomorphic encryption (HE) is a promising privacy-preserving solution to allow devices to securely share data with a cloud backend; however, its high memory consumption and computational overhead have limited its use on resource-constrained embedded devices. To address this problem, we present SEAL-Embedded, the first HE library targeted for embedded devices, featuring the CKKS approximate homomorphic encryption scheme. SEAL-Embedded employs several computational and algorithmic optimizations along with a detailed memory re-use scheme to achieve memory efficient, high performance CKKS encoding and encryption on embedded devices without any sacrifice of security. We additionally provide an “adapter” server module to convert data encrypted by SEAL-Embedded to be compatible with the Microsoft SEAL library for homomorphic encryption, enabling an end-to-end solution for building privacy-preserving applications. For a polynomial ring degree of 4096, using RNS primes of 30 or fewer bits, our library can be configured to use between 64–137 KB of RAM and 1–264 KB of flash data, depending on developer-selected configurations and tradeoffs. Using these parameters, we evaluate SEAL-Embedded on two different IoT platforms with high performance, memory efficient, and balanced configurations of the library for asymmetric and symmetric encryption. With 136 KB of RAM, SEAL-Embedded can perform asymmetric encryption of 2048 single-precision numbers in 77 ms on the Azure Sphere Cortex-A7 and 737 ms on the Nordic nRF52840 Cortex-M4.