Reviewerly/exHarmony huggingface dataset:
NLM at MEDIQA 2021: Transfer Learning-based Approaches for Consumer Question and Multi-Answer Summarization

The quest for seeking health information has swamped the web with consumers' healthrelated questions, which makes the need for efficient and reliable question answering systems more pressing.The consumers' questions, however, are very descriptive and contain several peripheral information (like patient's medical history, demographic information, etc.), that are often not required for answering the question.Furthermore, it contributes to the challenges of understanding natural language questions for automatic answer retrieval.Also, it is crucial to provide the consumers with the exact and relevant answers, rather than the entire pool of answer documents to their question.One of the cardinal tasks in achieving robust consumer health question answering systems is the question summarization and multi-document answer summarization.This paper describes the participation of the U.S. National Library of Medicine (NLM) in Consumer Question and Multi-Answer Summarization tasks of the MEDIQA 2021 challenge at NAACL-BioNLP workshop.In this work, we exploited the capabilities of pre-trained transformer models and introduced a transfer learning approach for the abstractive Question Summarization and extractive Multi-Answer Summarization tasks by first pre-training our model on a task-specific summarization dataset followed by fine-tuning it for both the tasks via incorporating medical entities.We achieved the second, sixth and the fourth position for the Question Summarization task in terms ROUGE-1, ROUGE-2 and ROUGE-L scores respectively.
Self-organizing migrating algorithm using covariance matrix adaptation evolution strategy for dynamic constrained optimization

The dynamic constrained optimization problems can be a challenge for the optimization algorithms. They must tackle global optimum detection, as well as the change of the environment. Recently, a novel test suite for dynamic constrained optimization was introduced. Furthermore, three well-performed evolutionary algorithms were compared based on it. The experimental results show that each algorithm performed best for a different type of optimization problem. The objective of our work was to develop an algorithm reflecting requirements arising from the novel test suite and regarding the results provided by the tested algorithms. In this work, we present a novel evolutionary algorithm for dynamic constrained optimization. The algorithm hybridizes the self-organizing migrating algorithm and the covariance matrix adaptation evolution strategy with constraints handling approach. To avoid premature convergence, the best solutions representing feasible regions do not affect the rest of the population. Two clustering methods, exclusion radius, and quantum particles are used to preserve population diversity. The performance is evaluated on the recently published test suite and compared to the three state-of-the-art algorithms. The presented algorithm outperformed these algorithms in most test cases, which indicates the efficiency of the utilized mechanisms.
The development of a low-cost photogrammetry-based 3D hand scanner

Acquiring an accurate 3D scan of the human hand is a challenging task, mainly due to the complicated geometry and the instability of the hand. In this paper, we present a low-cost photogrammetry-based scanner that is designed for scanning the human hand. The scanner has fifty modules, each has a Raspberry Pi with an 8-megapixels camera. They are uniformly positioned in two parallel frames and 96% of a hand surface can be viewed by at least 3 cameras. Using the timestamp method, we synchronize the shutters of the 50 cameras within the range of 80 ms to minimize the influence of the instability of the hand. Moreover, the scanner is easy to build with its modular design, and easy to operate with a laptop that is connected to the system by WiFi. Using a 3D printed prosthetic hand, we compared the 3D scanning accuracy of the proposed scanner with the Artec Spider® scanner. The mean absolute error between the two scans is 0.62 ± 0.28 mm. It is concluded that the proposed hand scanner can be used as a low-cost yet accurate tool in many applications, such as personalized product design.
Beyond Noise: Mitigating the Impact of Fine-grained Semantic Divergences on Neural Machine Translation

While it has been shown that Neural Machine Translation (NMT) is highly sensitive to noisy parallel training samples, prior work treats all types of mismatches between source and target as noise.As a result, it remains unclear how samples that are mostly equivalent but contain a small number of semantically divergent tokens impact NMT training.To close this gap, we analyze the impact of different types of fine-grained semantic divergences on Transformer models.We show that models trained on synthetic divergences output degenerated text more frequently and are less confident in their predictions.Based on these findings, we introduce a divergent-aware NMT framework that uses factors to help NMT recover from the degradation caused by naturally occurring divergences, improving both translation quality and model calibration on EN↔FR tasks.
LenSelect: Object Selection in Virtual Environments by Dynamic Object Scaling

Abstract We present a novel selection technique for VR called LenSelect . The main idea is to decrease the Index of Difficulty (ID) according to Fitts’ Law by dynamically increasing the size of the potentially selectable objects. This facilitates the selection process especially in cases of small, distant or partly occluded objects, but also for moving targets. In order to evaluate our method, we have defined a set of test scenarios that covers a broad range of use cases, in contrast to often used simpler scenes. Our test scenarios include practically relevant scenarios with realistic objects but also synthetic scenes, all of which are available for download. We have evaluated our method in a user study and compared the results to two state-of-the-art selection techniques and the standard ray-based selection. Our results show that LenSelect performs similar to the fastest method, which is ray-based selection, while significantly reducing the error rate by 44%.
ROSITA: Refined BERT cOmpreSsion with InTegrAted techniques

Pre-trained language models of the BERT family have defined the state-of-the-arts in a wide range of NLP tasks. However, the performance of BERT-based models is mainly driven by the enormous amount of parameters, which hinders their application to resource-limited scenarios. Faced with this problem, recent studies have been attempting to compress BERT into a small-scale model. However, most previous work primarily focuses on a single kind of compression technique, and few attention has been paid to the combination of different methods. When BERT is compressed with integrated techniques, a critical question is how to design the entire compression framework to obtain the optimal performance. In response to this question, we integrate three kinds of compression methods (weight pruning, low-rank factorization and knowledge distillation (KD)) and explore a range of designs concerning model architecture, KD strategy, pruning frequency and learning rate schedule. We find that a careful choice of the designs is crucial to the performance of the compressed model. Based on the empirical findings, our best compressed model, dubbed Refined BERT cOmpreSsion with InTegrAted techniques (ROSITA), is 7.5x smaller than BERT while maintains 98.5% of the performance on five tasks of the GLUE benchmark, outperforming the previous BERT compression methods with similar parameter budget.
EAML: ensemble self-attention-based mutual learning network for document image classification

In the recent past, complex deep neural networks have received huge interest in various document understanding tasks such as document image classification and document retrieval. As many document types have a distinct visual style, learning only visual features with deep CNNs to classify document images has encountered the problem of low inter-class discrimination, and high intra-class structural variations between its categories. In parallel, text-level understanding jointly learned with the corresponding visual properties within a given document image has considerably improved the classification performance in terms of accuracy. In this paper, we design a self-attention-based fusion module that serves as a block in our ensemble trainable network. It allows to simultaneously learn the discriminant features of image and text modalities throughout the training stage. Besides, we encourage mutual learning by transferring the positive knowledge between image and text modalities during the training stage. This constraint is realized by adding a truncated Kullback–Leibler divergence loss (Tr- $$\hbox {KLD}_{{\mathrm{Reg}}}$$ ) as a new regularization term, to the conventional supervised setting. To the best of our knowledge, this is the first time to leverage a mutual learning approach along with a self-attention-based fusion module to perform document image classification. The experimental results illustrate the effectiveness of our approach in terms of accuracy for the single-modal and multi-modal modalities. Thus, the proposed ensemble self-attention-based mutual learning model outperforms the state-of-the-art classification results based on the benchmark RVL-CDIP and Tobacco-3482 datasets.
Analysis of transaction flooding attacks against Monero

Monero was one of the first cryptocurrencies to address the problem of providing privacy-preserving digital asset trading. Currently, it has a market capitalization of over 2.5 billion US dollars and is among the 15 most valuable cryptocurrencies. This digital currency aims to protect users' identities and hide transaction information by using obfuscation mechanisms such as stealth addresses and ring signatures. However, in spite of the efforts to protect Monero's users' privacy, researchers have found ways to identify true payment keys within a ring signature in the past, making attacks against transaction privacy feasible. Since then, the system has received updates and adopted improved measures to provide privacy. This work presents an analysis on how an attacker can take advantage of the system's current settings to conduct both a high-profile transaction flooding attack and a stealthier version. Our results show that after flooding the network for 12 months, the attacker can identify the true spend of 46.24% of newly created transaction inputs by conducting the strongest attack and 14.47% by using the low-profile strategy.
Objective-Based Hierarchical Clustering of Deep Embedding Vectors

We initiate a comprehensive experimental study of objective-based hierarchical clustering methods on massive datasets consisting of deep embedding vectors from computer vision and NLP applications. This includes a large variety of image embedding (ImageNet, ImageNetV2, NaBirds), word embedding (Twitter, Wikipedia), and sentence embedding (SST-2) vectors from several popular recent models (e.g. ResNet, ResNext, Inception V3, SBERT). Our study includes datasets with up to 4.5 million data points with embedding dimensions up to 2048. In order to address the challenge of scaling up hierarchical clustering to such large datasets we propose a new practical hierarchical clustering algorithm B++&amp;C. It gives a 5%/20% improvement on average for the popular Moseley-Wang (MW)/ Cohen-Addad et al. (CKMM) objectives (normalized) compared to a wide range of classic methods and recent heuristics. We also introduce a theoretical algorithm B2SAT&amp;C which achieves a 0.74-approximation for the CKMM objective in polynomial time. This is the first substantial improvement over the trivial 2/3-approximation achieved by a random binary tree. Prior to this work, the best poly-time approximation of ≈2/3 + 0.0004 was due to Charikar et al. (SODA’19)
ABCnet: Adversarial bias correction network for infant brain MR images

Automatic correction of intensity nonuniformity (also termed as the bias correction) is an essential step in brain MR image analysis. Existing methods are typically developed for adult brain MR images based on the assumption that the image intensities within the same brain tissue are relatively uniform. However, this assumption is not valid in infant brain MR images, due to the dynamic and regionally-heterogeneous image contrast and appearance changes, which are caused by the underlying spatiotemporally-nonuniform myelination process. Therefore, it is not appropriate to directly use existing methods to correct the infant brain MR images. In this paper, we propose an end-to-end 3D adversarial bias correction network (ABCnet), tailored for direct prediction of bias fields from the input infant brain MR images for bias correction. The “ground-truth” bias fields for training our network are carefully defined by an improved N4 method, which integrates manually-corrected tissue segmentation maps as anatomical prior knowledge. The whole network is trained alternatively by minimizing generative and adversarial losses. To handle the heterogeneous intensity changes, our generative loss includes a tissue-aware local intensity uniformity term to reduce the local intensity variation in the corrected image. Besides, it also integrates two additional terms to enhance the smoothness of the estimated bias field and to improve the robustness of the proposed method, respectively. Comprehensive experiments with different sizes of training datasets have been carried out on a total of 1492 T1w and T2w MR images from neonates, infants, and adults, respectively. Both qualitative and quantitative evaluations on simulated and real datasets consistently demonstrate the superior performance of our ABCnet in both accuracy and efficiency, compared with popularly available methods.
Robust Value Iteration for Continuous Control Tasks

When transferring a control policy from simulation to a physical system, the policy needs to be robust to variations in the dynamics to perform well.Commonly, the optimal policy overfits to the approximate model and the corresponding statedistribution, often resulting in failure to trasnfer underlying distributional shifts.In this paper, we present Robust Fitted Value Iteration, which uses dynamic programming to compute the optimal value function on the compact state domain and incorporates adversarial perturbations of the system dynamics.The adversarial perturbations encourage a optimal policy that is robust to changes in the dynamics.Utilizing the continuoustime perspective of reinforcement learning, we derive the optimal perturbations for the states, actions, observations and model parameters in closed-form.Notably, the resulting algorithm does not require discretization of states or actions.Therefore, the optimal adversarial perturbations can be efficiently incorporated in the min-max value function update.We apply the resulting algorithm to the physical Furuta pendulum and cartpole.By changing the masses of the systems we evaluate the quantitative and qualitative performance across different model parameters.We show that robust value iteration is more robust compared to deep reinforcement learning algorithm and the non-robust version of the algorithm.Videos of the experiments are shown at https://sites.google.com/view/rfvi
An Empirical Study on Hyperparameter Optimization for Fine-Tuning Pre-trained Language Models

The performance of fine-tuning pre-trained language models largely depends on the hyperparameter configuration.In this paper, we investigate the performance of modern hyperparameter optimization methods (HPO) on finetuning pre-trained language models.First, we study and report three HPO algorithms' performances on fine-tuning two state-of-the-art language models on the GLUE dataset.We find that using the same time budget, HPO often fails to outperform grid search due to two reasons: insufficient time budget and overfitting.We propose two general strategies and an experimental procedure to systematically troubleshoot HPO's failure cases.By applying the procedure, we observe that HPO can succeed with more appropriate settings in the search space and time budget; however, in certain cases overfitting remains.Finally, we make suggestions for future work.Our implemen-
Core Challenges in Embodied Vision-Language Planning

Recent advances in the areas of multimodal machine learning and artificial intelligence (AI) have led to the development of challenging tasks at the intersection of Computer Vision, Natural Language Processing, and Embodied AI. Whereas many approaches and previous survey pursuits have characterised one or two of these dimensions, there has not been a holistic analysis at the center of all three. Moreover, even when combinations of these topics are considered, more focus is placed on describing, e.g., current architectural methods, as opposed to also illustrating high-level challenges and opportunities for the field. In this survey paper, we discuss Embodied Vision-Language Planning (EVLP) tasks, a family of prominent embodied navigation and manipulation problems that jointly use computer vision and natural language. We propose a taxonomy to unify these tasks and provide an in-depth analysis and comparison of the new and current algorithmic approaches, metrics, simulated environments, as well as the datasets used for EVLP tasks. Finally, we present the core challenges that we believe new EVLP works should seek to address, and we advocate for task construction that enables model generalizability and furthers real-world deployment.
Protecting the life and health of students from the negative impact of digital technologies

This article is devoted to the study of the negative aspects of the impact of the Internet on the life and health of persons who have not reached the age of majority in Russia (that is, the age of 18), as well as the criminal law and forensic possibilities of countering this. In the article, the authors goes by the types of information that are legally prohibited for distribution among minors in the Russian Federation. Having analyzed the types of this prohibited information, the authors identified exactly those types that can threaten the life and health of children. The article examines the problematic aspects of familiarizing minors with such information on the Internet, analyzes the norms of the Criminal Code of the Russian Federation, which provide for criminal liability for the distribution of this information on the Internet. The practical aspect of this study is expressed in the analysis of the issues of qualification of such crimes, their differentiation with similar elements of administrative offenses. As a result of the analysis of these criminal law norms, the authors have identified a number of gaps that need to be filled. In this regard, in order to increase the effectiveness of the criminal law struggle against the negative impact of the Internet on the life and health of minors, some amendments to the articles of the Criminal Code of the Russian Federation are proposed. The article also analyzes the forensic aspects applicable to this topic, namely, it examines some features and problems of identifying, disclosing and investigating crimes committed against minors with the use of the Internet. The author's conclusions and suggestions on this matter are expressed.
Correction to: Innovative deep learning models for EEG-based vigilance detection

A correction to this paper has been published: https://doi.org/10.1007/s00521-021-06187-0
The Case for Translation-Invariant Self-Attention in Transformer-Based Language Models

Mechanisms for encoding positional information are central for transformer-based language models.In this paper, we analyze the position embeddings of existing language models, finding strong evidence of translation invariance, both for the embeddings themselves and for their effect on self-attention.The degree of translation invariance increases during training and correlates positively with model performance.Our findings lead us to propose translation-invariant self-attention (TISA), which accounts for the relative position between tokens in an interpretable fashion without needing conventional position embeddings.Our proposal has several theoretical advantages over existing position-representation approaches.Experiments show that it improves on regular ALBERT on GLUE tasks, while only adding orders of magnitude less positional parameters.
Neural Extractive Search

Domain experts often need to extract structured information from large corpora.We advocate for a search paradigm called "extractive search", in which a search query is enriched with capture-slots, to allow for such rapid extraction.Such an extractive search system can be built around syntactic structures, resulting in high-precision, low-recall results.We show how the recall can be improved using neural retrieval and alignment.The goals of this paper are to concisely introduce the extractive-search paradigm; and to demonstrate a prototype neural retrieval system for extractive search and its benefits and potential.Our prototype is available at https://spike.neural-sim.apps.allenai.
Do Grammatical Error Correction Models Realize Grammatical Generalization?

There has been an increased interest in data generation approaches to grammatical error correction (GEC) using pseudo data.However, these approaches suffer from several issues that make them inconvenient for realworld deployment including a demand for large amounts of training data.On the other hand, some errors based on grammatical rules may not necessarily require a large amount of data if GEC models can realize grammatical generalization.This study explores to what extent GEC models generalize grammatical knowledge required for correcting errors.We introduce an analysis method using synthetic and real GEC datasets with controlled vocabularies to evaluate whether models can generalize to unseen errors.We found that a current standard Transformer-based GEC model fails to realize grammatical generalization even in simple settings with limited vocabulary and syntax, suggesting that it lacks the generalization ability required to correct errors from provided training examples.
XeroAlign: Zero-shot cross-lingual transformer alignment

The introduction of transformer-based crosslingual language models brought decisive improvements to multilingual NLP tasks.However, the lack of labelled data has necessitated a variety of methods that aim to close the gap to high-resource languages.Zero-shot methods in particular, often use translated task data as a training signal to bridge the performance gap between the source and target language(s).We introduce XeroAlign, a simple method for taskspecific alignment of cross-lingual pretrained transformers such as XLM-R.XeroAlign uses translated task data to encourage the model to generate similar sentence embeddings for different languages.The XeroAligned XLM-R, called XLM-RA, shows strong improvements over the baseline models to achieve state-ofthe-art zero-shot results on three multilingual natural language understanding tasks.XLM-RA performs on par with state-of-the-art models on a cross-lingual adversarial paraphrasing task and its text classification accuracy exceeds that of XLM-R trained with labelled data.