Reviewerly/exHarmony huggingface dataset:
Semantic visual navigation by watching YouTube videos

Semantic cues and statistical regularities in real-world environment layouts can improve efficiency for navigation in novel environments. This paper learns and leverages such semantic cues for navigating to objects of interest in novel environments, by simply watching YouTube videos. This is challenging because YouTube videos don't come with labels for actions or goals, and may not even showcase optimal behavior. Our method tackles these challenges through the use of Q-learning on pseudo-labeled transition quadruples (image, action, next image, reward). We show that such off-policy Q-learning from passive data is able to learn meaningful semantic cues for navigation. These cues, when used in a hierarchical navigation policy, lead to improved efficiency at the ObjectGoal task in visually realistic simulations. We observe a relative improvement of 15-83% over end-to-end RL, behavior cloning, and classical methods, while using minimal direct interaction.
Adversarial Style Mining for One-Shot Unsupervised Domain Adaptation

We aim at the problem named One-Shot Unsupervised Domain Adaptation. Unlike traditional Unsupervised Domain Adaptation, it assumes that only one unlabeled target sample can be available when learning to adapt. This setting is realistic but more challenging, in which conventional adaptation approaches are prone to failure due to the scarce of unlabeled target data. To this end, we propose a novel Adversarial Style Mining approach, which combines the style transfer module and task-specific module into an adversarial manner. Specifically, the style transfer module iteratively searches for harder stylized images around the one-shot target sample according to the current learning state, leading the task model to explore the potential styles that are difficult to solve in the almost unseen target domain, thus boosting the adaptation performance in a data-scarce scenario. The adversarial learning framework makes the style transfer module and task-specific module benefit each other during the competition. Extensive experiments on both cross-domain classification and segmentation benchmarks verify that ASM achieves state-of-the-art adaptation performance under the challenging one-shot setting.
Generating Image Descriptions via Sequential Cross-Modal Alignment Guided by Human Gaze

When speakers describe an image, they tend to look at objects before mentioning them.In this paper, we investigate such sequential crossmodal alignment by modelling the image description generation process computationally.We take as our starting point a state-of-theart image captioning system and develop several model variants that exploit information from human gaze patterns recorded during language production.In particular, we propose the first approach to image description generation where visual processing is modelled sequentially.Our experiments and analyses confirm that better descriptions can be obtained by exploiting gaze-driven attention and shed light on human cognitive processes by comparing different ways of aligning the gaze modality with language production.We find that processing gaze data sequentially leads to descriptions that are better aligned to those produced by speakers, more diverse, and more naturalparticularly when gaze is encoded with a dedicated recurrent component.
LOGAN: Local Group Bias Detection by Clustering

Machine learning techniques have been widely used in natural language processing (NLP). However, as revealed by many recent studies, machine learning models often inherit and amplify the societal biases in data. Various metrics have been proposed to quantify biases in model predictions. In particular, several of them evaluate disparity in model performance between protected groups and advantaged groups in the test corpus. However, we argue that evaluating bias at the corpus level is not enough for understanding how biases are embedded in a model. In fact, a model with similar aggregated performance between different groups on the entire data may behave differently on instances in a local region. To analyze and detect such local bias, we propose LOGAN, a new bias detection technique based on clustering. Experiments on toxicity classification and object classification tasks show that LOGAN identifies bias in a local region and allows us to better analyze the biases in model predictions.
Weakly Supervised Subevent Knowledge Acquisition

Subevents elaborate an event and widely exist in event descriptions.Subevent knowledge is useful for discourse analysis and event-centric applications.Acknowledging the scarcity of subevent knowledge, we propose a weakly supervised approach to extract subevent relation tuples from text and build the first large scale subevent knowledge base.We first obtain the initial set of event pairs that are likely to have the subevent relation, by exploiting two observations that 1) subevents are temporally contained by the parent event, and 2) the definitions of the parent event can be used to further guide the identification of subevents.Then, we collect rich weak supervision using the initial seed subevent pairs to train a contextual classifier using BERT and apply the classifier to identify new subevent pairs.The evaluation showed that the acquired subevent tuples (239K) are of high quality (90.1% accuracy) and cover a wide range of event types.The acquired subevent knowledge has been shown useful for discourse analysis and identifying a range of event-event relations 1 .
MedICaT: A Dataset of Medical Images, Captions, and Textual References

Understanding the relationship between figures and text is key to scientific document understanding.Medical figures in particular are quite complex, often consisting of several subfigures (75% of figures in our dataset), with detailed text describing their content.Previous work studying figures in scientific papers focused on classifying figure content rather than understanding how images relate to the text.To address challenges in figure retrieval and figure-to-text alignment, we introduce MED-ICAT, a dataset of medical images in context.MEDICAT consists of 217K images from 131K open access biomedical papers, and includes captions, inline references for 74% of figures, and manually annotated subfigures and subcaptions for a subset of figures.Using MEDICAT, we introduce the task of subfigure to subcaption alignment in compound figures and demonstrate the utility of inline references in image-text matching.Our data and code can be accessed at https://github.com/allenai/medicat.
Do We Need to Create Big Datasets to Learn a Task?

Deep Learning research has been largely accelerated by the development of huge datasets such as Imagenet.The general trend has been to create big datasets to make a deep neural network learn.A huge amount of resources is being spent in creating these big datasets, developing models, training them, and iterating this process to dominate leaderboards.We argue that the trend of creating bigger datasets needs to be revised by better leveraging the power of pre-trained language models.Since the language models have already been pretrained with huge amount of data and have basic linguistic knowledge, there is no need to create big datasets to learn a task.Instead, we need to create a dataset that is sufficient for the model to learn various task-specific terminologies, such as 'Entailment', 'Neutral', and 'Contradiction' for NLI.As evidence, we show that RoBERTA is able to achieve near-equal performance on ∼ 2% data of SNLI.We also observe competitive zero-shot generalization on several OOD datasets.In this paper, we propose a baseline algorithm to find the optimal dataset for learning a task.
Predicting Clinical Diagnosis from Patients Electronic Health Records Using BERT-Based Neural Networks

In this paper we study the problem of predicting clinical diagnoses from textual Electronic Health Records (EHR) data. We show the importance of this problem in medical community and present comprehensive historical review of the problem and proposed methods. As the main scientific contributions we present a modification of Bidirectional Encoder Representations from Transformers (BERT) model for sequence classification that implements a novel way of Fully-Connected (FC) layer composition and a BERT model pretrained only on domain data. To empirically validate our model, we use a large-scale Russian EHR dataset consisting of about 4 million unique patient visits. This is the largest such study for the Russian language and one of the largest globally. We performed a number of comparative experiments with other text representation models on the task of multiclass classification for 265 disease subset of ICD-10. The experiments demonstrate improved performance of our models compared to other baselines, including a fine-tuned Russian BERT (RuBERT) variant. We also show comparable performance of our model with a panel of experienced medical experts. This allows us to hope that implementation of this system will reduce misdiagnosis.
New insight into the Fourier-like and Darcy-like models in porous medium

In this study, we propose the general calculus operators based on the Richardson scaling law and Korcak scaling law. The Richardson-scaling-law calculus is considered to investigate the Fourier-like law for the scaling-law flow of the heat in the heat-transfer process. The Korcak-scaling-law calculus is used to model the Darcy-like law for describing the scaling-law flow of the fluid in porous medium. The formulas are as the special cases of the topology calculus proposed for descriptions of the fractal scaling-law behaviors in nature phenomena.
A simulation study on optimal scores for speaker recognition

Abstract In this article, we conduct a comprehensive simulation study for the optimal scores of speaker recognition systems that are based on speaker embedding. For that purpose, we first revisit the optimal scores for the speaker identification (SI) task and the speaker verification (SV) task in the sense of minimum Bayes risk (MBR) and show that the optimal scores for the two tasks can be formulated as a single form of normalized likelihood (NL). We show that when the underlying model is linear Gaussian, the NL score is mathematically equivalent to the PLDA likelihood ratio (LR), and the empirical scores based on cosine distance and Euclidean distance can be seen as approximations of this linear Gaussian NL score under some conditions.Based on the unified NL score, we conducted a comprehensive simulation study to investigate the behavior of the scoring component on both the SI task and SV task, in the case where the distribution of the speaker vectors perfectly matches the assumption of the NL model, as well as the case where some mismatch is involved. Importantly, our simulation is based on the statistics of speaker vectors derived from a practical speaker recognition system, hence reflecting the behavior of the NL scoring in real-life scenarios that are full of imperfection, including non-Gaussianality, non-homogeneity, and domain/condition mismatch.
Center-aligned domain adaptation network for image classification

For a target task where the labeled data are unavailable, unsupervised domain adaptive learning performs transfer learning from labeled source data to unlabeled target data. Previous deep domain adaption methods mainly learned the global domain shift between different domains, the global distributions are aligned without considering the correspondence information between the same class data of different domains. Recently, more and more researchers pay attention to semantic alignment that focuses on accurately aligning the distributions of the same class data from different domains. However, most of them ignore two points: the learning of the global distribution of the target domain data; the compactness of intra-class domain data and the discrimination of inter-class domain data, which lead to unsatisfying transfer learning performance. To resolve this problem, we propose a Center-aligned Domain Adaptation Network (CenterDA) to facilitate the semantic alignment, In this study, for each class in label space, we learn a common class center for all data with the same class label in the source and target domains, which allows us to learn the global distribution of the target domain data under the supervised learning of the source domain data. Furthermore, we minimize the distance between the deep features and its common class center to compact the feature representations of data. In this manner, we achieve the desired goals: The global distribution of the target domain data is learned by common class center. Second, the source and the target domain data of the same class are aligned near the common center. Third, we model the intra-class compactness and the inter-class separability modeling. Extensive experiments on three datasets show that our method achieves remarkable results on image classification and has comparable performance with the latest methods.
Learning Safe Neural Network Controllers with Barrier Certificates

We provide a novel approach to synthesize controllers for nonlinear continuous dynamical systems with control against safety properties. The controllers are based on neural networks (NNs). To certify the safety property we utilize barrier functions, which are also represented by NNs. We train controller-NN and barrier-NN simultaneously, achieving verification-in-the-loop synthesis. We provide a prototype tool nncontroller with a number of case studies. Preliminary experiment results confirm the feasibility and efficacy of our approach.
Race estimation with deep networks

Identifying race, which is a major physical feature in humans, is still a challenging task owing much to the lack of a concrete definition of race and the diversity of population across the globe. In this paper, we try to address the problem of race identification of four broad racial groups namely Caucasian, African, Asian and Indian. The newly developed BUPT Equalised Face dataset bearing about 1.3 M images in unrestricted environment is used to train our deep convolution network (R-Net) which achieves a state-of-the-art accuracy of 97%. To test the validity of this model it is evaluated on other datasets namely UTK and CFD. R-Net is also compared with fine-tuned VGG16 model for race estimation. Experimental results prove the robustness of this model for use in unconstrained environments. And finally, Gradient-weighted Class Activation Mapping (Grad-CAM) is applied to get a visual explanation of the deep learning model.
Object Detection Using Stacked YOLOv3

Object detection is a stimulating task in the applications of computer vision. It is gaining a lot of attention in many real-time applications such as detection of number plates of suspect cars, identifying trespassers under surveillance areas, detecting unmasked faces in security gates during the COVID-19 period, etc. Region-based Convolution Neural Networks(R-CNN), You only Look once (YOLO) based CNNs, etc., comes under Deep Learning approaches. In this proposed work, an improved stacked Yolov3 model is designed for the detection of objects by bounding boxes. Hyperparameters are tuned to get optimum performance. The proposed model evaluated using the COCO dataset, and the performance is better than other existing object detection models. Anchor boxes are used for overlapping objects. After removing all the predicted bounding boxes that have a low detection probability, bounding boxes with the highest detection probability are selected and eliminated all the bounding boxes whose Intersection Over Union value is higher than 0.4. Non-Maximal Suppression (NMS) is used to only keep the best bounding box. In this experimentation, we have tried with various range of values, but finally got better result at threshold 0.5.
Fingertip Gestures Recognition Using Leap Motion and Camera for Interaction with Virtual Environment

The emergence in computing and the latest hardware technologies realized the use of natural interaction with computers. Gesture-based interaction is one of the prominent fields of natural interactions. The recognition and application of hand gestures in virtual environments (VEs) need extensive calculations due to the complexities involved, which directly affect the performance and realism of interaction. In this paper, we propose a new interaction technique that uses single fingertip-based gestures for interaction with VEs. The objective of the study is to minimize the computational cost, increase performance, and improve usability. The interaction involves navigation, selection, translation, and release of objects. For this purpose, we propose a low-cost camera-based system that uses a colored fingertip for the fastest and accurate recognition of gestures. We also implemented the proposed interaction technique using the Leap Motion controller. We present a comparative analysis of the proposed system with the Leap Motion controller for gesture recognition and operation. A VE was developed for experimental purposes. Moreover, we conducted a comprehensive analysis of two different recognition setups including video camera and the Leap Motion sensor. The key parameters for analysis were task accuracy, interaction volume, update rate, and spatial distortion of accuracy. We used the Standard Usability Scale (SUS) for system usability analysis. The experiments revealed that camera implementation was found with good performance, less spatial distortion of accuracy, and large interaction volume as compared to the Leap Motion sensor. We also found the proposed interaction technique highly usable in terms of user satisfaction, user-friendliness, learning, and consistency.
G-DAUG: Generative data augmentation for commonsense reasoning

Recent advances in commonsense reasoning depend on large-scale human-annotated training data to achieve peak performance. However, manual curation of training examples is expensive and has been shown to introduce annotation artifacts that neural models can readily exploit and overfit on. We investigate G-DAUG^C, a novel generative data augmentation method that aims to achieve more accurate and robust learning in the low-resource setting. Our approach generates synthetic examples using pretrained language models, and selects the most informative and diverse set of examples for data augmentation. In experiments with multiple commonsense reasoning benchmarks, G-DAUG^C consistently outperforms existing data augmentation methods based on back-translation, and establishes a new state-of-the-art on WinoGrande, CODAH, and CommonsenseQA. Further, in addition to improvements in in-distribution accuracy, G-DAUG^C-augmented training also enhances out-of-distribution generalization, showing greater robustness against adversarial or perturbed examples. Our analysis demonstrates that G-DAUG^C produces a diverse set of fluent training examples, and that its selection and training approaches are important for performance. Our findings encourage future research toward generative data augmentation to enhance both in-distribution learning and out-of-distribution generalization.
A Neurophysiological Approach for Measuring Presence in Immersive Virtual Environments

Presence, the feeling of being there, is an important factor that affects the overall experience of Virtual Reality (VR). Higher presence commonly provides a better experience in VR than lower presence. However, presence is commonly measured subjectively through postexperience questionnaires, which can suffer from participant biases, dishonest answers, and fatigue. It can also be difficult for subjects to accurately remember their feelings of presence after they have left the VR experience. In this paper, we measured the effects of different levels of presence (high and low) in VR using physiological and neurological signals. The experiment involved 24 participants in a between-subjects design. Results indicated a significant effect of presence on both physiological and neurological signals. We noticed that higher presence results in higher heart rate, less visual stress, higher theta and beta activities in the frontal region, and higher alpha activities in the parietal region. These findings and insights could lead to an alternative objective measure of presence.
Assessing livelihood vulnerability using a Bayesian network: a case study in northern Laos

Agricultural transitions from subsistence to export-oriented production make households more reliant on volatile agricultural commodity markets and can increase households' exposure to crop price and yield shocks.At the same time, subsistence farming is also highly vulnerable to crop failures.In this work, we define household livelihood vulnerability as the probability of falling under an income threshold.We propose the use of a Bayesian network (BN) to calculate the income distribution based on household and community-level variables.BNs reflect relationships of dependence between variables and represent all variables as probability distributions, which allows for the explicit propagation of variability and uncertainty between variables.We focus on two agricultural frontier case study areas (CSAs) in northern Lao PDR that are at different stages in the transition from subsistence to export-oriented agriculture.Because agricultural production is the main livelihood activity in both CSAs, we develop a BN that calculates the probability distribution of net household agricultural production income.BN structure and parameterization are based on data collected in 110 household surveys across both CSAs, as well as interviews with villagers, government officials, and private sector actors.We analyze the effect of crop price and yield variability, land-use portfolio, and land holdings, on the probability of having a negative net agricultural income, which reflects a household's ability to meet its food consumption needs through cash crop sales.Results show that agricultural income is highly sensitive to rubber plantation area, rubber yield, and rubber price given the very large income potential of the crop.Households with larger agricultural areas have a lower probability of falling under an agricultural income threshold regardless of their diversification choices.Households that own more high-value cash crops are more buffered against rice yield shocks despite having higher agricultural income variability.However, low-income households are better off if they maintain a minimum level of rice sufficiency in combination with high-value cash crop production.Diversifying upland cash crops by increasing the share of cardamom (a lowvalue but low-volatility crop) at the expense of rubber (a highly lucrative crop with high price volatility) does not have a sizable beneficial impact, because returns from cardamom are significantly lower than for rubber.We show that BNs can be useful tools for the design and evaluation of rural development policies.