Reviewerly/exHarmony huggingface dataset:
PEVAE

Variational autoencoders (VAEs) have been widely applied in recommendations. One reason is that their amortized inferences are beneficial for overcoming the data sparsity. However, in explainable recommendation that generates natural language explanations, they are still rarely explored. Thus, we aim to extend VAE to explainable recommendation. In this task, we find that VAE can generate acceptable explanations for users with few relevant training samples, however, it tends to generate less personalized explanations for users with relatively sufficient samples than autoencoders (AEs). We conjecture that information shared by different users in VAE disturbs the information for a specific user. To deal with this problem, we present PErsonalized VAE (PEVAE) that generates personalized natural language explanations for explainable recommendation. Moreover, we propose two novel mechanisms to aid our model in generating more personalized explanations, including 1) Self-Adaption Fusion (SAF) manipulates the latent space in a self-adaption manner for controlling the influence of shared information. In this way, our model can enjoy the advantage of overcoming the sparsity of data while generating more personalized explanations for a user with relatively sufficient training samples. 2) DEpendence Maximization (DEM) strengthens dependence between recommendations and explanations by maximizing the mutual information. It makes the explanation more specific to the input user-item pair and thus improves the personalization of the generated explanations. Extensive experiments show PEVAE can generate more personalized explanations and further analyses demonstrate the practical effect of our proposed methods.
Simulation Dataset Preparation and Hybrid Training for Deep Learning in Defect Detection Using Digital Shearography

Since real experimental shearography images are usually few, the application of deep learning for defect detection in digital shearography is limited. A simulation dataset preparation method of shearography images is proposed in this paper. Firstly, deformation distributions are estimated by finite element analysis (FEA); secondly, phase maps are calculated according to the optical shearography system; finally, simulated shearography images are obtained after 2π modulus and gray transform. Various settings in the parameters of object, defect, load and shearing in those three steps could prepare a diverse simulation dataset for deep learning. Together with the real experimental images taken from a shearography setup, hybrid trainings of deep learning for defect detection are performed and discussed. The results show that a simulation dataset, generated without any real defective specimen, shearography system or manual experiment, can greatly improve the generalization of a deep learning network when the number of experimental training images is small.
Tracking Fast Trajectories with a Deformable Object using a Learned Model

We propose a method for robotic control of deformable objects using a learned nonlinear dynamics model. After collecting a dataset of trajectories from the real system, we train a recurrent neural network (RNN) to approximate its input-output behavior with a latent state-space model. The RNN internal state is low-dimensional enough to enable realtime nonlinear control methods. We demonstrate a closed-loop control scheme with the RNN model using a standard nonlinear state observer and model-predictive controller. We apply our method to track a highly dynamic trajectory with a point on the deformable object, in real time and on real hardware. Our experiments show that the RNN model captures the true system's frequency response and can be used to track trajectories outside the training distribution. In an ablation study, we find that the full method improves tracking accuracy compared to an open-loop version without the state observer.
FORTAP: Using Formulas for Numerical-Reasoning-Aware Table Pretraining

Tables store rich numerical data, but numerical reasoning over tables is still a challenge.In this paper, we find that the spreadsheet formula, a commonly used language to perform computations on numerical values in spreadsheets, is valuable supervision for numerical reasoning in tables.Considering large amounts of spreadsheets available on the web, we propose FORTAP , the first exploration to leverage spreadsheet formulas for table pretraining.Two novel self-supervised pretraining objectives are derived from formulas, numerical reference prediction (NRP) and numerical calculation prediction (NCP).While our proposed objectives are generic for encoders, to better capture spreadsheet table layouts and structures, we build FORTAP upon TUTA, the first transformer-based method for spread-sheet&web table pretraining with tree attention.FORTAP outperforms state-of-the-art methods by large margins on three representative datasets of formula prediction, question answering, and cell type classification, showing the great potential of leveraging formulas for table pretraining.The code will be released at https://github.com/microsoft/TUTA_table_understanding.
Integration Analysis of Heterogeneous Data on Mind Externalization of Elderly People at Home

As the aging population around the world, figuring out the reason for changes in the health status of elderly adults at home is pressing. In our research group, to comprehend the scientific self-care of elderly adults at home, a core concept called “mind externalization” that retrieves the elders’ thoughts as much as possible using spoken dialogue agent technology has developed swiftly. The purpose of this paper is to consider an approach to elucidating reasons for health changes in the elderly at home. Our key idea is to merge the results of health status, dialogue logs, and emotional values (recognized from images and audio during the spoken dialogue) into a time series. More specifically, we describe an approach for extracting the features of changes in health data (i.e., heart rate, stress, sleep quality, step, and activity level from the wearable device). It intends to add health data retrieved from a wearable device and unite heterogeneous data (i.e., health data and dialogue data). Based on the integration between health data and dialogue data (i.e., text logs, audio, and images), we discuss an approach to estimating the reasoning context before and after the period. In this way, assisting elderly adults at home by grasping their daily living in detail can be appreciated. Meanwhile, executing personalized self-management is promising.
Updated Headline Generation: Creating Updated Summaries for Evolving News Stories

We propose the task of updated headline generation, in which a system generates a headline for an updated article, considering both the previous article and headline.The system must identify the novel information in the article update, and modify the existing headline accordingly.We create data for this task using the NewsEdits corpus (Spangher and May, 2021) by automatically identifying contiguous article versions that are likely to require a substantive headline update.We find that models conditioned on the prior headline and body revisions produce headlines judged by humans to be as factual as gold headlines while making fewer unnecessary edits compared to a standard headline generation model.Our experiments establish benchmarks for this new contextual summarization task.
Deep Metric Learning on the SPD Manifold for Image Set Classification

Thanks to the efficacy of Symmetric Positive Definite (SPD) manifold in characterizing video sequences (image sets), image set-based visual classification has made remarkable progress. However, the issue of large intra-class diversity and inter-class similarity is still an open challenge for the research community. Although several recent studies have alleviated the above issue by constructing Riemannian neural networks for SPD matrix nonlinear processing, the degradation of structural information during multi-stage feature transformation impedes them from going deeper. Besides, a single cross-entropy loss is insufficient for discriminative learning as it neglects the peculiarities of data distribution. To this end, this paper develops a novel framework for image set classification. Specifically, we first choose a mainstream neural network built on the SPD manifold (SPDNet) [25] as the backbone with a stacked SPD manifold autoencoder (SSMAE) built on the tail to enrich the structured representations. Due to the associated reconstruction error terms, the embedding mechanism of both SSMAE and each SPD manifold autoencoder (SMAE) forms an approximate identity mapping, simplifying the training of the suggested deeper network. Then, the ReCov layer is introduced with a nonlinear function for the constructed architecture to narrow the discrepancy of the intra-class distributions from the perspective of regularizing the local statistical information of the SPD data. Afterward, two progressive metric learning stages are coupled with the proposed SSMAE to explicitly capture, encode, and analyze the geometric distributions of the generated deep representations during training. In consequence, not only a more powerful Riemannian network embedding but also effective classifiers can be obtained. Finally, a simple maximum voting strategy is applied to the outputs of the learned multiple classifiers for classification. The proposed model is evaluated on three typical visual classification tasks using widely adopted benchmarking datasets. Extensive experiments show its superiority over the state of the arts.
ParaUDA: Invariant Feature Learning With Auxiliary Synthetic Samples for Unsupervised Domain Adaptation

Recognizing and locating objects by algorithms are essential and challenging issues for Intelligent Transportation Systems. However, the increasing demand for much labeled data hinders the further application of deep learning-based object detection. One of the optimal solutions is to train the target model with an existing dataset and then adapt it to new scenes, namely Unsupervised Domain Adaptation (UDA). However, most of existing methods at the pixel level mainly focus on adapting the model from source domain to target domain and ignore the essence of UDA to learn domain-invariant feature learning. Meanwhile, almost all methods at the feature level ignore to make conditional distributions matched for UDA while conducting feature alignment between source and target domain. Considering these problems, this paper proposes the ParaUDA, a novel framework of learning invariant representations for UDA in two aspects: pixel level and feature level. At the pixel level, we adopt CycleGAN to conduct domain transfer and convert the problem of original unsupervised domain adaptation to supervised domain adaptation. At the feature level, we adopt an adversarial adaption model to learn domain-invariant representation by aligning the distributions of domains between different image pairs with same mixture distributions. We evaluate our proposed framework in different scenes, from synthetic scenes to real scenes, from normal weather to challenging weather, and from scenes across cameras. The results of all the above experiments show that ParaUDA is effective and robust for adapting object detection models from source scenes to target scenes.
Less Descriptive yet Discriminative: Quantifying the Properties of Multimodal Referring Utterances via CLIP

In this work, we use a transformer-based pretrained multimodal model, CLIP, to shed light on the mechanisms employed by human speakers when referring to visual entities.In particular, we use CLIP to quantify the degree of descriptiveness (how well an utterance describes an image in isolation) and discriminativeness (to what extent an utterance is effective in picking out a single image among similar images) of human referring utterances within multimodal dialogues.Overall, our results show that utterances become less descriptive over time while their discriminativeness remains unchanged.Through analysis, we propose that this trend could be due to participants relying on the previous mentions in the dialogue history, as well as being able to distill the most discriminative information from the visual context.In general, our study opens up the possibility of using this and similar models to quantify patterns in human data and shed light on the underlying cognitive mechanisms.
Crowd Counting via Unsupervised Cross-Domain Feature Adaptation

Given an image, crowd counting aims to estimate the amount of target objects in the image. With un-predictable installation situations of surveillance systems (or other equipments), crowd counting images from different data sets may exhibit severe discrepancies in viewing angle, scale, lighting condition, etc. As it is usually expensive and time-consuming to annotate each data set for model training, it has been an essential issue in crowd counting to transfer a well-trained model on a labeled data set (source domain) to a new data set (target domain). To tackle this problem, we propose a cross-domain learning network to learn the domain gaps in an unsupervised learning manner. The proposed network comprises of a Multi-granularity Feature-aware Discriminator (MFD) module, a Domain-invariant Feature Adaptation (DFA) module, and a Cross-domain Vanishing Bridge (CVB) module to remove domain-specific information from the extracted features and promote the mapping performances of the network. Unlike most existing methods that use only Global Feature Discriminator (GFD) to align features at image level, an additional Local Feature Discriminator (LFD) is inserted and together with GFD form the MFD module. As a complement to MFD, LFD refines features at pixel level and has the ability to align local features. The DFA module explicitly measures the distances between the source domain features and the target domain features and aligns the marginal distribution of their features with Maximum Mean Discrepancy (MMD). Finally, the CVB module provides an incremental capability of removing the impact of interfering part of the extracted features. Several well-known networks are adopted as the backbone of our algorithm to prove the effectiveness of the proposed adaptation structure. Comprehensive experiments demonstrate that our model achieves competitive performance to the state-of-the-art methods.
On the Importance of Data Size in Probing Fine-tuned Models

Several studies have investigated the reasons behind the effectiveness of fine-tuning, usually through the lens of probing.However, these studies often neglect the role of the size of the dataset on which the model is fine-tuned.In this paper, we highlight the importance of this factor and its undeniable role in probing performance.We show that the extent of encoded linguistic knowledge depends on the number of fine-tuning samples.The analysis also reveals that larger training data mainly affects higher layers, and that the extent of this change is a factor of the number of iterations updating the model during fine-tuning rather than the diversity of the training samples.Finally, we show through a set of experiments that finetuning data size affects the recoverability of the changes made to the model's linguistic knowledge.
Open data: The building block of 21st century (open) science

Abstract This paper identifies the potential benefits of data sharing and open science, supported by artificial intelligence tools and services, and dives into the challenges to make data open and findable, accessible, interoperable, and reusable (FAIR).
Synthetic Question Value Estimation for Domain Adaptation of Question Answering

Synthesizing QA pairs with a question generator (QG) on the target domain has become a popular approach for domain adaptation of question answering (QA) models.Since synthetic questions are often noisy in practice, existing work adapts scores from a pretrained QA (or QG) model as criteria to select highquality questions.However, these scores do not directly serve the ultimate goal of improving QA performance on the target domain.In this paper, we introduce a novel idea of training a question value estimator (QVE) that directly estimates the usefulness of synthetic questions for improving the target-domain QA performance.By conducting comprehensive experiments, we show that the synthetic questions selected by QVE can help achieve better target-domain QA performance, in comparison with existing techniques.We additionally show that by using such questions and only around 15% of the human annotations on the target domain, we can achieve comparable performance to the fully-supervised baselines. 1
Practical Vertical Federated Learning with Unsupervised Representation Learning

As societal concerns on data privacy recently increase, we have witnessed data silos among multiple parties in various applications. Federated learning emerges as a new learning paradigm that enables multiple parties to collaboratively train a machine learning model without sharing their raw data. Vertical federated learning, where each party owns different features of the same set of samples and only a single party has the label, is an important and challenging topic in federated learning. Communication costs among different parties have been a major hurdle for practical vertical learning systems. In this paper, we propose a novel communication-efficient vertical federated learning algorithm named FedOnce, which requires only one-shot communication among parties. To improve model accuracy and provide privacy guarantee, FedOnce features unsupervised learning representations in the federated setting and privacy-preserving techniques based on moments accountant. The comprehensive experiments on 10 datasets demonstrate that FedOnce achieves close performance compared to state-of-the-art vertical federated learning algorithms with much lower communication costs. Meanwhile, our privacy-preserving technique significantly outperforms the state-of-the-art approaches under the same privacy budget.
Learning When to Translate for Streaming Speech

How to find proper moments to generate partial sentence translation given a streaming speech input?Existing approaches waitingand-translating for a fixed duration often break the acoustic units in speech, since the boundaries between acoustic units in speech are not even.In this paper, we propose MoSST, a simple yet effective method for translating streaming speech content.Given a usually long speech sequence, we develop an efficient monotonic segmentation module inside an encoder-decoder model to accumulate acoustic information incrementally and detect proper speech unit boundaries for the input in speech translation task.Experiments on multiple translation directions of the MuST-C dataset show that MoSST outperforms existing methods and achieves the best trade-off between translation quality (BLEU) and latency.Our code is available at https://github.com/dqqcasia/mosst.
A global analysis of metrics used for measuring performance in natural language processing

Measuring the performance of natural language processing models is challenging.Traditionally used metrics, such as BLEU and ROUGE, originally devised for machine translation and summarization, have been shown to suffer from low correlation with human judgment and a lack of transferability to other tasks and languages.In the past 15 years, a wide range of alternative metrics have been proposed.However, it is unclear to what extent this has had an impact on NLP benchmarking efforts.Here we provide the first large-scale cross-sectional analysis of metrics used for measuring performance in natural language processing.We curated, mapped and systematized more than 3500 machine learning model performance results from the open repository 'Papers with Code' to enable a global and comprehensive analysis.Our results suggest that the large majority of natural language processing metrics currently used have properties that may result in an inadequate reflection of a models' performance.Furthermore, we found that ambiguities and inconsistencies in the reporting of metrics may lead to difficulties in interpreting and comparing model performances, impairing transparency and reproducibility in NLP research.
LSCDiscovery: A shared task on semantic change discovery and detection in Spanish

We present the first shared task on semantic change discovery and detection in Spanish and create the first dataset of Spanish words manually annotated for semantic change using the DURel framework (Schlechtweg et al., 2018).The task is divided in two phases: 1) Graded Change Discovery, and 2) Binary Change Detection.In addition to introducing a new language the main novelty with respect to the previous tasks consists in predicting and evaluating changes for all vocabulary words in the corpus.Six teams participated in phase 1 and seven teams in phase 2 of the shared task, and the best system obtained a Spearman rank correlation of 0.735 for phase 1 and an F1 score of 0.716 for phase 2. We describe the systems developed by the competing teams, highlighting the techniques that were particularly useful and discuss the limits of these approaches.
An ethico-legal assessment of intellectual property rights and their effect on COVID-19 vaccine distribution: an Australian case study

This article posits that Australia, as an affluent country with increasing capacity to manufacture vaccines, has an obligation to assist its regional (and global) counterparts in implementing vaccination programs that protect their populations. First, the article explores the capacity of high-income nations to meet their obligations, assist their neighbours and refrain from vaccine nationalism. This inquiry involves an analysis of the optimal ethical strategy for distributing vaccines globally, and the role that Australia might play in this distribution strategy. Secondly, the article examines the intellectual property landscape for vaccines in Australia, focusing on the patents that cover vaccine compositions and manufacturing techniques (recognizing the potential for know-how and access to materials as well as patents to affect manufacturing capacity). This article then discusses the strategies the Australian Government has at its disposal to counter potential intellectual property impediments whilst complying with existing obligations under the Agreement on Trade-related Aspects of Intellectual Property Rights (TRIPS), as an ethically appropriate response to the pandemic. This article also considers whether a so-called TRIPS waiver could provide better options and concludes that the challenge of compelling disclosure of know-how remains.
Affective word embedding in affective explanation generation for fine art paintings

Fine art paintings take an important place in human history and are one of the fundamental components of human culture. Even though deep learning in fine art paintings attracts increasing attention from researchers, few works focus on understanding the interplay between the visual content, its triggered affect, and language aiming to explain that affect. Most visual captioning models can only deal with tasks of generating captions describing objective affairs but lack the capabilities of generating affective explanations. In this paper, we introduce the use of the VAD (Valence, Arousal, and Dominance) dictionary in our model and propose a gated concatenation mechanism to construct word affective embedding. Corporating with the use of the affective loss function, our model outperforms the state-of-the-art in automatic evaluation metrics and subjective evaluations.