Miracle-dz/UniFAR-FTUs huggingface dataset:
Inflation from the relaxation of the cosmological constant

We propose a model of inflation driven by the relaxation of an initially Planckian cosmological constant due to diffusion. The model can solve the horizon and flatness problem, and generate a (approximately) scale invariant spectrum of (adiabatic) primordial perturbations with the correct amplitudes and blue tilt without an inflaton. The inhomogeneities observable in the CMB arise from those associated to the fundamental Planckian granularity that are imprinted into the standard model Higgs scalar fluctuations during the inflationary phase. The process admits a semiclassical interpretation and avoids the trans-Planckian problem of standard inflationary scenarios based on the role of vacuum fluctuations. The deviations from scale invariance observed in the CMB are controlled by known parameters of the standard model of particle physics. If an order one portion of these fluctuations collapses to form primordial black holes the model produces the amount of cold dark matter required by observations.
Identifying the Limits of Cross-Domain Knowledge Transfer for Pretrained Models

There is growing evidence that pretrained language models improve task-specific fine-tuning even where the task examples are radically different from those seen in training. We study an extreme case of transfer learning by providing a systematic exploration of how much transfer occurs when models are denied any information about word identity via random scrambling. In four classification tasks and two sequence labeling tasks, we evaluate LSTMs using GloVe embeddings, BERT, and baseline models. Among these models, we find that only BERT shows high rates of transfer into our scrambled domains, and for classification but not sequence labeling tasks. Our analyses seek to explain why transfer succeeds for some tasks but not others, to isolate the separate contributions of pretraining versus fine-tuning, to show that the fine-tuning process is not merely learning to unscramble the scrambled inputs, and to quantify the role of word frequency. Furthermore, our results suggest that current benchmarks may overestimate the degree to which current models actually understand language.
"Wikily" Neural Machine Translation Tailored to Cross-Lingual Tasks

We present a simple but effective approach for leveraging Wikipedia for neural machine translation as well as cross-lingual tasks of image captioning and dependency parsing without using any direct supervision from external parallel data or supervised models in the target language. We show that first sentences and titles of linked Wikipedia pages, as well as cross-lingual image captions, are strong signals for a seed parallel data to extract bilingual dictionaries and cross-lingual word embeddings for mining parallel text from Wikipedia. Our final model achieves high BLEU scores that are close to or sometimes higher than strong supervised baselines in low-resource languages; e.g. supervised BLEU of 4.0 versus 12.1 from our model in English-to-Kazakh. Moreover, we tailor our wikily translation models to unsupervised image captioning and cross-lingual dependency parser transfer. In image captioning, we train a multi-tasking machine translation and image captioning pipeline for Arabic and English from which the Arabic training data is a wikily translation of the English captioning data. Our captioning results in Arabic are slightly better than that of its supervised model. In dependency parsing, we translate a large amount of monolingual text, and use it as an artificial training data in an annotation projection framework. We show that our model outperforms recent work on cross-lingual transfer of dependency parsers.
Distributed Derivative-Free Learning Method for Stochastic Optimization Over a Network With Sparse Activity

This article addresses a distributed optimization problem in a communication network where nodes are active sporadically. Each active node applies some learning method to control its action to maximize the global utility function, which is defined as the sum of the local utility functions of active nodes. We deal with stochastic optimization problem with the setting that utility functions are disturbed by some nonadditive stochastic process. We consider a more challenging situation where the learning method has to be performed only based on a scalar approximation of the utility function, rather than its closed-form expression, so that the typical gradient descent method cannot be applied. This setting is quite realistic when the network is affected by some stochastic and time-varying process, and that each node cannot have the full knowledge of the network states. We propose a distributed optimization algorithm and prove its almost surely convergence to the optimum. Convergence rate is also derived with an additional assumption that the objective function is strongly concave. Numerical results are also presented to justify our claim.
Unsupervised Shape Completion via Deep Prior in the Neural Tangent Kernel Perspective

We present a novel approach for completing and reconstructing 3D shapes from incomplete scanned data by using deep neural networks. Rather than being trained on supervised completion tasks and applied on a testing shape, the network is optimized from scratch on the single testing shape to fully adapt to the shape and complete the missing data using contextual guidance from the known regions. The ability to complete missing data by an untrained neural network is usually referred to as the deep prior. In this article, we interpret the deep prior from a neural tangent kernel (NTK) perspective and show that the completed shape patches by the trained CNN are naturally similar to existing patches, as they are proximate in the kernel feature space induced by NTK. The interpretation allows us to design more efficient network structures and learning mechanisms for the shape completion and reconstruction task. Being more aware of structural regularities than both traditional and other unsupervised learning-based reconstruction methods, our approach completes large missing regions with plausible shapes and complements supervised learning-based methods that use database priors by requiring no extra training dataset and showing flexible adaptation to a particular shape instance.
Using Machine Learning at Scale in HPC Simulations with SmartSim: An Application to Ocean Climate Modeling

We demonstrate the first climate-scale, numerical ocean simulations improved through distributed, online inference of Deep Neural Networks (DNN) using SmartSim. SmartSim is a library dedicated to enabling online analysis and Machine Learning (ML) for traditional HPC simulations. In this paper, we detail the SmartSim architecture and provide benchmarks including online inference with a shared ML model on heterogeneous HPC systems. We demonstrate the capability of SmartSim by using it to run a 12-member ensemble of global-scale, high-resolution ocean simulations, each spanning 19 compute nodes, all communicating with the same ML architecture at each simulation timestep. In total, 970 billion inferences are collectively served by running the ensemble for a total of 120 simulated years. Finally, we show our solution is stable over the full duration of the model integrations, and that the inclusion of machine learning has minimal impact on the simulation runtimes.
Agent-Centric Representations for Multi-Agent Reinforcement Learning

Object-centric representations have recently enabled significant progress in tackling relational reasoning tasks. By building a strong object-centric inductive bias into neural architectures, recent efforts have improved generalization and data efficiency of machine learning algorithms for these problems. One problem class involving relational reasoning that still remains under-explored is multi-agent reinforcement learning (MARL). Here we investigate whether object-centric representations are also beneficial in the fully cooperative MARL setting. Specifically, we study two ways of incorporating an agent-centric inductive bias into our RL algorithm: 1. Introducing an agent-centric attention module with explicit connections across agents 2. Adding an agent-centric unsupervised predictive objective (i.e. not using action labels), to be used as an auxiliary loss for MARL, or as the basis of a pre-training step. We evaluate these approaches on the Google Research Football environment as well as DeepMind Lab 2D. Empirically, agent-centric representation learning leads to the emergence of more complex cooperation strategies between agents as well as enhanced sample efficiency and generalization.
Causal Program Dependence Analysis

We introduce Causal Program Dependence Analysis (CPDA), a dynamic dependence analysis that applies causal inference to model the strength of program dependence relations in a continuous space. CPDA observes the association between program elements by constructing and executing modified versions of a program. One advantage of CPDA is that this construction requires only light-weight parsing rather than sophisticated static analysis. The result is a collection of observations based on how often a change in the value produced by a mutated program element affects the behavior of other elements. From this set of observations, CPDA discovers a causal structure capturing the causal (i.e., dependence) relation between program elements. Qualitative evaluation finds that CPDA concisely expresses key dependence relationships between program elements. As an example application, we apply CPDA to the problem of fault localization. Using minimal test suites, our approach can rank twice as many faults compared to SBFL.
An Uncertainty-aware Hierarchical Probabilistic Network for Early Prediction, Quantification and Segmentation of Pulmonary Tumour Growth

Early detection and quantification of tumour growth would help clinicians to prescribe more accurate treatments and provide better surgical planning. However, the multifactorial and heterogeneous nature of lung tumour progression hampers identification of growth patterns. In this study, we present a novel method based on a deep hierarchical generative and probabilistic framework that, according to radiological guidelines, predicts tumour growth, quantifies its size and provides a semantic appearance of the future nodule. Unlike previous deterministic solutions, the generative characteristic of our approach also allows us to estimate the uncertainty in the predictions, especially important for complex and doubtful cases. Results of evaluating this method on an independent test set reported a tumour growth balanced accuracy of 74%, a tumour growth size MAE of 1.77 mm and a tumour segmentation Dice score of 78%. These surpassed the performances of equivalent deterministic and alternative generative solutions (i.e. probabilistic U-Net, Bayesian test dropout and Pix2Pix GAN) confirming the suitability of our approach.
On the Use of Context for Predicting Citation Worthiness of Sentences in Scholarly Articles

In this paper, we study the importance of context in predicting the citation worthiness of sentences in scholarly articles. We formulate this problem as a sequence labeling task solved using a hierarchical BiLSTM model. We contribute a new benchmark dataset containing over two million sentences and their corresponding labels. We preserve the sentence order in this dataset and perform document-level train/test splits, which importantly allows incorporating contextual information in the modeling process. We evaluate the proposed approach on three benchmark datasets. Our results quantify the benefits of using context and contextual embeddings for citation worthiness. Lastly, through error analysis, we provide insights into cases where context plays an essential role in predicting citation worthiness.
Who Responded to Whom: The Joint Effects of Latent Topics and Discourse in Conversation Structure

Numerous online conversations are produced on a daily basis, resulting in a pressing need to conversation understanding. As a basis to structure a discussion, we identify the responding relations in the conversation discourse, which link response utterances to their initiations. To figure out who responded to whom, here we explore how the consistency of topic contents and dependency of discourse roles indicate such interactions, whereas most prior work ignore the effects of latent factors underlying word occurrences. We propose a model to learn latent topics and discourse in word distributions, and predict pairwise initiation-response links via exploiting topic consistency and discourse dependency. Experimental results on both English and Chinese conversations show that our model significantly outperforms the previous state of the arts, such as 79 vs. 73 MRR on Chinese customer service dialogues. We further probe into our outputs and shed light on how topics and discourse indicate conversational user interactions.
Do admissions metrics predict PhD completion directly, or indirectly through graduate GPA?

Graduate admissions committees are slowly ending the use of GRE scores due to concerns that they restrict access to underrepresented groups but have limited predictive utility. Similar to previous studies, this analysis uses common admissions metrics such as undergraduate GPA (UGPA) and GRE scores to predict PhD completion. Now, using statistical methods recently advocated in causal inference literature, a counterfactual mediation framework is used to determine whether these metrics directly predict PhD completion or indirectly predict completion via an intermediate variable (graduate GPA). Consistent with prior work, results show that UGPA is a stronger predictor of completion than Physics GRE scores. However, the analysis reveals that these effects are fully mediated by graduate GPA, suggesting that admissions metrics are measuring traits linked to graduate course performance, which in turn affects graduate attrition. Poor course performance may negatively influence personal factors (e.g., self-efficacy, identity), limit access to research opportunities (e.g., repeating classes, ease of finding a research lab), or may indicate a lack of preparation for research, all of which could hinder PhD completion; however, more research is needed.
mT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs

Multilingual T5 pretrains a sequence-to-sequence model on massive monolingual texts, which has shown promising results on many cross-lingual tasks. In this paper, we improve multilingual text-to-text transfer Transformer with translation pairs (mT6). Specifically, we explore three cross-lingual text-to-text pre-training tasks, namely, machine translation, translation pair span corruption, and translation span corruption. In addition, we propose a partially non-autoregressive objective for text-to-text pre-training. We evaluate the methods on seven multilingual benchmark datasets, including sentence classification, named entity recognition, question answering, and abstractive summarization. Experimental results show that the proposed mT6 improves cross-lingual transferability over mT5.
Extract, Denoise, and Enforce: Evaluating and Predicting Lexical Constraints for Conditional Text Generation

Recently, pre-trained language models (PLMs) have dominated conditional text generation tasks. Given the impressive performance and prevalence of the PLMs, it is seemingly natu-ral to assume that they could ﬁgure out what to attend to in the input and what to include in the output via seq2seq learning without more guidance than the training input/output pairs. However, a rigorous study regarding the above assumption is still lacking. In this paper, we present a systematic analysis of conditional generation to study whether current PLMs are good enough for preserving important concepts in the input and to what extent explicitly guiding generation with lexical constraints is beneﬁcial. We conduct extensive analytical experiments on a range of conditional generation tasks and try to answer in what scenarios guiding generation with lexical constraints works well and why. We then propose a framework for automatic constraint extraction, denoising, and enforcement that is shown to perform comparably or better than unconstrained generation. We hope that our ﬁndings could serve as a reference when determining whether it is appropriate and worthwhile to use explicit constraints for a speciﬁc task or dataset. 1
A Mathematical Analysis of Learning Loss for Active Learning in Regression

Active learning continues to remain significant in the industry since it is data efficient. Not only is it cost effective on a constrained budget, continuous refinement of the model allows for early detection and resolution of failure scenarios during the model development stage. Identifying and fixing failures with the model is crucial as industrial applications demand that the underlying model performs accurately in all foreseeable use cases. One popular state-of-the-art technique that specializes in continuously refining the model via failure identification is Learning Loss [24]. Although simple and elegant, this approach is empirically motivated. Our paper develops a foundation for Learning Loss which enables us to propose a novel modification we call LearningLoss++. We show that gradients are crucial in interpreting how Learning Loss works, with rigorous analysis and comparison of the gradients between Learning Loss and LearningLoss++. We also propose a convolutional architecture that combines features at different scales to predict the loss. We validate LearningLoss++ for regression on the task of human pose estimation (using MPII and LSP datasets), as done in Learning Loss. We show that Learn-ingLoss++ outperforms in identifying scenarios where the model is likely to perform poorly, which on model refinement translates into reliable performance in the open world.
I Only Have Eyes for You: The Impact of Masks On Convolutional-Based Facial Expression Recognition

The current COVID-19 pandemic has shown us that we are still facing unpredictable challenges in our society. The necessary constrain on social interactions affected heavily how we envision and prepare the future of social robots and artificial agents in general. Adapting current affective perception models towards constrained perception based on the hard separation between facial perception and affective understanding would help us to provide robust systems. In this paper, we perform an in-depth analysis of how recognizing affect from persons with masks differs from general facial expression perception. We evaluate how the recently proposed FaceChannel adapts towards recognizing facial expressions from persons with masks. In Our analysis, we evaluate different training and fine-tuning schemes to understand better the impact of masked facial expressions. We also perform specific feature-level visualization to demonstrate how the inherent capabilities of the FaceChannel to learn and combine facial features change when in a constrained social interaction scenario.
Quantum criticality and spin liquid phase in the Shastry-Sutherland model

Using the density-matrix renormalization group method for the ground state and excitations of the Shastry- Sutherland spin model, we demonstrate the existence of a narrow quantum spin liquid phase between the previously known plaquette-singlet and antiferromagnetic states. Our conclusions are based on finite-size scaling of excited level crossings and order parameters. Together with previous results on candidate models for deconfined quantum criticality and spin liquid phases, our results point to a unified quantum phase diagram where the deconfined quantum-critical point separates a line of first-order transitions and a gapless spin liquid phase. The frustrated Shastry-Sutherland model is close to the critical point but slightly inside the spin liquid phase, while previously studied unfrustrated models cross the first-order line. We also argue that recent heat capacity measurements in SrCu2(BO3)2 show evidence of the proposed spin liquid at pressures between 2.6 and 3 GPa.
Improving Adversarial Robustness Using Proxy Distributions

We focus on the use of proxy distributions, i.e., approximations of the underlying distribution of the training dataset, in both understanding and improving the adversarial robustness in image classiﬁcation. While additional training data helps in adversarial training, curating a very large number of real-world images is challenging. In contrast, proxy distributions enable us to sample a potentially unlimited number of images and improve adversarial robustness using these samples. We ﬁrst ask the question: when does adversarial robustness beneﬁt from incorporating additional samples from the proxy distribution in the training stage? We prove that the diﬀerence between the robustness of a classiﬁer on the proxy and original training dataset distribution is upper bounded by the conditional Wasserstein distance between them. Our result conﬁrms the intuition that samples from a proxy distribution that closely approximates training dataset distribution should be able to boost adversarial robustness. Motivated by this ﬁnding, we leverage samples from state-of-the-art generative models, which can closely approximate training data distribution, to improve robustness. In particular, we improve robust accuracy by up to 6 . 1% and 5 . 7% in l ∞ and l 2 threat model, and certiﬁed robust accuracy by 6 . 7% over baselines not using proxy distributions on the CIFAR-10 dataset. Since we can sample an unlimited number of images from a proxy distribution, it also allows us to investigate the eﬀect of an increasing number of training samples on adversarial robustness. Here we provide the ﬁrst large scale empirical investigation of accuracy vs robustness trade-oﬀ and sample complexity of adversarial training by training deep neural networks on 2K to 10M images.
FedNLP: A Research Platform for Federated Learning in Natural Language Processing

Increasing concerns and regulations about data privacy, necessitate the study of privacy-preserving methods for natural language processing (NLP) applications. Federated learning (FL) provides promising methods for a large number of clients (i.e., personal devices or organizations) to collaboratively learn a shared global model to beneﬁt all clients, while allowing users to keep their data locally. To facilitate FL research in NLP, we present the FedNLP 1 , a research platform for federated learning in NLP. FedNLP supports various popular task formulations in NLP such as text classiﬁcation, sequence tagging, question answering, seq2seq generation, and language modeling. We also implement an interface between Transformer language models (e.g., BERT) and FL methods (e.g., FedAvg, Fe-dOpt, etc.) for distributed training. The evaluation protocol of this interface supports a comprehensive collection of non-IID partitioning strategies. Our preliminary experiments with FedNLP reveal that there exists a large performance gap between learning on decentralized and centralized datasets — opening intriguing and exciting future research directions aimed at developing FL methods suited to NLP tasks.