Miracle-dz/UniFAR-FTUs huggingface dataset:
Searching for dark energy with the Sun

General extensions of general relativity (GR) based on bona ﬁde degrees of freedom predict a ﬁfth force which operates within massive objects, opening up an exciting opportunity to perform precision tests of gravity at stellar scales. Here, focussing on general scalar-tensor theories for dark energy, we utilise the Sun as our laboratory and search for imprints of the ﬁfth-force e ﬀ ect on the solar equilibrium structure. With analytic results and numerical simulations, we explain how the di ﬀ erent solar regions o ﬀ er powerful ways to test gravity. Accounting for the delicate interplay between the ﬁfth force and solar microphysics such as opacity, di ﬀ usion, equation of state, and metallicity, we demonstrate that the ﬁfth force still leaves a sharp signature on the solar sound speed, in a region where simple estimates of input physics uncertainties become negligible. For general scalar-ﬁeld extensions of GR, known as (U-)DHOST, based solely on the observational helioseismic errors, our analysis at the equilibrium level allows us to place an approximate constraint on the ﬁfth-force coupling strength of − 10 − 3 (cid:46) Y (cid:46) 5 · 10 − 4 at 2 σ . This result improves previous stellar constraints by ∼ 3 orders of magnitude, and should be conﬁrmed and improved by future helioseismic inversions in modiﬁed gravity, combined with an elaborate accounting of theoretical uncertainties. Our analysis can be applied to a wide set of theories beyond GR, and also paves the way for helioseismic analyses in this context. In this regard, we discuss how the solar radiative and convective zone can be employed as promising laboratories to test generic theories of gravity.
Learning Dialogue Representations from Consecutive Utterances

Learning high-quality dialogue representations is essential for solving a variety of dialogue-oriented tasks, especially considering that dialogue systems often suffer from data scarcity. In this paper, we introduce Dialogue Sentence Embedding (DSE), a self-supervised contrastive learning method that learns effective dialogue representations suitable for a wide range of dialogue tasks. DSE learns from dialogues by taking consecutive utterances of the same dialogue as positive pairs for contrastive learning. Despite its simplicity, DSE achieves significantly better representation capability than other dialogue representation and universal sentence representation models. We evaluate DSE on five downstream dialogue tasks that examine dialogue representation at different semantic granularities. Experiments in few-shot and zero-shot settings show that DSE outperforms baselines by a large margin, for example, it achieves 13% average performance improvement over the strongest unsupervised baseline in 1-shot intent classification on 6 datasets. We also provide analyses on the benefits and limitations of our model.
Chapter 13 A Conceptual Framework for Studying a Technology Transfer from Academia to New Firms

Global technological competition has made technology transfer from academia to firms an important public policy issue (Rahm, 1994). Academia and individual academic institutions are a primary source of new knowledge production and innovation (Brennan & McGowan, 2007). It is widely acknowledged that the commercialization of scientific and technological knowledge produced in public funded research institutions, including universities and research centres, into the marketplace have a fundamental role to play in wealth creation, supporting economic growth and technological innovation, and plays a significant role in new venture creation, growth of existing firms, and new job creation (Mansfield, 1991; Harmon et al., 1997; Ndonzuau, Pirnay, & Surlemont, 2002; Siegel, Waldman, Atwater, & Link, 2003b; Steffensen, Rogers, & Speakman, 1999; Walter, Auer, & Ritter, 2006; Perez & Sanchez, 2003). Research by Acs, Audretsch, and Feldman (1992), Jaffe (1989), Mansfield (1991, 1998), and others indicates that technological change in important segments of the economy has been significantly based on knowledge that spin-off from academic research.
Forecasting change in conflict fatalities with dynamic elastic net

Abstract This article illustrates an approach to forecasting change in conflict fatalities designed to address the complexity of the drivers and processes of armed conflicts. The design of this approach is based on two main choices. First, to account for the specificity of conflict drivers and processes over time and space, we model conflicts in each individual country separately. Second, we draw on an adaptive model—Dynamic Elastic Net, DynENet—which is able to efficiently select relevant predictors among a large set of covariates. We include over 700 variables in our models, adding event data on top of the data features provided by the convenors of the forecasting competition. We show that our approach is suitable and computationally efficient enough to address the complexity of conflict dynamics. Moreover, the adaptive nature of our model brings a significant added value. Because for each country our model only selects the variables that are relevant to predict conflict intensity, the retained predictors can be analyzed to describe the dynamic configuration of conflict drivers both across countries and within countries over time. Countries can then be clustered to observe the emergence of broader patterns related to correlates of conflict. In this sense, our approach produces interpretable forecasts, addressing one key limitation of contemporary approaches to forecasting.
Cosmological constraints on decaying axion-like particles: a global analysis

Axion-like particles (ALPs) decaying into photons are known to affect a wide range of astrophysical and cosmological observables. In this study we focus on ALPs with masses in the keV–MeV range and lifetimes between 104 and 1013 seconds, corresponding to decays between the end of Big Bang Nucleosynthesis and the formation of the Cosmic Microwave Background (CMB). Using the CosmoBit module of the global fitting framework GAMBIT, we combine state-of-the-art calculations of the irreducible ALP freeze-in abundance, primordial element abundances (including photodisintegration through ALP decays), CMB spectral distortions and anisotropies, and constraints from supernovae and stellar cooling. This approach makes it possible for the first time to perform a global analysis of the ALP parameter space while varying the parameters of ΛCDM as well as several nuisance parameters. We find a lower bound on the ALP mass of around ma > 300 keV, which can only be evaded if ALPs are stable on cosmological timescales. Future observations of CMB spectral distortions with a PIXIE-like mission are expected to improve this bound by two orders of magnitude.
A Semidefinite Relaxation for Sums of Heterogeneous Quadratic Forms on the Stiefel Manifold

We study the maximization of sums of heterogeneous quadratic forms over the Stiefel manifold, a nonconvex problem that arises in several modern signal processing and machine learning applications such as heteroscedastic probabilistic principal component analysis (HPPCA). In this work, we derive a novel semidefinite program (SDP) relaxation and study a few of its theoretical properties. We prove a global optimality certificate for the original nonconvex problem via a dual certificate, which leads to a simple feasibility problem to certify global optimality of a candidate local solution on the Stiefel manifold. In addition, our relaxation reduces to an assignment linear program for jointly diagonalizable problems and is therefore known to be tight in that case. We generalize this result to show that it is also tight for close-to jointly diagonalizable problems, and we show that the HPPCA problem has this characteristic. Numerical results validate our global optimality certificate and sufficient conditions for when the SDP is tight in various problem settings.
HiJoNLP at SemEval-2022 Task 2: Detecting Idiomaticity of Multiword Expressions using Multilingual Pretrained Language Models

This paper describes an approach to detect idiomaticity only from the contextualized representation of a MWE over multilingual pretrained language models.Our experiments find that larger models are usually more effective in idiomaticity detection. However, using a higher layer of the model may not guarantee a better performance.In multilingual scenarios, the convergence of different languages are not consistent and rich-resource languages have big advantages over other languages.
Transformers from an Optimization Perspective

Deep learning models such as the Transformer are often constructed by heuristics and experience. To provide a complementary foundation, in this work we study the following problem: Is it possible to find an energy function underlying the Transformer model, such that descent steps along this energy correspond with the Transformer forward pass? By finding such a function, we can view Transformers as the unfolding of an interpretable optimization process across iterations. This unfolding perspective has been frequently adopted in the past to elucidate more straightforward deep models such as MLPs and CNNs; however, it has thus far remained elusive obtaining a similar equivalence for more complex models with self-attention mechanisms like the Transformer. To this end, we first outline several major obstacles before providing companion techniques to at least partially address them, demonstrating for the first time a close association between energy function minimization and deep layers with self-attention. This interpretation contributes to our intuition and understanding of Transformers, while potentially laying the ground-work for new model designs.
BagFlip: A Certified Defense against Data Poisoning

Machine learning models are vulnerable to data-poisoning attacks, in which an attacker maliciously modifies the training set to change the prediction of a learned model. In a trigger-less attack, the attacker can modify the training set but not the test inputs, while in a backdoor attack the attacker can also modify test inputs. Existing model-agnostic defense approaches either cannot handle backdoor attacks or do not provide effective certificates (i.e., a proof of a defense). We present BagFlip, a model-agnostic certified approach that can effectively defend against both trigger-less and backdoor attacks. We evaluate BagFlip on image classification and malware detection datasets. BagFlip is equal to or more effective than the state-of-the-art approaches for trigger-less attacks and more effective than the state-of-the-art approaches for backdoor attacks.
Lifting the Information Ratio: An Information-Theoretic Analysis of Thompson Sampling for Contextual Bandits

We study the Bayesian regret of the renowned Thompson Sampling algorithm in contextual bandits with binary losses and adversarially-selected contexts. We adapt the information-theoretic perspective of \cite{RvR16} to the contextual setting by considering a lifted version of the information ratio defined in terms of the unknown model parameter instead of the optimal action or optimal policy as done in previous works on the same setting. This allows us to bound the regret in terms of the entropy of the prior distribution through a remarkably simple proof, and with no structural assumptions on the likelihood or the prior. The extension to priors with infinite entropy only requires a Lipschitz assumption on the log-likelihood. An interesting special case is that of logistic bandits with $d$-dimensional parameters, $K$ actions, and Lipschitz logits, for which we provide a $\widetilde{O}(\sqrt{dKT})$ regret upper-bound that does not depend on the smallest slope of the sigmoid link function.
Can Foundation Models Help Us Achieve Perfect Secrecy?

A key promise of machine learning is the ability to assist users with personal tasks. Because the personal context required to make accurate predictions is often sensitive, we require systems that protect privacy. A gold standard privacy-preserving system will satisfy perfect secrecy, meaning that interactions with the system provably reveal no private information. However, privacy and quality appear to be in tension in existing systems for personal tasks. Neural models typically require copious amounts of training to perform well, while individual users typically hold a limited scale of data, so federated learning (FL) systems propose to learn from the aggregate data of multiple users. FL does not provide perfect secrecy, but rather practitioners apply statistical notions of privacy -- i.e., the probability of learning private information about a user should be reasonably low. The strength of the privacy guarantee is governed by privacy parameters. Numerous privacy attacks have been demonstrated on FL systems and it can be challenging to reason about the appropriate privacy parameters for a privacy-sensitive use case. Therefore our work proposes a simple baseline for FL, which both provides the stronger perfect secrecy guarantee and does not require setting any privacy parameters. We initiate the study of when and where an emerging tool in ML -- the in-context learning abilities of recent pretrained models -- can be an effective baseline alongside FL. We find in-context learning is competitive with strong FL baselines on 6 of 7 popular benchmarks from the privacy literature and a real-world case study, which is disjoint from the pretraining data. We release our code here: https://github.com/simran-arora/focus
Sample-Efficient Optimisation with Probabilistic Transformer Surrogates

Faced with problems of increasing complexity, recent research in Bayesian Optimisation (BO) has focused on adapting deep probabilistic models as flexible alternatives to Gaussian Processes (GPs). In a similar vein, this paper investigates the feasibility of employing state-of-the-art probabilistic transformers in BO. Upon further investigation, we observe two drawbacks stemming from their training procedure and loss definition, hindering their direct deployment as proxies in black-box optimisation. First, we notice that these models are trained on uniformly distributed inputs, which impairs predictive accuracy on non-uniform data - a setting arising from any typical BO loop due to exploration-exploitation trade-offs. Second, we realise that training losses (e.g., cross-entropy) only asymptotically guarantee accurate posterior approximations, i.e., after arriving at the global optimum, which generally cannot be ensured. At the stationary points of the loss function, however, we observe a degradation in predictive performance especially in exploratory regions of the input space. To tackle these shortcomings we introduce two components: 1) a BO-tailored training prior supporting non-uniformly distributed points, and 2) a novel approximate posterior regulariser trading-off accuracy and input sensitivity to filter favourable stationary points for improved predictive performance. In a large panel of experiments, we demonstrate, for the first time, that one transformer pre-trained on data sampled from random GP priors produces competitive results on 16 benchmark black-boxes compared to GP-based BO. Since our model is only pre-trained once and used in all tasks without any retraining and/or fine-tuning, we report an order of magnitude time-reduction, while matching and sometimes outperforming GPs.
Robust Counterfactual Explanations for Random Forests

Counterfactual explanations describe how to modify a feature vector in order to ﬂip the outcome of a trained classiﬁer. Several heuristic and optimal methods have been proposed to generate these explanations. However, the robustness of coun-terfactual explanations when the classiﬁer is re-trained has yet to be studied. Our goal is to obtain counterfactual explanations for random forests that are robust to algorithmic uncertainty. We study the link between the robustness of ensemble models and the robustness of base learners and frame the generation of robust counterfactual explanations as a chance-constrained optimization problem. We develop a practical method with good empirical performance and provide ﬁnite-sample and asymptotic guarantees for simple random forests of stumps. We show that existing methods give surprisingly low robustness: the validity of naive coun-terfactuals is below 50% on most data sets and can fall to 20% on large problem instances with many features. Even with high plausibility, counterfactual explanations often exhibit low robustness to algorithmic uncertainty. In contrast, our method achieves high robustness with only a small increase in the distance from counterfactual explanations to their initial observations. Furthermore, we high-light the connection between the robustness of counterfactual explanations and the predictive importance of features.
StereoKG: Data-Driven Knowledge Graph Construction For Cultural Knowledge and Stereotypes

Analyzing ethnic or religious bias is important for improving fairness, accountability, and transparency of natural language processing models. However, many techniques rely on human-compiled lists of bias terms, which are expensive to create and are limited in coverage. In this study, we present a fully data-driven pipeline for generating a knowledge graph (KG) of cultural knowledge and stereotypes. Our resulting KG covers 5 religious groups and 5 nationalities and can easily be extended to more entities. Our human evaluation shows that the majority (59.2%) of non-singleton entries are coherent and complete stereotypes. We further show that performing intermediate masked language model training on the verbalized KG leads to a higher level of cultural awareness in the model and has the potential to increase classification performance on knowledge-crucial samples on a related task, i.e., hate speech detection.
Patching Leaks in the Charformer for Efficient Character-Level Generation

Character-based representations have important advantages over subword-based ones for morphologically rich languages. They come with increased robustness to noisy input and do not need a separate tokenization step. However, they also have a crucial disadvantage: they notably increase the length of text sequences. The GBST method from Charformer groups (aka downsamples) characters to solve this, but allows information to leak when applied to a Transformer decoder. We solve this information leak issue, thereby enabling character grouping in the decoder. We show that Charformer downsampling has no apparent benefits in NMT over previous downsampling methods in terms of translation quality, however it can be trained roughly 30% faster. Promising performance on English--Turkish translation indicate the potential of character-level models for morphologically-rich languages.
Learning Markovian Homogenized Models in Viscoelasticity

Fully resolving dynamics of materials with rapidly-varying features involves expensive fine-scale computations which need to be conducted on macroscopic scales. The theory of homogenization provides an approach to derive effective macroscopic equations which eliminates the small scales by exploiting scale separation. An accurate homogenized model avoids the computationally-expensive task of numerically solving the underlying balance laws at a fine scale, thereby rendering a numerical solution of the balance laws more computationally tractable. In complex settings, homogenization only defines the constitutive model implicitly, and machine learning can be used to learn the constitutive model explicitly from localized fine-scale simulations. In the case of one-dimensional viscoelasticity, the linearity of the model allows for a complete analysis. We establish that the homogenized constitutive model may be approximated by a recurrent neural network (RNN) that captures the memory. The memory is encapsulated in the evolution of an appropriate finite set of internal variables, discovered through the learning process and dependent on the history of the strain. Simulations are presented which validate the theory. Guidance for the learning of more complex models, such as arise in plasticity, by similar techniques, is given.
Bias Reduction via Cooperative Bargaining in Synthetic Graph Dataset Generation

In general, to draw robust conclusions from a dataset, all the analyzed population must be represented on said dataset. Having a dataset that does not fulfill this condition normally leads to selection bias. Additionally, graphs have been used to model a wide variety of problems. Although synthetic graphs can be used to augment available real graph datasets to overcome selection bias, the generation of unbiased synthetic datasets is complex with current tools. In this work, we propose a method to find a synthetic graph dataset that has an even representation of graphs with different metrics. The resulting dataset can then be used, among others, for benchmarking graph processing techniques as the accuracy of different Graph Neural Network (GNN) models or the speedups obtained by different graph processing acceleration frameworks.
Efficient textual explanations for complex road and traffic scenarios based on semantic segmentation

The complex driving environment brings great challenges to the visual perception of autonomous vehicles. It’s essential to extract clear and explainable information from the complex road and traﬃc scenarios and oﬀer clues to decision and control. However, the previous scene explanation had been implemented as a separate model. The black box model makes it diﬃcult to interpret the driving environment. It cannot detect comprehensive textual information and requires a high computational load and time consumption. Thus, this study proposed a comprehensive and eﬃcient textual explanation model. From 336k video frames of the driving environment, critical images of complex road and traﬃc scenarios were selected into a dataset. Through transfer learning, this study established an accurate and eﬃcient segmentation model to obtain the critical traﬃc elements in the environment. Based on the XGBoost algorithm, a comprehensive model was developed. The model provided textual information about states of traﬃc elements, the motion of conﬂict objects, and scenario complexity. The approach was veriﬁed on the real-world road. It improved the perception accuracy of critical traﬃc elements to 78.8%. The time consumption reached 13 minutes for each epoch, which was 11.5 times more eﬃcient than the pre-trained network. The textual information analyzed from the model was also accordant with reality. The ﬁndings oﬀer clear and explainable information about the complex driving environment, which lays a foundation for subsequent decision and control. It can improve the visual perception ability and enrich the prior knowledge and judgments of complex traﬃc situations.
Client Selection in Nonconvex Federated Learning: Improved Convergence Analysis for Optimal Unbiased Sampling Strategy

Federated learning (FL) is a distributed machine learning paradigm that selects a subset of clients to participate in training to reduce communication burdens. However, partial client participation in FL causes objective inconsistency , which can hinder the convergence, while this objective inconsistency has not been analyzed in existing studies on sampling methods. To tackle this issue, we propose an improved analysis method that focuses on the convergence behavior of the practical participated client’s objective. Moreover, based on our convergence analysis, we give a novel unbiased sampling strategy, i.e., FedSRC-D, whose sampling probability is proportional to the client’s gradient diversity and local variance. FedSRC-D is provable the optimal unbiased sampling in non-convex settings for non-IID FL with respect to the given bounds. Speciﬁcally, FedSRC-D achieves O ( G 2 (cid:15) 2 + 1 (cid:15) 2 / 3 ) higher than SOTA convergence rate of FedAvg, and O ( G 2 (cid:15) 2 ) higher than other unbiased sampling methods. We corroborate our results with experiments on both synthetic and real data sets.