Miracle-dz/UniFAR-FTUs huggingface dataset:
Revisiting Calibration for Question Answering

Model calibration aims to adjust (calibrate) models’ confidence so that they match expected accuracy. We argue that the traditional evaluation of calibration (expected calibration error; ECE) does not reflect usefulness of the model confidence. For example, after conventional temperature scaling, confidence scores become similar for all predictions, which makes it hard for users to distinguish correct predictions from wrong ones, even though it achieves low ECE. Building on those observations, we propose a new calibration metric, MacroCE, that better captures whether the model assigns low confidence to wrong predictions and high confidence to correct predictions. We examine various conventional calibration methods including temperature scaling, feature-based classifier, neural answer reranking, and label smoothing, all of which do not bring significant gains under our new MacroCE metric. Towards more effective calibration, we propose a new calibration method based on the model’s prediction consistency along the training trajectory. This new method, which we name as consistency calibration, shows promise for better calibration.
Searching for outliers in the Chandra Source Catalog

Astronomers are increasingly faced with a deluge of information, and finding worthwhile targets of study in the sea of data can be difficult. Outlier identification studies are a method that can be used to focus investigations by presenting a smaller set of sources that could prove interesting because they do not follow the trends of the underlying population. We apply a principal component analysis (PCA) and an unsupervised random forest algorithm (uRF) to sources from the Chandra Source Catalog v.2 (CSC2). We present 119 high-significance sources that appear in all repeated applications of our outlier identification algorithm (OIA). We analyze the characteristics of our outlier sources and crossmatch them with the SIMBAD database. Our outliers contain several sources that were previously identified as having unusual or interesting features by studies. This OIA leads to the identification of interesting targets that could motivate more detailed study.
Estimating Waning of Vaccine Effectiveness: a Simulation Study.

BACKGROUND Developing accurate and reliable methods to estimate vaccine protection is a key goal in immunology and public health. While several statistical methods have been proposed, their potential inaccuracy in capturing fast intra-seasonal waning of vaccine-induced protection needs to be rigorously investigated. METHODS To compare statistical methods for vaccine effectiveness (VE) estimation, we generated simulated data using a multiscale agent-based model of an epidemic with an acute viral infection and differing extents of VE waning. We apply a previously proposed framework for VE measures based on the observational data richness to assess changes of vaccine-induced protection over time. RESULTS While VE measures based on hard-to-collect information (e.g. the exact timing of exposures) were accurate, usually VE studies rely on time-to-infection data and the Cox proportional hazard model. We found that its extension utilizing scaled Schoenfeld residuals, previously proposed for capturing VE waning, was unreliable in capturing both the degree of waning and its functional form and identified the mathematical factors contributing to this unreliability. We showed that partitioning time and including a time-vaccine interaction term in the Cox model significantly improved estimation of VE waning, even in the case of dramatic, rapid waning. We also proposed how to optimize the partitioning scheme. CONCLUSIONS While appropriate for rejecting the null hypothesis of no waning, scaled Schoenfeld residuals are unreliable for estimating the degree of waning. We propose a Cox-model-based method with a time-vaccine interaction term and further optimization of partitioning time. These findings may guide future analysis of VE waning data.
Differentially Private Data Generation Needs Better Features

Training even moderately-sized generative models with differentially-private stochastic gradient descent (DP-SGD) is difﬁcult: the required level of noise for reasonable levels of privacy is simply too large. We advocate instead building off a good, relevant representation on public data, then using private data only for “transfer learning. ” In particular, we minimize the maximum mean discrepancy (MMD) between private target data and the generated distribution, using a kernel based on perceptual features from a public dataset. With the MMD, we can simply privatize the data-dependent term once and for all, rather than introducing noise at each step of optimization as in DP-SGD. Our algorithm allows us to generate CIFAR10-level images faithfully with (cid:15) ≈ 2 , far surpassing the current state of the art, which only models MNIST and FashionMNIST at (cid:15) ≈ 10 . Our work introduces simple yet powerful foundations for reducing the gap between private and non-private deep generative models.
Interpretable Feature Engineering for Time Series Predictors using Attention Networks

Regression problems with time-series predictors are common in banking and many other areas of application. In this paper, we use multi-head attention networks to develop interpretable features and use them to achieve good predictive performance. The customized attention layer explicitly uses multiplicative interactions and builds feature-engineering heads that capture temporal dynamics in a parsimonious manner. Convolutional layers are used to combine multivariate time series. We also discuss methods for handling static covariates in the modeling process. Visualization and explanation tools are used to interpret the results and explain the relationship between the inputs and the extracted features. Both simulation and real dataset are used to illustrate the usefulness of the methodology. Keyword: Attention heads, Deep neural networks, Interpretable feature engineering
Nonlocal gluon condensates in QCD sum rules

Nonlocal gluon condensates are vacuum expectations of the product of gluon ﬁeld strength tensors. Short-distance expansions of two-, three-, and four-gluon condensates are presented up to dimension-8 local operators. We propose a method for calculating the Wilson coeﬃcients based on the presented expansions and the Feynman diagram technique in the background ﬁeld approach. The method is demonstrated using the glueball current correlators as examples. Methodological aspects of the background ﬁeld approach are discussed in relation to glueball studies within QCD sum rules. We conﬁrm the results for Operator Product Expansion (OPE) of the two-gluon 0 ± + glueball current correlators and calculate additional contributions coming from dimension-6 four-quark condensate and dimension-8 mixed quark-gluon condensates. The OPEs used in QCD sum rules for three-gluon 0 ± + glueballs are revisited up to dimension-6 order.
RobustLR: A Diagnostic Benchmark for Evaluating Logical Robustness of Deductive Reasoners

Transformers have been shown to be able to perform deductive reasoning on inputs containing rules and statements written in the English natural language. However, it is unclear if these models indeed follow rigorous logical reasoning to arrive at the prediction or rely on spurious correlation patterns in making decisions. A strong deductive reasoning model should consistently understand the semantics of different logical operators. To this end, we present RobustLR, a diagnostic benchmark that evaluates the robustness of language models to minimal logical edits in the inputs and different logical equivalence conditions. In our experiments with RoBERTa, T5, and GPT3 we show that the models trained on deductive reasoning datasets do not perform consistently on the RobustLR test set, thus showing that the models are not robust to our proposed logical perturbations. Further, we observe that the models find it especially hard to learn logical negation operators. Our results demonstrate the shortcomings of current language models in logical reasoning and call for the development of better inductive biases to teach the logical semantics to language models. All the datasets and code base have been made publicly available.
Skill Machines: Temporal Logic Composition in Reinforcement Learning

A major challenge in reinforcement learning is specifying tasks in a manner that is both interpretable and veriﬁable. One common approach is to specify tasks through reward machines—ﬁnite state machines that encode the task to be solved. We introduce skill machines, a representation that can be learned directly from these reward machines that encode the solution to such tasks. We propose a framework where an agent ﬁrst learns a set of base skills in a reward-free setting, and then combines these skills with the learned skill machine to produce composite behaviours speciﬁed by any regular language, such as linear temporal logics. This provides the agent with the ability to map from complex logical task speciﬁcations to near-optimal behaviours zero-shot. We demonstrate our approach in both a tabular and high-dimensional video game environment, where an agent is faced with several of these complex, long-horizon tasks. Our results indicate that the agent is capable of satisfying extremely complex task speciﬁcations, producing near optimal performance with no further learning. Finally, we demonstrate that the performance of skill machines can be improved with regular ofﬂine reinforcement learning algorithms when optimal behaviours are desired.
ProsocialDialog: A Prosocial Backbone for Conversational Agents

Most existing dialogue systems fail to respond properly to potentially unsafe user utterances by either ignoring or passively agreeing with them. To address this issue, we introduce ProsocialDialog, the first large-scale multi-turn dialogue dataset to teach conversational agents to respond to problematic content following social norms. Covering diverse unethical, problematic, biased, and toxic situations, ProsocialDialog contains responses that encourage prosocial behavior, grounded in commonsense social rules (i.e., rules-of-thumb, RoTs). Created via a human-AI collaborative framework, ProsocialDialog consists of 58K dialogues, with 331K utterances, 160K unique RoTs, and 497K dialogue safety labels accompanied by free-form rationales.With this dataset, we introduce a dialogue safety detection module, Canary, capable of generating RoTs given conversational context, and a socially-informed dialogue agent, Prost. Empirical results show that Prost generates more socially acceptable dialogues compared to other state-of-the-art language and dialogue models in both in-domain and out-of-domain settings. Additionally, Canary effectively guides conversational agents and off-the-shelf language models to generate significantly more prosocial responses. Our work highlights the promise and importance of creating and steering conversational AI to be socially responsible.
Eliciting Transferability in Multi-task Learning with Task-level Mixture-of-Experts

Recent work suggests that transformer models are capable of multi-task learning on diverse NLP tasks. However, the potential of these models may be limited as they use the same set of parameters for all tasks. In contrast, humans tackle tasks in a more ﬂexible way, by making proper presumptions on what skills and knowledge are relevant and executing only the necessary computations. Inspired by this, we propose to use task-level mixture-of-expert models, which has a collection of transformer layers ( i.e. , experts) and a router component to choose among these experts dynamically and ﬂexibly. We show that the learned routing decisions and experts partially rediscover human categorization of NLP tasks – certain experts are strongly associated with extractive tasks, some with classiﬁcation tasks, and some with tasks requiring world knowledge.
Toward the chiral phase transition in the Roberge-Weiss plane

We discuss the interplay between chiral and center sector phase transitions that occur in QCD with an imaginary quark chemical potential µ = i (2 n +1) πT/ 3. Based on a ﬁnite size scaling analysis in (2+1)-ﬂavor QCD using HISQ fermions with a physical strange quark mass and a range of light quark masses, we show that the endpoint of the line of ﬁrst-order Roberge-Weiss (RW) transitions between center sectors is second order for light quark masses m l ≥ m s / 320, and that it belongs to the 3-d, Z (2) universality class. The operator for the chiral condensate behaves like an energy-like operator in an eﬀective spin model for the RW phase transition. As a consequence, for any non-zero value of the quark mass, the chiral condensate will have an inﬁnite slope at the RW phase transition temperature, T RW . Its ﬂuctuation, the disconnected chiral susceptibility, behaves like the speciﬁc heat in Z (2) symmetric models and diverges in the inﬁnite volume limit at the RW phase transition temperature T RW for any non-zero value of the light quark masses. Our analysis suggests the critical temperatures for the RW phase transition and the chiral phase transition coincide in the RW plane. On lattices with temporal extent N τ = 4, we ﬁnd in the chiral limit T χ = T RW = 195(1) MeV.
Linear Connectivity Reveals Generalization Strategies

It is widely accepted in the mode connectivity literature that when two neural networks are trained similarly on the same data, they are connected by a path through parameter space over which test set accuracy is maintained. Under some circumstances, including transfer learning from pretrained models, these paths are presumed to be linear. In contrast to existing results, we find that among text classifiers (trained on MNLI, QQP, and CoLA), some pairs of finetuned models have large barriers of increasing loss on the linear paths between them. On each task, we find distinct clusters of models which are linearly connected on the test loss surface, but are disconnected from models outside the cluster -- models that occupy separate basins on the surface. By measuring performance on specially-crafted diagnostic datasets, we find that these clusters correspond to different generalization strategies: one cluster behaves like a bag of words model under domain shift, while another cluster uses syntactic heuristics. Our work demonstrates how the geometry of the loss surface can guide models towards different heuristic functions.
ReSmooth: Detecting and Utilizing OOD Samples when Training with Data Augmentation

Data augmentation (DA) is a widely used technique for enhancing the training of deep neural networks. Recent DA techniques which achieve state-of-the-art performance always meet the need for diversity in augmented training samples. However, an augmentation strategy that has a high diversity usually introduces out-of-distribution (OOD) augmented samples and these samples consequently impair the performance. To alleviate this issue, we propose ReSmooth, a framework that first detects OOD samples in augmented samples and then leverages them. To be specific, we first use a Gaussian mixture model (GMM) to fit the loss distribution of both the original and augmented samples and accordingly split these samples into in-distribution (ID) samples and OOD samples. Then we start a new training where ID and OOD samples are incorporated with different smooth labels. By treating ID samples and OOD samples unequally, we can make better use of the diverse augmented data. Furthermore, we incorporate our ReSmooth framework with negative DA (NDA) strategies. By properly handling their intentionally created OOD samples, the classification performance of NDAs is largely ameliorated. Experiments on several classification benchmarks show that ReSmooth can be easily extended to the existing augmentation strategies such as RandAugment (RA), rotate, and jigsaw and improve on them. Our code is available at https://github.com/Chenyang4/ReSmooth.
"Help! Can You Hear Me?": Understanding How Help-Seeking Posts are Overwhelmed on Social Media during a Natural Disaster

Posting help-seeking requests on social media has been broadly adopted by victims during natural disasters to look for urgent rescue and supplies. The help-seeking requests need to get sufficient public attention and be promptly routed to the intended target(s) for timely responses. However, the huge volume and diverse types of crisis-related posts on social media might limit help-seeking requests to receive adequate engagement and lead to their overwhelm. To understand this problem, this work proposes a mixed-methods approach to figure out the overwhelm situation of help-seeking requests, and individuals' and online communities' strategies to cope. We focused on the 2021 Henan Floods in China and collected 141,674 help-seeking posts with the keyword "Henan Rainstorm Mutual Aid" on a popular Chinese social media platform Weibo. The findings indicate that help-seeking posts confront critical challenges of both external overwhelm (i.e., an enormous number of non-help-seeking posts with the help-seeking-related keyword distracting public attention) and internal overwhelm (i.e., attention inequality with 5% help-seeking posts receiving more than 95% likes, comments, and shares). We discover linguistic and non-linguistic help-seeking strategies that could help to prevent the overwhelm, such as including contact information, disclosing situational vulnerabilities, using subjective narratives, and structuring help-seeking posts to a normalized syntax. We also illustrate how community members spontaneously work to prevent the overwhelm with their collective wisdom (e.g., norm development through discussion) and collaborative work (e.g., cross-community support). We reflect on how the findings enrich the literature in crisis informatics and raise design implications that facilitate effective help-seeking on social media during natural disasters.
First Experimental Confirmation of the CH3O + H2CO → CH3OH + HCO Reaction: Expanding the CH3OH Formation Mechanism in Interstellar Ices

The successive addition of H atoms to CO in the solid phase has been hitherto regarded as the primary route to form methanol in dark molecular clouds. However, recent Monte Carlo simulations of interstellar ices alternatively suggested the radical-molecule H-atom abstraction reaction CH3O + H2CO → CH3OH + HCO, in addition to CH3O + H → CH3OH, as a very promising and possibly dominating (70%–90%) final step to form CH3OH in those environments. Here, we compare the contributions of these two steps leading to methanol by experimentally investigating hydrogenation reactions on H2CO and D2CO ices, which ensures comparable starting points between the two scenarios. The experiments are performed under ultrahigh vacuum conditions and astronomically relevant temperatures, with H:H2CO (or D2CO) flux ratios of 10:1 and 30:1. The radical-molecule route in the partially deuterated scenario, CHD2O + D2CO → CHD2OD + DCO, is significantly hampered by the isotope effect in the D-abstraction process, and can thus be used as an artifice to probe the efficiency of this step. We observe a significantly smaller yield of D2CO + H products in comparison to H2CO + H, implying that the CH3O-induced abstraction route must play an important role in the formation of methanol in interstellar ices. Reflection-absorption infrared spectroscopy and temperature-programmed desorption-quadrupole mass spectrometry analyses are used to quantify the species in the ice. Both analytical techniques indicate constant contributions of ∼80% for the abstraction route in the 10–16 K interval, which agrees well with the Monte Carlo calculations. Additional H2CO + D experiments confirm these conclusions.
Anomaly in the dynamical quantum phase transition in a non-Hermitian system with extended gapless phases

The dynamical quantum phase transitions (DQPTs) and the associated winding numbers have been extensively studied in the context Hermitian system. We consider the non-Hermitian analogue of $p$-wave superconductor, supporting Hermitian gapless phase with complex hopping, in presence of on-site or superconducting loss term. This allows us to investigate the effect of non-Hermitian gapless phases on the DQPTs in addition to the Hermitian gapless phases. Our findings indicate that contour analysis of the underlying Hamiltonian, enclosing the origin and/or exceptional points, can predict the occurrences of DQPTs except the quench within the gapless phases. For the Hermitian case with initial and final Hamiltonians both being Hermitian, we find non-monotonic integer jump for the winding number as the hallmark signature of the gapless phase there. For the hybrid case with initial and final Hamiltonians being Hermitian and non-Hermitian respectively, winding number exhibits integer spike in addition to the non-monotonic integer jumps. For the non-Hermitian case with initial and final Hamiltonians both being non-Hermitian, the winding number show half-integer jumps for lossy superconcuctivity that does not have any Hermitian analogue. On the other hand, the integer jumps in winding number is observed for lossy chemical potential. We understand our findings by connecting them with the profile of Fisher zeros and number of exceptional points and/or origin.
The Dirac impenetrable barrier in the limit point of the Klein energy zone

We reanalyze the problem of a 1D Dirac single particle colliding with the electrostatic potential step of height $V_{0}$ with a positive incoming energy that tends to the limit point of the so-called Klein energy zone, i.e., $E\rightarrow V_{0}-\mathrm{m}c^{2}$, for a given $V_{0}$. In such a case, the particle is actually colliding with an impenetrable barrier. In fact, $V_{0}\rightarrow E+\mathrm{m}c^{2}$, for a given relativistic energy $E\,(<V_{0})$, is the maximum value that the height of the step can reach and that ensures the perfect impenetrability of the barrier. Nevertheless, we note that, unlike the nonrelativistic case, the entire eigensolution does not completely vanish, either at the barrier or in the region under the step, but its upper component does satisfy the Dirichlet boundary condition at the barrier. More importantly, by calculating the mean value of the force exerted by the impenetrable wall on the particle in this eigenstate and taking its nonrelativistic limit, we recover the required result. We use two different approaches to obtain the latter two results. In one of these approaches, the corresponding force on the particle is a type of boundary quantum force. Throughout the article, various issues related to the Klein energy zone, the transmitted solutions to this problem, and impenetrable barriers related to boundary conditions are also discussed. In particular, if the negative-energy transmitted solution is used, the lower component of the scattering solution satisfies the Dirichlet boundary condition at the barrier, but the mean value of the external force when $V_{0}\rightarrow E+\mathrm{m}c^{2}$ does not seem to be compatible with the existence of the impenetrable barrier.
LEPUS: Prompt-based Unsupervised Multi-hop Reranking for Open-domain QA

We study unsupervised multi-hop reranking for multi-hop QA (MQA) with open-domain questions. Since MQA requires piecing information from multiple documents, the main challenge thus resides in retrieving and reranking chains of passages that support the reasoning process. Our approach relies on LargE models with Prompt-Utilizing reranking Strategy (LEPUS): we construct an instruction-like prompt based on a candidate document path and compute a relevance score of the path as the probability of generating a given question, according to a pre-trained language model. Though unsupervised, LEPUS yields competitive reranking performance against state-of-the-art methods that are trained on thousands of examples. Adding a small number of samples (e.g., 2 ), we demonstrate further performance gain using in-context learning. Finally, we show that when integrated with a reader module, LEPUS can obtain competitive multi-hop QA performance, e.g., out-performing fully-supervised QA systems. 1