Miracle-dz/UniFAR-FTUs huggingface dataset:
Instruction Position Matters in Sequence Generation with Large Language Models

Large language models (LLMs) are capable of performing conditional sequence generation tasks, such as translation or summarization, through instruction fine-tuning. The fine-tuning data is generally sequentially concatenated from a specific task instruction, an input sentence, and the corresponding response. Considering the locality modeled by the self-attention mechanism of LLMs, these models face the risk of instruction forgetting when generating responses for long input sentences. To mitigate this issue, we propose enhancing the instruction-following capability of LLMs by shifting the position of task instructions after the input sentences. Theoretical analysis suggests that our straightforward method can alter the model's learning focus, thereby emphasizing the training of instruction-following capabilities. Concurrently, experimental results demonstrate that our approach consistently outperforms traditional settings across various model scales (1B / 7B / 13B) and different sequence generation tasks (translation and summarization), without any additional data or annotation costs. Notably, our method significantly improves the zero-shot performance on conditional sequence generation, e.g., up to 9.7 BLEU points on WMT zero-shot translation tasks.
A Benchmark Study on Calibration

Deep neural networks are increasingly utilized in various machine learning tasks. However, as these models grow in complexity, they often face calibration issues, despite enhanced prediction accuracy. Many studies have endeavored to improve calibration performance through the use of specific loss functions, data preprocessing and training frameworks. Yet, investigations into calibration properties have been somewhat overlooked. Our study leverages the Neural Architecture Search (NAS) search space, offering an exhaustive model architecture space for thorough calibration properties exploration. We specifically create a model calibration dataset. This dataset evaluates 90 bin-based and 12 additional calibration measurements across 117,702 unique neural networks within the widely employed NATS-Bench search space. Our analysis aims to answer several longstanding questions in the field, using our proposed dataset: (i) Can model calibration be generalized across different datasets? (ii) Can robustness be used as a calibration measurement? (iii) How reliable are calibration metrics? (iv) Does a post-hoc calibration method affect all models uniformly? (v) How does calibration interact with accuracy? (vi) What is the impact of bin size on calibration measurement? (vii) Which architectural designs are beneficial for calibration? Additionally, our study bridges an existing gap by exploring calibration within NAS. By providing this dataset, we enable further research into NAS calibration. As far as we are aware, our research represents the first large-scale investigation into calibration properties and the premier study of calibration issues within NAS. The project page can be found at https://www.taolinwei. com/calibration-study
Augmentations, Fillings, and Clusters for 2-Bridge Links

We produce the first examples relating non-orientable exact Lagrangian fillings of Legendrian links to cluster theory, showing that the ungraded augmentation variety of certain max-tb representatives of Legendrian $2$-bridge links is isomorphic to a product of $A_n$-type cluster varieties. As part of this construction, we describe a surjective map from the set of (possibly non-orientable) exact Lagrangian fillings to cluster seeds, producing a product of Catalan numbers of distinct fillings. We also relate the ruling stratification of the ungraded augmentation variety to Lam and Speyer's anticlique stratification of acyclic cluster varieties, showing that the two coincide in this context. As a corollary, we apply a result of Rutherford to show that the cluster-theoretic stratification encodes the information of the lowest $a$-degree term of the Kauffman polynomial of the smooth isotopy class of the $2$-bridge links we study.
Stabilizing RNN Gradients through Pre-training

Numerous theories of learning propose to prevent the gradient from exponential growth with depth or time, to stabilize and improve training. Typically, these analyses are conducted on feed-forward fully-connected neural networks or simple single-layer recurrent neural networks, given their mathematical tractability. In contrast, this study demonstrates that pre-training the network to local stability can be effective whenever the architectures are too complex for an analytical initialization. Furthermore, we extend known stability theories to encompass a broader family of deep recurrent networks, requiring minimal assumptions on data and parameter distribution, a theory we call the Local Stability Condition (LSC). Our investigation reveals that the classical Glorot, He, and Orthogonal initialization schemes satisfy the LSC when applied to feed-forward fully-connected neural networks. However, analysing deep recurrent networks, we identify a new additive source of exponential explosion that emerges from counting gradient paths in a rectangular grid in depth and time. We propose a new approach to mitigate this issue, that consists on giving a weight of a half to the time and depth contributions to the gradient, instead of the classical weight of one. Our empirical results confirm that pre-training both feed-forward and recurrent networks, for differentiable, neuromorphic and state-space models to fulfill the LSC, often results in improved final performance. This study contributes to the field by providing a means to stabilize networks of any complexity. Our approach can be implemented as an additional step before pre-training on large augmented datasets, and as an alternative to finding stable initializations analytically.
Neural oscillators for magnetic hysteresis modeling

Hysteresis is a ubiquitous phenomenon in science and engineering; its modeling and identification are crucial for understanding and optimizing the behavior of various systems. We develop an ordinary differential equation-based recurrent neural network (RNN) approach to model and quantify the hysteresis, which manifests itself in sequentiality and history-dependence. Our neural oscillator, HystRNN, draws inspiration from coupled-oscillatory RNN and phenomenological hysteresis models to update the hidden states. The performance of HystRNN is evaluated to predict generalized scenarios, involving first-order reversal curves and minor loops. The findings show the ability of HystRNN to generalize its behavior to previously untrained regions, an essential feature that hysteresis models must have. This research highlights the advantage of neural oscillators over the traditional RNN-based methods in capturing complex hysteresis patterns in magnetic materials, where traditional rate-dependent methods are inadequate to capture intrinsic nonlinearity.
Incremental Property Directed Reachability

Property Directed Reachability (PDR) is a widely used technique for formal verification of hardware and software systems. This paper presents an incremental version of PDR (IPDR), which enables the automatic verification of system instances of incremental complexity. The proposed algorithm leverages the concept of incremental SAT solvers to reuse verification results from previously verified system instances, thereby accelerating the verification process. The new algorithm supports both incremental constraining and relaxing; i.e., starting from an over-constrained instance that is gradually relaxed. To validate the effectiveness of the proposed algorithm, we implemented IPDR and experimentally evaluate it on two different problem domains. First, we consider a circuit pebbling problem, where the number of pebbles is both constrained and relaxed. Second, we explore parallel program instances, progressively increasing the allowed number of interleavings. The experimental results demonstrate significant performance improvements compared to Z3's PDR implementation SPACER. Experiments also show that the incremental approach succeeds in reusing a substantial amount of clauses between instances, for both the constraining and relaxing algorithm.
Expressive probabilistic sampling in recurrent neural networks

In sampling-based Bayesian models of brain function, neural activities are assumed to be samples from probability distributions that the brain uses for probabilistic computation. However, a comprehensive understanding of how mechanistic models of neural dynamics can sample from arbitrary distributions is still lacking. We use tools from functional analysis and stochastic differential equations to explore the minimum architectural requirements for $\textit{recurrent}$ neural circuits to sample from complex distributions. We first consider the traditional sampling model consisting of a network of neurons whose outputs directly represent the samples (sampler-only network). We argue that synaptic current and firing-rate dynamics in the traditional model have limited capacity to sample from a complex probability distribution. We show that the firing rate dynamics of a recurrent neural circuit with a separate set of output units can sample from an arbitrary probability distribution. We call such circuits reservoir-sampler networks (RSNs). We propose an efficient training procedure based on denoising score matching that finds recurrent and output weights such that the RSN implements Langevin sampling. We empirically demonstrate our model's ability to sample from several complex data distributions using the proposed neural dynamics and discuss its applicability to developing the next generation of sampling-based brain models.
Critical Learning Periods Emerge Even in Deep Linear Networks

Critical learning periods are periods early in development where temporary sensory deficits can have a permanent effect on behavior and learned representations. Despite the radical differences between biological and artificial networks, critical learning periods have been empirically observed in both systems. This suggests that critical periods may be fundamental to learning and not an accident of biology. Yet, why exactly critical periods emerge in deep networks is still an open question, and in particular it is unclear whether the critical periods observed in both systems depend on particular architectural or optimization details. To isolate the key underlying factors, we focus on deep linear network models, and show that, surprisingly, such networks also display much of the behavior seen in biology and artificial networks, while being amenable to analytical treatment. We show that critical periods depend on the depth of the model and structure of the data distribution. We also show analytically and in simulations that the learning of features is tied to competition between sources. Finally, we extend our analysis to multi-task learning to show that pre-training on certain tasks can damage the transfer performance on new tasks, and show how this depends on the relationship between tasks and the duration of the pre-training stage. To the best of our knowledge, our work provides the first analytically tractable model that sheds light into why critical learning periods emerge in biological and artificial networks.
Compressing CFI Graphs and Lower Bounds for the Weisfeiler-Leman Refinements

The k-dimensional Weisfeiler-Leman (k-WL) algorithm is a simple combinatorial algorithm that was originally designed as a graph isomorphism heuristic. It naturally finds applications in Babai’s quasipolynomial-time isomorphism algorithm, practical isomorphism solvers, and algebraic graph theory. However, it also has surprising connections to other areas such as logic, proof complexity, combinatorial optimization, and machine learning. The algorithm iteratively computes a coloring of the k-tuples of vertices of a graph. Since Fürer’s linear lower bound [ICALP 2001], it has been an open question whether there is a super-linear lower bound for the iteration number for k-WL on graphs. We answer this question affirmatively, establishing an $\Omega\left(n^{k / 2}\right)$-lower bound for all k.
Regular black holes from higher-derivative and nonlocal gravity: The smeared delta source approximation

In this work we study static spherically symmetric solutions of effective field equations related to local and nonlocal higher-derivative gravity models, based on the effective delta source approximation. We discuss several possibilities for the equations of state, and how they influence the solutions. In particular, we present an equation of state such that the solutions match the Newtonian-limit metric in both regimes of large and small $r$. A significant part of the work is dedicated to the study of the regularity of the solutions and the comparison with the linearized solutions. Explicit solutions are presented for some cases of local and nonlocal models. The results obtained here can be regarded as a possible link between the previous researches on linearized higher-derivative gravity and the solutions of the nonlinear complete field equations.
Heads in the Clouds: Measuring the Implications of Universities Migrating to Public Clouds

With the emergence of mandatory remote education and work in universities due to COVID-19, the `zoomification' of higher education, i.e., the migration of universities to the clouds, reached the public discourse. Ongoing discussions reason about how this shift will take control over students' data away from universities, and may ultimately prevent privacy from being an attainable goal for researchers and students alike. However, there has been no comprehensive measurement of universities' use of public clouds and reliance on Software-as-a-Service offerings to assess how far this migration has already progressed. In this paper, we perform a longitudinal study of the migration to public clouds among universities in the U.S. and Europe, as well as institutions listed in the Times Higher Education (THE) Top100 between January 2015 and June 2021. We find that cloud-adoption differs between countries, with one cluster (Germany, France, Austria, Switzerland) showing a limited move to clouds, while the other cluster (U.S., U.K., the Netherlands, THE Top100) frequently migrates universities' core functions and services to public clouds -- starting long before the COVID-19 pandemic. We attribute this clustering to several socio-economic factors in the respective countries, including the general culture of higher education and the administrative paradigm taken towards running universities. We then analyze and interpret our results, finding that the implications reach beyond individuals' privacy towards questions of academic independence and integrity.
Quasi-localized vibrational modes, Boson peak and sound attenuation in model mass-spring networks

We develop an algorithm to construct mass-spring networks differing in the correlation length of their local elastic properties. These networks reproduce Boson peak and quasilocalized vibrational modes observed in glasses and their relationship. The sound attenuation rate also behaves as in glasses, varying from a Rayleigh scattering to a disorder-broadening regime on increasing the frequency. We ﬁnd that networks modelling glasses with increased stability have a reduced elastic correlation length. Our results support a deep connection between the various vibrational features of disordered solids and clarify their relationship with the correlation length of the local elastic properties.
Effectiveness of social distancing measures and lockdowns for reducing transmission of COVID-19 in non-healthcare, community-based settings

Social distancing measures (SDMs) are community-level interventions that aim to reduce person-to-person contacts in the community. SDMs were a major part of the responses first to contain, then to mitigate, the spread of SARS-CoV-2 in the community. Common SDMs included limiting the size of gatherings, closing schools and/or workplaces, implementing work-from-home arrangements, or more stringent restrictions such as lockdowns. This systematic review summarized the evidence for the effectiveness of nine SDMs. Almost all of the studies included were observational in nature, which meant that there were intrinsic risks of bias that could have been avoided were conditions randomly assigned to study participants. There were no instances where only one form of SDM had been in place in a particular setting during the study period, making it challenging to estimate the separate effect of each intervention. The more stringent SDMs such as stay-at-home orders, restrictions on mass gatherings and closures were estimated to be most effective at reducing SARS-CoV-2 transmission. Most studies included in this review suggested that combinations of SDMs successfully slowed or even stopped SARS-CoV-2 transmission in the community. However, individual effects and optimal combinations of interventions, as well as the optimal timing for particular measures, require further investigation. This article is part of the theme issue 'The effectiveness of non-pharmaceutical interventions on the COVID-19 pandemic: the evidence'.
Faire: Repairing Fairness of Neural Networks via Neuron Condition Synthesis

Deep Neural Networks (DNNs) have achieved tremendous success in many applications, while it has been demonstrated that DNNs can exhibit some undesirable behaviors on concerns such as robustness, privacy, and other trustworthiness issues. Among them, fairness (i.e.,, non-discrimination) is one important property especially when they are applied to some sensitive applications (e.g.,, finance and employment). However, DNNs easily learn spurious correlations between protected attributes (e.g.,, age, gender, race) and the classification task and develop discriminatory behaviors if the training data is imbalanced. Such discriminatory decisions in sensitive applications would introduce severe social impacts. To expose potential discrimination problems in DNNs before putting them in use, some testing techniques have been proposed to identify the discriminatory instances (i.e.,, instances that show defined discrimination). However, how to repair DNNs after detecting such discrimination is still challenging. Existing techniques mainly rely on retraining on a large number of discriminatory instances generated by testing methods, which requires huge time overhead and makes the repairing inefficient. In this work, we propose a method Faire to effectively and efficiently repair the fairness issues of DNNs, without using additional data (e.g.,, discriminatory instances). Our basic idea is inspired by the traditional program repair method that synthesizes proper condition checking. To repair traditional programs, a typical method is to localize the program defects and repair the program logic by adding condition checking. Similarly, for DNNs, we try to understand the unfair logic and reformulate it with well-designed condition checking. In this paper, we synthesize the condition that can reduce the effect of features relevant to the protected attributes in the DNN. Specifically, we first perform the neuron-based analysis and check the functionalities of neurons to identify neurons whose outputs could be regarded as features relevant to protected attributes and original tasks. Then a new condition layer is added after each hidden layer to penalize neurons that are accountable for the protected features (i.e.,, intermediate features relevant to protected attributes) and promote neurons that are accountable for the non-protected features (i.e.,, intermediate features relevant to original tasks). In sum, the repair rate of Faire reaches up to more than \(99\% \) , which outperforms other methods, and the whole repairing process only takes no more than 340 seconds. The evaluation results demonstrate that our approach can effectively and efficiently repair the individual discriminatory instances of the target model.
Red light perception by the root is essential for gibberellin-induced primary-root elongation in tomato

The promoting effect of gibberellin (GA) on primary-root elongation is well-documented in several plant species, yet its influence in others, including tomato (Solanum lycopersicum), remains unclear. The role of GA in primary-root elongation has been studied in tomato using the GA-deficient mutants gib-1 and ga20-oxidase (ga20ox1) and various growth systems, including Dark (D)-root and D-shoot plates. GA application to these mutants following germination on vermiculite, promoted primary-root elongation. However, when the roots grew deeper into the dark environment the hormone had no effect. RNA-seq analysis of dark-grown roots, treated with GA, revealed typical transcriptional responses, but the output for cell expansion remained unaffected. When dark-grown roots were illuminated deep in the ground, the hormone promoted their elongation. The results suggest that activation of Phytochrome B (PhyB) in the root, by red light, is essential for GA-induced elongation. We propose that GA promotes tomato root elongation after germination, when roots are exposed to low light underground and this contributes to rapid seedling establishment. As roots penetrate deeper into the soil, insensitivity to GA due to the lack of light may be important for sustained root growth under fluctuating water availability, given that water deficiency suppresses GA accumulation.
Differential Impacts of Technology-Network Structures on Cost Efficiency: Knowledge Spillovers in Healthcare

ABSTRACT We examine how hospital cost efficiency can improve because of knowledge spillover effects, arising from the experiences of members of a healthcare system. We identify two types of functional technology networks—that we term repository-type and workflow-type networks—which are defined by the adoption patterns of different types of electronic health record (EHR) systems application functions by hospitals in a health system. Additionally, we examine how structural features of these two network types—namely, network centrality and interconnectedness—impact such knowledge spillover effects. By analyzing seven years of panel data for 1,420 U.S. hospitals across 216 health systems, and data obtained from surveys of healthcare experts, we found that greater centrality of nodes in repository-type networks enhances the influence of partners’ experiences on cost efficiency but has the opposite effect in workflow-type networks, while greater interconnectedness of workflow-type networks enhances the influence of partners’ experiences on cost efficiency. Interestingly, we found the opposite effect in repository-type networks, where greater interconnectedness diminishes the influence of partners’ experiences on cost efficiency. We discuss reasons for this surprising and counterintuitive finding. Overall, our results demonstrate that at least in healthcare settings where information technologies enable knowledge-sharing networks to be formed, benefits can accrue through knowledge spillover effects; moreover, variations across functional categories of technologies adopted in a network context can result in different technology-network structures with differential impacts on hospital cost efficiency because of their influence on knowledge spillover at both the hospital and system levels. Not only can these results explain mixed findings in the literature that has been dominated by a focus on cost efficiency outcomes dependent solely on the EHR adoption choices of individual hospitals, but they also provide the basis for guidelines for a network-based perspective of technology adoption. We discuss other possible settings to which our empirical findings can be reasonably generalized, including supply chain technologies and the Internet of Things.