Miracle-dz/UniFAR-FTUs huggingface dataset:
Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Models

Large language models (LLMs) are excellent in-context learners. However, the sensitivity of data contained in prompts raises privacy concerns. Our work first shows that these concerns are valid: we instantiate a simple but highly effective membership inference attack against the data used to prompt LLMs. To address this vulnerability, one could forego prompting and resort to fine-tuning LLMs with known algorithms for private gradient descent. However, this comes at the expense of the practicality and efficiency offered by prompting. Therefore, we propose to privately learn to prompt. We first show that soft prompts can be obtained privately through gradient descent on downstream data. However, this is not the case for discrete prompts. Thus, we orchestrate a noisy vote among an ensemble of LLMs presented with different prompts, i.e., a flock of stochastic parrots. The vote privately transfers the flock's knowledge into a single public prompt. We show that LLMs prompted with our private algorithms closely match the non-private baselines. For example, using GPT3 as the base model, we achieve a downstream accuracy of 92.7% on the sst2 dataset with ($\epsilon=0.147, \delta=10^{-6}$)-differential privacy vs. 95.2% for the non-private baseline. Through our experiments, we also show that our prompt-based approach is easily deployed with existing commercial APIs.
Steady-state quantum chaos in open quantum systems

We introduce the notion of steady-state quantum chaos as a general phenomenon in open quantum many-body systems. Classifying an isolated or open quantum system as integrable or chaotic relies in general on the properties of the equations governing its time evolution. This however may fail in predicting the actual nature of the quantum dynamics, that can be either regular or chaotic depending on the initial state. Chaos and integrability in the steady state of an open quantum system are instead uniquely determined by the spectral structure of the time evolution generator. To characterize steady-state quantum chaos we introduce a spectral analysis based on the spectral statistics of quantum trajectories (SSQT). We test the generality and reliability of the SSQT criterion on several dissipative systems, further showing that an open system with chaotic structure can evolve towards either a chaotic or integrable steady state. We study steady-state chaos in the driven-dissipative Bose-Hubbard model, a paradigmatic example of out-of-equilibrium bosonic system without particle number conservation. This system is widely employed as a building block in state-of-the-art noisy intermediate-scale quantum devices, with applications in quantum computation and sensing. Finally, our analysis shows the existence of an emergent dissipative quantum chaos, where the classical and semi-classical limits display an integrable behaviour. This emergent dissipative quantum chaos arises from the quantum and classical fluctuations associated with the dissipation mechanism. Our work establishes a fundamental understanding of the integrable and chaotic dynamics of open quantum systems and paves the way for the investigation of dissipative quantum chaos and its consequences on quantum technologies.
Exploring Automatically Perturbed Natural Language Explanations in Relation Extraction

Previous research has demonstrated that natural language explanations provide valuable inductive biases that guide models, thereby improving the generalization ability and data efficiency. In this paper, we undertake a systematic examination of the effectiveness of these explanations. Remarkably, we find that corrupted explanations with diminished inductive biases can achieve competitive or superior performance compared to the original explanations. Our findings furnish novel insights into the characteristics of natural language explanations in the following ways: (1) the impact of explanations varies across different training styles and datasets, with previously believed improvements primarily observed in frozen language models. (2) While previous research has attributed the effect of explanations solely to their inductive biases, our study shows that the effect persists even when the explanations are completely corrupted. We propose that the main effect is due to the provision of additional context space. (3) Utilizing the proposed automatic perturbed context, we were able to attain comparable results to annotated explanations, but with a significant increase in computational efficiency, 20-30 times faster.
Radio Galaxy Zoo: Towards building the first multi-purpose foundation model for radio astronomy with self-supervised learning

In this work, we apply self-supervised learning with instance differentiation to learn a robust, multi-purpose representation for image analysis of resolved extragalactic continuum images. We train a multi-use model which compresses our unlabelled data into a structured, low dimensional representation which can be used for a variety of downstream tasks (e.g. classification, similarity search). We exceed baseline supervised Fanaroff-Riley classification performance by a statistically significant margin, with our model reducing the test set error by up to half. Our model is also able to maintain high classification accuracy with very few labels, with only $7.79{{\%}}$ error when only using 145 labels. We further demonstrate that by using our foundation model, users can efficiently trade off compute, human labelling cost and test set accuracy according to their respective budgets, allowing for efficient classification in a wide variety of scenarios. We highlight the generalizability of our model by showing that it enables accurate classification in a label scarce regime with data from the new MIGHTEE survey without any hyper-parameter tuning, where it improves upon the baseline by $\sim 8{{\%}}$ . Visualizations of our labelled and un-labelled data show that our model’s representation space is structured with respect to physical properties of the sources, such as angular source extent. We show that the learned representation is scientifically useful even if no labels are available by performing a similarity search, finding hybrid sources in the RGZ DR1 data-set without any labels. We show that good augmentation design and hyper-parameter choice can help achieve peak performance, while emphasising that optimal hyper-parameters are not required to obtain benefits from self-supervised pre-training.
VanillaKD: Revisit the Power of Vanilla Knowledge Distillation from Small Scale to Large Scale

The tremendous success of large models trained on extensive datasets demonstrates that scale is a key ingredient in achieving superior results. Therefore, the reflection on the rationality of designing knowledge distillation (KD) approaches for limited-capacity architectures solely based on small-scale datasets is now deemed imperative. In this paper, we identify the \emph{small data pitfall} that presents in previous KD methods, which results in the underestimation of the power of vanilla KD framework on large-scale datasets such as ImageNet-1K. Specifically, we show that employing stronger data augmentation techniques and using larger datasets can directly decrease the gap between vanilla KD and other meticulously designed KD variants. This highlights the necessity of designing and evaluating KD approaches in the context of practical scenarios, casting off the limitations of small-scale datasets. Our investigation of the vanilla KD and its variants in more complex schemes, including stronger training strategies and different model capacities, demonstrates that vanilla KD is elegantly simple but astonishingly effective in large-scale scenarios. Without bells and whistles, we obtain state-of-the-art ResNet-50, ViT-S, and ConvNeXtV2-T models for ImageNet, which achieve 83.1\%, 84.3\%, and 85.0\% top-1 accuracy, respectively. PyTorch code and checkpoints can be found at https://github.com/Hao840/vanillaKD.
Theoretical Guarantees of Learning Ensembling Strategies with Applications to Time Series Forecasting

Ensembling is among the most popular tools in machine learning (ML) due to its effectiveness in minimizing variance and thus improving generalization. Most ensembling methods for black-box base learners fall under the umbrella of"stacked generalization,"namely training an ML algorithm that takes the inferences from the base learners as input. While stacking has been widely applied in practice, its theoretical properties are poorly understood. In this paper, we prove a novel result, showing that choosing the best stacked generalization from a (finite or finite-dimensional) family of stacked generalizations based on cross-validated performance does not perform"much worse"than the oracle best. Our result strengthens and significantly extends the results in Van der Laan et al. (2007). Inspired by the theoretical analysis, we further propose a particular family of stacked generalizations in the context of probabilistic forecasting, each one with a different sensitivity for how much the ensemble weights are allowed to vary across items, timestamps in the forecast horizon, and quantiles. Experimental results demonstrate the performance gain of the proposed method.
A Mechanism for Solving Relational Tasks in Transformer Language Models

A primary criticism towards language models (LMs) is their inscrutability. This paper presents evidence that, despite their size and complexity, LMs sometimes exploit a simple computational mechanism to solve one-to-one relational tasks (e.g., capital_of(Poland)=Warsaw). We investigate a range of language model sizes (from 124M parameters to 176B parameters) in an in-context learning setting, and find that for a variety of tasks (involving capital cities, upper-casing, and past-tensing) a key part of the mechanism reduces to a simple linear update typically applied by the feedforward (FFN) networks. These updates also tend to promote the output of the relation in a content-independent way (e.g., encoding Poland:Warsaw::China:Beijing), revealing a predictable pattern that these models take in solving these tasks. We further show that this mechanism is specific to tasks that require retrieval from pretraining memory, rather than retrieval from local context. Our results contribute to a growing body of work on the mechanistic interpretability of LLMs, and offer reason to be optimistic that, despite the massive and non-linear nature of the models, the strategies they ultimately use to solve tasks can sometimes reduce to familiar and even intuitive algorithms.
Cosmic energy density: particles, fields and the vacuum

We revisit the cosmic evolution of the energy density of a quantized free scalar field and assess under what conditions the particle production and classical field approximations reproduce its correct value. Because the unrenormalized energy-momentum tensor diverges in the ultraviolet, it is necessary to frame our discussion within an appropriate regularization and renormalization scheme. Pauli-Villars avoids some of the drawbacks of adiabatic subtraction and dimensional regularization and is particularly convenient in this context. In some cases, we can predict the evolution of the energy density irrespectively of the quantum state of the field modes. To further illustrate our results we focus however on the in vacuum, the preferred quantum state singled out by inflation, and explore to what extent the latter determines the subsequent evolution of the energy density regardless of the unknown details of reheating. We contrast this discussion with examples of transitions to radiation domination that avoid some of the problems of the one commonly studied in the literature, and point out some instances in which the particle production or the classical field approximations lead to the incorrect energy density. Along the way, we also elaborate on the connection of our analysis to dynamical dark energy models and axion-like dark matter candidates.
The Nonlinear Theory of Sound

We prove the existence of ``pure tone'' nonlinear sound waves of all frequencies. These are smooth, space and time periodic, oscillatory solutions of the $3\times3$ compressible Euler equations in one space dimension. Being perturbations of solutions of a linear wave equation, they provide a rigorous justification for the centuries old theory of Acoustics. In particular, Riemann's celebrated 1860 proof that compressions always form shocks holds for isentropic and barotropic flows, but for generic entropy profiles, shock-free periodic solutions containing nontrivial compressions and rarefactions exist for every wavenumber $k$.
Interactive Segment Anything NeRF with Feature Imitation

This paper investigates the potential of enhancing Neural Radiance Fields (NeRF) with semantics to expand their applications. Although NeRF has been proven useful in real-world applications like VR and digital creation, the lack of semantics hinders interaction with objects in complex scenes. We propose to imitate the backbone feature of off-the-shelf perception models to achieve zero-shot semantic segmentation with NeRF. Our framework reformulates the segmentation process by directly rendering semantic features and only applying the decoder from perception models. This eliminates the need for expensive backbones and benefits 3D consistency. Furthermore, we can project the learned semantics onto extracted mesh surfaces for real-time interaction. With the state-of-the-art Segment Anything Model (SAM), our framework accelerates segmentation by 16 times with comparable mask quality. The experimental results demonstrate the efficacy and computational advantages of our approach. Project page: \url{https://me.kiui. moe/san/}.
Alternating quantum-emitter chains: Exceptional-point phase transition, edge state, and quantum walks

We study the long-range hopping limit of a one-dimensional array of $N$ equal-distanced quantum emitters in free space, where the hopping amplitude of emitter excitation is proportional to the inverse of the distance and equals the lattice dimension. For two species of emitters in an alternating arrangement, the single excitation sector exhibits non-Hermitian spectral singularities known as exceptional points. We unveil an unconventional phase transition, dubbed exceptional-point phase transition, from the collective to individual spontaneous emission behaviors. At the transition point, the $N \times N$ Hamiltonian fragments into $N/2-1$ many two-dimensional non-diagonalizable blocks. The remaining diagonalizable block contains a dissipation-induced edge state with algebraically localized profiles, and we provide numerical evidence for its existence in the infinite-array limit. We demonstrate that the edge state can be eliminated via a continuous deformation, consistent with the ill-definedness of bulk topological invariant. We also propose a spatially resolved character to quantify the incoherent flow and loss in the non-unitary quantum walks of single atomic excitations.
More than Words: Twitter Chatter and Financial Market Sentiment

We build a new measure of credit and financial market sentiment using Natural Language Processing on Twitter data. We find that the Twitter Financial Sentiment Index (TFSI) correlates highly with corporate bond spreads and other price- and survey-based measures of financial conditions. We document that overnight Twitter financial sentiment helps predict next day stock market returns. Most notably, we show that the index contains information that helps forecast changes in the U.S. monetary policy stance: a deterioration in Twitter financial sentiment the day ahead of an FOMC statement release predicts the size of restrictive monetary policy shocks. Finally, we document that sentiment worsens in response to an unexpected tightening of monetary policy.
Learn to Not Link: Exploring NIL Prediction in Entity Linking

Entity linking models have achieved significant success via utilizing pretrained language models to capture semantic features. However, the NIL prediction problem, which aims to identify mentions without a corresponding entity in the knowledge base, has received insufficient attention. We categorize mentions linking to NIL into Missing Entity and Non-Entity Phrase, and propose an entity linking dataset NEL that focuses on the NIL prediction problem. NEL takes ambiguous entities as seeds, collects relevant mention context in the Wikipedia corpus, and ensures the presence of mentions linking to NIL by human annotation and entity masking. We conduct a series of experiments with the widely used bi-encoder and cross-encoder entity linking models, results show that both types of NIL mentions in training data have a significant influence on the accuracy of NIL prediction. Our code and dataset can be accessed at https://github.com/solitaryzero/NIL_EL
Communication quality in extreme environments affects performance of astronauts and their support teams through increases in workload: Insights from the AMADEE-20 analog Mars mission

Astronaut crews and ground control support teams are highly interdependent teams that need to communicate effectively to achieve a safe mission - despite being separated by large distances. Team communication quality with its facets clarity of objectives and information flow is a key coordination process to achieve high team performance and task satisfaction. Especially in interdependent teams working in extreme environments with time-delayed communications, the team's success depends on effective communication. We hypothesized that communication quality affects two key team outcomes, performance and task satisfaction, and that these effects can be explained by increased workload (effort and frustration). Hypotheses were tested during the AMADEE-20 analog Mars mission of the Austrian Space Forum. The analog astronauts (AA) were supported by an On-Site-Support (OSS) team and a remote Mission-Support-Centre (MSC) team. The MSC was the only contact for both AA and OSS, and the communication between them had a one-way delay of 10 minutes. Our study consisted of three runs in which members of each team had to exchange information to solve an interdependent task. We measured communication quality, effort and frustration, task satisfaction, and team performance. Results show that clarity of objectives and information flow positively impacted multiteam system performance. Furthermore, clarity of objectives reduced experienced effort which in turn enhanced team performance. High levels of information flow reduced experienced frustration, in turn enhancing task satisfaction. Our findings show that these facets of communication quality are essential for multiteam systems that work separated from each other. We stress that specific (team) communication training for astronauts and support personnel will be key to effective teamwork during future Mars missions, and thus to overall mission success.
Learning Occupancy for Monocular 3D Object Detection

Monocular 3D detection is a challenging task due to the lack of accurate 3D information. Existing approaches typically rely on geometry constraints and dense depth estimates to facilitate the learning, but often fail to fully exploit the benefits of three-dimensional feature extraction in frustum and 3D space. In this paper, we propose \textbf{OccupancyM3D}, a method of learning occupancy for monocular 3D detection. It directly learns occupancy in frustum and 3D space, leading to more discriminative and informative 3D features and representations. Specifically, by using synchronized raw sparse LiDAR point clouds, we define the space status and generate voxel-based occupancy labels. We formulate occupancy prediction as a simple classification problem and design associated occupancy losses. Resulting occupancy estimates are employed to enhance original frustum/3D features. As a result, experiments on KITTI and Waymo open datasets demonstrate that the proposed method achieves a new state of the art and surpasses other methods by a significant margin. Codes and pre-trained models will be available at \url{https://github.com/SPengLiang/OccupancyM3D}.
Linear Layouts of Bipartite Planar Graphs

A linear layout of a graph $ G $ consists of a linear order $\prec$ of the vertices and a partition of the edges. A part is called a queue (stack) if no two edges nest (cross), that is, two edges $ (v,w) $ and $ (x,y) $ with $ v \prec x \prec y \prec w $ ($ v \prec x \prec w \prec y $) may not be in the same queue (stack). The best known lower and upper bounds for the number of queues needed for planar graphs are 4 [Alam et al., Algorithmica 2020] and 42 [Bekos et al., Algorithmica 2022], respectively. While queue layouts of special classes of planar graphs have received increased attention following the breakthrough result of [Dujmovi\'c et al., J. ACM 2020], the meaningful class of bipartite planar graphs has remained elusive so far, explicitly asked for by Bekos et al. In this paper we investigate bipartite planar graphs and give an improved upper bound of 28 by refining existing techniques. In contrast, we show that two queues or one queue together with one stack do not suffice; the latter answers an open question by Pupyrev [GD 2018]. We further investigate subclasses of bipartite planar graphs and give improved upper bounds; in particular we construct 5-queue layouts for 2-degenerate quadrangulations.
Vehicle-in-Virtual-Environment (VVE)

The current approach to connected and autonomous driving function development and evaluation uses model-in-the-loop simulation, hardware-in-the-loop simulation, and limited proving ground work followed by public road deployment of beta version of software and technology. The rest of the road users are involuntarily forced into taking part in the development and evaluation of these connected and autonomous driving functions in this approach. This is an unsafe, costly and inefficient method. Motivated by these shortcomings, this paper introduces the Vehicle-in-Virtual-Environment (VVE) method of safe, efficient and low cost connected and autonomous driving function development, evaluation and demonstration. The VVE method is compared to the existing state-of-the-art. Its basic implementation for a path following task is used to explain the method where the actual autonomous vehicle operates in a large empty area with its sensor feeds being replaced by realistic sensor feeds corresponding to its location and pose in the virtual environment. It is possible to easily change the development virtual environment and inject rare and difficult events which can be tested very safely. Vehicle-to-Pedestrian (V2P) communication based pedestrian safety is chosen as the application use case for VVE and corresponding experimental results are presented and discussed. It is noted that actual pedestrians and other vulnerable road users can be used very safely in this approach.
ConvGQR: Generative Query Reformulation for Conversational Search

In conversational search, the user’s real search intent for the current conversation turn is dependent on the previous conversation history. It is challenging to determine a good search query from the whole conversation context. To avoid the expensive re-training of the query encoder, most existing methods try to learn a rewriting model to de-contextualize the current query by mimicking the manual query rewriting.However, manually rewritten queries are not always the best search queries.Thus, training a rewriting model on them would lead to sub-optimal queries. Another useful information to enhance the search query is the potential answer to the question. In this paper, we propose ConvGQR, a new framework to reformulate conversational queries based on generative pre-trained language models (PLMs), one for query rewriting and another for generating potential answers.By combining both, ConvGQR can produce better search queries.In addition, to relate query reformulation to the retrieval task, we propose a knowledge infusion mechanism to optimize both query reformulation and retrieval. Extensive experiments on four conversational search datasets demonstrate the effectiveness of ConvGQR.