Miracle-dz/UniFAR-FTUs huggingface dataset:
Language-Guided World Models: A Model-Based Approach to AI Control

Installing probabilistic world models into artificial agents opens an efficient channel for humans to communicate with and control these agents. In addition to updating agent policies, humans can modify their internal world models in order to influence their decisions. The challenge, however, is that currently existing world models are difficult for humans to adapt because they lack a natural communication interface. Aimed at addressing this shortcoming, we develop Language-Guided World Models (LWMs), which can capture environment dynamics by reading language descriptions. These models enhance agent communication efficiency, allowing humans to simultaneously alter their behavior on multiple tasks with concise language feedback. They also enable agents to self-learn from texts originally written to instruct humans. To facilitate the development of LWMs, we design a challenging benchmark based on the game of MESSENGER (Hanjie et al., 2021), requiring compositional generalization to new language descriptions and environment dynamics. Our experiments reveal that the current state-of-the-art Transformer architecture performs poorly on this benchmark, motivating us to design a more robust architecture. To showcase the practicality of our proposed LWMs, we simulate a scenario where these models augment the interpretability and safety of an agent by enabling it to generate and discuss plans with a human before execution. By effectively incorporating language feedback on the plan, the models boost the agent performance in the real environment by up to three times without collecting any interactive experiences in this environment.
Physics-Inspired Degradation Models for Hyperspectral Image Fusion

The fusion of a low-spatial-resolution hyperspectral image (LR-HSI) with a high-spatial-resolution multispectral image (HR-MSI) has garnered increasing research interest. However, most fusion methods solely focus on the fusion algorithm itself and overlook the degradation models, which results in unsatisfactory performance in practical scenarios. To fill this gap, we propose physics-inspired degradation models (PIDM) to model the degradation of LR-HSI and HR-MSI, which comprises a spatial degradation network (SpaDN) and a spectral degradation network (SpeDN). SpaDN and SpeDN are designed based on two insights. First, we employ spatial warping and spectral modulation operations to simulate lens aberrations, thereby introducing non-uniformity into the spatial and spectral degradation processes. Second, we utilize asymmetric downsampling and parallel downsampling operations to separately reduce the spatial and spectral resolutions of the images, thus ensuring the matching of spatial and spectral degradation processes with specific physical characteristics. Once SpaDN and SpeDN are established, we adopt a self-supervised training strategy to optimize the network parameters and provide a plug-and-play solution for fusion methods. Comprehensive experiments demonstrate that our proposed PIDM can boost the fusion performance of existing fusion methods in practical scenarios.
Contrasting twisted bilayer graphene and transition metal dichalcogenides for fractional Chern insulators: an emergent gauge picture

The recent experimental discovery of the zero-field fractional Chern insulator (FCI) in twisted $\mathrm{MoTe_2}$ moir\'e superlattices has sparked immense interest in this exotic topological quantum state. The FCI has also been observed in previous experiments in magic angle twisted bilayer graphene (TBG) under a finite magnetic field of about 5 Tesla. Generally, the stabilization of FCI requires fine-tuning the topological band to satisfy certain conditions. It would still be helpful to have an intuitive picture to understand the different behaviors in twisted $\mathrm{MoTe_2}$ and TBG. Here, we compare them through the lens of emergent gauge fields. In TBG, the system can be mapped to two Dirac fermions coupled to emergent gauge fields with opposite signs. In contrast, the twisted $\mathrm{MoTe_2}$ reduces to a hole with parabolic dispersion coupled to an emergent gauge field. This contrasting gauge structure provides a new perspective on the observed difference: the zero-field FCI is stable in $\mathrm{MoTe_2}$ but absent in TBG. Based on this understanding, we will explore potential strategies for stabilizing FCI in both moir\'e superlattices.
Lattice-dynamics and in-plane antiferromagnetism in MnxZn1_xPS3 across the entire composition range

Alloyed MnxZn1_xPS3 samples have been grown covering the whole compositional range and studied by means of Raman spectroscopy at temperatures covering from 4K up to 850K. Our results, supported by SQUID magnetic measurements, allowed, from one hand, to complete the magnetic phase diagram of MnxZn1_xPS3 and establish x>0.3 as the composition at which the alloy retains antiferromagnetism and, from the other hand, to identify the Raman signatures indicative of a magnetic transition. The origin of these Raman signatures is discussed in terms of spin-phonon coupling resulting in the appearance of low- and high-frequency zone-folded phonon modes. For the alloy, an assignment of the 1st and 2nd order modes is provided with the aid of first-principle lattice-dynamical calculations. The compositional dependence of all phonon modes is described and the presence of zone-folded modes is shown to take place for both, the alloy and MnPS3. Finally, a comparison of the Raman spectra of ZnPS3 to other compounds of the transition-metal phosphorous trisulfide family allowed shows that low-frequency phonon peaks exhibit an abnormally large broadening. This is consistent with previous claims on the occurrence of a second-order Jahn-Teller effect that takes place for ZnPS3 and Zn-rich MnxZn1_xPS3.
Aligner: Achieving Efficient Alignment through Weak-to-Strong Correction

Efforts to align Large Language Models (LLMs) are mainly conducted via Reinforcement Learning from Human Feedback (RLHF) methods. However, RLHF encounters major challenges including training reward models, actor-critic engineering, and importantly, it requires access to LLM parameters. Here we introduce Aligner, a new efficient alignment paradigm that bypasses the whole RLHF process by learning the correctional residuals between the aligned and the unaligned answers. Our Aligner offers several key advantages. Firstly, it is an autoregressive seq2seq model that is trained on the query-answer-correction dataset via supervised learning; this offers a parameter-efficient alignment solution with minimal resources. Secondly, the Aligner facilitates weak-to-strong generalization; finetuning large pretrained models by Aligner's supervisory signals demonstrates strong performance boost. Thirdly, Aligner functions as a model-agnostic plug-and-play module, allowing for its direct application on different open-source and API-based models. Remarkably, Aligner-7B improves 11 different LLMs by 21.9% in helpfulness and 23.8% in harmlessness on average (GPT-4 by 17.5% and 26.9%). When finetuning (strong) Llama2-70B with (weak) Aligner-13B's supervision, we can improve Llama2 by 8.2% in helpfulness and 61.6% in harmlessness. See our dataset and code at https://aligner2024.github. io
English Prompts are Better for NLI-based Zero-Shot Emotion Classification than Target-Language Prompts

Emotion classification in text is a challenging task due to the processes involved when interpreting a textual description of a potential emotion stimulus. In addition, the set of emotion categories is highly domain-specific. For instance, literature analysis might require the use of aesthetic emotions (e.g., finding something beautiful), and social media analysis could benefit from fine-grained sets (e.g., separating anger from annoyance) than only those that represent basic categories as they have been proposed by Paul Ekman (anger, disgust, fear, joy, surprise, sadness). This renders the task an interesting field for zero-shot classifications, in which the label set is not known at model development time. Unfortunately, most resources for emotion analysis are English, and therefore, most studies on emotion analysis have been performed in English, including those that involve prompting language models for text labels. This leaves us with a research gap that we address in this paper: In which language should we prompt for emotion labels on non-English texts? This is particularly of interest when we have access to a multilingual large language model, because we could request labels with English prompts even for non-English data. Our experiments with natural language inference-based language models show that it is consistently better to use English prompts even if the data is in a different language.
Video Editing for Video Retrieval

Though pre-training vision-language models have demonstrated significant benefits in boosting video-text retrieval performance from large-scale web videos, fine-tuning still plays a critical role with manually annotated clips with start and end times, which requires considerable human effort. To address this issue, we explore an alternative cheaper source of annotations, single timestamps, for video-text retrieval. We initialise clips from timestamps in a heuristic way to warm up a retrieval model. Then a video clip editing method is proposed to refine the initial rough boundaries to improve retrieval performance. A student-teacher network is introduced for video clip editing. The teacher model is employed to edit the clips in the training set whereas the student model trains on the edited clips. The teacher weights are updated from the student's after the student's performance increases. Our method is model agnostic and applicable to any retrieval models. We conduct experiments based on three state-of-the-art retrieval models, COOT, VideoCLIP and CLIP4Clip. Experiments conducted on three video retrieval datasets, YouCook2, DiDeMo and ActivityNet-Captions show that our edited clips consistently improve retrieval performance over initial clips across all the three retrieval models.
A flexible Bayesian g-formula for causal survival analyses with time-dependent confounding

In longitudinal observational studies with a time-to-event outcome, a common objective in causal analysis is to estimate the causal survival curve under hypothetical intervention scenarios within the study cohort. The g-formula is a particularly useful tool for this analysis. To enhance the traditional parametric g-formula approach, we developed a more adaptable Bayesian g-formula estimator. This estimator facilitates both longitudinal predictive and causal inference. It incorporates Bayesian additive regression trees in the modeling of the time-evolving generative components, aiming to mitigate bias due to model misspecification. Specifically, we introduce a more general class of g-formulas for discrete survival data. These formulas can incorporate the longitudinal balancing scores, which serve as an effective method for dimension reduction and are vital when dealing with an expanding array of time-varying confounders. The minimum sufficient formulation of these longitudinal balancing scores is linked to the nature of treatment regimes, whether static or dynamic. For each type of treatment regime, we provide posterior sampling algorithms, which are grounded in the Bayesian additive regression trees framework. We have conducted simulation studies to illustrate the empirical performance of our proposed Bayesian g-formula estimators, and to compare them with existing parametric estimators. We further demonstrate the practical utility of our methods in real-world scenarios using data from the Yale New Haven Health System's electronic health records.
Nonlinear potential theoretic methods in nonuniformly ellliptic problems

Nonuniform ellipticity is a classical topic in the theory of partial differential equations. While several results in regularity theory have been adding up over decades, many basic issues, as for instance the validity of Schauder theory and sharp dependence of regularity upon data, remained opened for a while. In these notes we give an overview of recent results and techniques about the topic, that, via a novel use of nonlinear potential theoretic methods, allow to answer several of the above questions.
Subsurface Thermophysical Properties of Europa’s Leading and Trailing Hemispheres as Revealed by ALMA

We present best-fit values of porosity—and the corresponding effective thermal inertiae—determined from three different depths in Europa’s near-subsurface (∼1–20 cm). The porosity of the upper ∼20 cm of Europa’s subsurface varies between 75% and 50% (Γeff ≈ 50–140 J m−2 K−1 s−1/2) on the leading hemisphere and 50%–40% (Γeff ≈ 140–180 J m−2 K−1 s−1/2) on the trailing hemisphere. Residual maps produced by comparison with these models reveal thermally anomalous features that cannot be reproduced by globally homogeneous porosity models. These regions are compared to Europa’s surface terrain and known compositional variations. We find that some instances of warm thermal anomalies are co-located with known geographical or compositional features on both the leading and trailing hemisphere; cool temperature anomalies are well correlated with surfaces previously observed to contain pure, crystalline water ice and the expansive rays of Pwyll crater. Anomalous regions correspond to locations with subsurface properties different from those of our best-fit models, such as potentially elevated thermal inertia, decreased emissivity, or more porous regolith. We also find that ALMA observations at ∼3 mm sound below the thermal skin depth of Europa (∼10–15 cm) for a range of porosity values, and thus do not exhibit features indicative of diurnal variability or residuals similar to other frequency bands. Future observations of Europa at higher angular resolution may reveal additional locations of variable subsurface thermophysical properties, while those at other wavelengths will inform our understanding of the regolith compaction length and the effects of external processes on the shallow subsurface.
Solving Hierarchical Information-Sharing Dec-POMDPs: An Extensive-Form Game Approach

A recent theory shows that a multi-player decentralized partially observable Markov decision process can be transformed into an equivalent single-player game, enabling the application of \citeauthor{bellman}'s principle of optimality to solve the single-player game by breaking it down into single-stage subgames. However, this approach entangles the decision variables of all players at each single-stage subgame, resulting in backups with a double-exponential complexity. This paper demonstrates how to disentangle these decision variables while maintaining optimality under hierarchical information sharing, a prominent management style in our society. To achieve this, we apply the principle of optimality to solve any single-stage subgame by breaking it down further into smaller subgames, enabling us to make single-player decisions at a time. Our approach reveals that extensive-form games always exist with solutions to a single-stage subgame, significantly reducing time complexity. Our experimental results show that the algorithms leveraging these findings can scale up to much larger multi-player games without compromising optimality.
Sub-cycle resolved strong field ionization of chiral molecules and the origin of chiral photoelectron asymmetries

We report on strong field ionization of S- and R-propylene oxide in circularly polarized two-color laser fields. We find that the relative helicity of the two single color laser fields affects the photoelectron circular dichroism (PECD). Further, we observe that PECD is modulated as a function of the sub-cycle release time of the electron. Our experimental observations are successfully described by a heuristic model based on electrons in chiral initial states, which are selectively liberated by the laser field and, after tunneling, interact with an achiral Coulomb potential.
Bootstrapping Audio-Visual Segmentation by Strengthening Audio Cues

How to effectively interact audio with vision has garnered considerable interest within the multi-modality research field. Recently, a novel audio-visual segmentation (AVS) task has been proposed, aiming to segment the sounding objects in video frames under the guidance of audio cues. However, most existing AVS methods are hindered by a modality imbalance where the visual features tend to dominate those of the audio modality, due to a unidirectional and insufficient integration of audio cues. This imbalance skews the feature representation towards the visual aspect, impeding the learning of joint audio-visual representations and potentially causing segmentation inaccuracies. To address this issue, we propose AVSAC. Our approach features a Bidirectional Audio-Visual Decoder (BAVD) with integrated bidirectional bridges, enhancing audio cues and fostering continuous interplay between audio and visual modalities. This bidirectional interaction narrows the modality imbalance, facilitating more effective learning of integrated audio-visual representations. Additionally, we present a strategy for audio-visual frame-wise synchrony as fine-grained guidance of BAVD. This strategy enhances the share of auditory components in visual features, contributing to a more balanced audio-visual representation learning. Extensive experiments show that our method attains new benchmarks in AVS performance.
Good Teachers Explain: Explanation-Enhanced Knowledge Distillation

Knowledge Distillation (KD) has proven effective for compressing large teacher models into smaller student models. While it is well known that student models can achieve similar accuracies as the teachers, it has also been shown that they nonetheless often do not learn the same function. It is, however, often highly desirable that the student's and teacher's functions share similar properties such as basing the prediction on the same input features, as this ensures that students learn the 'right features' from the teachers. In this work, we explore whether this can be achieved by not only optimizing the classic KD loss but also the similarity of the explanations generated by the teacher and the student. Despite the idea being simple and intuitive, we find that our proposed 'explanation-enhanced' KD (e$^2$KD) (1) consistently provides large gains in terms of accuracy and student-teacher agreement, (2) ensures that the student learns from the teacher to be right for the right reasons and to give similar explanations, and (3) is robust with respect to the model architectures, the amount of training data, and even works with 'approximate', pre-computed explanations.
Deconstructing the spin susceptibility of a cuprate superconductor

A major obstacle to understanding high-Tc cuprates is that superconductivity precludes observing normal-state properties at low temperatures. One prime example is the normal-state spin susceptibility: although its decrease upon cooling far above Tc typifies pseudogap behavior, its behavior at low temperatures is generally unknown. Here, our measurements in high magnetic fields expose the spin susceptibility of YBa2Cu3Oy down to low temperatures. Even though superconductivity is suppressed by the field, we uncover two thermally-activated contributions alongside a residual susceptibility at T=0 due to gapless excitations. We relate these two distinct gaps to short-range charge-density waves and to the formation of spin singlets similar to those found in certain quantum spin systems. These phenomena thus collectively contribute to the pseudogap in the spin susceptibility at low temperature, supplementing short-lived antiferromagnetism known to initiate pseudogap behavior at high temperatures. We therefore propose that the pseudogap should be regarded as a composite property.
IGUANe: a 3D generalizable CycleGAN for multicenter harmonization of brain MR images

In MRI studies, the aggregation of imaging data from multiple acquisition sites enhances sample size but may introduce site-related variabilities that hinder consistency in subsequent analyses. Deep learning methods for image translation have emerged as a solution for harmonizing MR images across sites. In this study, we introduce IGUANe (Image Generation with Unified Adversarial Networks), an original 3D model that leverages the strengths of domain translation and straightforward application of style transfer methods for multicenter brain MR image harmonization. IGUANe extends CycleGAN architecture by integrating an arbitrary number of domains for training through a many-to-one strategy. During inference, the model can be applied to any image, even from an unknown acquisition site, making it a universal generator for harmonization. Trained on a dataset comprising T1-weighted images from 11 different scanners, IGUANe was evaluated on data from unseen sites. The assessments included the transformation of MR images with traveling subjects, the preservation of pairwise distances between MR images within domains, the evolution of volumetric patterns related to age and Alzheimer$^\prime$s disease (AD), and the performance in age regression and patient classification tasks. Comparisons with other harmonization and normalization methods suggest that IGUANe better preserves individual information in MR images and is more suitable for maintaining and reinforcing variabilities related to age and AD. Future studies may further assess IGUANe in other multicenter contexts, either using the same model or retraining it for applications to different image modalities.
Few-Shot Scenario Testing for Autonomous Vehicles Based on Neighborhood Coverage and Similarity

Testing and evaluating the safety performance of autonomous vehicles (AVs) is essential before the large-scale deployment. Practically, the acceptable cost of testing specific AV model can be restricted within an extremely small limit because of testing cost or time. With existing testing methods, the limitations imposed by strictly restricted testing numbers often result in significant uncertainties or challenges in quantifying testing results. In this paper, we formulate this problem for the first time the"few-shot testing"(FST) problem and propose a systematic FST framework to address this challenge. To alleviate the considerable uncertainty inherent in a small testing scenario set and optimize scenario utilization, we frame the FST problem as an optimization problem and search for a small scenario set based on neighborhood coverage and similarity. By leveraging the prior information on surrogate models (SMs), we dynamically adjust the testing scenario set and the contribution of each scenario to the testing result under the guidance of better generalization ability on AVs. With certain hypotheses on SMs, a theoretical upper bound of testing error is established to verify the sufficiency of testing accuracy within given limited number of tests. The experiments of the cut-in scenario using FST method demonstrate a notable reduction in testing error and variance compared to conventional testing methods, especially for situations with a strict limitation on the number of scenarios.
A Case Study on Filtering for End-to-End Speech Translation

It is relatively easy to mine a large parallel corpus for any machine learning task, such as speech-to-text or speech-to-speech translation. Although these mined corpora are large in volume, their quality is questionable. This work shows that the simplest filtering technique can trim down these big, noisy datasets to a more manageable, clean dataset. We also show that using this clean dataset can improve the model's performance, as in the case of the multilingual-to-English Speech Translation (ST) model, where, on average, we obtain a 4.65 BLEU score improvement.