Miracle-dz/UniFAR-FTUs huggingface dataset:
Supervised Knowledge Makes Large Language Models Better In-context Learners

Large Language Models (LLMs) exhibit emerging in-context learning abilities through prompt engineering. The recent progress in large-scale generative models has further expanded their use in real-world language applications. However, the critical challenge of improving the generalizability and factuality of LLMs in natural language understanding and question answering remains under-explored. While previous in-context learning research has focused on enhancing models to adhere to users' specific instructions and quality expectations, and to avoid undesired outputs, little to no work has explored the use of task-Specific fine-tuned Language Models (SLMs) to improve LLMs' in-context learning during the inference stage. Our primary contribution is the establishment of a simple yet effective framework that enhances the reliability of LLMs as it: 1) generalizes out-of-distribution data, 2) elucidates how LLMs benefit from discriminative models, and 3) minimizes hallucinations in generative tasks. Using our proposed plug-in method, enhanced versions of Llama 2 and ChatGPT surpass their original versions regarding generalizability and factuality. We offer a comprehensive suite of resources, including 16 curated datasets, prompts, model checkpoints, and LLM outputs across 9 distinct tasks. Our empirical analysis sheds light on the advantages of incorporating discriminative models into LLMs and highlights the potential of our methodology in fostering more reliable LLMs.
A graph-based multimodal framework to predict gentrification

Gentrification--the transformation of a low-income urban area caused by the influx of affluent residents--has many revitalizing benefits. However, it also poses extremely concerning challenges to low-income residents. To help policymakers take targeted and early action in protecting low-income residents, researchers have recently proposed several machine learning models to predict gentrification using socioeconomic and image features. Building upon previous studies, we propose a novel graph-based multimodal deep learning framework to predict gentrification based on urban networks of tracts and essential facilities (e.g., schools, hospitals, and subway stations). We train and test the proposed framework using data from Chicago, New York City, and Los Angeles. The model successfully predicts census-tract level gentrification with 0.9 precision on average. Moreover, the framework discovers a previously unexamined strong relationship between schools and gentrification, which provides a basis for further exploration of social factors affecting gentrification.
PERP: Rethinking the Prune-Retrain Paradigm in the Era of LLMs

Neural Networks can be efficiently compressed through pruning, significantly reducing storage and computational demands while maintaining predictive performance. Simple yet effective methods like Iterative Magnitude Pruning (IMP, Han et al., 2015) remove less important parameters and require a costly retraining procedure to recover performance after pruning. However, with the rise of Large Language Models (LLMs), full retraining has become infeasible due to memory and compute constraints. In this study, we challenge the practice of retraining all parameters by demonstrating that updating only a small subset of highly expressive parameters is often sufficient to recover or even improve performance compared to full retraining. Surprisingly, retraining as little as 0.27%-0.35% of the parameters of GPT-architectures achieves comparable performance to One Shot IMP across various sparsity levels. Our approach, Parameter-Efficient Retraining after Pruning (PERP), drastically reduces compute and memory demands, enabling pruning and retraining of up to 30 billion parameter models on a single NVIDIA A100 GPU within minutes. Despite magnitude pruning being considered as unsuited for pruning LLMs, our findings show that PERP positions it as a strong contender against state-of-the-art retraining-free approaches such as Wanda (Sun et al., 2023) and SparseGPT (Frantar&Alistarh, 2023), opening up a promising alternative to avoiding retraining.
Evolution of Tidal Disruption Event Disks with Magnetically Driven Winds

We present a time-dependent, one-dimensional magnetically driven disk-wind model based on magnetohydrodynamical (MHD) equations in the context of tidal disruption events (TDEs). We assume that the disk is geometrically thin and gas-pressure dominant and explicitly take into account magnetic braking as well as turbulent viscosity by an extended alpha viscosity prescription. We find a particular wind solution for a set of basic equations that satisfies the necessary and sufficient conditions for vertically unbound MHD flows. The solution demonstrates that the disk evolves with losing mass due to wind and accretion from the initial Gaussian density distribution. We confirm that the mass accretion rate follows the power law of time $t^{-19/16}$ at late times if the wind is absent, which corresponds to the classical solution of Cannizzo et al. (1990). We find that the mass accretion rate is steeper than the $t^{-19/16}$ curve if the disk wind is present. This is because the wind removes a significant fraction of the mass and angular momentum. Mass accretion is further induced by magnetic braking, known as a wind-driven accretion mechanism, resulting in more rapid decay with time in both the mass accretion and loss rates. The ultraviolet (UV) luminosity is the highest among the optical, UV, and X-ray luminosities from early to late evolutionary phases, suggesting optical and X-ray emissions from the disk are observationally insignificant due to magnetically driven winds in TDEs. Our model predicts that late-time bolometric light curves steeper than $t^{-19/16}$ in UV-bright TDEs potentially serve as compelling indicators for magnetically driven winds.
Decentralized Monte Carlo Tree Search for Partially Observable Multi-agent Pathfinding

The Multi-Agent Pathfinding (MAPF) problem involves finding a set of conflict-free paths for a group of agents confined to a graph. In typical MAPF scenarios, the graph and the agents' starting and ending vertices are known beforehand, allowing the use of centralized planning algorithms. However, in this study, we focus on the decentralized MAPF setting, where the agents may observe the other agents only locally and are restricted in communications with each other. Specifically, we investigate the lifelong variant of MAPF, where new goals are continually assigned to the agents upon completion of previous ones. Drawing inspiration from the successful AlphaZero approach, we propose a decentralized multi-agent Monte Carlo Tree Search (MCTS) method for MAPF tasks. Our approach utilizes the agent's observations to recreate the intrinsic Markov decision process, which is then used for planning with a tailored for multi-agent tasks version of neural MCTS. The experimental results show that our approach outperforms state-of-the-art learnable MAPF solvers. The source code is available at https://github.com/AIRI-Institute/mats-lp.
Enhanced Ferromagnetism in Monolayer Cr2Te3 via Topological Insulator Coupling

Exchange-coupled interfaces are pivotal in exploiting two-dimensional (2D) ferromagnetism. Due to the extraordinary correlations among charge, spin, orbital and lattice degrees of freedom, layered magnetic transition metal chalcogenides (TMCs) bode well for exotic topological phenomena. Here we report the realization of wafer-scale Cr2Te3 down to monolayer (ML) on insulating SrTiO3(111) substrates using molecular beam epitaxy. Robust ferromagnetism emerges in 2D Cr2Te3 ML with a Curie temperature TC = 17 K. Moreover, when Cr2Te3 is proximitized with topological insulator (TI) (Bi,Sb)2Te3, the magnetism becomes stronger -- for 1 ML, TC increases to 30 K, while for 2 ML it boosts from 65 K to 82 K. Our experiments and theory strongly indicate that the Bloembergen-Rowland interaction is likely a universal aspect of TC enhancement in TI-coupled magnetic heterostructures. The topological-surface-enhanced magnetism in 2D TMC enables further exchange coupling physics and quantum hybrid studies, including paving the way to realize interface-modulated topological electronics.
A Theory of Non-Acyclic Generative Flow Networks

GFlowNets is a novel flow-based method for learning a stochastic policy to generate objects via a sequence of actions and with probability proportional to a given positive reward. We contribute to relaxing hypotheses limiting the application range of GFlowNets, in particular: acyclicity (or lack thereof). To this end, we extend the theory of GFlowNets on measurable spaces which includes continuous state spaces without cycle restrictions, and provide a generalization of cycles in this generalized context. We show that losses used so far push flows to get stuck into cycles and we define a family of losses solving this issue. Experiments on graphs and continuous tasks validate those principles.
A Study in Zucker: Insights on Interactions Between Humans and Small Service Robots

Despite recent advancements in human-robot interaction (HRI), there is still limited knowledge about how humans interact and behave in the presence of small service indoor robots and, subsequently, about the human-centered behavior of such robots. This also raises concerns about the applicability of current trajectory prediction methods to indoor HRI settings as well as the accuracy of existing crowd simulation models in shared environments. To address these issues, we introduce a new HRI dataset focusing on interactions between humans and small differential drive robots running different types of controllers. Our analysis shows that anticipatory and non-anticipatory robot controllers impose similar constraints to humans' safety and efficiency. Additionally, we found that current state-of-the-art models for human trajectory prediction can adequately extend to indoor HRI settings. Finally, we show that humans respond differently to small differential drives than to other humans when collisions are imminent, since interacting with small robots can only cause a finite level of social discomfort as compared to human-human interactions.
Making Large Language Models A Better Foundation For Dense Retrieval

Dense retrieval needs to learn discriminative text embeddings to represent the semantic relationship between query and document. It may benefit from the using of large language models (LLMs), given LLMs' strong capability on semantic understanding. However, the LLMs are pre-trained by text generation tasks, whose working pattern is completely different from representing texts as embeddings. As a result, it is imperative to study how to adapt LLMs properly so that they can be effectively initialized as the backbone encoder for dense retrieval. In this paper, we propose a novel approach, called LLaRA (LLM adapted for dense RetrievAl), which works as a post-hoc adaptation of LLM for the dense retrieval application. LLaRA consists of two pretext tasks: EBAE (Embedding-Based Auto-Encoding) and EBAR (Embedding-Based Auto-Regression), where the text embeddings from LLM are used to reconstruct the tokens for the input sentence and predict the tokens for the next sentence, respectively. LLaRA turns out to be simple, lightweight, and highly effective. It is applied to adapt LLaMA-2-7B (base) on the Wikipedia corpus, where it substantially improves the model's fine-tuned performances on a variety of dense retrieval benchmarks, like MSMARCO and BEIR. Our model and code will be made publicly available at BGE repository.
Towards a Unified Multimodal Reasoning Framework

Recent advancements in deep learning have led to the development of powerful language models (LMs) that excel in various tasks. Despite these achievements, there is still room for improvement, particularly in enhancing reasoning abilities and incorporating multimodal data. This report investigates the potential impact of combining Chain-of-Thought (CoT) reasoning and Visual Question Answering (VQA) techniques to improve LM's accuracy in solving multiple-choice questions. By employing TextVQA and ScienceQA datasets, we assessed the effectiveness of three text embedding methods and three visual embedding approaches. Our experiments aimed to fill the gap in current research by investigating the combined impact of CoT and VQA, contributing to the understanding of how these techniques can improve the reasoning capabilities of state-of-the-art models like GPT-4. Results from our experiments demonstrated the potential of these approaches in enhancing LM's reasoning and question-answering capabilities, providing insights for further research and development in the field, and paving the way for more accurate and reliable AI systems that can handle complex reasoning tasks across multiple modalities.
Attention-aware Social Graph Transformer Networks for Stochastic Trajectory Prediction

Trajectory prediction is fundamental to various intelligent technologies, such as autonomous driving and robotics. The motion prediction of pedestrians and vehicles helps emergency braking, reduces collisions, and improves traffic safety. Current trajectory prediction research faces problems of complex social interactions, high dynamics and multi-modality. Especially, it still has limitations in long-time prediction. We propose Attention-aware Social Graph Transformer Networks for multi-modal trajectory prediction. We combine Graph Convolutional Networks and Transformer Networks by generating stable resolution pseudo-images from Spatio-temporal graphs through a designed stacking and interception method. Furthermore, we design the attention-aware module to handle social interaction information in scenarios involving mixed pedestrian-vehicle traffic. Thus, we maintain the advantages of the Graph and Transformer, i.e., the ability to aggregate information over an arbitrary number of neighbors and the ability to perform complex time-dependent data processing. We conduct experiments on datasets involving pedestrian, vehicle, and mixed trajectories, respectively. Our results demonstrate that our model minimizes displacement errors across various metrics and significantly reduces the likelihood of collisions. It is worth noting that our model effectively reduces the final displacement error, illustrating the ability of our model to predict for a long time.
Dynamic Syntax Mapping: A New Approach to Unsupervised Syntax Parsing

The intricate hierarchical structure of syntax is fundamental to the intricate and systematic nature of human language. This study investigates the premise that language models, specifically their attention distributions, can encapsulate syntactic dependencies. We introduce Dynamic Syntax Mapping (DSM), an innovative approach for the agnostic induction of these structures. Our method diverges from traditional syntax models which rely on predefined annotation schemata. Instead, we focus on a core characteristic inherent in dependency relations: syntactic substitutability. This concept refers to the interchangeability of words within the same syntactic category at either end of a dependency. By leveraging this property, we generate a collection of syntactically invariant sentences, which serve as the foundation for our parsing framework. Our findings reveal that the use of an increasing array of substitutions notably enhances parsing precision on natural language data. Specifically, in the context of long-distance subject-verb agreement, DSM exhibits a remarkable advancement over prior methodologies. Furthermore, DSM's adaptability is demonstrated through its successful application in varied parsing scenarios, underscoring its broad applicability.
Radiometric propulsion: Advancing with the order-of-magnitude enhancement through graphene aerogel-coated vanes

Radiometer is a light-induced aerodynamic propulsive device under the rarefied gas environment, which holds great potential for the next-gen near-space flight. However, its practical applications are hindered by the weak propulsion forces on the conventional radiometer vanes. Herein, this material-aerodynamics cross-disciplinary study develops novel radiometer vanes with graphene aerogel coatings, which for the first time realize an order of magnitude enhancement in radiometric propulsion. The improvement is manifested as up to 29.7 times faster rotation speed at a low pressure of 0.2 Pa, 13.8 times faster at the pressure (1.5 Pa) with maximum speeds, and 4 orders of magnitude broader operating pressure range (10E-4 - 10E2 Pa). Direct Simulation Monte Carlo calculations reveal that the outstanding performance is ascribed to the improved temperature gradient and gas-solid momentum transfer efficiency tailored by surface porous microstructures. Moreover, we demonstrate a stable and long-term levitation prototype under both 1 sun irradiation and a rarefied gas environment.
Federated Hyperdimensional Computing

Federated learning (FL) enables a loose set of participating clients to collaboratively learn a global model via coordination by a central server and with no need for data sharing. Existing FL approaches that rely on complex algorithms with massive models, such as deep neural networks (DNNs), suffer from computation and communication bottlenecks. In this paper, we first propose FedHDC, a federated learning framework based on hyperdimensional computing (HDC). FedHDC allows for fast and light-weight local training on clients, provides robust learning, and has smaller model communication overhead compared to learning with DNNs. However, current HDC algorithms get poor accuracy when classifying larger&more complex images, such as CIFAR10. To address this issue, we design FHDnn, which complements FedHDC with a self-supervised contrastive learning feature extractor. We avoid the transmission of the DNN and instead train only the HDC learner in a federated manner, which accelerates learning, reduces transmission cost, and utilizes the robustness of HDC to tackle network errors. We present a formal analysis of the algorithm and derive its convergence rate both theoretically, and show experimentally that FHDnn converges 3$\times$ faster vs. DNNs. The strategies we propose to improve the communication efficiency enable our design to reduce communication costs by 66$\times$ vs. DNNs, local client compute and energy consumption by ~1.5 - 6$\times$, while being highly robust to network errors. Finally, our proposed strategies for improving the communication efficiency have up to 32$\times$ lower communication costs with good accuracy.
Isotrivial Lagrangian fibrations of compact hyper-K\"ahler manifolds

This article initiates the study of isotrivial Lagrangian fibrations of compact hyper-K\"ahler manifolds. We present four foundational results that extend well-known facts about isotrivial elliptic fibrations of K3 surfaces. First, we prove that smooth fibers of an isotrivial Lagrangian fibration are isogenous to a power of an elliptic curve. Second, we exhibit a dichotomy between two types of isotrivial Lagrangian fibrations, which we call A and B. Third, we give a classification result for type A isotrivial Lagrangian fibrations assuming a well-known conjecture on the base of Lagrangian fibrations. Namely, if a type A isotrivial Lagrangian fibration admits a rational section, then it is birational to one of two straightforward examples of isotrivial fibrations of hyper-K\"ahler manifolds of $\text{K3}^{[n]}$-type and $\text{Kum}_n$-type. Finally, we prove that a genericity assumption on the smooth fiber of an isotrivial Lagrangian fibration ensures that the fibration is of type A.
SSFlowNet: Semi-supervised Scene Flow Estimation On Point Clouds With Pseudo Label

In the domain of supervised scene flow estimation, the process of manual labeling is both time-intensive and financially demanding. This paper introduces SSFlowNet, a semi-supervised approach for scene flow estimation, that utilizes a blend of labeled and unlabeled data, optimizing the balance between the cost of labeling and the precision of model training. SSFlowNet stands out through its innovative use of pseudo-labels, mainly reducing the dependency on extensively labeled datasets while maintaining high model accuracy. The core of our model is its emphasis on the intricate geometric structures of point clouds, both locally and globally, coupled with a novel spatial memory feature. This feature is adept at learning the geometric relationships between points over sequential time frames. By identifying similarities between labeled and unlabeled points, SSFlowNet dynamically constructs a correlation matrix to evaluate scene flow dependencies at individual point level. Furthermore, the integration of a flow consistency module within SSFlowNet enhances its capability to consistently estimate flow, an essential aspect for analyzing dynamic scenes. Empirical results demonstrate that SSFlowNet surpasses existing methods in pseudo-label generation and shows adaptability across varying data volumes. Moreover, our semi-supervised training technique yields promising outcomes even with different smaller ratio labeled data, marking a substantial advancement in the field of scene flow estimation.
HyperMix: Out-of-Distribution Detection and Classification in Few-Shot Settings

Out-of-distribution (OOD) detection is an important topic for real-world machine learning systems, but settings with limited in-distribution samples have been underexplored. Such few-shot OOD settings are challenging, as models have scarce opportunities to learn the data distribution before being tasked with identifying OOD samples. Indeed, we demonstrate that recent state-of-the-art OOD methods fail to outperform simple baselines in the few-shot setting. We thus propose a hypernetwork framework called HyperMix, using Mixup on the generated classifier parameters, as well as a natural out-of-episode outlier exposure technique that does not require an additional outlier dataset. We conduct experiments on CIFAR-FS and MiniImageNet, significantly outperforming other OOD methods in the few-shot regime.
IMEX-RK methods for Landau-Lifshitz equation with arbitrary damping

Magnetization dynamics in ferromagnetic materials is modeled by the Landau-Lifshitz (LL) equation, a nonlinear system of partial differential equations. Among the numerical approaches, semi-implicit schemes are widely used in the micromagnetics simulation, due to a nice compromise between accuracy and efficiency. At each time step, only a linear system needs to be solved and a projection is then applied to preserve the length of magnetization. However, this linear system contains variable coefficients and a non-symmetric structure, and thus an efficient linear solver is highly desired. If the damping parameter becomes large, it has been realized that efficient solvers are only available to a linear system with constant, symmetric, and positive definite (SPD) structure. In this work, based on the implicit-explicit Runge-Kutta (IMEX-RK) time discretization, we introduce an artificial damping term, which is treated implicitly. The remaining terms are treated explicitly. This strategy leads to a semi-implicit scheme with the following properties: (1) only a few linear system with constant and SPD structure needs to be solved at each time step; (2) it works for the LL equation with arbitrary damping parameter; (3) high-order accuracy can be obtained with high-order IMEX-RK time discretization. Numerically, second-order and third-order IMEX-RK methods are designed in both the 1-D and 3-D domains. A comparison with the backward differentiation formula scheme is undertaken, in terms of accuracy and efficiency. The robustness of both numerical methods is tested on the first benchmark problem from National Institute of Standards and Technology. The linearized stability estimate and optimal rate convergence analysis are provided for an alternate IMEX-RK2 numerical scheme as well.