Miracle-dz/UniFAR-FTUs huggingface dataset:
m&m's: A Benchmark to Evaluate Tool-Use for multi-step multi-modal Tasks

Real-world multi-modal problems are rarely solved by a single machine learning model, and often require multi-step computational plans that involve stitching several models. Tool-augmented LLMs hold tremendous promise for automating the generation of such computational plans. However, the lack of standardized benchmarks for evaluating LLMs as planners for multi-step multi-modal tasks has prevented a systematic study of planner design decisions. Should LLMs generate a full plan in a single shot or step-by-step? Should they invoke tools directly with Python code or through structured data formats like JSON? Does feedback improve planning? To answer these questions and more, we introduce m&m's: a benchmark containing 4K+ multi-step multi-modal tasks involving 33 tools that include multi-modal models, (free) public APIs, and image processing modules. For each of these task queries, we provide automatically generated plans using this realistic toolset. We further provide a high-quality subset of 1,565 task plans that are human-verified and correctly executable. With m&m's, we evaluate 6 popular LLMs with 2 planning strategies (multi-step vs. step-by-step planning), 2 plan formats (JSON vs. code), and 3 types of feedback (parsing/verification/execution). Finally, we summarize takeaways from our extensive experiments. Our dataset and code are available on HuggingFace (https://huggingface.co/datasets/zixianma/mnms) and Github (https://github.com/RAIVNLab/mnms).
Wait to be Faster: a Smart Pooling Framework for Dynamic Ridesharing

Ridesharing services, such as Uber or Didi, have attracted considerable attention in recent years due to their positive impact on environmental protection and the economy. Existing studies require quick responses to orders, which lack the flexibility to accommodate longer wait times for better grouping opportunities. In this paper, we address a NP-hard ridesharing problem, called Minimal Extra Time RideSharing (METRS), which balances waiting time and group quality (i.e., detour time) to improve riders' satisfaction. To tackle this problem, we propose a novel approach called WATTER (WAit To be fasTER), which leverages an order pooling management algorithm allowing orders to wait until they can be matched with suitable groups. The key challenge is to customize the extra time threshold for each order by reducing the original optimization objective into a convex function of threshold, thus offering a theoretical guarantee to be optimized efficiently. We model the dispatch process using a Markov Decision Process (MDP) with a carefully designed value function to learn the threshold. Through extensive experiments on three real datasets, we demonstrate the efficiency and effectiveness of our proposed approaches.
Why are Gasoline Prices Sticky? A Test of Alternative Models of Price Adjustment

Macroeconomic models of business cycles rely on the assumption that firms adjust prices infrequently to generate the short-run non-neutrality of money documented by the monetary transmission literature. They posit different mechanisms to generate price stickiness, with correspondingly different implications for inflation dynamics. Using an autoregressive conditional binomial model, we test which mechanism is most consistent with the pattern of price adjustment found in daily wholesale gasoline price data. Our results lead us to reject menu costs and information-processing delays but suggest that strategic considerations related to the idea of 'fair pricing' play an important role in accounting for price stickiness. Copyright © 2009 John Wiley & Sons, Ltd.
Prior-dependent analysis of posterior sampling reinforcement learning with function approximation

This work advances randomized exploration in reinforcement learning (RL) with function approximation modeled by linear mixture MDPs. We establish the first prior-dependent Bayesian regret bound for RL with function approximation; and refine the Bayesian regret analysis for posterior sampling reinforcement learning (PSRL), presenting an upper bound of ${\mathcal{O}}(d\sqrt{H^3 T \log T})$, where $d$ represents the dimensionality of the transition kernel, $H$ the planning horizon, and $T$ the total number of interactions. This signifies a methodological enhancement by optimizing the $\mathcal{O}(\sqrt{\log T})$ factor over the previous benchmark (Osband and Van Roy, 2014) specified to linear mixture MDPs. Our approach, leveraging a value-targeted model learning perspective, introduces a decoupling argument and a variance reduction technique, moving beyond traditional analyses reliant on confidence sets and concentration inequalities to formalize Bayesian regret bounds more effectively.
Rules still work for Open Information Extraction

Open information extraction (OIE) aims to extract surface relations and their corresponding arguments from natural language text, irrespective of domain. This paper presents an innovative OIE model, APRCOIE, tailored for Chinese text. Diverging from previous models, our model generates extraction patterns autonomously. The model defines a new pattern form for Chinese OIE and proposes an automated pattern generation methodology. In that way, the model can handle a wide array of complex and diverse Chinese grammatical phenomena. We design a preliminary filter based on tensor computing to conduct the extraction procedure efficiently. To train the model, we manually annotated a large-scale Chinese OIE dataset. In the comparative evaluation, we demonstrate that APRCOIE outperforms state-of-the-art Chinese OIE models and significantly expands the boundaries of achievable OIE performance. The code of APRCOIE and the annotated dataset are released on GitHub (https://github.com/jialin666/APRCOIE_v1)
Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoning

This paper introduces Scene-LLM, a 3D-visual-language model that enhances embodied agents' abilities in interactive 3D indoor environments by integrating the reasoning strengths of Large Language Models (LLMs). Scene-LLM adopts a hybrid 3D visual feature representation, that incorporates dense spatial information and supports scene state updates. The model employs a projection layer to efficiently project these features in the pre-trained textual embedding space, enabling effective interpretation of 3D visual information. Unique to our approach is the integration of both scene-level and ego-centric 3D information. This combination is pivotal for interactive planning, where scene-level data supports global planning and ego-centric data is important for localization. Notably, we use ego-centric 3D frame features for feature alignment, an efficient technique that enhances the model's ability to align features of small objects within the scene. Our experiments with Scene-LLM demonstrate its strong capabilities in dense captioning, question answering, and interactive planning. We believe Scene-LLM advances the field of 3D visual understanding and reasoning, offering new possibilities for sophisticated agent interactions in indoor settings.
A Fully Dynamical Description of Time-Resolved Resonant Inelastic X-ray Scattering of Pyrazine

Recent advancements in ultrashort and intense X-ray sources have enabled the utilisation of resonant inelastic X-ray scattering (RIXS) as a probing technique for monitoring photoinduced dynamics in molecular systems. To account for dynamic phenomena like non-adiabatic transitions across the entire electronic state manifold, a time-dependent framework is crucial. Here, we introduce a fully time-dependent approach for calculating transient RIXS spectra using wavepacket dynamics simulations, alongside an explicit treatment of the X-ray probe pulse that surpasses Kramers-Heisenberg-Dirac constraints. Our analysis of pyrazine at the nitrogen K-edge underscores the importance of considering motion effects in all electronic states involved in the transient RIXS process. As a result, we propose a numerically exact approach to computationally support and predict cutting-edge time-resolved RIXS experiments.
A New Hybrid Viscoelastic Soft Tissue Model based on Meshless Method for Haptic Surgical Simulation

This paper proposes a hybrid soft tissue model that consists of a multilayer structure and many spheres for surgical simulation system based on meshless. To improve accuracy of the model, tension is added to the three-parameter viscoelastic structure that connects the two spheres. By using haptic device, the three-parameter viscoelastic model (TPM) produces accurate deformationand also has better stress-strain, stress relaxation and creep properties. Stress relaxation and creep formulas have been obtained by mathematical formula derivation. Comparing with the experimental results of the real pig liver which were reported by Evren et al. and Amy et al., the curve lines of stress-strain, stress relaxation and creep of TPM are close to the experimental data of the real liver. Simulated results show that TPM has better real-time, stability and accuracy.
Reducing State of OpenFlow Switches in Mobile Core Networks by Flow Rule Aggregation

While bringing many advantages, Software-Defined Networking (SDN) is accompanied by potential scalability issues that should be considered in the design of SDN-based networks. Specifically, SDN hardware switches based on Ternary Content-Addressable Memory (TCAM) can only store a few thousands of rules, imposing thus severe limits on the number of flows they can serve/process. Current proposals to deal with this problem mainly focus on optimal placement of flows that complies with the given constraints on TCAM size. We argue in this paper that flow routing not only should be but also can be independent of TCAM size constraints. We introduce two flow rule aggregation algorithms: One performs the "per-outport'' aggregation of the paths between access nodes in the network. It is optimal in that it holds the flow table sizes at the minimum, but has a drawback that it produces long identifiers of the path endpoints (access nodes), which cannot fit the IP address size. The other algorithm is its approximation under the constraint that the generated identifiers fit the limit imposed by the addressing scheme (e.g., IPv4). We study the performance of our algorithms analytically and through a set of experiments. While the optimal solution always keeps the flow table sizes at the minimum, we show that the approximate algorithm reduces the flow table sizes by a factor of 2 to 10 compared to the state of the art solution, under a reasonable constraint on the address length (e.g., 32 bits in case of IPv4).
Environmental perturbations influence telomere dynamics in long-lived birds in their natural habitat

Telomeres are regarded as markers of biological or cellular ageing because they shorten with the degree of stress exposure. Accordingly, telomere lengths should show different rates of change when animals are faced with different intensities of environmental challenges. However, a relationship between telomere length and the environment has not yet been tested within a natural setting. Here, we report longitudinal telomere dynamics in free-living, black-tailed gulls (Larus crassirostris) through the recapture of birds of a known age over 2–5 consecutive years. The rate of change in telomere lengths differed with respect to year but not sex or age. The years when gulls showed stable telomere lengths or increases in telomere lengths (from 2009 to 2010) and decreases in telomere lengths (from 2010 to 2011) were characterized by El Niño and the Great Japan Earthquake, respectively. Both events are suspected to have had long-lasting effects on food availability and/or weather conditions. Thus, our findings that telomere dynamics in long-lived birds are influenced by dramatic changes in environmental conditions highlight the importance of environmental fluctuations in affecting stress and lifespan.
Domain-Guided Masked Autoencoders for Unique Player Identification

Unique player identification is a fundamental module in vision-driven sports analytics. Identifying players from broadcast videos can aid with various downstream tasks such as player assessment, in-game analysis, and broadcast production. However, automatic detection of jersey numbers using deep features is challenging primarily due to: a) motion blur, b) low resolution video feed, and c) occlusions. With their recent success in various vision tasks, masked autoencoders (MAEs) have emerged as a superior alternative to conventional feature extractors. However, most MAEs simply zero-out image patches either randomly or focus on where to mask rather than how to mask. Motivated by human vision, we devise a novel domain-guided masking policy for MAEs termed d-MAE to facilitate robust feature extraction in the presence of motion blur for player identification. We further introduce a new spatio-temporal network leveraging our novel d-MAE for unique player identification. We conduct experiments on three large-scale sports datasets, including a curated baseball dataset, the SoccerNet dataset, and an in-house ice hockey dataset. We preprocess the datasets using an upgraded keyframe identification (KfID) module by focusing on frames containing jersey numbers. Additionally, we propose a keyframe-fusion technique to augment keyframes, preserving spatial and temporal context. Our spatio-temporal network showcases significant improvements, surpassing the current state-of-the-art by 8.58%, 4.29%, and 1.20% in the test set accuracies, respectively. Rigorous ablations highlight the effectiveness of our domain-guided masking approach and the refined KfID module, resulting in performance enhancements of 1.48% and 1.84% respectively, compared to original architectures.
Semi-Supervised Learning for Anomaly Traffic Detection via Bidirectional Normalizing Flows

With the rapid development of the Internet, various types of anomaly traffic are threatening network security. We consider the problem of anomaly network traffic detection and propose a three-stage anomaly detection framework using only normal traffic. Our framework can generate pseudo anomaly samples without prior knowledge of anomalies to achieve the detection of anomaly data. Firstly, we employ a reconstruction method to learn the deep representation of normal samples. Secondly, these representations are normalized to a standard normal distribution using a bidirectional flow module. To simulate anomaly samples, we add noises to the normalized representations which are then passed through the generation direction of the bidirectional flow module. Finally, a simple classifier is trained to differentiate the normal samples and pseudo anomaly samples in the latent space. During inference, our framework requires only two modules to detect anomalous samples, leading to a considerable reduction in model size. According to the experiments, our method achieves the state of-the-art results on the common benchmarking datasets of anomaly network traffic detection. The code is given in the https://github.com/ZxuanDang/ATD-via-Flows.git
Quantum nonlinear optics on the edge of small lattice fractional quantum Hall fluids

We study the quantum dynamics of the edge modes of lattice fractional quantum Hall liquids in response to time-dependent external potentials. We show that the nonlinear chiral Luttinger liquid theory provides a quantitatively accurate description even for the small lattice geometries away from the continuum limit that are available in state-of-the-art experiments. Experimentally accessible signatures of the quantized value of the bulk transverse Hall conductivity are identified both in the linear and the non-linear response to an external excitation. The strong nonlinearity induced by the open boundaries is responsible for sizable quantum blockade effects, leading to the generation of nonclassical states of the edge modes.
Improving Dialogue Agents by Decomposing One Global Explicit Annotation with Local Implicit Multimodal Feedback

We describe an approach for aligning an LLM-based dialogue agent based on global (i.e., dialogue-level) rewards, while also taking into account naturally-occurring multimodal signals. At a high level, our approach (dubbed GELI) learns a local, turn-level reward model by decomposing the human-provided Global Explicit (GE) session-level reward, using Local Implicit (LI} multimodal reward signals to crossmodally shape the reward decomposition step. This decomposed reward model is then used as part of the standard RHLF pipeline improve an LLM-based dialog agent. We run quantitative and qualitative human studies to evaluate the performance of our GELI approach, and find that it shows consistent improvements across various conversational metrics compared to baseline methods.
Exclusive interplay between topological quasiparticles and strongly correlated fermions

The low-energy excitations of topological quantum matter have been increasingly comprehended. However, the impact of these excitations on strongly correlated fermions remains rarely explored. Here, we report the discovery of an incompatible dynamic of 4$f$ quasiparticle and topological quasiparticle. The former forms through screening processes, while the latter suppresses the screening channel. By employing ab initio many-body perturbation theory combined with dynamical mean field theory, we show that this effect prompts two types of topological semimetals. The type-I semimetal PrPtBi exhibits a topological quasiparticle away from the Fermi level due to the formation of a 4$f$ quasiparticle. The type-II semimetals HoPtBi and PrAlGe feature a robust topological quasiparticle at the Fermi level, which pushes the 4$f$ quasiparticle away. Our work provides an avenue to harness semimetals' anomalous quantum effects from topological quasiparticle to heavy-fermion behavior by the incorporation of 4$f$ quasiparticles.
VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding

We explore how reconciling several foundation models (large language models and vision-language models) with a novel unified memory mechanism could tackle the challenging video understanding problem, especially capturing the long-term temporal relations in lengthy videos. In particular, the proposed multimodal agent VideoAgent: 1) constructs a structured memory to store both the generic temporal event descriptions and object-centric tracking states of the video; 2) given an input task query, it employs tools including video segment localization and object memory querying along with other visual foundation models to interactively solve the task, utilizing the zero-shot tool-use ability of LLMs. VideoAgent demonstrates impressive performances on several long-horizon video understanding benchmarks, an average increase of 6.6% on NExT-QA and 26.0% on EgoSchema over baselines, closing the gap between open-sourced models and private counterparts including Gemini 1.5 Pro.
Hessian-Free Laplace in Bayesian Deep Learning

The Laplace approximation (LA) of the Bayesian posterior is a Gaussian distribution centered at the maximum a posteriori estimate. Its appeal in Bayesian deep learning stems from the ability to quantify uncertainty post-hoc (i.e., after standard network parameter optimization), the ease of sampling from the approximate posterior, and the analytic form of model evidence. However, an important computational bottleneck of LA is the necessary step of calculating and inverting the Hessian matrix of the log posterior. The Hessian may be approximated in a variety of ways, with quality varying with a number of factors including the network, dataset, and inference task. In this paper, we propose an alternative framework that sidesteps Hessian calculation and inversion. The Hessian-free Laplace (HFL) approximation uses curvature of both the log posterior and network prediction to estimate its variance. Only two point estimates are needed: the standard maximum a posteriori parameter and the optimal parameter under a loss regularized by the network prediction. We show that, under standard assumptions of LA in Bayesian deep learning, HFL targets the same variance as LA, and can be efficiently amortized in a pre-trained network. Experiments demonstrate comparable performance to that of exact and approximate Hessians, with excellent coverage for in-between uncertainty.
On Analyzing User Ratings and Directional Trusts in Epinions.com

Recently, product review sites that contain user ratings and trust statements such as Epinions.com have become popular, and many attempts have been made to analyze user behavior on those sites. In this paper, we first define three different user groups for each user, i.e. Trustee, Trust or, and Irrelevance which describe different kinds of trust statements. Then we investigate the relationship between each user and users in his/her different user groups by comparing how their ratings are similar on Epinions data sets. Most existing works simply ignored Trust or group when considering a certain user. However, our observations suggest that Trust or group still meaningful that should not be simply disregarded.
Robust scheduling for flexible processing networks

Abstract Modern processing networks often consist of heterogeneous servers with widely varying capabilities, and process job flows with complex structure and requirements. A major challenge in designing efficient scheduling policies in these networks is the lack of reliable estimates of system parameters, and an attractive approach for addressing this challenge is to design robust policies, i.e. policies that do not use system parameters such as arrival and/or service rates for making scheduling decisions. In this paper we propose a general framework for the design of robust policies. The main technical novelty is the use of a stochastic gradient projection method that reacts to queue-length changes in order to find a balanced allocation of service resources to incoming tasks. We illustrate our approach on two broad classes of processing systems, namely the flexible fork-join networks and the flexible queueing networks, and prove the rate stability of our proposed policies for these networks under nonrestrictive assumptions.
Entity Alignment with Unlabeled Dangling Cases

We investigate the entity alignment problem with unlabeled dangling cases, meaning that there are entities in the source or target graph having no counterparts in the other, and those entities remain unlabeled. The problem arises when the source and target graphs are of different scales, and it is much cheaper to label the matchable pairs than the dangling entities. To solve the issue, we propose a novel GNN-based dangling detection and entity alignment framework. While the two tasks share the same GNN and are trained together, the detected dangling entities are removed in the alignment. Our framework is featured by a designed entity and relation attention mechanism for selective neighborhood aggregation in representation learning, as well as a positive-unlabeled learning loss for an unbiased estimation of dangling entities. Experimental results have shown that each component of our design contributes to the overall alignment performance which is comparable or superior to baselines, even if the baselines additionally have 30\% of the dangling entities labeled as training data.