Orionnnn/hub-stats huggingface dataset:
Vector-ICL: In-context Learning with Continuous Vector Representations
Authors: name: Yufan Zhuang, name: Chandan Singh, name: Liyuan Liu, name: Jingbo Shang, name: Jianfeng Gao
Published: 2024-10-08 02:25:38

Large language models (LLMs) have shown remarkable in-context learning (ICL)
capabilities on textual data. We explore whether these capabilities can be
extended to continuous vectors from diverse domains, obtained from black-box
pretrained encoders. By aligning input data with an LLM's embedding space
through lightweight projectors, we observe that LLMs can effectively process
and learn from these projected vectors, which we term Vector-ICL. In
particular, we find that pretraining projectors with general language modeling
objectives enables Vector-ICL, while task-specific finetuning further enhances
performance. In our experiments across various tasks and modalities, including
text reconstruction, numerical function regression, text classification,
summarization, molecule captioning, time-series classification, graph
classification, and fMRI decoding, Vector-ICL often surpasses both few-shot ICL
and domain-specific model or tuning. We further conduct analyses and case
studies, indicating the potential of LLMs to process vector representations
beyond traditional token-based paradigms.
Zebra: In-Context and Generative Pretraining for Solving Parametric PDEs
Authors: name: Louis Serrano, name: Armand Kassaï Koupaï, name: Thomas X Wang, name: Pierre Erbacher, name: Patrick Gallinari
Published: 2024-10-04 13:52:02

Solving time-dependent parametric partial differential equations (PDEs) is
challenging, as models must adapt to variations in parameters such as
coefficients, forcing terms, and boundary conditions. Data-driven neural
solvers either train on data sampled from the PDE parameters distribution in
the hope that the model generalizes to new instances or rely on gradient-based
adaptation and meta-learning to implicitly encode the dynamics from
observations. This often comes with increased inference complexity. Inspired by
the in-context learning capabilities of large language models (LLMs), we
introduce Zebra, a novel generative auto-regressive transformer designed to
solve parametric PDEs without requiring gradient adaptation at inference. By
leveraging in-context information during both pre-training and inference, Zebra
dynamically adapts to new tasks by conditioning on input sequences that
incorporate context trajectories or preceding states. This approach enables
Zebra to flexibly handle arbitrarily sized context inputs and supports
uncertainty quantification through the sampling of multiple solution
trajectories. We evaluate Zebra across a variety of challenging PDE scenarios,
demonstrating its adaptability, robustness, and superior performance compared
to existing approaches.
DART: Denoising Autoregressive Transformer for Scalable Text-to-Image
  Generation
Authors: name: Jiatao Gu, name: Yuyang Wang, name: Yizhe Zhang, name: Qihang Zhang, name: Dinghuai Zhang, name: Navdeep Jaitly, name: Josh Susskind, name: Shuangfei Zhai
Published: 2024-10-10 17:41:54

Diffusion models have become the dominant approach for visual generation.
They are trained by denoising a Markovian process that gradually adds noise to
the input. We argue that the Markovian property limits the models ability to
fully utilize the generation trajectory, leading to inefficiencies during
training and inference. In this paper, we propose DART, a transformer-based
model that unifies autoregressive (AR) and diffusion within a non-Markovian
framework. DART iteratively denoises image patches spatially and spectrally
using an AR model with the same architecture as standard language models. DART
does not rely on image quantization, enabling more effective image modeling
while maintaining flexibility. Furthermore, DART seamlessly trains with both
text and image data in a unified model. Our approach demonstrates competitive
performance on class-conditioned and text-to-image generation tasks, offering a
scalable, efficient alternative to traditional diffusion models. Through this
unified framework, DART sets a new benchmark for scalable, high-quality image
synthesis.
Everything Everywhere All at Once: LLMs can In-Context Learn Multiple
  Tasks in Superposition
Authors: name: Zheyang Xiong, name: Ziyang Cai, name: John Cooper, name: Albert Ge, name: Vasilis Papageorgiou, name: Zack Sifakis, name: Angeliki Giannou, name: Ziqian Lin, name: Liu Yang, name: Saurabh Agarwal, name: Grigorios G Chrysos, name: Samet Oymak, name: Kangwook Lee, name: Dimitris Papailiopoulos
Published: 2024-10-08 01:28:57

Large Language Models (LLMs) have demonstrated remarkable in-context learning
(ICL) capabilities. In this study, we explore a surprising phenomenon related
to ICL: LLMs can perform multiple, computationally distinct ICL tasks
simultaneously, during a single inference call, a capability we term "task
superposition". We provide empirical evidence of this phenomenon across various
LLM families and scales and show that this phenomenon emerges even if we train
the model to in-context learn one task at a time. We offer theoretical
explanations that this capability is well within the expressive power of
transformers. We also explore how LLMs internally compose task vectors during
superposition. Furthermore, we show that larger models can solve more ICL tasks
in parallel, and better calibrate their output distribution. Our findings offer
insights into the latent capabilities of LLMs, further substantiate the
perspective of "LLMs as superposition of simulators", and raise questions about
the mechanisms enabling simultaneous task execution.
Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Rates
Authors: name: Xiaosen Zheng, name: Tianyu Pang, name: Chao Du, name: Qian Liu, name: Jing Jiang, name: Min Lin
Published: 2024-10-09 17:53:06

Automatic LLM benchmarks, such as AlpacaEval 2.0, Arena-Hard-Auto, and
MT-Bench, have become popular for evaluating language models due to their
cost-effectiveness and scalability compared to human evaluation. Achieving high
win rates on these benchmarks can significantly boost the promotional impact of
newly released language models. This promotional benefit may motivate tricks,
such as manipulating model output length or style to game win rates, even
though several mechanisms have been developed to control length and disentangle
style to reduce gameability. Nonetheless, we show that even a "null model" that
always outputs a constant response (irrelevant to input instructions) can cheat
automatic benchmarks and achieve top-ranked win rates: an 86.5% LC win rate on
AlpacaEval 2.0; an 83.0 score on Arena-Hard-Auto; and a 9.55 score on MT-Bench.
Moreover, the crafted cheating outputs are transferable because we assume that
the instructions of these benchmarks (e.g., 805 samples of AlpacaEval 2.0) are
private and cannot be accessed. While our experiments are primarily
proof-of-concept, an adversary could use LLMs to generate more imperceptible
cheating responses, unethically benefiting from high win rates and promotional
impact. Our findings call for the development of anti-cheating mechanisms for
reliable automatic benchmarks. The code is available at
https://github.com/sail-sg/Cheating-LLM-Benchmarks.
LPZero: Language Model Zero-cost Proxy Search from Zero
Authors: name: Peijie Dong, name: Lujun Li, name: Xiang Liu, name: Zhenheng Tang, name: Xuebo Liu, name: Qiang Wang, name: Xiaowen Chu
Published: 2024-10-07 07:41:48

In spite of the outstanding performance, Neural Architecture Search (NAS) is
criticized for massive computation. Recently, Zero-shot NAS has emerged as a
promising approach by exploiting Zero-cost (ZC) proxies, which markedly reduce
computational demands. Despite this, existing ZC proxies heavily rely on expert
knowledge and incur significant trial-and-error costs. Particularly in NLP
tasks, most existing ZC proxies fail to surpass the performance of the naive
baseline. To address these challenges, we introduce a novel framework,
LPZero, which is the first to automatically design ZC proxies for
various tasks, achieving higher ranking consistency than human-designed
proxies. Specifically, we model the ZC proxy as a symbolic equation and
incorporate a unified proxy search space that encompasses existing ZC proxies,
which are composed of a predefined set of mathematical symbols. To
heuristically search for the best ZC proxy, LPZero incorporates genetic
programming to find the optimal symbolic composition. We propose a
Rule-based Pruning Strategy (RPS), which preemptively eliminates
unpromising proxies, thereby mitigating the risk of proxy degradation.
Extensive experiments on FlexiBERT, GPT-2, and LLaMA-7B demonstrate LPZero's
superior ranking ability and performance on downstream tasks compared to
current approaches.
GLOV: Guided Large Language Models as Implicit Optimizers for Vision
  Language Models
Authors: name: M. Jehanzeb Mirza, name: Mengjie Zhao, name: Zhuoyuan Mao, name: Sivan Doveh, name: Wei Lin, name: Paul Gavrikov, name: Michael Dorkenwald, name: Shiqi Yang, name: Saurav Jha, name: Hiromi Wakaki, name: Yuki Mitsufuji, name: Horst Possegger, name: Rogerio Feris, name: Leonid Karlinsky, name: James Glass
Published: 2024-10-08 15:55:40

In this work, we propose a novel method (GLOV) enabling Large Language Models
(LLMs) to act as implicit Optimizers for Vision-Langugage Models (VLMs) to
enhance downstream vision tasks. Our GLOV meta-prompts an LLM with the
downstream task description, querying it for suitable VLM prompts (e.g., for
zero-shot classification with CLIP). These prompts are ranked according to a
purity measure obtained through a fitness function. In each respective
optimization step, the ranked prompts are fed as in-context examples (with
their accuracies) to equip the LLM with the knowledge of the type of text
prompts preferred by the downstream VLM. Furthermore, we also explicitly steer
the LLM generation process in each optimization step by specifically adding an
offset difference vector of the embeddings from the positive and negative
solutions found by the LLM, in previous optimization steps, to the intermediate
layer of the network for the next generation step. This offset vector steers
the LLM generation toward the type of language preferred by the downstream VLM,
resulting in enhanced performance on the downstream vision tasks. We
comprehensively evaluate our GLOV on 16 diverse datasets using two families of
VLMs, i.e., dual-encoder (e.g., CLIP) and encoder-decoder (e.g., LLaVa) models
-- showing that the discovered solutions can enhance the recognition
performance by up to 15.0% and 57.5% (3.8% and 21.6% on average) for these
models.
Benchmarking Agentic Workflow Generation
Authors: name: Shuofei Qiao, name: Runnan Fang, name: Zhisong Qiu, name: Xiaobin Wang, name: Ningyu Zhang, name: Yong Jiang, name: Pengjun Xie, name: Fei Huang, name: Huajun Chen
Published: 2024-10-10 12:41:19

Large Language Models (LLMs), with their exceptional ability to handle a wide
range of tasks, have driven significant advancements in tackling reasoning and
planning tasks, wherein decomposing complex problems into executable workflows
is a crucial step in this process. Existing workflow evaluation frameworks
either focus solely on holistic performance or suffer from limitations such as
restricted scenario coverage, simplistic workflow structures, and lax
evaluation standards. To this end, we introduce WorFBench, a unified workflow
generation benchmark with multi-faceted scenarios and intricate graph workflow
structures. Additionally, we present WorFEval, a systemic evaluation protocol
utilizing subsequence and subgraph matching algorithms to accurately quantify
the LLM agent's workflow generation capabilities. Through comprehensive
evaluations across different types of LLMs, we discover distinct gaps between
the sequence planning capabilities and graph planning capabilities of LLM
agents, with even GPT-4 exhibiting a gap of around 15%. We also train two
open-source models and evaluate their generalization abilities on held-out
tasks. Furthermore, we observe that the generated workflows can enhance
downstream tasks, enabling them to achieve superior performance with less time
during inference. Code and dataset will be available at
https://github.com/zjunlp/WorFBench.
MotionGS: Exploring Explicit Motion Guidance for Deformable 3D Gaussian
  Splatting
Authors: name: Ruijie Zhu, name: Yanzhe Liang, name: Hanzhi Chang, name: Jiacheng Deng, name: Jiahao Lu, name: Wenfei Yang, name: Tianzhu Zhang, name: Yongdong Zhang
Published: 2024-10-10 08:19:47

Dynamic scene reconstruction is a long-term challenge in the field of 3D
vision. Recently, the emergence of 3D Gaussian Splatting has provided new
insights into this problem. Although subsequent efforts rapidly extend static
3D Gaussian to dynamic scenes, they often lack explicit constraints on object
motion, leading to optimization difficulties and performance degradation. To
address the above issues, we propose a novel deformable 3D Gaussian splatting
framework called MotionGS, which explores explicit motion priors to guide the
deformation of 3D Gaussians. Specifically, we first introduce an optical flow
decoupling module that decouples optical flow into camera flow and motion flow,
corresponding to camera movement and object motion respectively. Then the
motion flow can effectively constrain the deformation of 3D Gaussians, thus
simulating the motion of dynamic objects. Additionally, a camera pose
refinement module is proposed to alternately optimize 3D Gaussians and camera
poses, mitigating the impact of inaccurate camera poses. Extensive experiments
in the monocular dynamic scenes validate that MotionGS surpasses
state-of-the-art methods and exhibits significant superiority in both
qualitative and quantitative results. Project page:
https://ruijiezhu94.github.io/MotionGS_page
MathCoder2: Better Math Reasoning from Continued Pretraining on
  Model-translated Mathematical Code
Authors: name: Zimu Lu, name: Aojun Zhou, name: Ke Wang, name: Houxing Ren, name: Weikang Shi, name: Junting Pan, name: Mingjie Zhan, name: Hongsheng Li
Published: 2024-10-10 17:58:40

Code has been shown to be effective in enhancing the mathematical reasoning
abilities of large language models due to its precision and accuracy. Previous
works involving continued mathematical pretraining often include code that
utilizes math-related packages, which are primarily designed for fields such as
engineering, machine learning, signal processing, or module testing, rather
than being directly focused on mathematical reasoning. In this paper, we
introduce a novel method for generating mathematical code accompanied with
corresponding reasoning steps for continued pretraining. Our approach begins
with the construction of a high-quality mathematical continued pretraining
dataset by incorporating math-related web data, code using mathematical
packages, math textbooks, and synthetic data. Next, we construct reasoning
steps by extracting LaTeX expressions, the conditions needed for the
expressions, and the results of the expressions from the previously collected
dataset. Based on this extracted information, we generate corresponding code to
accurately capture the mathematical reasoning process. Appending the generated
code to each reasoning step results in data consisting of paired natural
language reasoning steps and their corresponding code. Combining this data with
the original dataset results in a 19.2B-token high-performing mathematical
pretraining corpus, which we name MathCode-Pile. Training several popular base
models with this corpus significantly improves their mathematical abilities,
leading to the creation of the MathCoder2 family of models. All of our data
processing and training code is open-sourced, ensuring full transparency and
easy reproducibility of the entire data collection and training pipeline. The
code is released at https://github.com/mathllm/MathCoder2 .
SFTMix: Elevating Language Model Instruction Tuning with Mixup Recipe
Authors: name: Yuxin Xiao, name: Shujian Zhang, name: Wenxuan Zhou, name: Marzyeh Ghassemi, name: Sanqiang Zhao
Published: 2024-10-07 17:52:21

To induce desired behaviors in large language models (LLMs) for
interaction-driven tasks, the instruction-tuning stage typically trains LLMs on
instruction-response pairs using the next-token prediction (NTP) loss. Previous
work aiming to improve instruction-tuning performance often emphasizes the need
for higher-quality supervised fine-tuning (SFT) datasets, which typically
involves expensive data filtering with proprietary LLMs or labor-intensive data
generation by human annotators. However, these approaches do not fully leverage
the datasets' intrinsic properties, resulting in high computational and labor
costs, thereby limiting scalability and performance gains. In this paper, we
propose SFTMix, a novel recipe that elevates instruction-tuning performance
beyond the conventional NTP paradigm, without the need for well-curated
datasets. Observing that LLMs exhibit uneven confidence across the semantic
representation space, we argue that examples with different confidence levels
should play distinct roles during the instruction-tuning process. Based on this
insight, SFTMix leverages training dynamics to identify examples with varying
confidence levels, then applies a Mixup-based regularization to mitigate
overfitting on confident examples while propagating supervision signals to
improve learning on relatively unconfident ones. This approach enables SFTMix
to significantly outperform NTP across a wide range of instruction-following
and healthcare domain-specific SFT tasks, demonstrating its adaptability to
diverse LLM families and scalability to datasets of any size. Comprehensive
ablation studies further verify the robustness of SFTMix's design choices,
underscoring its versatility in consistently enhancing performance across
different LLMs and datasets in broader natural language processing
applications.
Agent S: An Open Agentic Framework that Uses Computers Like a Human
Authors: name: Saaket Agashe, name: Jiuzhou Han, name: Shuyu Gan, name: Jiachen Yang, name: Ang Li, name: Xin Eric Wang
Published: 2024-10-10 17:43:51

We present Agent S, an open agentic framework that enables autonomous
interaction with computers through a Graphical User Interface (GUI), aimed at
transforming human-computer interaction by automating complex, multi-step
tasks. Agent S aims to address three key challenges in automating computer
tasks: acquiring domain-specific knowledge, planning over long task horizons,
and handling dynamic, non-uniform interfaces. To this end, Agent S introduces
experience-augmented hierarchical planning, which learns from external
knowledge search and internal experience retrieval at multiple levels,
facilitating efficient task planning and subtask execution. In addition, it
employs an Agent-Computer Interface (ACI) to better elicit the reasoning and
control capabilities of GUI agents based on Multimodal Large Language Models
(MLLMs). Evaluation on the OSWorld benchmark shows that Agent S outperforms the
baseline by 9.37% on success rate (an 83.6% relative improvement) and achieves
a new state-of-the-art. Comprehensive analysis highlights the effectiveness of
individual components and provides insights for future improvements.
Furthermore, Agent S demonstrates broad generalizability to different operating
systems on a newly-released WindowsAgentArena benchmark. Code available at
https://github.com/simular-ai/Agent-S.
Intriguing Properties of Large Language and Vision Models
Authors: name: Young-Jun Lee, name: Byungsoo Ko, name: Han-Gyu Kim, name: Yechan Hwang, name: Ho-Jin Choi
Published: 2024-10-07 05:07:01

Recently, large language and vision models (LLVMs) have received significant
attention and development efforts due to their remarkable generalization
performance across a wide range of tasks requiring perception and cognitive
abilities. A key factor behind their success is their simple architecture,
which consists of a vision encoder, a projector, and a large language model
(LLM). Despite their achievements in advanced reasoning tasks, their
performance on fundamental perception-related tasks (e.g., MMVP) remains
surprisingly low. This discrepancy raises the question of how LLVMs truly
perceive images and exploit the advantages of the vision encoder. To address
this, we systematically investigate this question regarding several aspects:
permutation invariance, robustness, math reasoning, alignment preserving and
importance, by evaluating the most common LLVM's families (i.e., LLaVA) across
10 evaluation benchmarks. Our extensive experiments reveal several intriguing
properties of current LLVMs: (1) they internally process the image in a global
manner, even when the order of visual patch sequences is randomly permuted; (2)
they are sometimes able to solve math problems without fully perceiving
detailed numerical information; (3) the cross-modal alignment is overfitted to
complex reasoning tasks, thereby, causing them to lose some of the original
perceptual capabilities of their vision encoder; (4) the representation space
in the lower layers (<25%) plays a crucial role in determining performance and
enhancing visual understanding. Lastly, based on the above observations, we
suggest potential future directions for building better LLVMs and constructing
more challenging evaluation benchmarks.
Towards Self-Improvement of LLMs via MCTS: Leveraging Stepwise Knowledge
  with Curriculum Preference Learning
Authors: name: Xiyao Wang, name: Linfeng Song, name: Ye Tian, name: Dian Yu, name: Baolin Peng, name: Haitao Mi, name: Furong Huang, name: Dong Yu
Published: 2024-10-09 03:20:02

Monte Carlo Tree Search (MCTS) has recently emerged as a powerful technique
for enhancing the reasoning capabilities of LLMs. Techniques such as SFT or DPO
have enabled LLMs to distill high-quality behaviors from MCTS, improving their
reasoning performance. However, existing distillation methods underutilize the
rich trajectory information generated by MCTS, limiting the potential for
improvements in LLM reasoning. In this paper, we propose AlphaLLM-CPL, a novel
pairwise training framework that enables LLMs to self-improve through MCTS
behavior distillation. AlphaLLM-CPL efficiently leverages MCTS trajectories via
two key innovations: (1) AlphaLLM-CPL constructs stepwise trajectory pairs from
child nodes sharing the same parent in the search tree, providing step-level
information for more effective MCTS behavior distillation. (2) AlphaLLM-CPL
introduces curriculum preference learning, dynamically adjusting the training
sequence of trajectory pairs in each offline training epoch to prioritize
critical learning steps and mitigate overfitting. Experimental results on
mathematical reasoning tasks demonstrate that AlphaLLM-CPL significantly
outperforms previous MCTS behavior distillation methods, substantially boosting
the reasoning capabilities of LLMs.
Progressive Autoregressive Video Diffusion Models
Authors: name: Desai Xie, name: Zhan Xu, name: Yicong Hong, name: Hao Tan, name: Difan Liu, name: Feng Liu, name: Arie Kaufman, name: Yang Zhou
Published: 2024-10-10 17:36:15

Current frontier video diffusion models have demonstrated remarkable results
at generating high-quality videos. However, they can only generate short video
clips, normally around 10 seconds or 240 frames, due to computation limitations
during training. In this work, we show that existing models can be naturally
extended to autoregressive video diffusion models without changing the
architectures. Our key idea is to assign the latent frames with progressively
increasing noise levels rather than a single noise level, which allows for
fine-grained condition among the latents and large overlaps between the
attention windows. Such progressive video denoising allows our models to
autoregressively generate video frames without quality degradation or abrupt
scene changes. We present state-of-the-art results on long video generation at
1 minute (1440 frames at 24 FPS). Videos from this paper are available at
https://desaixie.github.io/pa-vdm/.