Orionnnn/hub-stats huggingface dataset:
Mixture-of-Agents Enhances Large Language Model Capabilities
Authors: name: Junlin Wang, name: Jue Wang, name: Ben Athiwaratkun, name: Ce Zhang, name: James Zou
Published: 2024-06-07 07:04:10

Recent advances in large language models (LLMs) demonstrate substantial
capabilities in natural language understanding and generation tasks. With the
growing number of LLMs, how to harness the collective expertise of multiple
LLMs is an exciting open direction. Toward this goal, we propose a new approach
that leverages the collective strengths of multiple LLMs through a
Mixture-of-Agents (MoA) methodology. In our approach, we construct a layered
MoA architecture wherein each layer comprises multiple LLM agents. Each agent
takes all the outputs from agents in the previous layer as auxiliary
information in generating its response. MoA models achieves state-of-art
performance on AlpacaEval 2.0, MT-Bench and FLASK, surpassing GPT-4 Omni. For
example, our MoA using only open-source LLMs is the leader of AlpacaEval 2.0 by
a substantial gap, achieving a score of 65.1% compared to 57.5% by GPT-4 Omni.
NATURAL PLAN: Benchmarking LLMs on Natural Language Planning
Authors: name: Huaixiu Steven Zheng, name: Swaroop Mishra, name: Hugh Zhang, name: Xinyun Chen, name: Minmin Chen, name: Azade Nova, name: Le Hou, name: Heng-Tze Cheng, name: Quoc V. Le, name: Ed H. Chi, name: Denny Zhou
Published: 2024-06-06 21:27:35

We introduce NATURAL PLAN, a realistic planning benchmark in natural language
containing 3 key tasks: Trip Planning, Meeting Planning, and Calendar
Scheduling. We focus our evaluation on the planning capabilities of LLMs with
full information on the task, by providing outputs from tools such as Google
Flights, Google Maps, and Google Calendar as contexts to the models. This
eliminates the need for a tool-use environment for evaluating LLMs on Planning.
We observe that NATURAL PLAN is a challenging benchmark for state of the art
models. For example, in Trip Planning, GPT-4 and Gemini 1.5 Pro could only
achieve 31.1% and 34.8% solve rate respectively. We find that model performance
drops drastically as the complexity of the problem increases: all models
perform below 5% when there are 10 cities, highlighting a significant gap in
planning in natural language for SoTA LLMs. We also conduct extensive ablation
studies on NATURAL PLAN to further shed light on the (in)effectiveness of
approaches such as self-correction, few-shot generalization, and in-context
planning with long-contexts on improving LLM planning.
GenAI Arena: An Open Evaluation Platform for Generative Models
Authors: name: Dongfu Jiang, name: Max Ku, name: Tianle Li, name: Yuansheng Ni, name: Shizhuo Sun, name: Rongqi Fan, name: Wenhu Chen
Published: 2024-06-06 20:15:42

Generative AI has made remarkable strides to revolutionize fields such as
image and video generation. These advancements are driven by innovative
algorithms, architecture, and data. However, the rapid proliferation of
generative models has highlighted a critical gap: the absence of trustworthy
evaluation metrics. Current automatic assessments such as FID, CLIP, FVD, etc
often fail to capture the nuanced quality and user satisfaction associated with
generative outputs. This paper proposes an open platform GenAI-Arena to
evaluate different image and video generative models, where users can actively
participate in evaluating these models. By leveraging collective user feedback
and votes, GenAI-Arena aims to provide a more democratic and accurate measure
of model performance. It covers three arenas for text-to-image generation,
text-to-video generation, and image editing respectively. Currently, we cover a
total of 27 open-source generative models. GenAI-Arena has been operating for
four months, amassing over 6000 votes from the community. We describe our
platform, analyze the data, and explain the statistical methods for ranking the
models. To further promote the research in building model-based evaluation
metrics, we release a cleaned version of our preference data for the three
tasks, namely GenAI-Bench. We prompt the existing multi-modal models like
Gemini, GPT-4o to mimic human voting. We compute the correlation between model
voting with human voting to understand their judging abilities. Our results
show existing multimodal models are still lagging in assessing the generated
visual content, even the best model GPT-4o only achieves a Pearson correlation
of 0.22 in the quality subscore, and behaves like random guessing in others.
ShareGPT4Video: Improving Video Understanding and Generation with Better
  Captions
Authors: name: Lin Chen, name: Xilin Wei, name: Jinsong Li, name: Xiaoyi Dong, name: Pan Zhang, name: Yuhang Zang, name: Zehui Chen, name: Haodong Duan, name: Bin Lin, name: Zhenyu Tang, name: Li Yuan, name: Yu Qiao, name: Dahua Lin, name: Feng Zhao, name: Jiaqi Wang
Published: 2024-06-06 17:58:54

We present the ShareGPT4Video series, aiming to facilitate the video
understanding of large video-language models (LVLMs) and the video generation
of text-to-video models (T2VMs) via dense and precise captions. The series
comprises: 1) ShareGPT4Video, 40K GPT4V annotated dense captions of videos with
various lengths and sources, developed through carefully designed data
filtering and annotating strategy. 2) ShareCaptioner-Video, an efficient and
capable captioning model for arbitrary videos, with 4.8M high-quality aesthetic
videos annotated by it. 3) ShareGPT4Video-8B, a simple yet superb LVLM that
reached SOTA performance on three advancing video benchmarks. To achieve this,
taking aside the non-scalable costly human annotators, we find using GPT4V to
caption video with a naive multi-frame or frame-concatenation input strategy
leads to less detailed and sometimes temporal-confused results. We argue the
challenge of designing a high-quality video captioning strategy lies in three
aspects: 1) Inter-frame precise temporal change understanding. 2) Intra-frame
detailed content description. 3) Frame-number scalability for arbitrary-length
videos. To this end, we meticulously designed a differential video captioning
strategy, which is stable, scalable, and efficient for generating captions for
videos with arbitrary resolution, aspect ratios, and length. Based on it, we
construct ShareGPT4Video, which contains 40K high-quality videos spanning a
wide range of categories, and the resulting captions encompass rich world
knowledge, object attributes, camera movements, and crucially, detailed and
precise temporal descriptions of events. Based on ShareGPT4Video, we further
develop ShareCaptioner-Video, a superior captioner capable of efficiently
generating high-quality captions for arbitrary videos...
pOps: Photo-Inspired Diffusion Operators
Authors: name: Elad Richardson, name: Yuval Alaluf, name: Ali Mahdavi-Amiri, name: Daniel Cohen-Or
Published: 2024-06-03 13:09:32

Text-guided image generation enables the creation of visual content from
textual descriptions. However, certain visual concepts cannot be effectively
conveyed through language alone. This has sparked a renewed interest in
utilizing the CLIP image embedding space for more visually-oriented tasks
through methods such as IP-Adapter. Interestingly, the CLIP image embedding
space has been shown to be semantically meaningful, where linear operations
within this space yield semantically meaningful results. Yet, the specific
meaning of these operations can vary unpredictably across different images. To
harness this potential, we introduce pOps, a framework that trains specific
semantic operators directly on CLIP image embeddings. Each pOps operator is
built upon a pretrained Diffusion Prior model. While the Diffusion Prior model
was originally trained to map between text embeddings and image embeddings, we
demonstrate that it can be tuned to accommodate new input conditions, resulting
in a diffusion operator. Working directly over image embeddings not only
improves our ability to learn semantic operations but also allows us to
directly use a textual CLIP loss as an additional supervision when needed. We
show that pOps can be used to learn a variety of photo-inspired operators with
distinct semantic meanings, highlighting the semantic diversity and potential
of our proposed approach.
Step-aware Preference Optimization: Aligning Preference with Denoising
  Performance at Each Step
Authors: name: Zhanhao Liang, name: Yuhui Yuan, name: Shuyang Gu, name: Bohan Chen, name: Tiankai Hang, name: Ji Li, name: Liang Zheng
Published: 2024-06-06 17:57:09

Recently, Direct Preference Optimization (DPO) has extended its success from
aligning large language models (LLMs) to aligning text-to-image diffusion
models with human preferences. Unlike most existing DPO methods that assume all
diffusion steps share a consistent preference order with the final generated
images, we argue that this assumption neglects step-specific denoising
performance and that preference labels should be tailored to each step's
contribution. To address this limitation, we propose Step-aware Preference
Optimization (SPO), a novel post-training approach that independently evaluates
and adjusts the denoising performance at each step, using a step-aware
preference model and a step-wise resampler to ensure accurate step-aware
supervision. Specifically, at each denoising step, we sample a pool of images,
find a suitable win-lose pair, and, most importantly, randomly select a single
image from the pool to initialize the next denoising step. This step-wise
resampler process ensures the next win-lose image pair comes from the same
image, making the win-lose comparison independent of the previous step. To
assess the preferences at each step, we train a separate step-aware preference
model that can be applied to both noisy and clean images. Our experiments with
Stable Diffusion v1.5 and SDXL demonstrate that SPO significantly outperforms
the latest Diffusion-DPO in aligning generated images with complex, detailed
prompts and enhancing aesthetics, while also achieving more than 20x times
faster in training efficiency. Code and model:
https://rockeycoss.github.io/spo.github.io/
BitsFusion: 1.99 bits Weight Quantization of Diffusion Model
Authors: name: Yang Sui, name: Yanyu Li, name: Anil Kag, name: Yerlan Idelbayev, name: Junli Cao, name: Ju Hu, name: Dhritiman Sagar, name: Bo Yuan, name: Sergey Tulyakov, name: Jian Ren
Published: 2024-06-06 17:59:23

Diffusion-based image generation models have achieved great success in recent
years by showing the capability of synthesizing high-quality content. However,
these models contain a huge number of parameters, resulting in a significantly
large model size. Saving and transferring them is a major bottleneck for
various applications, especially those running on resource-constrained devices.
In this work, we develop a novel weight quantization method that quantizes the
UNet from Stable Diffusion v1.5 to 1.99 bits, achieving a model with 7.9X
smaller size while exhibiting even better generation quality than the original
one. Our approach includes several novel techniques, such as assigning optimal
bits to each layer, initializing the quantized model for better performance,
and improving the training strategy to dramatically reduce quantization error.
Furthermore, we extensively evaluate our quantized model across various
benchmark datasets and through human evaluation to demonstrate its superior
generation quality.
Buffer of Thoughts: Thought-Augmented Reasoning with Large Language
  Models
Authors: name: Ling Yang, name: Zhaochen Yu, name: Tianjun Zhang, name: Shiyi Cao, name: Minkai Xu, name: Wentao Zhang, name: Joseph E. Gonzalez, name: Bin Cui
Published: 2024-06-06 17:22:08

We introduce Buffer of Thoughts (BoT), a novel and versatile
thought-augmented reasoning approach for enhancing accuracy, efficiency and
robustness of large language models (LLMs). Specifically, we propose
meta-buffer to store a series of informative high-level thoughts, namely
thought-template, distilled from the problem-solving processes across various
tasks. Then for each problem, we retrieve a relevant thought-template and
adaptively instantiate it with specific reasoning structures to conduct
efficient reasoning. To guarantee the scalability and stability, we further
propose buffer-manager to dynamically update the meta-buffer, thus enhancing
the capacity of meta-buffer as more tasks are solved. We conduct extensive
experiments on 10 challenging reasoning-intensive tasks, and achieve
significant performance improvements over previous SOTA methods: 11% on Game of
24, 20% on Geometric Shapes and 51% on Checkmate-in-One. Further analysis
demonstrate the superior generalization ability and model robustness of our
BoT, while requiring only 12% of the cost of multi-query prompting methods
(e.g., tree/graph of thoughts) on average. Notably, we find that our
Llama3-8B+BoT has the potential to surpass Llama3-70B model. Our project is
available at: https://github.com/YangLing0818/buffer-of-thought-llm
AgentGym: Evolving Large Language Model-based Agents across Diverse
  Environments
Authors: name: Zhiheng Xi, name: Yiwen Ding, name: Wenxiang Chen, name: Boyang Hong, name: Honglin Guo, name: Junzhe Wang, name: Dingwen Yang, name: Chenyang Liao, name: Xin Guo, name: Wei He, name: Songyang Gao, name: Lu Chen, name: Rui Zheng, name: Yicheng Zou, name: Tao Gui, name: Qi Zhang, name: Xipeng Qiu, name: Xuanjing Huang, name: Zuxuan Wu, name: Yu-Gang Jiang
Published: 2024-06-06 15:15:41

Building generalist agents that can handle diverse tasks and evolve
themselves across different environments is a long-term goal in the AI
community. Large language models (LLMs) are considered a promising foundation
to build such agents due to their generalized capabilities. Current approaches
either have LLM-based agents imitate expert-provided trajectories step-by-step,
requiring human supervision, which is hard to scale and limits environmental
exploration; or they let agents explore and learn in isolated environments,
resulting in specialist agents with limited generalization. In this paper, we
take the first step towards building generally-capable LLM-based agents with
self-evolution ability. We identify a trinity of ingredients: 1) diverse
environments for agent exploration and learning, 2) a trajectory set to equip
agents with basic capabilities and prior knowledge, and 3) an effective and
scalable evolution method. We propose AgentGym, a new framework featuring a
variety of environments and tasks for broad, real-time, uni-format, and
concurrent agent exploration. AgentGym also includes a database with expanded
instructions, a benchmark suite, and high-quality trajectories across
environments. Next, we propose a novel method, AgentEvol, to investigate the
potential of agent self-evolution beyond previously seen data across tasks and
environments. Experimental results show that the evolved agents can achieve
results comparable to SOTA models. We release the AgentGym suite, including the
platform, dataset, benchmark, checkpoints, and algorithm implementations. The
AgentGym suite is available on https://github.com/WooooDyy/AgentGym.
Open-Endedness is Essential for Artificial Superhuman Intelligence
Authors: name: Edward Hughes, name: Michael Dennis, name: Jack Parker-Holder, name: Feryal Behbahani, name: Aditi Mavalankar, name: Yuge Shi, name: Tom Schaul, name: Tim Rocktaschel
Published: 2024-06-06 17:15:02

In recent years there has been a tremendous surge in the general capabilities
of AI systems, mainly fuelled by training foundation models on internetscale
data. Nevertheless, the creation of openended, ever self-improving AI remains
elusive. In this position paper, we argue that the ingredients are now in place
to achieve openendedness in AI systems with respect to a human observer.
Furthermore, we claim that such open-endedness is an essential property of any
artificial superhuman intelligence (ASI). We begin by providing a concrete
formal definition of open-endedness through the lens of novelty and
learnability. We then illustrate a path towards ASI via open-ended systems
built on top of foundation models, capable of making novel, humanrelevant
discoveries. We conclude by examining the safety implications of
generally-capable openended AI. We expect that open-ended foundation models
will prove to be an increasingly fertile and safety-critical area of research
in the near future.
SF-V: Single Forward Video Generation Model
Authors: name: Zhixing Zhang, name: Yanyu Li, name: Yushu Wu, name: Yanwu Xu, name: Anil Kag, name: Ivan Skorokhodov, name: Willi Menapace, name: Aliaksandr Siarohin, name: Junli Cao, name: Dimitris Metaxas, name: Sergey Tulyakov, name: Jian Ren
Published: 2024-06-06 17:58:27

Diffusion-based video generation models have demonstrated remarkable success
in obtaining high-fidelity videos through the iterative denoising process.
However, these models require multiple denoising steps during sampling,
resulting in high computational costs. In this work, we propose a novel
approach to obtain single-step video generation models by leveraging
adversarial training to fine-tune pre-trained video diffusion models. We show
that, through the adversarial training, the multi-steps video diffusion model,
i.e., Stable Video Diffusion (SVD), can be trained to perform single forward
pass to synthesize high-quality videos, capturing both temporal and spatial
dependencies in the video data. Extensive experiments demonstrate that our
method achieves competitive generation quality of synthesized videos with
significantly reduced computational overhead for the denoising process (i.e.,
around 23times speedup compared with SVD and 6times speedup compared with
existing works, with even better generation quality), paving the way for
real-time video synthesis and editing. More visualization results are made
publicly available at https://snap-research.github.io/SF-V.
VideoTetris: Towards Compositional Text-to-Video Generation
Authors: name: Ye Tian, name: Ling Yang, name: Haotian Yang, name: Yuan Gao, name: Yufan Deng, name: Jingmin Chen, name: Xintao Wang, name: Zhaochen Yu, name: Xin Tao, name: Pengfei Wan, name: Di Zhang, name: Bin Cui
Published: 2024-06-06 17:25:33

Diffusion models have demonstrated great success in text-to-video (T2V)
generation. However, existing methods may face challenges when handling complex
(long) video generation scenarios that involve multiple objects or dynamic
changes in object numbers. To address these limitations, we propose
VideoTetris, a novel framework that enables compositional T2V generation.
Specifically, we propose spatio-temporal compositional diffusion to precisely
follow complex textual semantics by manipulating and composing the attention
maps of denoising networks spatially and temporally. Moreover, we propose an
enhanced video data preprocessing to enhance the training data regarding motion
dynamics and prompt understanding, equipped with a new reference frame
attention mechanism to improve the consistency of auto-regressive video
generation. Extensive experiments demonstrate that our VideoTetris achieves
impressive qualitative and quantitative results in compositional T2V
generation. Code is available at: https://github.com/YangLing0818/VideoTetris
Mobile-Agent-v2: Mobile Device Operation Assistant with Effective
  Navigation via Multi-Agent Collaboration
Authors: name: Junyang Wang, name: Haiyang Xu, name: Haitao Jia, name: Xi Zhang, name: Ming Yan, name: Weizhou Shen, name: Ji Zhang, name: Fei Huang, name: Jitao Sang
Published: 2024-06-03 05:50:00

Mobile device operation tasks are increasingly becoming a popular multi-modal
AI application scenario. Current Multi-modal Large Language Models (MLLMs),
constrained by their training data, lack the capability to function effectively
as operation assistants. Instead, MLLM-based agents, which enhance capabilities
through tool invocation, are gradually being applied to this scenario. However,
the two major navigation challenges in mobile device operation tasks, task
progress navigation and focus content navigation, are significantly complicated
under the single-agent architecture of existing work. This is due to the overly
long token sequences and the interleaved text-image data format, which limit
performance. To address these navigation challenges effectively, we propose
Mobile-Agent-v2, a multi-agent architecture for mobile device operation
assistance. The architecture comprises three agents: planning agent, decision
agent, and reflection agent. The planning agent generates task progress, making
the navigation of history operations more efficient. To retain focus content,
we design a memory unit that updates with task progress. Additionally, to
correct erroneous operations, the reflection agent observes the outcomes of
each operation and handles any mistakes accordingly. Experimental results
indicate that Mobile-Agent-v2 achieves over a 30% improvement in task
completion compared to the single-agent architecture of Mobile-Agent. The code
is open-sourced at https://github.com/X-PLUG/MobileAgent.
Searching Priors Makes Text-to-Video Synthesis Better
Authors: name: Haoran Cheng, name: Liang Peng, name: Linxuan Xia, name: Yuepeng Hu, name: Hengjia Li, name: Qinglin Lu, name: Xiaofei He, name: Boxi Wu
Published: 2024-06-05 12:53:28

Significant advancements in video diffusion models have brought substantial
progress to the field of text-to-video (T2V) synthesis. However, existing T2V
synthesis model struggle to accurately generate complex motion dynamics,
leading to a reduction in video realism. One possible solution is to collect
massive data and train the model on it, but this would be extremely expensive.
To alleviate this problem, in this paper, we reformulate the typical T2V
generation process as a search-based generation pipeline. Instead of scaling up
the model training, we employ existing videos as the motion prior database.
Specifically, we divide T2V generation process into two steps: (i) For a given
prompt input, we search existing text-video datasets to find videos with text
labels that closely match the prompt motions. We propose a tailored search
algorithm that emphasizes object motion features. (ii) Retrieved videos are
processed and distilled into motion priors to fine-tune a pre-trained base T2V
model, followed by generating desired videos using input prompt. By utilizing
the priors gleaned from the searched videos, we enhance the realism of the
generated videos' motion. All operations can be finished on a single NVIDIA RTX
4090 GPU. We validate our method against state-of-the-art T2V models across
diverse prompt inputs. The code will be public.
Audio Mamba: Bidirectional State Space Model for Audio Representation
  Learning
Authors: name: Mehmet Hamza Erol, name: Arda Senocak, name: Jiu Feng, name: Joon Son Chung
Published: 2024-06-05 15:00:59

Transformers have rapidly become the preferred choice for audio
classification, surpassing methods based on CNNs. However, Audio Spectrogram
Transformers (ASTs) exhibit quadratic scaling due to self-attention. The
removal of this quadratic self-attention cost presents an appealing direction.
Recently, state space models (SSMs), such as Mamba, have demonstrated potential
in language and vision tasks in this regard. In this study, we explore whether
reliance on self-attention is necessary for audio classification tasks. By
introducing Audio Mamba (AuM), the first self-attention-free, purely SSM-based
model for audio classification, we aim to address this question. We evaluate
AuM on various audio datasets - comprising six different benchmarks - where it
achieves comparable or better performance compared to well-established AST
model.