Orionnnn/hub-stats huggingface dataset:
MMAU: A Holistic Benchmark of Agent Capabilities Across Diverse Domains
Authors: name: Guoli Yin, name: Haoping Bai, name: Shuang Ma, name: Feng Nan, name: Yanchao Sun, name: Zhaoyang Xu, name: Shen Ma, name: Jiarui Lu, name: Xiang Kong, name: Aonan Zhang, name: Dian Ang Yap, name: Yizhe zhang, name: Karsten Ahnert, name: Vik Kamath, name: Mathias Berglund, name: Dominic Walsh, name: Tobias Gindele, name: Juergen Wiest, name: Zhengfeng Lai, name: Xiaoming Wang, name: Jiulong Shan, name: Meng Cao, name: Ruoming Pang, name: Zirui Wang
Published: 2024-07-18 00:58:41

Recent advances in large language models (LLMs) have increased the demand for
comprehensive benchmarks to evaluate their capabilities as human-like agents.
Existing benchmarks, while useful, often focus on specific application
scenarios, emphasizing task completion but failing to dissect the underlying
skills that drive these outcomes. This lack of granularity makes it difficult
to deeply discern where failures stem from. Additionally, setting up these
environments requires considerable effort, and issues of unreliability and
reproducibility sometimes arise, especially in interactive tasks. To address
these limitations, we introduce the Massive Multitask Agent Understanding
(MMAU) benchmark, featuring comprehensive offline tasks that eliminate the need
for complex environment setups. It evaluates models across five domains,
including teal{Tool-use}, teal{Directed Acyclic Graph
(DAG) QA}, teal{Data Science and Machine Learning coding},
teal{Contest-level programming} and teal{Mathematics},
and covers five essential capabilities: orange{Understanding},
orange{Reasoning}, orange{Planning},
orange{Problem-solving}, and orange{Self-correction}.
With a total of 20 meticulously designed tasks encompassing over 3K distinct
prompts, MMAU provides a comprehensive framework for evaluating the strengths
and limitations of LLM agents. By testing 18 representative models on MMAU, we
provide deep and insightful analyses. Ultimately, MMAU not only sheds light on
the capabilities and limitations of LLM agents but also enhances the
interpretability of their performance. Datasets and evaluation scripts of MMAU
are released at https://github.com/apple/axlearn/docs/research/mmau.
WalkTheDog: Cross-Morphology Motion Alignment via Phase Manifolds
Authors: name: Peizhuo Li, name: Sebastian Starke, name: Yuting Ye, name: Olga Sorkine-Hornung
Published: 2024-07-11 09:31:05

We present a new approach for understanding the periodicity structure and
semantics of motion datasets, independently of the morphology and skeletal
structure of characters. Unlike existing methods using an overly sparse
high-dimensional latent, we propose a phase manifold consisting of multiple
closed curves, each corresponding to a latent amplitude. With our proposed
vector quantized periodic autoencoder, we learn a shared phase manifold for
multiple characters, such as a human and a dog, without any supervision. This
is achieved by exploiting the discrete structure and a shallow network as
bottlenecks, such that semantically similar motions are clustered into the same
curve of the manifold, and the motions within the same component are aligned
temporally by the phase variable. In combination with an improved motion
matching framework, we demonstrate the manifold's capability of timing and
semantics alignment in several applications, including motion retrieval,
transfer and stylization. Code and pre-trained models for this paper are
available at https://peizhuoli.github.io/walkthedog.
FreeLong: Training-Free Long Video Generation with SpectralBlend
  Temporal Attention
Authors: name: Yu Lu, name: Yuanzhi Liang, name: Linchao Zhu, name: Yi Yang
Published: 2024-07-29 11:52:07

Video diffusion models have made substantial progress in various video
generation applications. However, training models for long video generation
tasks require significant computational and data resources, posing a challenge
to developing long video diffusion models. This paper investigates a
straightforward and training-free approach to extend an existing short video
diffusion model (e.g. pre-trained on 16-frame videos) for consistent long video
generation (e.g. 128 frames). Our preliminary observation has found that
directly applying the short video diffusion model to generate long videos can
lead to severe video quality degradation. Further investigation reveals that
this degradation is primarily due to the distortion of high-frequency
components in long videos, characterized by a decrease in spatial
high-frequency components and an increase in temporal high-frequency
components. Motivated by this, we propose a novel solution named FreeLong to
balance the frequency distribution of long video features during the denoising
process. FreeLong blends the low-frequency components of global video features,
which encapsulate the entire video sequence, with the high-frequency components
of local video features that focus on shorter subsequences of frames. This
approach maintains global consistency while incorporating diverse and
high-quality spatiotemporal details from local videos, enhancing both the
consistency and fidelity of long video generation. We evaluated FreeLong on
multiple base video diffusion models and observed significant improvements.
Additionally, our method supports coherent multi-prompt generation, ensuring
both visual coherence and seamless transitions between scenes.
Self-Training with Direct Preference Optimization Improves
  Chain-of-Thought Reasoning
Authors: name: Tianduo Wang, name: Shichen Li, name: Wei Lu
Published: 2024-07-25 17:59:16

Effective training of language models (LMs) for mathematical reasoning tasks
demands high-quality supervised fine-tuning data. Besides obtaining annotations
from human experts, a common alternative is sampling from larger and more
powerful LMs. However, this knowledge distillation approach can be costly and
unstable, particularly when relying on closed-source, proprietary LMs like
GPT-4, whose behaviors are often unpredictable. In this work, we demonstrate
that the reasoning abilities of small-scale LMs can be enhanced through
self-training, a process where models learn from their own outputs. We also
show that the conventional self-training can be further augmented by a
preference learning algorithm called Direct Preference Optimization (DPO). By
integrating DPO into self-training, we leverage preference data to guide LMs
towards more accurate and diverse chain-of-thought reasoning. We evaluate our
method across various mathematical reasoning tasks using different base models.
Our experiments show that this approach not only improves LMs' reasoning
performance but also offers a more cost-effective and scalable solution
compared to relying on large proprietary LMs.
3D Question Answering for City Scene Understanding
Authors: name: Penglei Sun, name: Yaoxian Song, name: Xiang Liu, name: Xiaofei Yang, name: Qiang Wang, name: Tiefeng Li, name: Yang Yang, name: Xiaowen Chu
Published: 2024-07-24 16:22:27

3D multimodal question answering (MQA) plays a crucial role in scene
understanding by enabling intelligent agents to comprehend their surroundings
in 3D environments. While existing research has primarily focused on indoor
household tasks and outdoor roadside autonomous driving tasks, there has been
limited exploration of city-level scene understanding tasks. Furthermore,
existing research faces challenges in understanding city scenes, due to the
absence of spatial semantic information and human-environment interaction
information at the city level.To address these challenges, we investigate 3D
MQA from both dataset and method perspectives. From the dataset perspective, we
introduce a novel 3D MQA dataset named City-3DQA for city-level scene
understanding, which is the first dataset to incorporate scene semantic and
human-environment interactive tasks within the city. From the method
perspective, we propose a Scene graph enhanced City-level Understanding method
(Sg-CityU), which utilizes the scene graph to introduce the spatial semantic. A
new benchmark is reported and our proposed Sg-CityU achieves accuracy of 63.94
% and 63.76 % in different settings of City-3DQA. Compared to indoor 3D MQA
methods and zero-shot using advanced large language models (LLMs), Sg-CityU
demonstrates state-of-the-art (SOTA) performance in robustness and
generalization.
SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal
  Domain
Authors: name: Pierre Colombo, name: Telmo Pires, name: Malik Boudiaf, name: Rui Melo, name: Dominic Culver, name: Sofia Morgado, name: Etienne Malaboeuf, name: Gabriel Hautreux, name: Johanne Charpentier, name: Michael Desa
Published: 2024-07-28 20:50:53

In this paper, we introduce SaulLM-54B and SaulLM-141B, two large language
models (LLMs) tailored for the legal sector. These models, which feature
architectures of 54 billion and 141 billion parameters, respectively, are based
on the Mixtral architecture. The development of SaulLM-54B and SaulLM-141B is
guided by large-scale domain adaptation, divided into three strategies: (1) the
exploitation of continued pretraining involving a base corpus that includes
over 540 billion of legal tokens, (2) the implementation of a specialized legal
instruction-following protocol, and (3) the alignment of model outputs with
human preferences in legal interpretations. The integration of synthetically
generated data in the second and third steps enhances the models' capabilities
in interpreting and processing legal texts, effectively reaching
state-of-the-art performance and outperforming previous open-source models on
LegalBench-Instruct. This work explores the trade-offs involved in
domain-specific adaptation at this scale, offering insights that may inform
future studies on domain adaptation using strong decoder models. Building upon
SaulLM-7B, this study refines the approach to produce an LLM better equipped
for legal tasks. We are releasing base, instruct, and aligned versions on top
of SaulLM-54B and SaulLM-141B under the MIT License to facilitate reuse and
collaborative research.
mGTE: Generalized Long-Context Text Representation and Reranking Models
  for Multilingual Text Retrieval
Authors: name: Xin Zhang, name: Yanzhao Zhang, name: Dingkun Long, name: Wen Xie, name: Ziqi Dai, name: Jialong Tang, name: Huan Lin, name: Baosong Yang, name: Pengjun Xie, name: Fei Huang, name: Meishan Zhang, name: Wenjie Li, name: Min Zhang
Published: 2024-07-29 03:12:28

We present systematic efforts in building long-context multilingual text
representation model (TRM) and reranker from scratch for text retrieval. We
first introduce a text encoder (base size) enhanced with RoPE and unpadding,
pre-trained in a native 8192-token context (longer than 512 of previous
multilingual encoders). Then we construct a hybrid TRM and a cross-encoder
reranker by contrastive learning. Evaluations show that our text encoder
outperforms the same-sized previous state-of-the-art XLM-R. Meanwhile, our TRM
and reranker match the performance of large-sized state-of-the-art BGE-M3
models and achieve better results on long-context retrieval benchmarks. Further
analysis demonstrate that our proposed models exhibit higher efficiency during
both training and inference. We believe their efficiency and effectiveness
could benefit various researches and industrial applications.
Floating No More: Object-Ground Reconstruction from a Single Image
Authors: name: Yunze Man, name: Yichen Sheng, name: Jianming Zhang, name: Liang-Yan Gui, name: Yu-Xiong Wang
Published: 2024-07-26 17:59:56

Recent advancements in 3D object reconstruction from single images have
primarily focused on improving the accuracy of object shapes. Yet, these
techniques often fail to accurately capture the inter-relation between the
object, ground, and camera. As a result, the reconstructed objects often appear
floating or tilted when placed on flat surfaces. This limitation significantly
affects 3D-aware image editing applications like shadow rendering and object
pose manipulation. To address this issue, we introduce ORG (Object
Reconstruction with Ground), a novel task aimed at reconstructing 3D object
geometry in conjunction with the ground surface. Our method uses two compact
pixel-level representations to depict the relationship between camera, object,
and ground. Experiments show that the proposed ORG model can effectively
reconstruct object-ground geometry on unseen data, significantly enhancing the
quality of shadow generation and pose manipulation compared to conventional
single-image 3D reconstruction techniques.
Wolf: Captioning Everything with a World Summarization Framework
Authors: name: Boyi Li, name: Ligeng Zhu, name: Ran Tian, name: Shuhan Tan, name: Yuxiao Chen, name: Yao Lu, name: Yin Cui, name: Sushant Veer, name: Max Ehrlich, name: Jonah Philion, name: Xinshuo Weng, name: Fuzhao Xue, name: Andrew Tao, name: Ming-Yu Liu, name: Sanja Fidler, name: Boris Ivanovic, name: Trevor Darrell, name: Jitendra Malik, name: Song Han, name: Marco Pavone
Published: 2024-07-26 17:59:09

We propose Wolf, a WOrLd summarization Framework for accurate video
captioning. Wolf is an automated captioning framework that adopts a
mixture-of-experts approach, leveraging complementary strengths of Vision
Language Models (VLMs). By utilizing both image and video models, our framework
captures different levels of information and summarizes them efficiently. Our
approach can be applied to enhance video understanding, auto-labeling, and
captioning. To evaluate caption quality, we introduce CapScore, an LLM-based
metric to assess the similarity and quality of generated captions compared to
the ground truth captions. We further build four human-annotated datasets in
three domains: autonomous driving, general scenes, and robotics, to facilitate
comprehensive comparisons. We show that Wolf achieves superior captioning
performance compared to state-of-the-art approaches from the research community
(VILA1.5, CogAgent) and commercial solutions (Gemini-Pro-1.5, GPT-4V). For
instance, in comparison with GPT-4V, Wolf improves CapScore both quality-wise
by 55.6% and similarity-wise by 77.4% on challenging driving videos. Finally,
we establish a benchmark for video captioning and introduce a leaderboard,
aiming to accelerate advancements in video understanding, captioning, and data
alignment. Leaderboard: https://wolfv0.github.io/leaderboard.html.
SHIC: Shape-Image Correspondences with no Keypoint Supervision
Authors: name: Aleksandar Shtedritski, name: Christian Rupprecht, name: Andrea Vedaldi
Published: 2024-07-26 17:58:59

Canonical surface mapping generalizes keypoint detection by assigning each
pixel of an object to a corresponding point in a 3D template. Popularised by
DensePose for the analysis of humans, authors have since attempted to apply the
concept to more categories, but with limited success due to the high cost of
manual supervision. In this work, we introduce SHIC, a method to learn
canonical maps without manual supervision which achieves better results than
supervised methods for most categories. Our idea is to leverage foundation
computer vision models such as DINO and Stable Diffusion that are open-ended
and thus possess excellent priors over natural categories. SHIC reduces the
problem of estimating image-to-template correspondences to predicting
image-to-image correspondences using features from the foundation models. The
reduction works by matching images of the object to non-photorealistic renders
of the template, which emulates the process of collecting manual annotations
for this task. These correspondences are then used to supervise high-quality
canonical maps for any object of interest. We also show that image generators
can further improve the realism of the template views, which provide an
additional source of supervision for the model.
Lessons from Learning to Spin "Pens"
Authors: name: Jun Wang, name: Ying Yuan, name: Haichuan Che, name: Haozhi Qi, name: Yi Ma, name: Jitendra Malik, name: Xiaolong Wang
Published: 2024-07-26 17:56:01

In-hand manipulation of pen-like objects is an important skill in our daily
lives, as many tools such as hammers and screwdrivers are similarly shaped.
However, current learning-based methods struggle with this task due to a lack
of high-quality demonstrations and the significant gap between simulation and
the real world. In this work, we push the boundaries of learning-based in-hand
manipulation systems by demonstrating the capability to spin pen-like objects.
We first use reinforcement learning to train an oracle policy with privileged
information and generate a high-fidelity trajectory dataset in simulation. This
serves two purposes: 1) pre-training a sensorimotor policy in simulation; 2)
conducting open-loop trajectory replay in the real world. We then fine-tune the
sensorimotor policy using these real-world trajectories to adapt it to the real
world dynamics. With less than 50 trajectories, our policy learns to rotate
more than ten pen-like objects with different physical properties for multiple
revolutions. We present a comprehensive analysis of our design choices and
share the lessons learned during development.
AppWorld: A Controllable World of Apps and People for Benchmarking
  Interactive Coding Agents
Authors: name: Harsh Trivedi, name: Tushar Khot, name: Mareike Hartmann, name: Ruskin Manku, name: Vinty Dong, name: Edward Li, name: Shashank Gupta, name: Ashish Sabharwal, name: Niranjan Balasubramanian
Published: 2024-07-26 17:55:45

Autonomous agents that address day-to-day digital tasks (e.g., ordering
groceries for a household), must not only operate multiple apps (e.g., notes,
messaging, shopping app) via APIs, but also generate rich code with complex
control flow in an iterative manner based on their interaction with the
environment. However, existing benchmarks for tool use are inadequate, as they
only cover tasks that require a simple sequence of API calls.
  To remedy this gap, we built AppWorld Engine, a high-quality
execution environment (60K lines of code) of 9 day-to-day apps operable via 457
APIs and populated with realistic digital activities simulating the lives of
~100 fictitious users. We then created AppWorld Benchmark (40K lines
of code), a suite of 750 natural, diverse, and challenging autonomous agent
tasks requiring rich and interactive code generation. It supports robust
programmatic evaluation with state-based unit tests, allowing for different
ways of completing a task while also checking for unexpected changes, i.e.,
collateral damage. The state-of-the-art LLM, GPT-4o, solves only ~49% of our
'normal' tasks and ~30% of 'challenge' tasks, while other models solve at least
16% fewer. This highlights the benchmark's difficulty and AppWorld's potential
to push the frontiers of interactive coding agents. The project website is
available at https://appworld.dev/.
VSSD: Vision Mamba with Non-Casual State Space Duality
Authors: name: Yuheng Shi, name: Minjing Dong, name: Mingjia Li, name: Chang Xu
Published: 2024-07-26 07:16:52

Vision transformers have significantly advanced the field of computer vision,
offering robust modeling capabilities and global receptive field. However,
their high computational demands limit their applicability in processing long
sequences. To tackle this issue, State Space Models (SSMs) have gained
prominence in vision tasks as they offer linear computational complexity.
Recently, State Space Duality (SSD), an improved variant of SSMs, was
introduced in Mamba2 to enhance model performance and efficiency. However, the
inherent causal nature of SSD/SSMs restricts their applications in non-causal
vision tasks. To address this limitation, we introduce Visual State Space
Duality (VSSD) model, which has a non-causal format of SSD. Specifically, we
propose to discard the magnitude of interactions between the hidden state and
tokens while preserving their relative weights, which relieves the dependencies
of token contribution on previous tokens. Together with the involvement of
multi-scan strategies, we show that the scanning results can be integrated to
achieve non-causality, which not only improves the performance of SSD in vision
tasks but also enhances its efficiency. We conduct extensive experiments on
various benchmarks including image classification, detection, and segmentation,
where VSSD surpasses existing state-of-the-art SSM-based models. Code and
weights are available at https://github.com/YuHengsss/VSSD.
Text-Driven Neural Collaborative Filtering Model for Paper Source
  Tracing
Authors: name: Aobo Xu, name: Bingyu Chang, name: Qingpeng Liu, name: Ling Jian
Published: 2024-07-25 02:48:56

Identifying significant references within the complex interrelations of a
citation knowledge graph is challenging, which encompasses connections through
citations, authorship, keywords, and other relational attributes. The Paper
Source Tracing (PST) task seeks to automate the identification of pivotal
references for given scholarly articles utilizing advanced data mining
techniques. In the KDD CUP 2024, we design a recommendation-based framework
tailored for the PST task. This framework employs the Neural Collaborative
Filtering (NCF) model to generate final predictions. To process the textual
attributes of the papers and extract input features for the model, we utilize
SciBERT, a pre-trained language model. According to the experimental results,
our method achieved a score of 0.37814 on the Mean Average Precision (MAP)
metric, outperforming baseline models and ranking 11th among all participating
teams. The source code is publicly available at
https://github.com/MyLove-XAB/KDDCupFinal.
LKCell: Efficient Cell Nuclei Instance Segmentation with Large
  Convolution Kernels
Authors: name: Ziwei Cui, name: Jingfeng Yao, name: Lunbin Zeng, name: Juan Yang, name: Wenyu Liu, name: Xinggang Wang
Published: 2024-07-25 14:07:49

The segmentation of cell nuclei in tissue images stained with the blood dye
hematoxylin and eosin (H&E) is essential for various clinical applications
and analyses. Due to the complex characteristics of cellular morphology, a
large receptive field is considered crucial for generating high-quality
segmentation. However, previous methods face challenges in achieving a balance
between the receptive field and computational burden. To address this issue, we
propose LKCell, a high-accuracy and efficient cell segmentation method. Its
core insight lies in unleashing the potential of large convolution kernels to
achieve computationally efficient large receptive fields. Specifically, (1) We
transfer pre-trained large convolution kernel models to the medical domain for
the first time, demonstrating their effectiveness in cell segmentation. (2) We
analyze the redundancy of previous methods and design a new segmentation
decoder based on large convolution kernels. It achieves higher performance
while significantly reducing the number of parameters. We evaluate our method
on the most challenging benchmark and achieve state-of-the-art results (0.5080
mPQ) in cell nuclei instance segmentation with only 21.6% FLOPs compared with
the previous leading method. Our source code and models are available at
https://github.com/hustvl/LKCell.