Orionnnn/hub-stats huggingface dataset:
gsplat: An Open-Source Library for Gaussian Splatting
Authors: name: Vickie Ye, name: Ruilong Li, name: Justin Kerr, name: Matias Turkulainen, name: Brent Yi, name: Zhuoyang Pan, name: Otto Seiskari, name: Jianbo Ye, name: Jeffrey Hu, name: Matthew Tancik, name: Angjoo Kanazawa
Published: 2024-09-10 17:57:38

gsplat is an open-source library designed for training and developing
Gaussian Splatting methods. It features a front-end with Python bindings
compatible with the PyTorch library and a back-end with highly optimized CUDA
kernels. gsplat offers numerous features that enhance the optimization of
Gaussian Splatting models, which include optimization improvements for speed,
memory, and convergence times. Experimental results demonstrate that gsplat
achieves up to 10% less training time and 4x less memory than the original
implementation. Utilized in several research projects, gsplat is actively
maintained on GitHub. Source code is available at
https://github.com/nerfstudio-project/gsplat under Apache License 2.0. We
welcome contributions from the open-source community.
Agent Workflow Memory
Authors: name: Zora Zhiruo Wang, name: Jiayuan Mao, name: Daniel Fried, name: Graham Neubig
Published: 2024-09-11 17:21:00

Despite the potential of language model-based agents to solve real-world
tasks such as web navigation, current methods still struggle with long-horizon
tasks with complex action trajectories. In contrast, humans can flexibly solve
complex tasks by learning reusable task workflows from past experiences and
using them to guide future actions. To build agents that can similarly benefit
from this process, we introduce Agent Workflow Memory (AWM), a method for
inducing commonly reused routines, i.e., workflows, and selectively providing
workflows to the agent to guide subsequent generations. AWM flexibly applies to
both offline and online scenarios, where agents induce workflows from training
examples beforehand or from test queries on the fly. We experiment on two major
web navigation benchmarks -- Mind2Web and WebArena -- that collectively cover
1000+ tasks from 200+ domains across travel, shopping, and social media, among
others. AWM substantially improves the baseline results by 24.6% and 51.1%
relative success rate on Mind2Web and WebArena while reducing the number of
steps taken to solve WebArena tasks successfully. Furthermore, online AWM
robustly generalizes in cross-task, website, and domain evaluations, surpassing
baselines from 8.9 to 14.0 absolute points as train-test task distribution gaps
widen.
Self-Harmonized Chain of Thought
Authors: name: Ziqi Jin, name: Wei Lu
Published: 2024-09-06 06:57:04

Chain-of-Thought (CoT) prompting reveals that large language models are
capable of performing complex reasoning via intermediate steps. CoT prompting
is primarily categorized into three approaches. The first approach utilizes
straightforward prompts like ``Let's think step by step'' to generate a
sequential thought process before yielding an answer. The second approach makes
use of human-crafted, step-by-step demonstrations to guide the model's
reasoning process. The third automates the generation of reasoned
demonstrations with the 'Let's think step by step'.This approach sometimes
leads to reasoning errors, highlighting the need to diversify demonstrations to
mitigate its misleading effects. However, diverse demonstrations pose
challenges for effective representations. In this work, we propose ECHO, a
self-harmonized chain-of-thought prompting method. It consolidates diverse
solution paths into a uniform and effective solution pattern.ECHO demonstrates
the best overall performance across three reasoning domains.
Gated Slot Attention for Efficient Linear-Time Sequence Modeling
Authors: name: Yu Zhang, name: Songlin Yang, name: Ruijie Zhu, name: Yue Zhang, name: Leyang Cui, name: Yiqiao Wang, name: Bolun Wang, name: Freda Shi, name: Bailin Wang, name: Wei Bi, name: Peng Zhou, name: Guohong Fu
Published: 2024-09-11 09:49:50

Linear attention Transformers and their gated variants, celebrated for
enabling parallel training and efficient recurrent inference, still fall short
in recall-intensive tasks compared to traditional Transformers and demand
significant resources for training from scratch. This paper introduces Gated
Slot Attention (GSA), which enhances Attention with Bounded-memory-Control
(ABC) by incorporating a gating mechanism inspired by Gated Linear Attention
(GLA). Essentially, GSA comprises a two-layer GLA linked via softmax, utilizing
context-aware memory reading and adaptive forgetting to improve memory capacity
while maintaining compact recurrent state size. This design greatly enhances
both training and inference efficiency through GLA's hardware-efficient
training algorithm and reduced state size. Additionally, retaining the softmax
operation is particularly beneficial in "finetuning pretrained Transformers to
RNNs" (T2R) settings, reducing the need for extensive training from scratch.
Extensive experiments confirm GSA's superior performance in scenarios requiring
in-context recall and in T2R settings.
Hi3D: Pursuing High-Resolution Image-to-3D Generation with Video
  Diffusion Models
Authors: name: Haibo Yang, name: Yang Chen, name: Yingwei Pan, name: Ting Yao, name: Zhineng Chen, name: Chong-Wah Ngo, name: Tao Mei
Published: 2024-09-11 17:58:57

Despite having tremendous progress in image-to-3D generation, existing
methods still struggle to produce multi-view consistent images with
high-resolution textures in detail, especially in the paradigm of 2D diffusion
that lacks 3D awareness. In this work, we present High-resolution Image-to-3D
model (Hi3D), a new video diffusion based paradigm that redefines a single
image to multi-view images as 3D-aware sequential image generation (i.e.,
orbital video generation). This methodology delves into the underlying temporal
consistency knowledge in video diffusion model that generalizes well to
geometry consistency across multiple views in 3D generation. Technically, Hi3D
first empowers the pre-trained video diffusion model with 3D-aware prior
(camera pose condition), yielding multi-view images with low-resolution texture
details. A 3D-aware video-to-video refiner is learnt to further scale up the
multi-view images with high-resolution texture details. Such high-resolution
multi-view images are further augmented with novel views through 3D Gaussian
Splatting, which are finally leveraged to obtain high-fidelity meshes via 3D
reconstruction. Extensive experiments on both novel view synthesis and single
view reconstruction demonstrate that our Hi3D manages to produce superior
multi-view consistency images with highly-detailed textures. Source code and
data are available at https://github.com/yanghb22-fdu/Hi3D-Official.
GroUSE: A Benchmark to Evaluate Evaluators in Grounded Question
  Answering
Authors: name: Sacha Muller, name: António Loison, name: Bilel Omrani, name: Gautier Viaud
Published: 2024-09-10 15:39:32

Retrieval-Augmented Generation (RAG) has emerged as a common paradigm to use
Large Language Models (LLMs) alongside private and up-to-date knowledge bases.
In this work, we address the challenges of using LLM-as-a-Judge when evaluating
grounded answers generated by RAG systems. To assess the calibration and
discrimination capabilities of judge models, we identify 7 generator failure
modes and introduce GroUSE (Grounded QA Unitary Scoring of Evaluators), a
meta-evaluation benchmark of 144 unit tests. This benchmark reveals that
existing automated RAG evaluation frameworks often overlook important failure
modes, even when using GPT-4 as a judge.
  To improve on the current design of automated RAG evaluation frameworks, we
propose a novel pipeline and find that while closed models perform well on
GroUSE, state-of-the-art open-source judges do not generalize to our proposed
criteria, despite strong correlation with GPT-4's judgement. Our findings
suggest that correlation with GPT-4 is an incomplete proxy for the practical
performance of judge models and should be supplemented with evaluations on unit
tests for precise failure mode detection.
  We further show that finetuning Llama-3 on GPT-4's reasoning traces
significantly boosts its evaluation capabilities, improving upon both
correlation with GPT-4's evaluations and calibration on reference situations.
LEIA: Latent View-invariant Embeddings for Implicit 3D Articulation
Authors: name: Archana Swaminathan, name: Anubhav Gupta, name: Kamal Gupta, name: Shishira R. Maiya, name: Vatsal Agarwal, name: Abhinav Shrivastava
Published: 2024-09-10 17:59:53

Neural Radiance Fields (NeRFs) have revolutionized the reconstruction of
static scenes and objects in 3D, offering unprecedented quality. However,
extending NeRFs to model dynamic objects or object articulations remains a
challenging problem. Previous works have tackled this issue by focusing on
part-level reconstruction and motion estimation for objects, but they often
rely on heuristics regarding the number of moving parts or object categories,
which can limit their practical use. In this work, we introduce LEIA, a novel
approach for representing dynamic 3D objects. Our method involves observing the
object at distinct time steps or "states" and conditioning a hypernetwork on
the current state, using this to parameterize our NeRF. This approach allows us
to learn a view-invariant latent representation for each state. We further
demonstrate that by interpolating between these states, we can generate novel
articulation configurations in 3D space that were previously unseen. Our
experimental results highlight the effectiveness of our method in articulating
objects in a manner that is independent of the viewing angle and joint
configuration. Notably, our approach outperforms previous methods that rely on
motion information for articulation registration.
LLaMA-Omni: Seamless Speech Interaction with Large Language Models
Authors: name: Qingkai Fang, name: Shoutao Guo, name: Yan Zhou, name: Zhengrui Ma, name: Shaolei Zhang, name: Yang Feng
Published: 2024-09-10 17:34:34

Models like GPT-4o enable real-time interaction with large language models
(LLMs) through speech, significantly enhancing user experience compared to
traditional text-based interaction. However, there is still a lack of
exploration on how to build speech interaction models based on open-source
LLMs. To address this, we propose LLaMA-Omni, a novel model architecture
designed for low-latency and high-quality speech interaction with LLMs.
LLaMA-Omni integrates a pretrained speech encoder, a speech adaptor, an LLM,
and a streaming speech decoder. It eliminates the need for speech
transcription, and can simultaneously generate text and speech responses
directly from speech instructions with extremely low latency. We build our
model based on the latest Llama-3.1-8B-Instruct model. To align the model with
speech interaction scenarios, we construct a dataset named InstructS2S-200K,
which includes 200K speech instructions and corresponding speech responses.
Experimental results show that compared to previous speech-language models,
LLaMA-Omni provides better responses in both content and style, with a response
latency as low as 226ms. Additionally, training LLaMA-Omni takes less than 3
days on just 4 GPUs, paving the way for the efficient development of
speech-language models in the future.
INTRA: Interaction Relationship-aware Weakly Supervised Affordance
  Grounding
Authors: name: Ji Ha Jang, name: Hoigi Seo, name: Se Young Chun
Published: 2024-09-10 04:31:51

Affordance denotes the potential interactions inherent in objects. The
perception of affordance can enable intelligent agents to navigate and interact
with new environments efficiently. Weakly supervised affordance grounding
teaches agents the concept of affordance without costly pixel-level
annotations, but with exocentric images. Although recent advances in weakly
supervised affordance grounding yielded promising results, there remain
challenges including the requirement for paired exocentric and egocentric image
dataset, and the complexity in grounding diverse affordances for a single
object. To address them, we propose INTeraction Relationship-aware weakly
supervised Affordance grounding (INTRA). Unlike prior arts, INTRA recasts this
problem as representation learning to identify unique features of interactions
through contrastive learning with exocentric images only, eliminating the need
for paired datasets. Moreover, we leverage vision-language model embeddings for
performing affordance grounding flexibly with any text, designing
text-conditioned affordance map generation to reflect interaction relationship
for contrastive learning and enhancing robustness with our text synonym
augmentation. Our method outperformed prior arts on diverse datasets such as
AGD20K, IIT-AFF, CAD and UMD. Additionally, experimental results demonstrate
that our method has remarkable domain scalability for synthesized images /
illustrations and is capable of performing affordance grounding for novel
interactions and objects.
Draw an Audio: Leveraging Multi-Instruction for Video-to-Audio Synthesis
Authors: name: Qi Yang, name: Binjie Mao, name: Zili Wang, name: Xing Nie, name: Pengfei Gao, name: Ying Guo, name: Cheng Zhen, name: Pengfei Yan, name: Shiming Xiang
Published: 2024-09-10 01:07:20

Foley is a term commonly used in filmmaking, referring to the addition of
daily sound effects to silent films or videos to enhance the auditory
experience. Video-to-Audio (V2A), as a particular type of automatic foley task,
presents inherent challenges related to audio-visual synchronization. These
challenges encompass maintaining the content consistency between the input
video and the generated audio, as well as the alignment of temporal and
loudness properties within the video. To address these issues, we construct a
controllable video-to-audio synthesis model, termed Draw an Audio, which
supports multiple input instructions through drawn masks and loudness signals.
To ensure content consistency between the synthesized audio and target video,
we introduce the Mask-Attention Module (MAM), which employs masked video
instruction to enable the model to focus on regions of interest. Additionally,
we implement the Time-Loudness Module (TLM), which uses an auxiliary loudness
signal to ensure the synthesis of sound that aligns with the video in both
loudness and temporal dimensions. Furthermore, we have extended a large-scale
V2A dataset, named VGGSound-Caption, by annotating caption prompts. Extensive
experiments on challenging benchmarks across two large-scale V2A datasets
verify Draw an Audio achieves the state-of-the-art. Project page:
https://yannqi.github.io/Draw-an-Audio/.
SongCreator: Lyrics-based Universal Song Generation
Authors: name: Shun Lei, name: Yixuan Zhou, name: Boshi Tang, name: Max W. Y. Lam, name: Feng Liu, name: Hangyu Liu, name: Jingcheng Wu, name: Shiyin Kang, name: Zhiyong Wu, name: Helen Meng
Published: 2024-09-09 19:37:07

Music is an integral part of human culture, embodying human intelligence and
creativity, of which songs compose an essential part. While various aspects of
song generation have been explored by previous works, such as singing voice,
vocal composition and instrumental arrangement, etc., generating songs with
both vocals and accompaniment given lyrics remains a significant challenge,
hindering the application of music generation models in the real world. In this
light, we propose SongCreator, a song-generation system designed to tackle this
challenge. The model features two novel designs: a meticulously designed
dual-sequence language model (DSLM) to capture the information of vocals and
accompaniment for song generation, and an additional attention mask strategy
for DSLM, which allows our model to understand, generate and edit songs, making
it suitable for various song-related generation tasks. Extensive experiments
demonstrate the effectiveness of SongCreator by achieving state-of-the-art or
competitive performances on all eight tasks. Notably, it surpasses previous
works by a large margin in lyrics-to-song and lyrics-to-vocals. Additionally,
it is able to independently control the acoustic conditions of the vocals and
accompaniment in the generated song through different prompts, exhibiting its
potential applicability. Our samples are available at
https://songcreator.github.io/.
SaRA: High-Efficient Diffusion Model Fine-tuning with Progressive Sparse
  Low-Rank Adaptation
Authors: name: Teng Hu, name: Jiangning Zhang, name: Ran Yi, name: Hongrui Huang, name: Yabiao Wang, name: Lizhuang Ma
Published: 2024-09-10 16:44:47

In recent years, the development of diffusion models has led to significant
progress in image and video generation tasks, with pre-trained models like the
Stable Diffusion series playing a crucial role. Inspired by model pruning which
lightens large pre-trained models by removing unimportant parameters, we
propose a novel model fine-tuning method to make full use of these ineffective
parameters and enable the pre-trained model with new task-specified
capabilities. In this work, we first investigate the importance of parameters
in pre-trained diffusion models, and discover that the smallest 10% to 20% of
parameters by absolute values do not contribute to the generation process.
Based on this observation, we propose a method termed SaRA that re-utilizes
these temporarily ineffective parameters, equating to optimizing a sparse
weight matrix to learn the task-specific knowledge. To mitigate overfitting, we
propose a nuclear-norm-based low-rank sparse training scheme for efficient
fine-tuning. Furthermore, we design a new progressive parameter adjustment
strategy to make full use of the re-trained/finetuned parameters. Finally, we
propose a novel unstructural backpropagation strategy, which significantly
reduces memory costs during fine-tuning. Our method enhances the generative
capabilities of pre-trained models in downstream applications and outperforms
traditional fine-tuning methods like LoRA in maintaining model's generalization
ability. We validate our approach through fine-tuning experiments on SD models,
demonstrating significant improvements. SaRA also offers a practical advantage
that requires only a single line of code modification for efficient
implementation and is seamlessly compatible with existing methods.
Robot Utility Models: General Policies for Zero-Shot Deployment in New
  Environments
Authors: name: Haritheja Etukuru, name: Norihito Naka, name: Zijin Hu, name: Seungjae Lee, name: Julian Mehu, name: Aaron Edsinger, name: Chris Paxton, name: Soumith Chintala, name: Lerrel Pinto, name: Nur Muhammad Mahi Shafiullah
Published: 2024-09-09 17:59:50

Robot models, particularly those trained with large amounts of data, have
recently shown a plethora of real-world manipulation and navigation
capabilities. Several independent efforts have shown that given sufficient
training data in an environment, robot policies can generalize to demonstrated
variations in that environment. However, needing to finetune robot models to
every new environment stands in stark contrast to models in language or vision
that can be deployed zero-shot for open-world problems. In this work, we
present Robot Utility Models (RUMs), a framework for training and deploying
zero-shot robot policies that can directly generalize to new environments
without any finetuning. To create RUMs efficiently, we develop new tools to
quickly collect data for mobile manipulation tasks, integrate such data into a
policy with multi-modal imitation learning, and deploy policies on-device on
Hello Robot Stretch, a cheap commodity robot, with an external mLLM verifier
for retrying. We train five such utility models for opening cabinet doors,
opening drawers, picking up napkins, picking up paper bags, and reorienting
fallen objects. Our system, on average, achieves 90% success rate in unseen,
novel environments interacting with unseen objects. Moreover, the utility
models can also succeed in different robot and camera set-ups with no further
data, training, or fine-tuning. Primary among our lessons are the importance of
training data over training algorithm and policy class, guidance about data
scaling, necessity for diverse yet high-quality demonstrations, and a recipe
for robot introspection and retrying to improve performance on individual
environments. Our code, data, models, hardware designs, as well as our
experiment and deployment videos are open sourced and can be found on our
project website: https://robotutilitymodels.com
POINTS: Improving Your Vision-language Model with Affordable Strategies
Authors: name: Yuan Liu, name: Zhongyin Zhao, name: Ziyuan Zhuang, name: Le Tian, name: Xiao Zhou, name: Jie Zhou
Published: 2024-09-07 13:41:37

In recent years, vision-language models have made significant strides,
excelling in tasks like optical character recognition and geometric
problem-solving. However, several critical issues remain: 1) Proprietary models
often lack transparency about their architectures, while open-source models
need more detailed ablations of their training strategies. 2) Pre-training data
in open-source works is under-explored, with datasets added empirically, making
the process cumbersome. 3) Fine-tuning often focuses on adding datasets,
leading to diminishing returns. To address these issues, we propose the
following contributions: 1) We trained a robust baseline model using the latest
advancements in vision-language models, introducing effective improvements and
conducting comprehensive ablation and validation for each technique. 2)
Inspired by recent work on large language models, we filtered pre-training data
using perplexity, selecting the lowest perplexity data for training. This
approach allowed us to train on a curated 1M dataset, achieving competitive
performance. 3) During visual instruction tuning, we used model soup on
different datasets when adding more datasets yielded marginal improvements.
These innovations resulted in a 9B parameter model that performs competitively
with state-of-the-art models. Our strategies are efficient and lightweight,
making them easily adoptable by the community.
UniDet3D: Multi-dataset Indoor 3D Object Detection
Authors: name: Maksim Kolodiazhnyi, name: Anna Vorontsova, name: Matvey Skripkin, name: Danila Rukhovich, name: Anton Konushin
Published: 2024-09-06 12:40:19

Growing customer demand for smart solutions in robotics and augmented reality
has attracted considerable attention to 3D object detection from point clouds.
Yet, existing indoor datasets taken individually are too small and
insufficiently diverse to train a powerful and general 3D object detection
model. In the meantime, more general approaches utilizing foundation models are
still inferior in quality to those based on supervised training for a specific
task. In this work, we propose , a simple yet effective 3D object
detection model, which is trained on a mixture of indoor datasets and is
capable of working in various indoor environments. By unifying different label
spaces,  enables learning a strong representation across multiple
datasets through a supervised joint training scheme. The proposed network
architecture is built upon a vanilla transformer encoder, making it easy to
run, customize and extend the prediction pipeline for practical use. Extensive
experiments demonstrate that  obtains significant gains over existing 3D
object detection methods in 6 indoor benchmarks: ScanNet (+1.1 mAP50),
ARKitScenes (+19.4 mAP25), S3DIS (+9.1 mAP50), MultiScan (+9.3 mAP50), 3RScan
(+3.2 mAP50), and ScanNet++ (+2.7 mAP50). Code is available at
https://github.com/filapro/unidet3d .