Orionnnn/hub-stats huggingface dataset:
Robot See Robot Do: Imitating Articulated Object Manipulation with
  Monocular 4D Reconstruction
Authors: name: Justin Kerr, name: Chung Min Kim, name: Mingxuan Wu, name: Brent Yi, name: Qianqian Wang, name: Ken Goldberg, name: Angjoo Kanazawa
Published: 2024-09-26 17:57:16

Humans can learn to manipulate new objects by simply watching others;
providing robots with the ability to learn from such demonstrations would
enable a natural interface specifying new behaviors. This work develops Robot
See Robot Do (RSRD), a method for imitating articulated object manipulation
from a single monocular RGB human demonstration given a single static
multi-view object scan. We first propose 4D Differentiable Part Models
(4D-DPM), a method for recovering 3D part motion from a monocular video with
differentiable rendering. This analysis-by-synthesis approach uses part-centric
feature fields in an iterative optimization which enables the use of geometric
regularizers to recover 3D motions from only a single video. Given this 4D
reconstruction, the robot replicates object trajectories by planning bimanual
arm motions that induce the demonstrated object part motion. By representing
demonstrations as part-centric trajectories, RSRD focuses on replicating the
demonstration's intended behavior while considering the robot's own
morphological limits, rather than attempting to reproduce the hand's motion. We
evaluate 4D-DPM's 3D tracking accuracy on ground truth annotated 3D part
trajectories and RSRD's physical execution performance on 9 objects across 10
trials each on a bimanual YuMi robot. Each phase of RSRD achieves an average of
87% success rate, for a total end-to-end success rate of 60% across 90 trials.
Notably, this is accomplished using only feature fields distilled from large
pretrained vision models -- without any task-specific training, fine-tuning,
dataset collection, or annotation. Project page:
https://robot-see-robot-do.github.io
EMOVA: Empowering Language Models to See, Hear and Speak with Vivid
  Emotions
Authors: name: Kai Chen, name: Yunhao Gou, name: Runhui Huang, name: Zhili Liu, name: Daxin Tan, name: Jing Xu, name: Chunwei Wang, name: Yi Zhu, name: Yihan Zeng, name: Kuo Yang, name: Dingdong Wang, name: Kun Xiang, name: Haoyuan Li, name: Haoli Bai, name: Jianhua Han, name: Xiaohui Li, name: Weike Jin, name: Nian Xie, name: Yu Zhang, name: James T. Kwok, name: Hengshuang Zhao, name: Xiaodan Liang, name: Dit-Yan Yeung, name: Xiao Chen, name: Zhenguo Li, name: Wei Zhang, name: Qun Liu, name: Lanqing Hong, name: Lu Hou, name: Hang Xu
Published: 2024-09-26 16:44:02

GPT-4o, an omni-modal model that enables vocal conversations with diverse
emotions and tones, marks a milestone for omni-modal foundation models.
However, empowering Large Language Models to perceive and generate images,
texts, and speeches end-to-end with publicly available data remains challenging
in the open-source community. Existing vision-language models rely on external
tools for the speech processing, while speech-language models still suffer from
limited or even without vision-understanding abilities. To address this gap, we
propose EMOVA (EMotionally Omni-present Voice Assistant), to enable Large
Language Models with end-to-end speech capabilities while maintaining the
leading vision-language performance. With a semantic-acoustic disentangled
speech tokenizer, we notice surprisingly that omni-modal alignment can further
enhance vision-language and speech abilities compared with the corresponding
bi-modal aligned counterparts. Moreover, a lightweight style module is proposed
for flexible speech style controls (e.g., emotions and pitches). For the first
time, EMOVA achieves state-of-the-art performance on both the vision-language
and speech benchmarks, and meanwhile, supporting omni-modal spoken dialogue
with vivid emotions.
MaskLLM: Learnable Semi-Structured Sparsity for Large Language Models
Authors: name: Gongfan Fang, name: Hongxu Yin, name: Saurav Muralidharan, name: Greg Heinrich, name: Jeff Pool, name: Jan Kautz, name: Pavlo Molchanov, name: Xinchao Wang
Published: 2024-09-26 02:37:41

Large Language Models (LLMs) are distinguished by their massive parameter
counts, which typically result in significant redundancy. This work introduces
MaskLLM, a learnable pruning method that establishes Semi-structured (or
``N:M'') Sparsity in LLMs, aimed at reducing computational overhead during
inference. Instead of developing a new importance criterion, MaskLLM explicitly
models N:M patterns as a learnable distribution through Gumbel Softmax
sampling. This approach facilitates end-to-end training on large-scale datasets
and offers two notable advantages: 1) High-quality Masks - our method
effectively scales to large datasets and learns accurate masks; 2)
Transferability - the probabilistic modeling of mask distribution enables the
transfer learning of sparsity across domains or tasks. We assessed MaskLLM
using 2:4 sparsity on various LLMs, including LLaMA-2, Nemotron-4, and GPT-3,
with sizes ranging from 843M to 15B parameters, and our empirical results show
substantial improvements over state-of-the-art methods. For instance, leading
approaches achieve a perplexity (PPL) of 10 or greater on Wikitext compared to
the dense model's 5.12 PPL, but MaskLLM achieves a significantly lower 6.72 PPL
solely by learning the masks with frozen weights. Furthermore, MaskLLM's
learnable nature allows customized masks for lossless application of 2:4
sparsity to downstream tasks or domains. Code is available at
https://github.com/NVlabs/MaskLLM.
Discovering the Gems in Early Layers: Accelerating Long-Context LLMs
  with 1000x Input Token Reduction
Authors: name: Zhenmei Shi, name: Yifei Ming, name: Xuan-Phi Nguyen, name: Yingyu Liang, name: Shafiq Joty
Published: 2024-09-25 23:14:47

Large Language Models (LLMs) have demonstrated remarkable capabilities in
handling long context inputs, but this comes at the cost of increased
computational resources and latency. Our research introduces a novel approach
for the long context bottleneck to accelerate LLM inference and reduce GPU
memory consumption. Our research demonstrates that LLMs can identify relevant
tokens in the early layers before generating answers to a query. Leveraging
this insight, we propose an algorithm that uses early layers of an LLM as
filters to select and compress input tokens, significantly reducing the context
length for subsequent processing. Our method, GemFilter, demonstrates
substantial improvements in both speed and memory efficiency compared to
existing techniques, such as standard attention and SnapKV/H2O. Notably, it
achieves a 2.4times speedup and 30\% reduction in GPU memory usage compared
to SOTA methods. Evaluation on the Needle in a Haystack task shows that
GemFilter significantly outperforms standard attention, SnapKV and demonstrates
comparable performance on the LongBench challenge. GemFilter is simple,
training-free, and broadly applicable across different LLMs. Crucially, it
provides interpretability by allowing humans to inspect the selected input
sequence. These findings not only offer practical benefits for LLM deployment,
but also enhance our understanding of LLM internal mechanisms, paving the way
for further optimizations in LLM design and inference. Our code is available at
https://github.com/SalesforceAIResearch/GemFilter.
Disco4D: Disentangled 4D Human Generation and Animation from a Single
  Image
Authors: name: Hui En Pang, name: Shuai Liu, name: Zhongang Cai, name: Lei Yang, name: Tianwei Zhang, name: Ziwei Liu
Published: 2024-09-25 18:46:06

We present Disco4D, a novel Gaussian Splatting framework for 4D
human generation and animation from a single image. Different from existing
methods, Disco4D distinctively disentangles clothings (with Gaussian models)
from the human body (with SMPL-X model), significantly enhancing the generation
details and flexibility. It has the following technical innovations.
1) Disco4D learns to efficiently fit the clothing Gaussians over the
SMPL-X Gaussians. 2) It adopts diffusion models to enhance the 3D
generation process, e.g., modeling occluded parts not visible in the
input image. 3) It learns an identity encoding for each clothing
Gaussian to facilitate the separation and extraction of clothing assets.
Furthermore, Disco4D naturally supports 4D human animation with vivid dynamics.
Extensive experiments demonstrate the superiority of Disco4D on 4D human
generation and animation tasks. Our visualizations can be found in
https://disco-4d.github.io/.
Self-Supervised Any-Point Tracking by Contrastive Random Walks
Authors: name: Ayush Shrivastava, name: Andrew Owens
Published: 2024-09-24 17:59:56

We present a simple, self-supervised approach to the Tracking Any Point (TAP)
problem. We train a global matching transformer to find cycle consistent tracks
through video via contrastive random walks, using the transformer's
attention-based global matching to define the transition matrices for a random
walk on a space-time graph. The ability to perform "all pairs" comparisons
between points allows the model to obtain high spatial precision and to obtain
a strong contrastive learning signal, while avoiding many of the complexities
of recent approaches (such as coarse-to-fine matching). To do this, we propose
a number of design decisions that allow global matching architectures to be
trained through self-supervision using cycle consistency. For example, we
identify that transformer-based methods are sensitive to shortcut solutions,
and propose a data augmentation scheme to address them. Our method achieves
strong performance on the TapVid benchmarks, outperforming previous
self-supervised tracking methods, such as DIFT, and is competitive with several
supervised methods.
NoTeeline: Supporting Real-Time Notetaking from Keypoints with Large
  Language Models
Authors: name: Faria Huq, name: Abdus Samee, name: David Chuan-en Lin, name: Xiaodi Alice Tang, name: Jeffrey P. Bigham
Published: 2024-09-24 22:36:44

Video has become a popular media form for information sharing and
consumption. However, taking notes while watching a video requires significant
time and effort. To address this, we propose a novel interactive system,
NoTeeline, for taking real-time, personalized notes. NoTeeline lets users
quickly jot down keypoints (micronotes), which are automatically expanded into
full-fledged notes that capture the content of the user's micronotes and are
consistent with the user's writing style. In a within-subjects study (N=12), we
found that NoTeeline helps users create high-quality notes that capture the
essence of their micronotes with a higher factual correctness (93.2%) while
accurately reflecting their writing style. While using NoTeeline, participants
experienced significantly reduced mental effort, captured satisfactory notes
while writing 47% less text, and completed notetaking with 43.9% less time
compared to a manual notetaking baseline.
HyperAgent: Generalist Software Engineering Agents to Solve Coding Tasks
  at Scale
Authors: name: Huy Nhat Phan, name: Phong X. Nguyen, name: Nghi D. Q. Bui
Published: 2024-09-09 19:35:34

Large Language Models (LLMs) have revolutionized software engineering (SE),
demonstrating remarkable capabilities in various coding tasks. While recent
efforts have produced autonomous software agents based on LLMs for end-to-end
development tasks, these systems are typically designed for specific SE tasks.
We introduce HyperAgent, a novel generalist multi-agent system designed to
address a wide spectrum of SE tasks across different programming languages by
mimicking human developers' workflows. Comprising four specialized agents -
Planner, Navigator, Code Editor, and Executor. HyperAgent manages the full
lifecycle of SE tasks, from initial conception to final verification. Through
extensive evaluations, HyperAgent achieves state-of-the-art performance across
diverse SE tasks: it attains a 25.01% success rate on SWE-Bench-Lite and 31.40%
on SWE-Bench-Verified for GitHub issue resolution, surpassing existing methods.
Furthermore, HyperAgent demonstrates SOTA performance in repository-level code
generation (RepoExec), and in fault localization and program repair
(Defects4J), often outperforming specialized systems. This work represents a
significant advancement towards versatile, autonomous agents capable of
handling complex, multi-step SE tasks across various domains and languages,
potentially transforming AI-assisted software development practices.
Game4Loc: A UAV Geo-Localization Benchmark from Game Data
Authors: name: Yuxiang Ji, name: Boyong He, name: Zhuoyue Tan, name: Liaoni Wu
Published: 2024-09-25 13:33:28

The vision-based geo-localization technology for UAV, serving as a secondary
source of GPS information in addition to the global navigation satellite
systems (GNSS), can still operate independently in the GPS-denied environment.
Recent deep learning based methods attribute this as the task of image matching
and retrieval. By retrieving drone-view images in geo-tagged satellite image
database, approximate localization information can be obtained. However, due to
high costs and privacy concerns, it is usually difficult to obtain large
quantities of drone-view images from a continuous area. Existing drone-view
datasets are mostly composed of small-scale aerial photography with a strong
assumption that there exists a perfect one-to-one aligned reference image for
any query, leaving a significant gap from the practical localization scenario.
In this work, we construct a large-range contiguous area UAV geo-localization
dataset named GTA-UAV, featuring multiple flight altitudes, attitudes, scenes,
and targets using modern computer games. Based on this dataset, we introduce a
more practical UAV geo-localization task including partial matches of
cross-view paired data, and expand the image-level retrieval to the actual
localization in terms of distance (meters). For the construction of drone-view
and satellite-view pairs, we adopt a weight-based contrastive learning
approach, which allows for effective learning while avoiding additional
post-processing matching steps. Experiments demonstrate the effectiveness of
our data and training method for UAV geo-localization, as well as the
generalization capabilities to real-world scenarios.
AIM 2024 Sparse Neural Rendering Challenge: Dataset and Benchmark
Authors: name: Michal Nazarczuk, name: Thomas Tanay, name: Sibi Catley-Chandar, name: Richard Shaw, name: Radu Timofte, name: Eduardo Pérez-Pellitero
Published: 2024-09-23 14:10:06

Recent developments in differentiable and neural rendering have made
impressive breakthroughs in a variety of 2D and 3D tasks, e.g. novel view
synthesis, 3D reconstruction. Typically, differentiable rendering relies on a
dense viewpoint coverage of the scene, such that the geometry can be
disambiguated from appearance observations alone. Several challenges arise when
only a few input views are available, often referred to as sparse or few-shot
neural rendering. As this is an underconstrained problem, most existing
approaches introduce the use of regularisation, together with a diversity of
learnt and hand-crafted priors. A recurring problem in sparse rendering
literature is the lack of an homogeneous, up-to-date, dataset and evaluation
protocol. While high-resolution datasets are standard in dense reconstruction
literature, sparse rendering methods often evaluate with low-resolution images.
Additionally, data splits are inconsistent across different manuscripts, and
testing ground-truth images are often publicly available, which may lead to
over-fitting. In this work, we propose the Sparse Rendering (SpaRe) dataset and
benchmark. We introduce a new dataset that follows the setup of the DTU MVS
dataset. The dataset is composed of 97 new scenes based on synthetic,
high-quality assets. Each scene has up to 64 camera views and 7 lighting
configurations, rendered at 1600x1200 resolution. We release a training split
of 82 scenes to foster generalizable approaches, and provide an online
evaluation platform for the validation and test sets, whose ground-truth images
remain hidden. We propose two different sparse configurations (3 and 9 input
images respectively). This provides a powerful and convenient tool for
reproducible evaluation, and enable researchers easy access to a public
leaderboard with the state-of-the-art performance scores. Available at:
https://sparebenchmark.github.io/
Boosting Healthcare LLMs Through Retrieved Context
Authors: name: Jordi Bayarri-Planas, name: Ashwin Kumar Gururajan, name: Dario Garcia-Gasulla
Published: 2024-09-23 15:33:38

Large Language Models (LLMs) have demonstrated remarkable capabilities in
natural language processing, and yet, their factual inaccuracies and
hallucinations limits their application, particularly in critical domains like
healthcare. Context retrieval methods, by introducing relevant information as
input, have emerged as a crucial approach for enhancing LLM factuality and
reliability. This study explores the boundaries of context retrieval methods
within the healthcare domain, optimizing their components and benchmarking
their performance against open and closed alternatives. Our findings reveal how
open LLMs, when augmented with an optimized retrieval system, can achieve
performance comparable to the biggest private solutions on established
healthcare benchmarks (multiple-choice question answering). Recognizing the
lack of realism of including the possible answers within the question (a setup
only found in medical exams), and after assessing a strong LLM performance
degradation in the absence of those options, we extend the context retrieval
system in that direction. In particular, we propose OpenMedPrompt a pipeline
that improves the generation of more reliable open-ended answers, moving this
technology closer to practical application.
Degradation-Guided One-Step Image Super-Resolution with Diffusion Priors
Authors: name: Aiping Zhang, name: Zongsheng Yue, name: Renjing Pei, name: Wenqi Ren, name: Xiaochun Cao
Published: 2024-09-25 16:15:21

Diffusion-based image super-resolution (SR) methods have achieved remarkable
success by leveraging large pre-trained text-to-image diffusion models as
priors. However, these methods still face two challenges: the requirement for
dozens of sampling steps to achieve satisfactory results, which limits
efficiency in real scenarios, and the neglect of degradation models, which are
critical auxiliary information in solving the SR problem. In this work, we
introduced a novel one-step SR model, which significantly addresses the
efficiency issue of diffusion-based SR methods. Unlike existing fine-tuning
strategies, we designed a degradation-guided Low-Rank Adaptation (LoRA) module
specifically for SR, which corrects the model parameters based on the
pre-estimated degradation information from low-resolution images. This module
not only facilitates a powerful data-dependent or degradation-dependent SR
model but also preserves the generative prior of the pre-trained diffusion
model as much as possible. Furthermore, we tailor a novel training pipeline by
introducing an online negative sample generation strategy. Combined with the
classifier-free guidance strategy during inference, it largely improves the
perceptual quality of the super-resolution results. Extensive experiments have
demonstrated the superior efficiency and effectiveness of the proposed model
compared to recent state-of-the-art methods.
Molmo and PixMo: Open Weights and Open Data for State-of-the-Art
  Multimodal Models
Authors: name: Matt Deitke, name: Christopher Clark, name: Sangho Lee, name: Rohun Tripathi, name: Yue Yang, name: Jae Sung Park, name: Mohammadreza Salehi, name: Niklas Muennighoff, name: Kyle Lo, name: Luca Soldaini, name: Jiasen Lu, name: Taira Anderson, name: Erin Bransom, name: Kiana Ehsani, name: Huong Ngo, name: YenSung Chen, name: Ajay Patel, name: Mark Yatskar, name: Chris Callison-Burch, name: Andrew Head, name: Rose Hendrix, name: Favyen Bastani, name: Eli VanderBilt, name: Nathan Lambert, name: Yvonne Chou, name: Arnavi Chheda, name: Jenna Sparks, name: Sam Skjonsberg, name: Michael Schmitz, name: Aaron Sarnat, name: Byron Bischoff, name: Pete Walsh, name: Chris Newell, name: Piper Wolters, name: Tanmay Gupta, name: Kuo-Hao Zeng, name: Jon Borchardt, name: Dirk Groeneveld, name: Jen Dumas, name: Crystal Nam, name: Sophie Lebrecht, name: Caitlin Wittlif, name: Carissa Schoenick, name: Oscar Michel, name: Ranjay Krishna, name: Luca Weihs, name: Noah A. Smith, name: Hannaneh Hajishirzi, name: Ross Girshick, name: Ali Farhadi, name: Aniruddha Kembhavi
Published: 2024-09-25 17:59:51

Today's most advanced multimodal models remain proprietary. The strongest
open-weight models rely heavily on synthetic data from proprietary VLMs to
achieve good performance, effectively distilling these closed models into open
ones. As a result, the community is still missing foundational knowledge about
how to build performant VLMs from scratch. We present Molmo, a new family of
VLMs that are state-of-the-art in their class of openness. Our key innovation
is a novel, highly detailed image caption dataset collected entirely from human
annotators using speech-based descriptions. To enable a wide array of user
interactions, we also introduce a diverse dataset mixture for fine-tuning that
includes in-the-wild Q&A and innovative 2D pointing data. The success of our
approach relies on careful choices for the model architecture details, a
well-tuned training pipeline, and, most critically, the quality of our newly
collected datasets, all of which will be released. The best-in-class 72B model
within the Molmo family not only outperforms others in the class of open weight
and data models but also compares favorably against proprietary systems like
GPT-4o, Claude 3.5, and Gemini 1.5 on both academic benchmarks and human
evaluation.
  We will be releasing all of our model weights, captioning and fine-tuning
data, and source code in the near future. Select model weights, inference code,
and demo are available at https://molmo.allenai.org.
DreamWaltz-G: Expressive 3D Gaussian Avatars from Skeleton-Guided 2D
  Diffusion
Authors: name: Yukun Huang, name: Jianan Wang, name: Ailing Zeng, name: Zheng-Jun Zha, name: Lei Zhang, name: Xihui Liu
Published: 2024-09-25 17:59:45

Leveraging pretrained 2D diffusion models and score distillation sampling
(SDS), recent methods have shown promising results for text-to-3D avatar
generation. However, generating high-quality 3D avatars capable of expressive
animation remains challenging. In this work, we present DreamWaltz-G, a novel
learning framework for animatable 3D avatar generation from text. The core of
this framework lies in Skeleton-guided Score Distillation and Hybrid 3D
Gaussian Avatar representation. Specifically, the proposed skeleton-guided
score distillation integrates skeleton controls from 3D human templates into 2D
diffusion models, enhancing the consistency of SDS supervision in terms of view
and human pose. This facilitates the generation of high-quality avatars,
mitigating issues such as multiple faces, extra limbs, and blurring. The
proposed hybrid 3D Gaussian avatar representation builds on the efficient 3D
Gaussians, combining neural implicit fields and parameterized 3D meshes to
enable real-time rendering, stable SDS optimization, and expressive animation.
Extensive experiments demonstrate that DreamWaltz-G is highly effective in
generating and animating 3D avatars, outperforming existing methods in both
visual quality and animation expressiveness. Our framework further supports
diverse applications, including human video reenactment and multi-subject scene
composition.