Orionnnn/hub-stats huggingface dataset:
Embodied-RAG: General non-parametric Embodied Memory for Retrieval and
  Generation
Authors: name: Quanting Xie, name: So Yeon Min, name: Tianyi Zhang, name: Aarav Bajaj, name: Ruslan Salakhutdinov, name: Matthew Johnson-Roberson, name: Yonatan Bisk
Published: 2024-09-26 21:44:11

There is no limit to how much a robot might explore and learn, but all of
that knowledge needs to be searchable and actionable. Within language research,
retrieval augmented generation (RAG) has become the workhouse of large-scale
non-parametric knowledge, however existing techniques do not directly transfer
to the embodied domain, which is multimodal, data is highly correlated, and
perception requires abstraction.
  To address these challenges, we introduce Embodied-RAG, a framework that
enhances the foundational model of an embodied agent with a non-parametric
memory system capable of autonomously constructing hierarchical knowledge for
both navigation and language generation. Embodied-RAG handles a full range of
spatial and semantic resolutions across diverse environments and query types,
whether for a specific object or a holistic description of ambiance. At its
core, Embodied-RAG's memory is structured as a semantic forest, storing
language descriptions at varying levels of detail. This hierarchical
organization allows the system to efficiently generate context-sensitive
outputs across different robotic platforms. We demonstrate that Embodied-RAG
effectively bridges RAG to the robotics domain, successfully handling over 200
explanation and navigation queries across 19 environments, highlighting its
promise for general-purpose non-parametric system for embodied agents.
What the Harm? Quantifying the Tangible Impact of Gender Bias in Machine
  Translation with a Human-centered Study
Authors: name: Beatrice Savoldi, name: Sara Papi, name: Matteo Negri, name: Ana Guerberof, name: Luisa Bentivogli
Published: 2024-10-01 09:38:34

Gender bias in machine translation (MT) is recognized as an issue that can
harm people and society. And yet, advancements in the field rarely involve
people, the final MT users, or inform how they might be impacted by biased
technologies. Current evaluations are often restricted to automatic methods,
which offer an opaque estimate of what the downstream impact of gender
disparities might be. We conduct an extensive human-centered study to examine
if and to what extent bias in MT brings harms with tangible costs, such as
quality of service gaps across women and men. To this aim, we collect
behavioral data from 90 participants, who post-edited MT outputs to ensure
correct gender translation. Across multiple datasets, languages, and types of
users, our study shows that feminine post-editing demands significantly more
technical and temporal effort, also corresponding to higher financial costs.
Existing bias measurements, however, fail to reflect the found disparities. Our
findings advocate for human-centered approaches that can inform the societal
impact of bias.
Visual Context Window Extension: A New Perspective for Long Video
  Understanding
Authors: name: Hongchen Wei, name: Zhenzhong Chen
Published: 2024-09-30 07:25:16

Large Multimodal Models (LMMs) have demonstrated impressive performance in
short video understanding tasks but face great challenges when applied to long
video understanding. In contrast, Large Language Models (LLMs) exhibit
outstanding capabilities in modeling long texts. Existing work attempts to
address this issue by introducing long video-text pairs during training.
However, these approaches require substantial computational and data resources.
In this paper, we tackle the challenge of long video understanding from the
perspective of context windows, aiming to apply LMMs to long video tasks
without retraining on long video datasets. We first conduct an in-depth
analysis of why pretrained LMMs struggle to understand lengthy video content,
identifying that discrepancies between visual and language modalities lead to
different context windows for visual and language tokens, making it difficult
to directly extend the visual tokens to match the language context window.
Based on this, we propose to adapt LMMs for long video understanding tasks by
extending the visual context window, eliminating the need for retraining on
large scalelong video datasets. To further mitigate the significant memory
consumption caused by long sequences, we introduce a progressive pooling
inference strategy that selectively adjusts the spatial resolution of frame
embeddings, reducing the number of visual tokens while retaining important
spatial information. Across multiple long video understanding benchmarks, our
method consistently improves the performance as the number of video frames
increases. On the MLVU benchmark, our method outperforms GPT-4o, even though
our model size is only 7B. Additionally, in the 256-frame setting, our method
reduces memory usage by approximately 45% compared to the baseline, without
introducing any performance loss.
SyntheOcc: Synthesize Geometric-Controlled Street View Images through 3D
  Semantic MPIs
Authors: name: Leheng Li, name: Weichao Qiu, name: Yingjie Cai, name: Xu Yan, name: Qing Lian, name: Bingbing Liu, name: Ying-Cong Chen
Published: 2024-10-01 02:29:24

The advancement of autonomous driving is increasingly reliant on high-quality
annotated datasets, especially in the task of 3D occupancy prediction, where
the occupancy labels require dense 3D annotation with significant human effort.
In this paper, we propose SyntheOcc, which denotes a diffusion model that
Synthesize photorealistic and geometric-controlled images by conditioning
Occupancy labels in driving scenarios. This yields an unlimited amount of
diverse, annotated, and controllable datasets for applications like training
perception models and simulation. SyntheOcc addresses the critical challenge of
how to efficiently encode 3D geometric information as conditional input to a 2D
diffusion model. Our approach innovatively incorporates 3D semantic multi-plane
images (MPIs) to provide comprehensive and spatially aligned 3D scene
descriptions for conditioning. As a result, SyntheOcc can generate
photorealistic multi-view images and videos that faithfully align with the
given geometric labels (semantics in 3D voxel space). Extensive qualitative and
quantitative evaluations of SyntheOcc on the nuScenes dataset prove its
effectiveness in generating controllable occupancy datasets that serve as an
effective data augmentation to perception models.
Posterior-Mean Rectified Flow: Towards Minimum MSE Photo-Realistic Image
  Restoration
Authors: name: Guy Ohayon, name: Tomer Michaeli, name: Michael Elad
Published: 2024-10-01 05:54:07

Photo-realistic image restoration algorithms are typically evaluated by
distortion measures (e.g., PSNR, SSIM) and by perceptual quality measures
(e.g., FID, NIQE), where the desire is to attain the lowest possible distortion
without compromising on perceptual quality. To achieve this goal, current
methods typically attempt to sample from the posterior distribution, or to
optimize a weighted sum of a distortion loss (e.g., MSE) and a perceptual
quality loss (e.g., GAN). Unlike previous works, this paper is concerned
specifically with the optimal estimator that minimizes the MSE under a
constraint of perfect perceptual index, namely where the distribution of the
reconstructed images is equal to that of the ground-truth ones. A recent
theoretical result shows that such an estimator can be constructed by optimally
transporting the posterior mean prediction (MMSE estimate) to the distribution
of the ground-truth images. Inspired by this result, we introduce
Posterior-Mean Rectified Flow (PMRF), a simple yet highly effective algorithm
that approximates this optimal estimator. In particular, PMRF first predicts
the posterior mean, and then transports the result to a high-quality image
using a rectified flow model that approximates the desired optimal transport
map. We investigate the theoretical utility of PMRF and demonstrate that it
consistently outperforms previous methods on a variety of image restoration
tasks.
One Token to Seg Them All: Language Instructed Reasoning Segmentation in
  Videos
Authors: name: Zechen Bai, name: Tong He, name: Haiyang Mei, name: Pichao Wang, name: Ziteng Gao, name: Joya Chen, name: Lei Liu, name: Zheng Zhang, name: Mike Zheng Shou
Published: 2024-09-29 07:47:15

We introduce VideoLISA, a video-based multimodal large language model
designed to tackle the problem of language-instructed reasoning segmentation in
videos. Leveraging the reasoning capabilities and world knowledge of large
language models, and augmented by the Segment Anything Model, VideoLISA
generates temporally consistent segmentation masks in videos based on language
instructions. Existing image-based methods, such as LISA, struggle with video
tasks due to the additional temporal dimension, which requires temporal dynamic
understanding and consistent segmentation across frames. VideoLISA addresses
these challenges by integrating a Sparse Dense Sampling strategy into the
video-LLM, which balances temporal context and spatial detail within
computational constraints. Additionally, we propose a One-Token-Seg-All
approach using a specially designed <TRK> token, enabling the model to segment
and track objects across multiple frames. Extensive evaluations on diverse
benchmarks, including our newly introduced ReasonVOS benchmark, demonstrate
VideoLISA's superior performance in video object segmentation tasks involving
complex reasoning, temporal understanding, and object tracking. While optimized
for videos, VideoLISA also shows promising generalization to image
segmentation, revealing its potential as a unified foundation model for
language-instructed object segmentation. Code and model will be available at:
https://github.com/showlab/VideoLISA.
Atlas-Chat: Adapting Large Language Models for Low-Resource Moroccan
  Arabic Dialect
Authors: name: Guokan Shang, name: Hadi Abdine, name: Yousef Khoubrane, name: Amr Mohamed, name: Yassine Abbahaddou, name: Sofiane Ennadir, name: Imane Momayiz, name: Xuguang Ren, name: Eric Moulines, name: Preslav Nakov, name: Michalis Vazirgiannis, name: Eric Xing
Published: 2024-09-26 14:56:38

We introduce Atlas-Chat, the first-ever collection of large language models
specifically developed for dialectal Arabic. Focusing on Moroccan Arabic, also
known as Darija, we construct our instruction dataset by consolidating existing
Darija language resources, creating novel datasets both manually and
synthetically, and translating English instructions with stringent quality
control. Atlas-Chat-9B and 2B models, fine-tuned on the dataset, exhibit
superior ability in following Darija instructions and performing standard NLP
tasks. Notably, our models outperform both state-of-the-art and
Arabic-specialized LLMs like LLaMa, Jais, and AceGPT, e.g., achieving a 13%
performance boost over a larger 13B model on DarijaMMLU, in our newly
introduced evaluation suite for Darija covering both discriminative and
generative tasks. Furthermore, we perform an experimental analysis of various
fine-tuning strategies and base model choices to determine optimal
configurations. All our resources are publicly accessible, and we believe our
work offers comprehensive design methodologies of instruction-tuning for
low-resource language variants, which are often neglected in favor of data-rich
languages by contemporary LLMs.
Illustrious: an Open Advanced Illustration Model
Authors: name: Sang Hyun Park, name: Jun Young Koh, name: Junha Lee, name: Joy Song, name: Dongha Kim, name: Hoyeon Moon, name: Hyunju Lee, name: Min Song
Published: 2024-09-30 04:59:12

In this work, we share the insights for achieving state-of-the-art quality in
our text-to-image anime image generative model, called Illustrious. To achieve
high resolution, dynamic color range images, and high restoration ability, we
focus on three critical approaches for model improvement. First, we delve into
the significance of the batch size and dropout control, which enables faster
learning of controllable token based concept activations. Second, we increase
the training resolution of images, affecting the accurate depiction of
character anatomy in much higher resolution, extending its generation
capability over 20MP with proper methods. Finally, we propose the refined
multi-level captions, covering all tags and various natural language captions
as a critical factor for model development. Through extensive analysis and
experiments, Illustrious demonstrates state-of-the-art performance in terms of
animation style, outperforming widely-used models in illustration domains,
propelling easier customization and personalization with nature of open source.
We plan to publicly release updated Illustrious model series sequentially as
well as sustainable plans for improvements.
Law of the Weakest Link: Cross Capabilities of Large Language Models
Authors: name: Ming Zhong, name: Aston Zhang, name: Xuewei Wang, name: Rui Hou, name: Wenhan Xiong, name: Chenguang Zhu, name: Zhengxing Chen, name: Liang Tan, name: Chloe Bi, name: Mike Lewis, name: Sravya Popuri, name: Sharan Narang, name: Melanie Kambadur, name: Dhruv Mahajan, name: Sergey Edunov, name: Jiawei Han, name: Laurens van der Maaten
Published: 2024-09-30 05:12:01

The development and evaluation of Large Language Models (LLMs) have largely
focused on individual capabilities. However, this overlooks the intersection of
multiple abilities across different types of expertise that are often required
for real-world tasks, which we term cross capabilities. To systematically
explore this concept, we first define seven core individual capabilities and
then pair them to form seven common cross capabilities, each supported by a
manually constructed taxonomy. Building on these definitions, we introduce
CrossEval, a benchmark comprising 1,400 human-annotated prompts, with 100
prompts for each individual and cross capability. To ensure reliable
evaluation, we involve expert annotators to assess 4,200 model responses,
gathering 8,400 human ratings with detailed explanations to serve as reference
examples. Our findings reveal that, in both static evaluations and attempts to
enhance specific abilities, current LLMs consistently exhibit the "Law of the
Weakest Link," where cross-capability performance is significantly constrained
by the weakest component. Specifically, across 58 cross-capability scores from
17 models, 38 scores are lower than all individual capabilities, while 20 fall
between strong and weak, but closer to the weaker ability. These results
highlight the under-performance of LLMs in cross-capability tasks, making the
identification and improvement of the weakest capabilities a critical priority
for future research to optimize performance in complex, multi-dimensional
scenarios.
Flex3D: Feed-Forward 3D Generation With Flexible Reconstruction Model
  And Input View Curation
Authors: name: Junlin Han, name: Jianyuan Wang, name: Andrea Vedaldi, name: Philip Torr, name: Filippos Kokkinos
Published: 2024-10-01 17:29:43

Generating high-quality 3D content from text, single images, or sparse view
images remains a challenging task with broad applications.Existing methods
typically employ multi-view diffusion models to synthesize multi-view images,
followed by a feed-forward process for 3D reconstruction. However, these
approaches are often constrained by a small and fixed number of input views,
limiting their ability to capture diverse viewpoints and, even worse, leading
to suboptimal generation results if the synthesized views are of poor quality.
To address these limitations, we propose Flex3D, a novel two-stage framework
capable of leveraging an arbitrary number of high-quality input views. The
first stage consists of a candidate view generation and curation pipeline. We
employ a fine-tuned multi-view image diffusion model and a video diffusion
model to generate a pool of candidate views, enabling a rich representation of
the target 3D object. Subsequently, a view selection pipeline filters these
views based on quality and consistency, ensuring that only the high-quality and
reliable views are used for reconstruction. In the second stage, the curated
views are fed into a Flexible Reconstruction Model (FlexRM), built upon a
transformer architecture that can effectively process an arbitrary number of
inputs. FlemRM directly outputs 3D Gaussian points leveraging a tri-plane
representation, enabling efficient and detailed 3D generation. Through
extensive exploration of design and training strategies, we optimize FlexRM to
achieve superior performance in both reconstruction and generation tasks. Our
results demonstrate that Flex3D achieves state-of-the-art performance, with a
user study winning rate of over 92% in 3D generation tasks when compared to
several of the latest feed-forward 3D generative models.
TPI-LLM: Serving 70B-scale LLMs Efficiently on Low-resource Edge Devices
Authors: name: Zonghang Li, name: Wenjiao Feng, name: Mohsen Guizani, name: Hongfang Yu
Published: 2024-10-01 09:18:56

Large model inference is shifting from cloud to edge due to concerns about
the privacy of user interaction data. However, edge devices often struggle with
limited computing power, memory, and bandwidth, requiring collaboration across
multiple devices to run and speed up LLM inference. Pipeline parallelism, the
mainstream solution, is inefficient for single-user scenarios, while tensor
parallelism struggles with frequent communications. In this paper, we argue
that tensor parallelism can be more effective than pipeline on low-resource
devices, and present a compute- and memory-efficient tensor parallel inference
system, named TPI-LLM, to serve 70B-scale models. TPI-LLM keeps sensitive raw
data local in the users' devices and introduces a sliding window memory
scheduler to dynamically manage layer weights during inference, with disk I/O
latency overlapped with the computation and communication. This allows larger
models to run smoothly on memory-limited devices. We analyze the communication
bottleneck and find that link latency, not bandwidth, emerges as the main
issue, so a star-based allreduce algorithm is implemented. Through extensive
experiments on both emulated and real testbeds, TPI-LLM demonstrated over 80%
less time-to-first-token and token latency compared to Accelerate, and over 90%
compared to Transformers and Galaxy, while cutting the peak memory footprint of
Llama 2-70B by 90%, requiring only 3.1 GB of memory for 70B-scale models.
Helpful DoggyBot: Open-World Object Fetching using Legged Robots and
  Vision-Language Models
Authors: name: Qi Wu, name: Zipeng Fu, name: Xuxin Cheng, name: Xiaolong Wang, name: Chelsea Finn
Published: 2024-09-30 20:58:38

Learning-based methods have achieved strong performance for quadrupedal
locomotion. However, several challenges prevent quadrupeds from learning
helpful indoor skills that require interaction with environments and humans:
lack of end-effectors for manipulation, limited semantic understanding using
only simulation data, and low traversability and reachability in indoor
environments. We present a system for quadrupedal mobile manipulation in indoor
environments. It uses a front-mounted gripper for object manipulation, a
low-level controller trained in simulation using egocentric depth for agile
skills like climbing and whole-body tilting, and pre-trained vision-language
models (VLMs) with a third-person fisheye and an egocentric RGB camera for
semantic understanding and command generation. We evaluate our system in two
unseen environments without any real-world data collection or training. Our
system can zero-shot generalize to these environments and complete tasks, like
following user's commands to fetch a randomly placed stuff toy after climbing
over a queen-sized bed, with a 60% success rate. Project website:
https://helpful-doggybot.github.io/
DressRecon: Freeform 4D Human Reconstruction from Monocular Video
Authors: name: Jeff Tan, name: Donglai Xiang, name: Shubham Tulsiani, name: Deva Ramanan, name: Gengshan Yang
Published: 2024-09-30 17:59:15

We present a method to reconstruct time-consistent human body models from
monocular videos, focusing on extremely loose clothing or handheld object
interactions. Prior work in human reconstruction is either limited to tight
clothing with no object interactions, or requires calibrated multi-view
captures or personalized template scans which are costly to collect at scale.
Our key insight for high-quality yet flexible reconstruction is the careful
combination of generic human priors about articulated body shape (learned from
large-scale training data) with video-specific articulated "bag-of-bones"
deformation (fit to a single video via test-time optimization). We accomplish
this by learning a neural implicit model that disentangles body versus clothing
deformations as separate motion model layers. To capture subtle geometry of
clothing, we leverage image-based priors such as human body pose, surface
normals, and optical flow during optimization. The resulting neural fields can
be extracted into time-consistent meshes, or further optimized as explicit 3D
Gaussians for high-fidelity interactive rendering. On datasets with highly
challenging clothing deformations and object interactions, DressRecon yields
higher-fidelity 3D reconstructions than prior art. Project page:
https://jefftan969.github.io/dressrecon/
ACE: All-round Creator and Editor Following Instructions via Diffusion
  Transformer
Authors: name: Zhen Han, name: Zeyinzi Jiang, name: Yulin Pan, name: Jingfeng Zhang, name: Chaojie Mao, name: Chenwei Xie, name: Yu Liu, name: Jingren Zhou
Published: 2024-09-30 17:56:27

Diffusion models have emerged as a powerful generative technology and have
been found to be applicable in various scenarios. Most existing foundational
diffusion models are primarily designed for text-guided visual generation and
do not support multi-modal conditions, which are essential for many visual
editing tasks. This limitation prevents these foundational diffusion models
from serving as a unified model in the field of visual generation, like GPT-4
in the natural language processing field. In this work, we propose ACE, an
All-round Creator and Editor, which achieves comparable performance compared to
those expert models in a wide range of visual generation tasks. To achieve this
goal, we first introduce a unified condition format termed Long-context
Condition Unit (LCU), and propose a novel Transformer-based diffusion model
that uses LCU as input, aiming for joint training across various generation and
editing tasks. Furthermore, we propose an efficient data collection approach to
address the issue of the absence of available training data. It involves
acquiring pairwise images with synthesis-based or clustering-based pipelines
and supplying these pairs with accurate textual instructions by leveraging a
fine-tuned multi-modal large language model. To comprehensively evaluate the
performance of our model, we establish a benchmark of manually annotated pairs
data across a variety of visual generation tasks. The extensive experimental
results demonstrate the superiority of our model in visual generation fields.
Thanks to the all-in-one capabilities of our model, we can easily build a
multi-modal chat system that responds to any interactive request for image
creation using a single model to serve as the backend, avoiding the cumbersome
pipeline typically employed in visual agents. Code and models will be available
on the project page: https://ali-vilab.github.io/ace-page/.