Orionnnn/hub-stats huggingface dataset:
Towards Building Specialized Generalist AI with System 1 and System 2
  Fusion
Authors: name: Kaiyan Zhang, name: Biqing Qi, name: Bowen Zhou
Published: 2024-07-11 16:23:16

In this perspective paper, we introduce the concept of Specialized Generalist
Artificial Intelligence (SGAI or simply SGI) as a crucial milestone toward
Artificial General Intelligence (AGI). Compared to directly scaling general
abilities, SGI is defined as AI that specializes in at least one task,
surpassing human experts, while also retaining general abilities. This fusion
path enables SGI to rapidly achieve high-value areas. We categorize SGI into
three stages based on the level of mastery over professional skills and
generality performance. Additionally, we discuss the necessity of SGI in
addressing issues associated with large language models, such as their
insufficient generality, specialized capabilities, uncertainty in innovation,
and practical applications. Furthermore, we propose a conceptual framework for
developing SGI that integrates the strengths of Systems 1 and 2 cognitive
processing. This framework comprises three layers and four key components,
which focus on enhancing individual abilities and facilitating collaborative
evolution. We conclude by summarizing the potential challenges and suggesting
future directions. We hope that the proposed SGI will provide insights into
further research and applications towards achieving AGI.
Map It Anywhere (MIA): Empowering Bird's Eye View Mapping using
  Large-scale Public Data
Authors: name: Cherie Ho, name: Jiaye Zou, name: Omar Alama, name: Sai Mitheran Jagadesh Kumar, name: Benjamin Chiang, name: Taneesh Gupta, name: Chen Wang, name: Nikhil Keetha, name: Katia Sycara, name: Sebastian Scherer
Published: 2024-07-11 17:57:22

Top-down Bird's Eye View (BEV) maps are a popular representation for ground
robot navigation due to their richness and flexibility for downstream tasks.
While recent methods have shown promise for predicting BEV maps from
First-Person View (FPV) images, their generalizability is limited to small
regions captured by current autonomous vehicle-based datasets. In this context,
we show that a more scalable approach towards generalizable map prediction can
be enabled by using two large-scale crowd-sourced mapping platforms, Mapillary
for FPV images and OpenStreetMap for BEV semantic maps. We introduce Map It
Anywhere (MIA), a data engine that enables seamless curation and modeling of
labeled map prediction data from existing open-source map platforms. Using our
MIA data engine, we display the ease of automatically collecting a dataset of
1.2 million pairs of FPV images & BEV maps encompassing diverse geographies,
landscapes, environmental factors, camera models & capture scenarios. We
further train a simple camera model-agnostic model on this data for BEV map
prediction. Extensive evaluations using established benchmarks and our dataset
show that the data curated by MIA enables effective pretraining for
generalizable BEV map prediction, with zero-shot performance far exceeding
baselines trained on existing datasets by 35%. Our analysis highlights the
promise of using large-scale public maps for developing & testing generalizable
BEV perception, paving the way for more robust autonomous navigation.
Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive
  Low-Rank Gradients
Authors: name: Zhenyu Zhang, name: Ajay Jaiswal, name: Lu Yin, name: Shiwei Liu, name: Jiawei Zhao, name: Yuandong Tian, name: Zhangyang Wang
Published: 2024-07-11 08:42:58

Training Large Language Models (LLMs) is memory-intensive due to the large
number of parameters and associated optimization states. GaLore, a recent
method, reduces memory usage by projecting weight gradients into a low-rank
subspace without compromising performance. However, GaLore relies on
time-consuming Singular Value Decomposition (SVD) operations to identify the
subspace, and the frequent subspace updates lead to significant training time
overhead. Moreover, GaLore offers minimal improvements in accuracy and
efficiency compared to LoRA in more accessible fine-tuning scenarios. To
address these limitations, we introduce Q-Galore, a novel approach that
substantially reduces memory usage by combining quantization and low-rank
projection, surpassing the benefits of GaLore. Our method is based on two key
observations: (i) the gradient subspace exhibits diverse properties, with some
layers converging early in training while others are subject to frequent
changes; (ii) the projection matrices are highly resilient to low-bit
quantization. Leveraging these insights, Q-GaLore adaptively updates the
gradient subspace based on its convergence statistics, achieving comparable
performance while significantly reducing the number of SVD operations. We
maintain the projection matrices in INT4 format and weights in INT8 format,
incorporating stochastic rounding to capture accumulated gradient information.
This approach enables a high-precision training trajectory using only
low-precision weights. We demonstrate that Q-GaLore achieves highly competitive
performance with exceptional memory efficiency. At pre-training, Q-GaLore
facilitates training a LLaMA-7B model from scratch on a single NVIDIA RTX 4060
Ti with only 16 GB memory. At fine-tuning, it reduces memory consumption by up
to 50% compared to LoRA and GaLore, while consistently outperforming QLoRA at
the same memory cost.
GTA: A Benchmark for General Tool Agents
Authors: name: Jize Wang, name: Zerun Ma, name: Yining Li, name: Songyang Zhang, name: Cailian Chen, name: Kai Chen, name: Xinyi Le
Published: 2024-07-11 17:50:09

Significant focus has been placed on integrating large language models (LLMs)
with various tools in developing general-purpose agents. This poses a challenge
to LLMs' tool-use capabilities. However, there are evident gaps between
existing tool-use evaluations and real-world scenarios. Current evaluations
often use AI-generated queries, single-step tasks, dummy tools, and text-only
interactions, failing to reveal the agents' real-world problem-solving
abilities effectively. To address this, we propose GTA, a benchmark for General
Tool Agents, featuring three main aspects: (i) Real user queries: human-written
queries with simple real-world objectives but implicit tool-use, requiring the
LLM to reason the suitable tools and plan the solution steps. (ii) Real
deployed tools: an evaluation platform equipped with tools across perception,
operation, logic, and creativity categories to evaluate the agents' actual task
execution performance. (iii) Real multimodal inputs: authentic image files,
such as spatial scenes, web page screenshots, tables, code snippets, and
printed/handwritten materials, used as the query contexts to align with
real-world scenarios closely. We design 229 real-world tasks and executable
tool chains to evaluate mainstream LLMs. Our findings show that real-world user
queries are challenging for existing LLMs, with GPT-4 completing less than 50%
of the tasks and most LLMs achieving below 25%. This evaluation reveals the
bottlenecks in the tool-use capabilities of current LLMs in real-world
scenarios, which provides future direction for advancing general-purpose tool
agents. The code and dataset are available at
https://github.com/open-compass/GTA.
MAVIS: Mathematical Visual Instruction Tuning
Authors: name: Renrui Zhang, name: Xinyu Wei, name: Dongzhi Jiang, name: Yichi Zhang, name: Ziyu Guo, name: Chengzhuo Tong, name: Jiaming Liu, name: Aojun Zhou, name: Bin Wei, name: Shanghang Zhang, name: Peng Gao, name: Hongsheng Li
Published: 2024-07-11 17:59:47

Multi-modal Large Language Models (MLLMs) have recently emerged as a
significant focus in academia and industry. Despite their proficiency in
general multi-modal scenarios, the mathematical problem-solving capabilities in
visual contexts remain insufficiently explored. We identify three key areas
within MLLMs that need to be improved: visual encoding of math diagrams,
diagram-language alignment, and mathematical reasoning skills. This draws forth
an urgent demand for large-scale, high-quality data and training pipelines in
visual mathematics. In this paper, we propose MAVIS, the first MAthematical
VISual instruction tuning paradigm for MLLMs, involving a series of
mathematical visual datasets and specialized MLLMs. Targeting the three issues,
MAVIS contains three progressive training stages from scratch. First, we curate
MAVIS-Caption, consisting of 558K diagram-caption pairs, to fine-tune a
math-specific vision encoder (CLIP-Math) through contrastive learning, tailored
for improved diagram visual encoding. Second, we utilize MAVIS-Caption to align
the CLIP-Math with a large language model (LLM) by a projection layer,
enhancing vision-language alignment in mathematical domains. Third, we
introduce MAVIS-Instruct, including 900K meticulously collected and annotated
visual math problems, which is adopted to finally instruct-tune the MLLM for
robust mathematical reasoning skills. In MAVIS-Instruct, we incorporate
complete chain-of-thought (CoT) rationales for each problem, and minimize
textual redundancy, thereby concentrating the model towards the visual
elements. Data and Models are released at https://github.com/ZrrSkywalker/MAVIS
Skywork-Math: Data Scaling Laws for Mathematical Reasoning in Large
  Language Models -- The Story Goes On
Authors: name: Liang Zeng, name: Liangjun Zhong, name: Liang Zhao, name: Tianwen Wei, name: Liu Yang, name: Jujie He, name: Cheng Cheng, name: Rui Hu, name: Yang Liu, name: Shuicheng Yan, name: Han Fang, name: Yahui Zhou
Published: 2024-07-11 09:56:51

In this paper, we investigate the underlying factors that potentially enhance
the mathematical reasoning capabilities of large language models (LLMs). We
argue that the data scaling law for math reasoning capabilities in modern LLMs
is far from being saturated, highlighting how the model's quality improves with
increases in data quantity. To support this claim, we introduce the
Skywork-Math model series, supervised fine-tuned (SFT) on common 7B LLMs using
our proposed 2.5M-instance Skywork-MathQA dataset. Skywork-Math 7B has achieved
impressive accuracies of 51.2% on the competition-level MATH benchmark and
83.9% on the GSM8K benchmark using only SFT data, outperforming an early
version of GPT-4 on MATH. The superior performance of Skywork-Math models
contributes to our novel two-stage data synthesis and model SFT pipelines,
which include three different augmentation methods and a diverse seed problem
set, ensuring both the quantity and quality of Skywork-MathQA dataset across
varying difficulty levels. Most importantly, we provide several practical
takeaways to enhance math reasoning abilities in LLMs for both research and
industry applications.
Video Diffusion Alignment via Reward Gradients
Authors: name: Mihir Prabhudesai, name: Russell Mendonca, name: Zheyang Qin, name: Katerina Fragkiadaki, name: Deepak Pathak
Published: 2024-07-11 17:59:45

We have made significant progress towards building foundational video
diffusion models. As these models are trained using large-scale unsupervised
data, it has become crucial to adapt these models to specific downstream tasks.
Adapting these models via supervised fine-tuning requires collecting target
datasets of videos, which is challenging and tedious. In this work, we utilize
pre-trained reward models that are learned via preferences on top of powerful
vision discriminative models to adapt video diffusion models. These models
contain dense gradient information with respect to generated RGB pixels, which
is critical to efficient learning in complex search spaces, such as videos. We
show that backpropagating gradients from these reward models to a video
diffusion model can allow for compute and sample efficient alignment of the
video diffusion model. We show results across a variety of reward models and
video diffusion models, demonstrating that our approach can learn much more
efficiently in terms of reward queries and computation than prior gradient-free
approaches. Our code, model weights,and more visualization are available at
https://vader-vid.github.io.
OmniNOCS: A unified NOCS dataset and model for 3D lifting of 2D objects
Authors: name: Akshay Krishnan, name: Abhijit Kundu, name: Kevis-Kokitsi Maninis, name: James Hays, name: Matthew Brown
Published: 2024-07-11 17:49:05

We propose OmniNOCS, a large-scale monocular dataset with 3D Normalized
Object Coordinate Space (NOCS) maps, object masks, and 3D bounding box
annotations for indoor and outdoor scenes. OmniNOCS has 20 times more object
classes and 200 times more instances than existing NOCS datasets (NOCS-Real275,
Wild6D). We use OmniNOCS to train a novel, transformer-based monocular NOCS
prediction model (NOCSformer) that can predict accurate NOCS, instance masks
and poses from 2D object detections across diverse classes. It is the first
NOCS model that can generalize to a broad range of classes when prompted with
2D boxes. We evaluate our model on the task of 3D oriented bounding box
prediction, where it achieves comparable results to state-of-the-art 3D
detection methods such as Cube R-CNN. Unlike other 3D detection methods, our
model also provides detailed and accurate 3D object shape and segmentation. We
propose a novel benchmark for the task of NOCS prediction based on OmniNOCS,
which we hope will serve as a useful baseline for future work in this area. Our
dataset and code will be at the project website: https://omninocs.github.io.
Live2Diff: Live Stream Translation via Uni-directional Attention in
  Video Diffusion Models
Authors: name: Zhening Xing, name: Gereon Fox, name: Yanhong Zeng, name: Xingang Pan, name: Mohamed Elgharib, name: Christian Theobalt, name: Kai Chen
Published: 2024-07-11 17:34:51

Large Language Models have shown remarkable efficacy in generating streaming
data such as text and audio, thanks to their temporally uni-directional
attention mechanism, which models correlations between the current token and
previous tokens. However, video streaming remains much less explored, despite a
growing need for live video processing. State-of-the-art video diffusion models
leverage bi-directional temporal attention to model the correlations between
the current frame and all the surrounding (i.e. including future) frames, which
hinders them from processing streaming videos. To address this problem, we
present Live2Diff, the first attempt at designing a video diffusion model with
uni-directional temporal attention, specifically targeting live streaming video
translation. Compared to previous works, our approach ensures temporal
consistency and smoothness by correlating the current frame with its
predecessors and a few initial warmup frames, without any future frames.
Additionally, we use a highly efficient denoising scheme featuring a KV-cache
mechanism and pipelining, to facilitate streaming video translation at
interactive framerates. Extensive experiments demonstrate the effectiveness of
the proposed attention mechanism and pipeline, outperforming previous methods
in terms of temporal smoothness and/or efficiency.
The Synergy between Data and Multi-Modal Large Language Models: A Survey
  from Co-Development Perspective
Authors: name: Zhen Qin, name: Daoyuan Chen, name: Wenhao Zhang, name: Liuyi Yao, name: Yilun Huang, name: Bolin Ding, name: Yaliang Li, name: Shuiguang Deng
Published: 2024-07-11 15:08:11

The rapid development of large language models (LLMs) has been witnessed in
recent years. Based on the powerful LLMs, multi-modal LLMs (MLLMs) extend the
modality from text to a broader spectrum of domains, attracting widespread
attention due to the broader range of application scenarios. As LLMs and MLLMs
rely on vast amounts of model parameters and data to achieve emergent
capabilities, the importance of data is receiving increasingly widespread
attention and recognition. Tracing and analyzing recent data-oriented works for
MLLMs, we find that the development of models and data is not two separate
paths but rather interconnected. On the one hand, vaster and higher-quality
data contribute to better performance of MLLMs, on the other hand, MLLMs can
facilitate the development of data. The co-development of multi-modal data and
MLLMs requires a clear view of 1) at which development stage of MLLMs can
specific data-centric approaches be employed to enhance which capabilities, and
2) by utilizing which capabilities and acting as which roles can models
contribute to multi-modal data. To promote the data-model co-development for
MLLM community, we systematically review existing works related to MLLMs from
the data-model co-development perspective. A regularly maintained project
associated with this survey is accessible at
https://github.com/modelscope/data-juicer/blob/main/docs/awesome_llm_data.md.
SEED-Story: Multimodal Long Story Generation with Large Language Model
Authors: name: Shuai Yang, name: Yuying Ge, name: Yang Li, name: Yukang Chen, name: Yixiao Ge, name: Ying Shan, name: Yingcong Chen
Published: 2024-07-11 17:21:03

With the remarkable advancements in image generation and open-form text
generation, the creation of interleaved image-text content has become an
increasingly intriguing field. Multimodal story generation, characterized by
producing narrative texts and vivid images in an interleaved manner, has
emerged as a valuable and practical task with broad applications. However, this
task poses significant challenges, as it necessitates the comprehension of the
complex interplay between texts and images, and the ability to generate long
sequences of coherent, contextually relevant texts and visuals. In this work,
we propose SEED-Story, a novel method that leverages a Multimodal Large
Language Model (MLLM) to generate extended multimodal stories. Our model, built
upon the powerful comprehension capability of MLLM, predicts text tokens as
well as visual tokens, which are subsequently processed with an adapted visual
de-tokenizer to produce images with consistent characters and styles. We
further propose multimodal attention sink mechanism to enable the generation of
stories with up to 25 sequences (only 10 for training) in a highly efficient
autoregressive manner. Additionally, we present a large-scale and
high-resolution dataset named StoryStream for training our model and
quantitatively evaluating the task of multimodal story generation in various
aspects.
Generalizable Implicit Motion Modeling for Video Frame Interpolation
Authors: name: Zujin Guo, name: Wei Li, name: Chen Change Loy
Published: 2024-07-11 17:13:15

Motion modeling is critical in flow-based Video Frame Interpolation (VFI).
Existing paradigms either consider linear combinations of bidirectional flows
or directly predict bilateral flows for given timestamps without exploring
favorable motion priors, thus lacking the capability of effectively modeling
spatiotemporal dynamics in real-world videos. To address this limitation, in
this study, we introduce Generalizable Implicit Motion Modeling (GIMM), a novel
and effective approach to motion modeling for VFI. Specifically, to enable GIMM
as an effective motion modeling paradigm, we design a motion encoding pipeline
to model spatiotemporal motion latent from bidirectional flows extracted from
pre-trained flow estimators, effectively representing input-specific motion
priors. Then, we implicitly predict arbitrary-timestep optical flows within two
adjacent input frames via an adaptive coordinate-based neural network, with
spatiotemporal coordinates and motion latent as inputs. Our GIMM can be
smoothly integrated with existing flow-based VFI works without further
modifications. We show that GIMM performs better than the current state of the
art on the VFI benchmarks.
Autoregressive Speech Synthesis without Vector Quantization
Authors: name: Lingwei Meng, name: Long Zhou, name: Shujie Liu, name: Sanyuan Chen, name: Bing Han, name: Shujie Hu, name: Yanqing Liu, name: Jinyu Li, name: Sheng Zhao, name: Xixin Wu, name: Helen Meng, name: Furu Wei
Published: 2024-07-11 14:36:53

We present MELLE, a novel continuous-valued tokens based language modeling
approach for text to speech synthesis (TTS). MELLE autoregressively generates
continuous mel-spectrogram frames directly from text condition, bypassing the
need for vector quantization, which are originally designed for audio
compression and sacrifice fidelity compared to mel-spectrograms. Specifically,
(i) instead of cross-entropy loss, we apply regression loss with a proposed
spectrogram flux loss function to model the probability distribution of the
continuous-valued tokens. (ii) we have incorporated variational inference into
MELLE to facilitate sampling mechanisms, thereby enhancing the output diversity
and model robustness. Experiments demonstrate that, compared to the two-stage
codec language models VALL-E and its variants, the single-stage MELLE mitigates
robustness issues by avoiding the inherent flaws of sampling discrete codes,
achieves superior performance across multiple metrics, and, most importantly,
offers a more streamlined paradigm. See https://aka.ms/melle for demos of our
work.
WildGaussians: 3D Gaussian Splatting in the Wild
Authors: name: Jonas Kulhanek, name: Songyou Peng, name: Zuzana Kukelova, name: Marc Pollefeys, name: Torsten Sattler
Published: 2024-07-11 12:41:32

While the field of 3D scene reconstruction is dominated by NeRFs due to their
photorealistic quality, 3D Gaussian Splatting (3DGS) has recently emerged,
offering similar quality with real-time rendering speeds. However, both methods
primarily excel with well-controlled 3D scenes, while in-the-wild data -
characterized by occlusions, dynamic objects, and varying illumination -
remains challenging. NeRFs can adapt to such conditions easily through
per-image embedding vectors, but 3DGS struggles due to its explicit
representation and lack of shared parameters. To address this, we introduce
WildGaussians, a novel approach to handle occlusions and appearance changes
with 3DGS. By leveraging robust DINO features and integrating an appearance
modeling module within 3DGS, our method achieves state-of-the-art results. We
demonstrate that WildGaussians matches the real-time rendering speed of 3DGS
while surpassing both 3DGS and NeRF baselines in handling in-the-wild data, all
within a simple architectural framework.
DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal
  Perception
Authors: name: Xiaotong Li, name: Fan Zhang, name: Haiwen Diao, name: Yueze Wang, name: Xinlong Wang, name: Ling-Yu Duan
Published: 2024-07-11 08:48:06

Existing Multimodal Large Language Models (MLLMs) increasingly emphasize
complex understanding of various visual elements, including multiple objects,
text information, and spatial relations. Their development for comprehensive
visual perception hinges on the availability of high-quality image-text
datasets that offer diverse visual elements and throughout image descriptions.
However, the scarcity of such hyper-detailed datasets currently hinders
progress within the MLLM community. The bottleneck stems from the limited
perceptual capabilities of current caption engines, which fall short in
providing complete and accurate annotations. To facilitate the cutting-edge
research of MLLMs on comprehensive vision perception, we thereby propose
Perceptual Fusion, using a low-budget but highly effective caption engine for
complete and accurate image descriptions. Specifically, Perceptual Fusion
integrates diverse perception experts as image priors to provide explicit
information on visual elements and adopts an efficient MLLM as a centric pivot
to mimic advanced MLLMs' perception abilities. We carefully select 1M highly
representative images from uncurated LAION dataset and generate dense
descriptions using our engine, dubbed DenseFusion-1M. Extensive experiments
validate that our engine outperforms its counterparts, where the resulting
dataset significantly improves the perception and cognition abilities of
existing MLLMs across diverse vision-language benchmarks, especially with
high-resolution images as inputs. The dataset and code are publicly available
at https://github.com/baaivision/DenseFusion.