Orionnnn/hub-stats huggingface dataset:
PanGu-Draw: Advancing Resource-Efficient Text-to-Image Synthesis with
  Time-Decoupled Training and Reusable Coop-Diffusion
Authors: name: Guansong Lu, name: Yuanfan Guo, name: Jianhua Han, name: Minzhe Niu, name: Yihan Zeng, name: Songcen Xu, name: Zeyi Huang, name: Zhao Zhong, name: Wei Zhang, name: Hang Xu
Published: 2023-12-27 09:21:45

Current large-scale diffusion models represent a giant leap forward in
conditional image synthesis, capable of interpreting diverse cues like text,
human poses, and edges. However, their reliance on substantial computational
resources and extensive data collection remains a bottleneck. On the other
hand, the integration of existing diffusion models, each specialized for
different controls and operating in unique latent spaces, poses a challenge due
to incompatible image resolutions and latent space embedding structures,
hindering their joint use. Addressing these constraints, we present
"PanGu-Draw", a novel latent diffusion model designed for resource-efficient
text-to-image synthesis that adeptly accommodates multiple control signals. We
first propose a resource-efficient Time-Decoupling Training Strategy, which
splits the monolithic text-to-image model into structure and texture
generators. Each generator is trained using a regimen that maximizes data
utilization and computational efficiency, cutting data preparation by 48% and
reducing training resources by 51%. Secondly, we introduce "Coop-Diffusion", an
algorithm that enables the cooperative use of various pre-trained diffusion
models with different latent spaces and predefined resolutions within a unified
denoising process. This allows for multi-control image synthesis at arbitrary
resolutions without the necessity for additional data or retraining. Empirical
validations of Pangu-Draw show its exceptional prowess in text-to-image and
multi-control image generation, suggesting a promising direction for future
model training efficiencies and generation versatility. The largest 5B T2I
PanGu-Draw model is released on the Ascend platform. Project page:
https://pangu-draw.github.io
I2V-Adapter: A General Image-to-Video Adapter for Video Diffusion Models
Authors: name: Xun Guo, name: Mingwu Zheng, name: Liang Hou, name: Yuan Gao, name: Yufan Deng, name: Chongyang Ma, name: Weiming Hu, name: Zhengjun Zha, name: Haibin Huang, name: Pengfei Wan, name: Di Zhang
Published: 2023-12-27 19:11:50

In the rapidly evolving domain of digital content generation, the focus has
shifted from text-to-image (T2I) models to more advanced video diffusion
models, notably text-to-video (T2V) and image-to-video (I2V). This paper
addresses the intricate challenge posed by I2V: converting static images into
dynamic, lifelike video sequences while preserving the original image fidelity.
Traditional methods typically involve integrating entire images into diffusion
processes or using pretrained encoders for cross attention. However, these
approaches often necessitate altering the fundamental weights of T2I models,
thereby restricting their reusability. We introduce a novel solution, namely
I2V-Adapter, designed to overcome such limitations. Our approach preserves the
structural integrity of T2I models and their inherent motion modules. The
I2V-Adapter operates by processing noised video frames in parallel with the
input image, utilizing a lightweight adapter module. This module acts as a
bridge, efficiently linking the input to the model's self-attention mechanism,
thus maintaining spatial details without requiring structural changes to the
T2I model. Moreover, I2V-Adapter requires only a fraction of the parameters of
conventional models and ensures compatibility with existing community-driven
T2I models and controlling tools. Our experimental results demonstrate
I2V-Adapter's capability to produce high-quality video outputs. This
performance, coupled with its versatility and reduced need for trainable
parameters, represents a substantial advancement in the field of AI-driven
video generation, particularly for creative applications.
The LLM Surgeon
Authors: name: Tycho F. A. van der Ouderaa, name: Markus Nagel, name: Mart van Baalen, name: Yuki M. Asano, name: Tijmen Blankevoort
Published: 2023-12-28 18:59:09

State-of-the-art language models are becoming increasingly large in an effort
to achieve the highest performance on large corpora of available textual data.
However, the sheer size of the Transformer architectures makes it difficult to
deploy models within computational, environmental or device-specific
constraints. We explore data-driven compression of existing pretrained models
as an alternative to training smaller models from scratch. To do so, we scale
Kronecker-factored curvature approximations of the target loss landscape to
large language models. In doing so, we can compute both the dynamic allocation
of structures that can be removed as well as updates of remaining weights that
account for the removal. We provide a general framework for unstructured,
semi-structured and structured pruning and improve upon weight updates to
capture more correlations between weights, while remaining computationally
efficient. Experimentally, our method can prune rows and columns from a range
of OPT models and Llamav2-7B by 20%-30%, with a negligible loss in performance,
and achieve state-of-the-art results in unstructured and semi-structured
pruning of large language models.
Generative AI for Math: Part I -- MathPile: A Billion-Token-Scale
  Pretraining Corpus for Math
Authors: name: Zengzhi Wang, name: Rui Xia, name: Pengfei Liu
Published: 2023-12-28 16:55:40

High-quality, large-scale corpora are the cornerstone of building foundation
models. In this work, we introduce MathPile, a diverse and
high-quality math-centric corpus comprising about 9.5 billion tokens.
Throughout its creation, we adhered to the principle of ``less is
more'', firmly believing in the supremacy of data quality over quantity, even
in the pre-training phase. Our meticulous data collection and processing
efforts included a complex suite of preprocessing, prefiltering, language
identification, cleaning, filtering, and deduplication, ensuring the high
quality of our corpus. Furthermore, we performed data contamination detection
on downstream benchmark test sets to eliminate duplicates. We hope our
MathPile can help to enhance the mathematical reasoning abilities of
language models. We plan to open-source different versions of \mathpile with
the scripts used for processing, to facilitate future developments in this
field.
MobileVLM : A Fast, Reproducible and Strong Vision Language Assistant
  for Mobile Devices
Authors: name: Xiangxiang Chu, name: Limeng Qiao, name: Xinyang Lin, name: Shuang Xu, name: Yang Yang, name: Yiming Hu, name: Fei Wei, name: Xinyu Zhang, name: Bo Zhang, name: Xiaolin Wei, name: Chunhua Shen
Published: 2023-12-28 08:21:24

We present MobileVLM, a competent multimodal vision language model (MMVLM)
targeted to run on mobile devices. It is an amalgamation of a myriad of
architectural designs and techniques that are mobile-oriented, which comprises
a set of language models at the scale of 1.4B and 2.7B parameters, trained from
scratch, a multimodal vision model that is pre-trained in the CLIP fashion,
cross-modality interaction via an efficient projector. We evaluate MobileVLM on
several typical VLM benchmarks. Our models demonstrate on par performance
compared with a few much larger models. More importantly, we measure the
inference speed on both a Qualcomm Snapdragon 888 CPU and an NVIDIA Jeston Orin
GPU, and we obtain state-of-the-art performance of 21.5 tokens and 65.3 tokens
per second, respectively. Our code will be made available at:
https://github.com/Meituan-AutoML/MobileVLM.
TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbones
Authors: name: Zhengqing Yuan, name: Zhaoxu Li, name: Lichao Sun
Published: 2023-12-28 07:11:41

In the era of advanced multimodel learning, multimodal large language models
(MLLMs) such as GPT-4V have made remarkable strides towards bridging language
and visual elements. However, the closed-source nature and considerable
computational demand present notable challenges for universal usage and
modifications. This is where open-source MLLMs like LLaVA and MiniGPT-4 come
in, presenting groundbreaking achievements across tasks. Despite these
accomplishments, computational efficiency remains an unresolved issue, as these
models, like LLaVA-v1.5-13B, require substantial resources. Addressing these
issues, we introduce TinyGPT-V, a new-wave model marrying impressive
performance with commonplace computational capacity. It stands out by requiring
merely a 24G GPU for training and an 8G GPU or CPU for inference. Built upon
Phi-2, TinyGPT-V couples an effective language backbone with pre-trained vision
modules from BLIP-2 or CLIP. TinyGPT-V's 2.8B parameters can undergo a unique
quantisation process, suitable for local deployment and inference tasks on 8G
various devices. Our work fosters further developments for designing
cost-effective, efficient, and high-performing MLLMs, expanding their
applicability in a broad array of real-world scenarios. Furthermore this paper
proposed a new paradigm of Multimodal Large Language Model via small backbones.
Our code and training weights are placed at:
https://github.com/DLYuanGod/TinyGPT-V and
https://huggingface.co/Tyrannosaurus/TinyGPT-V respectively.
UniRef++: Segment Every Reference Object in Spatial and Temporal Spaces
Authors: name: Jiannan Wu, name: Yi Jiang, name: Bin Yan, name: Huchuan Lu, name: Zehuan Yuan, name: Ping Luo
Published: 2023-12-25 12:54:11

The reference-based object segmentation tasks, namely referring image
segmentation (RIS), few-shot image segmentation (FSS), referring video object
segmentation (RVOS), and video object segmentation (VOS), aim to segment a
specific object by utilizing either language or annotated masks as references.
Despite significant progress in each respective field, current methods are
task-specifically designed and developed in different directions, which hinders
the activation of multi-task capabilities for these tasks. In this work, we end
the current fragmented situation and propose UniRef++ to unify the four
reference-based object segmentation tasks with a single architecture. At the
heart of our approach is the proposed UniFusion module which performs
multiway-fusion for handling different tasks with respect to their specified
references. And a unified Transformer architecture is then adopted for
achieving instance-level segmentation. With the unified designs, UniRef++ can
be jointly trained on a broad range of benchmarks and can flexibly complete
multiple tasks at run-time by specifying the corresponding references. We
evaluate our unified models on various benchmarks. Extensive experimental
results indicate that our proposed UniRef++ achieves state-of-the-art
performance on RIS and RVOS, and performs competitively on FSS and VOS with a
parameter-shared network. Moreover, we showcase that the proposed UniFusion
module could be easily incorporated into the current advanced foundation model
SAM and obtain satisfactory results with parameter-efficient finetuning. Codes
and models are available at https://github.com/FoundationVision/UniRef.
LangSplat: 3D Language Gaussian Splatting
Authors: name: Minghan Qin, name: Wanhua Li, name: Jiawei Zhou, name: Haoqian Wang, name: Hanspeter Pfister
Published: 2023-12-26 15:14:37

Human lives in a 3D world and commonly uses natural language to interact with
a 3D scene. Modeling a 3D language field to support open-ended language queries
in 3D has gained increasing attention recently. This paper introduces
LangSplat, which constructs a 3D language field that enables precise and
efficient open-vocabulary querying within 3D spaces. Unlike existing methods
that ground CLIP language embeddings in a NeRF model, LangSplat advances the
field by utilizing a collection of 3D Gaussians, each encoding language
features distilled from CLIP, to represent the language field. By employing a
tile-based splatting technique for rendering language features, we circumvent
the costly rendering process inherent in NeRF. Instead of directly learning
CLIP embeddings, LangSplat first trains a scene-wise language autoencoder and
then learns language features on the scene-specific latent space, thereby
alleviating substantial memory demands imposed by explicit modeling. Existing
methods struggle with imprecise and vague 3D language fields, which fail to
discern clear boundaries between objects. We delve into this issue and propose
to learn hierarchical semantics using SAM, thereby eliminating the need for
extensively querying the language field across various scales and the
regularization of DINO features. Extensive experiments on open-vocabulary 3D
object localization and semantic segmentation demonstrate that LangSplat
significantly outperforms the previous state-of-the-art method LERF by a large
margin. Notably, LangSplat is extremely efficient, achieving a {\speed}
times speedup compared to LERF at the resolution of 1440 times 1080. We
strongly recommend readers to check out our video results at
https://langsplat.github.io
Human101: Training 100+FPS Human Gaussians in 100s from 1 View
Authors: name: Mingwei Li, name: Jiachen Tao, name: Zongxin Yang, name: Yi Yang
Published: 2023-12-23 13:41:56

Reconstructing the human body from single-view videos plays a pivotal role in
the virtual reality domain. One prevalent application scenario necessitates the
rapid reconstruction of high-fidelity 3D digital humans while simultaneously
ensuring real-time rendering and interaction. Existing methods often struggle
to fulfill both requirements. In this paper, we introduce Human101, a novel
framework adept at producing high-fidelity dynamic 3D human reconstructions
from 1-view videos by training 3D Gaussians in 100 seconds and rendering in
100+ FPS. Our method leverages the strengths of 3D Gaussian Splatting, which
provides an explicit and efficient representation of 3D humans. Standing apart
from prior NeRF-based pipelines, Human101 ingeniously applies a Human-centric
Forward Gaussian Animation method to deform the parameters of 3D Gaussians,
thereby enhancing rendering speed (i.e., rendering 1024-resolution images at an
impressive 60+ FPS and rendering 512-resolution images at 100+ FPS).
Experimental results indicate that our approach substantially eclipses current
methods, clocking up to a 10 times surge in frames per second and delivering
comparable or superior rendering quality. Code and demos will be released at
https://github.com/longxiang-ai/Human101.
Audiobox: Unified Audio Generation with Natural Language Prompts
Authors: name: Apoorv Vyas, name: Bowen Shi, name: Matthew Le, name: Andros Tjandra, name: Yi-Chiao Wu, name: Baishan Guo, name: Jiemin Zhang, name: Xinyue Zhang, name: Robert Adkins, name: William Ngan, name: Jeff Wang, name: Ivan Cruz, name: Bapi Akula, name: Akinniyi Akinyemi, name: Brian Ellis, name: Rashel Moritz, name: Yael Yungster, name: Alice Rakotoarison, name: Liang Tan, name: Chris Summers, name: Carleigh Wood, name: Joshua Lane, name: Mary Williamson, name: Wei-Ning Hsu
Published: 2023-12-25 22:24:49

Audio is an essential part of our life, but creating it often requires
expertise and is time-consuming. Research communities have made great progress
over the past year advancing the performance of large scale audio generative
models for a single modality (speech, sound, or music) through adopting more
powerful generative models and scaling data. However, these models lack
controllability in several aspects: speech generation models cannot synthesize
novel styles based on text description and are limited on domain coverage such
as outdoor environments; sound generation models only provide coarse-grained
control based on descriptions like "a person speaking" and would only generate
mumbling human voices. This paper presents Audiobox, a unified model based on
flow-matching that is capable of generating various audio modalities. We design
description-based and example-based prompting to enhance controllability and
unify speech and sound generation paradigms. We allow transcript, vocal, and
other audio styles to be controlled independently when generating speech. To
improve model generalization with limited labels, we adapt a self-supervised
infilling objective to pre-train on large quantities of unlabeled audio.
Audiobox sets new benchmarks on speech and sound generation (0.745 similarity
on Librispeech for zero-shot TTS; 0.77 FAD on AudioCaps for text-to-sound) and
unlocks new methods for generating audio with novel vocal and acoustic styles.
We further integrate Bespoke Solvers, which speeds up generation by over 25
times compared to the default ODE solver for flow-matching, without loss of
performance on several tasks. Our demo is available at
https://audiobox.metademolab.com/
HarmonyView: Harmonizing Consistency and Diversity in One-Image-to-3D
Authors: name: Sangmin Woo, name: Byeongjun Park, name: Hyojun Go, name: Jin-Young Kim, name: Changick Kim
Published: 2023-12-26 10:15:28

Recent progress in single-image 3D generation highlights the importance of
multi-view coherency, leveraging 3D priors from large-scale diffusion models
pretrained on Internet-scale images. However, the aspect of novel-view
diversity remains underexplored within the research landscape due to the
ambiguity in converting a 2D image into 3D content, where numerous potential
shapes can emerge. Here, we aim to address this research gap by simultaneously
addressing both consistency and diversity. Yet, striking a balance between
these two aspects poses a considerable challenge due to their inherent
trade-offs. This work introduces HarmonyView, a simple yet effective diffusion
sampling technique adept at decomposing two intricate aspects in single-image
3D generation: consistency and diversity. This approach paves the way for a
more nuanced exploration of the two critical dimensions within the sampling
process. Moreover, we propose a new evaluation metric based on CLIP image and
text encoders to comprehensively assess the diversity of the generated views,
which closely aligns with human evaluators' judgments. In experiments,
HarmonyView achieves a harmonious balance, demonstrating a win-win scenario in
both consistency and diversity.
One-dimensional Adapter to Rule Them All: Concepts, Diffusion Models and
  Erasing Applications
Authors: name: Mengyao Lyu, name: Yuhong Yang, name: Haiwen Hong, name: Hui Chen, name: Xuan Jin, name: Yuan He, name: Hui Xue, name: Jungong Han, name: Guiguang Ding
Published: 2023-12-26 18:08:48

The prevalent use of commercial and open-source diffusion models (DMs) for
text-to-image generation prompts risk mitigation to prevent undesired
behaviors. Existing concept erasing methods in academia are all based on full
parameter or specification-based fine-tuning, from which we observe the
following issues: 1) Generation alternation towards erosion: Parameter drift
during target elimination causes alternations and potential deformations across
all generations, even eroding other concepts at varying degrees, which is more
evident with multi-concept erased; 2) Transfer inability & deployment
inefficiency: Previous model-specific erasure impedes the flexible combination
of concepts and the training-free transfer towards other models, resulting in
linear cost growth as the deployment scenarios increase. To achieve
non-invasive, precise, customizable, and transferable elimination, we ground
our erasing framework on one-dimensional adapters to erase multiple concepts
from most DMs at once across versatile erasing applications. The
concept-SemiPermeable structure is injected as a Membrane (SPM) into any DM to
learn targeted erasing, and meantime the alteration and erosion phenomenon is
effectively mitigated via a novel Latent Anchoring fine-tuning strategy. Once
obtained, SPMs can be flexibly combined and plug-and-play for other DMs without
specific re-tuning, enabling timely and efficient adaptation to diverse
scenarios. During generation, our Facilitated Transport mechanism dynamically
regulates the permeability of each SPM to respond to different input prompts,
further minimizing the impact on other concepts. Quantitative and qualitative
results across ~40 concepts, 7 DMs and 4 erasing applications have demonstrated
the superior erasing of SPM. Our code and pre-tuned SPMs will be available on
the project page https://lyumengyao.github.io/projects/spm.
Make-A-Character: High Quality Text-to-3D Character Generation within
  Minutes
Authors: name: Jianqiang Ren, name: Chao He, name: Lin Liu, name: Jiahao Chen, name: Yutong Wang, name: Yafei Song, name: Jianfang Li, name: Tangli Xue, name: Siqi Hu, name: Tao Chen, name: Kunkun Zheng, name: Jianjing Xiang, name: Liefeng Bo
Published: 2023-12-24 08:11:39

There is a growing demand for customized and expressive 3D characters with
the emergence of AI agents and Metaverse, but creating 3D characters using
traditional computer graphics tools is a complex and time-consuming task. To
address these challenges, we propose a user-friendly framework named
Make-A-Character (Mach) to create lifelike 3D avatars from text descriptions.
The framework leverages the power of large language and vision models for
textual intention understanding and intermediate image generation, followed by
a series of human-oriented visual perception and 3D generation modules. Our
system offers an intuitive approach for users to craft controllable, realistic,
fully-realized 3D characters that meet their expectations within 2 minutes,
while also enabling easy integration with existing CG pipeline for dynamic
expressiveness. For more information, please visit the project page at
https://human3daigc.github.io/MACH/.
A Recipe for Scaling up Text-to-Video Generation with Text-free Videos
Authors: name: Xiang Wang, name: Shiwei Zhang, name: Hangjie Yuan, name: Zhiwu Qing, name: Biao Gong, name: Yingya Zhang, name: Yujun Shen, name: Changxin Gao, name: Nong Sang
Published: 2023-12-25 16:37:39

Diffusion-based text-to-video generation has witnessed impressive progress in
the past year yet still falls behind text-to-image generation. One of the key
reasons is the limited scale of publicly available data (e.g., 10M video-text
pairs in WebVid10M vs. 5B image-text pairs in LAION), considering the high cost
of video captioning. Instead, it could be far easier to collect unlabeled clips
from video platforms like YouTube. Motivated by this, we come up with a novel
text-to-video generation framework, termed TF-T2V, which can directly learn
with text-free videos. The rationale behind is to separate the process of text
decoding from that of temporal modeling. To this end, we employ a content
branch and a motion branch, which are jointly optimized with weights shared.
Following such a pipeline, we study the effect of doubling the scale of
training set (i.e., video-only WebVid10M) with some randomly collected
text-free videos and are encouraged to observe the performance improvement (FID
from 9.67 to 8.19 and FVD from 484 to 441), demonstrating the scalability of
our approach. We also find that our model could enjoy sustainable performance
gain (FID from 8.19 to 7.64 and FVD from 441 to 366) after reintroducing some
text labels for training. Finally, we validate the effectiveness and
generalizability of our ideology on both native text-to-video generation and
compositional video synthesis paradigms. Code and models will be publicly
available at https://tf-t2v.github.io/.