Orionnnn/hub-stats huggingface dataset:
VastGaussian: Vast 3D Gaussians for Large Scene Reconstruction
Authors: name: Jiaqi Lin, name: Zhihao Li, name: Xiao Tang, name: Jianzhuang Liu, name: Shiyong Liu, name: Jiayue Liu, name: Yangdi Lu, name: Xiaofei Wu, name: Songcen Xu, name: Youliang Yan, name: Wenming Yang
Published: 2024-02-27 11:40:50

Existing NeRF-based methods for large scene reconstruction often have
limitations in visual quality and rendering speed. While the recent 3D Gaussian
Splatting works well on small-scale and object-centric scenes, scaling it up to
large scenes poses challenges due to limited video memory, long optimization
time, and noticeable appearance variations. To address these challenges, we
present VastGaussian, the first method for high-quality reconstruction and
real-time rendering on large scenes based on 3D Gaussian Splatting. We propose
a progressive partitioning strategy to divide a large scene into multiple
cells, where the training cameras and point cloud are properly distributed with
an airspace-aware visibility criterion. These cells are merged into a complete
scene after parallel optimization. We also introduce decoupled appearance
modeling into the optimization process to reduce appearance variations in the
rendered images. Our approach outperforms existing NeRF-based methods and
achieves state-of-the-art results on multiple large scene datasets, enabling
fast optimization and high-fidelity real-time rendering.
Disentangled 3D Scene Generation with Layout Learning
Authors: name: Dave Epstein, name: Ben Poole, name: Ben Mildenhall, name: Alexei A. Efros, name: Aleksander Holynski
Published: 2024-02-26 18:54:15

We introduce a method to generate 3D scenes that are disentangled into their
component objects. This disentanglement is unsupervised, relying only on the
knowledge of a large pretrained text-to-image model. Our key insight is that
objects can be discovered by finding parts of a 3D scene that, when rearranged
spatially, still produce valid configurations of the same scene. Concretely,
our method jointly optimizes multiple NeRFs from scratch - each representing
its own object - along with a set of layouts that composite these objects into
scenes. We then encourage these composited scenes to be in-distribution
according to the image generator. We show that despite its simplicity, our
approach successfully generates 3D scenes decomposed into individual objects,
enabling new capabilities in text-to-3D content creation. For results and an
interactive demo, see our project page at https://dave.ml/layoutlearning/
Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion
  Latent Aligners
Authors: name: Yazhou Xing, name: Yingqing He, name: Zeyue Tian, name: Xintao Wang, name: Qifeng Chen
Published: 2024-02-27 17:57:04

Video and audio content creation serves as the core technique for the movie
industry and professional users. Recently, existing diffusion-based methods
tackle video and audio generation separately, which hinders the technique
transfer from academia to industry. In this work, we aim at filling the gap,
with a carefully designed optimization-based framework for cross-visual-audio
and joint-visual-audio generation. We observe the powerful generation ability
of off-the-shelf video or audio generation models. Thus, instead of training
the giant models from scratch, we propose to bridge the existing strong models
with a shared latent representation space. Specifically, we propose a
multimodality latent aligner with the pre-trained ImageBind model. Our latent
aligner shares a similar core as the classifier guidance that guides the
diffusion denoising process during inference time. Through carefully designed
optimization strategy and loss functions, we show the superior performance of
our method on joint video-audio generation, visual-steered audio generation,
and audio-steered visual generation tasks. The project website can be found at
https://yzxing87.github.io/Seeing-and-Hearing/
Playground v2.5: Three Insights towards Enhancing Aesthetic Quality in
  Text-to-Image Generation
Authors: name: Daiqing Li, name: Aleks Kamko, name: Ehsan Akhgari, name: Ali Sabet, name: Linmiao Xu, name: Suhail Doshi
Published: 2024-02-27 06:31:52

In this work, we share three insights for achieving state-of-the-art
aesthetic quality in text-to-image generative models. We focus on three
critical aspects for model improvement: enhancing color and contrast, improving
generation across multiple aspect ratios, and improving human-centric fine
details. First, we delve into the significance of the noise schedule in
training a diffusion model, demonstrating its profound impact on realism and
visual fidelity. Second, we address the challenge of accommodating various
aspect ratios in image generation, emphasizing the importance of preparing a
balanced bucketed dataset. Lastly, we investigate the crucial role of aligning
model outputs with human preferences, ensuring that generated images resonate
with human perceptual expectations. Through extensive analysis and experiments,
Playground v2.5 demonstrates state-of-the-art performance in terms of aesthetic
quality under various conditions and aspect ratios, outperforming both
widely-used open-source models like SDXL and Playground v2, and closed-source
commercial systems such as DALLE 3 and Midjourney v5.2. Our model is
open-source, and we hope the development of Playground v2.5 provides valuable
guidelines for researchers aiming to elevate the aesthetic quality of
diffusion-based image generation models.
OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist
  Autonomous Agents for Desktop and Web
Authors: name: Raghav Kapoor, name: Yash Parag Butala, name: Melisa Russak, name: Jing Yu Koh, name: Kiran Kamble, name: Waseem Alshikh, name: Ruslan Salakhutdinov
Published: 2024-02-27 14:47:53

For decades, human-computer interaction has fundamentally been manual. Even
today, almost all productive work done on the computer necessitates human input
at every step. Autonomous virtual agents represent an exciting step in
automating many of these menial tasks. Virtual agents would empower users with
limited technical proficiency to harness the full possibilities of computer
systems. They could also enable the efficient streamlining of numerous computer
tasks, ranging from calendar management to complex travel bookings, with
minimal human intervention. In this paper, we introduce OmniACT, the
first-of-a-kind dataset and benchmark for assessing an agent's capability to
generate executable programs to accomplish computer tasks. Our scope extends
beyond traditional web automation, covering a diverse range of desktop
applications. The dataset consists of fundamental tasks such as "Play the next
song", as well as longer horizon tasks such as "Send an email to John Doe
mentioning the time and place to meet". Specifically, given a pair of screen
image and a visually-grounded natural language task, the goal is to generate a
script capable of fully executing the task. We run several strong baseline
language model agents on our benchmark. The strongest baseline, GPT-4, performs
the best on our benchmark However, its performance level still reaches only 15%
of the human proficiency in generating executable scripts capable of completing
the task, demonstrating the challenge of our task for conventional web agents.
Our benchmark provides a platform to measure and evaluate the progress of
language model agents in automating computer tasks and motivates future work
towards building multimodal models that bridge large language models and the
visual grounding of computer screens.
EMO: Emote Portrait Alive - Generating Expressive Portrait Videos with
  Audio2Video Diffusion Model under Weak Conditions
Authors: name: Linrui Tian, name: Qi Wang, name: Bang Zhang, name: Liefeng Bo
Published: 2024-02-27 13:10:11

In this work, we tackle the challenge of enhancing the realism and
expressiveness in talking head video generation by focusing on the dynamic and
nuanced relationship between audio cues and facial movements. We identify the
limitations of traditional techniques that often fail to capture the full
spectrum of human expressions and the uniqueness of individual facial styles.
To address these issues, we propose EMO, a novel framework that utilizes a
direct audio-to-video synthesis approach, bypassing the need for intermediate
3D models or facial landmarks. Our method ensures seamless frame transitions
and consistent identity preservation throughout the video, resulting in highly
expressive and lifelike animations. Experimental results demonsrate that EMO is
able to produce not only convincing speaking videos but also singing videos in
various styles, significantly outperforming existing state-of-the-art
methodologies in terms of expressiveness and realism.
Sora: A Review on Background, Technology, Limitations, and Opportunities
  of Large Vision Models
Authors: name: Yixin Liu, name: Kai Zhang, name: Yuan Li, name: Zhiling Yan, name: Chujie Gao, name: Ruoxi Chen, name: Zhengqing Yuan, name: Yue Huang, name: Hanchi Sun, name: Jianfeng Gao, name: Lifang He, name: Lichao Sun
Published: 2024-02-27 03:30:58

Sora is a text-to-video generative AI model, released by OpenAI in February
2024. The model is trained to generate videos of realistic or imaginative
scenes from text instructions and show potential in simulating the physical
world. Based on public technical reports and reverse engineering, this paper
presents a comprehensive review of the model's background, related
technologies, applications, remaining challenges, and future directions of
text-to-video AI models. We first trace Sora's development and investigate the
underlying technologies used to build this "world simulator". Then, we describe
in detail the applications and potential impact of Sora in multiple industries
ranging from film-making and education to marketing. We discuss the main
challenges and limitations that need to be addressed to widely deploy Sora,
such as ensuring safe and unbiased video generation. Lastly, we discuss the
future development of Sora and video generation models in general, and how
advancements in the field could enable new ways of human-AI interaction,
boosting productivity and creativity of video generation.
Sora Generates Videos with Stunning Geometrical Consistency
Authors: name: Xuanyi Li, name: Daquan Zhou, name: Chenxu Zhang, name: Shaodong Wei, name: Qibin Hou, name: Ming-Ming Cheng
Published: 2024-02-27 10:49:05

The recently developed Sora model [1] has exhibited remarkable capabilities
in video generation, sparking intense discussions regarding its ability to
simulate real-world phenomena. Despite its growing popularity, there is a lack
of established metrics to evaluate its fidelity to real-world physics
quantitatively. In this paper, we introduce a new benchmark that assesses the
quality of the generated videos based on their adherence to real-world physics
principles. We employ a method that transforms the generated videos into 3D
models, leveraging the premise that the accuracy of 3D reconstruction is
heavily contingent on the video quality. From the perspective of 3D
reconstruction, we use the fidelity of the geometric constraints satisfied by
the constructed 3D models as a proxy to gauge the extent to which the generated
videos conform to real-world physics rules. Project page:
https://sora-geometrical-consistency.github.io/
Video as the New Language for Real-World Decision Making
Authors: name: Sherry Yang, name: Jacob Walker, name: Jack Parker-Holder, name: Yilun Du, name: Jake Bruce, name: Andre Barreto, name: Pieter Abbeel, name: Dale Schuurmans
Published: 2024-02-27 02:05:29

Both text and video data are abundant on the internet and support large-scale
self-supervised learning through next token or frame prediction. However, they
have not been equally leveraged: language models have had significant
real-world impact, whereas video generation has remained largely limited to
media entertainment. Yet video data captures important information about the
physical world that is difficult to express in language. To address this gap,
we discuss an under-appreciated opportunity to extend video generation to solve
tasks in the real world. We observe how, akin to language, video can serve as a
unified interface that can absorb internet knowledge and represent diverse
tasks. Moreover, we demonstrate how, like language models, video generation can
serve as planners, agents, compute engines, and environment simulators through
techniques such as in-context learning, planning and reinforcement learning. We
identify major impact opportunities in domains such as robotics, self-driving,
and science, supported by recent work that demonstrates how such advanced
capabilities in video generation are plausibly within reach. Lastly, we
identify key challenges in video generation that mitigate progress. Addressing
these challenges will enable video generation models to demonstrate unique
value alongside language models in a wider array of AI applications.
MobiLlama: Towards Accurate and Lightweight Fully Transparent GPT
Authors: name: Omkar Thawakar, name: Ashmal Vayani, name: Salman Khan, name: Hisham Cholakal, name: Rao M. Anwer, name: Michael Felsberg, name: Tim Baldwin, name: Eric P. Xing, name: Fahad Shahbaz Khan
Published: 2024-02-26 18:59:03

"Bigger the better" has been the predominant trend in recent Large Language
Models (LLMs) development. However, LLMs do not suit well for scenarios that
require on-device processing, energy efficiency, low memory footprint, and
response efficiency. These requisites are crucial for privacy, security, and
sustainable deployment. This paper explores the "less is more" paradigm by
addressing the challenge of designing accurate yet efficient Small Language
Models (SLMs) for resource constrained devices. Our primary contribution is the
introduction of an accurate and fully transparent open-source 0.5 billion
(0.5B) parameter SLM, named MobiLlama, catering to the specific needs of
resource-constrained computing with an emphasis on enhanced performance with
reduced resource demands. MobiLlama is a SLM design that initiates from a
larger model and applies a careful parameter sharing scheme to reduce both the
pre-training and the deployment cost. Our work strives to not only bridge the
gap in open-source SLMs but also ensures full transparency, where complete
training data pipeline, training code, model weights, and over 300 checkpoints
along with evaluation codes is available at :
https://github.com/mbzuai-oryx/MobiLlama.
Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts
Authors: name: Mikayel Samvelyan, name: Sharath Chandra Raparthy, name: Andrei Lupu, name: Eric Hambro, name: Aram H. Markosyan, name: Manish Bhatt, name: Yuning Mao, name: Minqi Jiang, name: Jack Parker-Holder, name: Jakob Foerster, name: Tim Rocktäschel, name: Roberta Raileanu
Published: 2024-02-26 18:47:27

As large language models (LLMs) become increasingly prevalent across many
real-world applications, understanding and enhancing their robustness to user
inputs is of paramount importance. Existing methods for identifying adversarial
prompts tend to focus on specific domains, lack diversity, or require extensive
human annotations. To address these limitations, we present Rainbow Teaming, a
novel approach for producing a diverse collection of adversarial prompts.
Rainbow Teaming casts adversarial prompt generation as a quality-diversity
problem, and uses open-ended search to generate prompts that are both effective
and diverse. It can uncover a model's vulnerabilities across a broad range of
domains including, in this paper, safety, question answering, and
cybersecurity. We also demonstrate that fine-tuning on synthetic data generated
by Rainbow Teaming improves the safety of state-of-the-art LLMs without hurting
their general capabilities and helpfulness, paving the path to open-ended
self-improvement.
StructLM: Towards Building Generalist Models for Structured Knowledge
  Grounding
Authors: name: Alex Zhuang, name: Ge Zhang, name: Tianyu Zheng, name: Xinrun Du, name: Junjie Wang, name: Weiming Ren, name: Stephen W. Huang, name: Jie Fu, name: Xiang Yue, name: Wenhu Chen
Published: 2024-02-26 15:47:01

Structured data sources, such as tables, graphs, and databases, are
ubiquitous knowledge sources. Despite the demonstrated capabilities of large
language models (LLMs) on plain text, their proficiency in interpreting and
utilizing structured data remains limited. Our investigation reveals a notable
deficiency in LLMs' ability to process structured data, e.g., ChatGPT lags
behind state-of-the-art (SoTA) model by an average of 35%. To augment the
Structured Knowledge Grounding (SKG) capabilities in LLMs, we have developed a
comprehensive instruction tuning dataset comprising 1.1 million examples.
Utilizing this dataset, we train a series of models, referred to as StructLM,
based on the Code-LLaMA architecture, ranging from 7B to 34B parameters. Our
StructLM series surpasses task-specific models on 14 out of 18 evaluated
datasets and establishes new SoTA achievements on 7 SKG tasks. Furthermore,
StructLM demonstrates exceptional generalization across 6 novel SKG tasks.
Contrary to expectations, we observe that scaling model size offers marginal
benefits, with StructLM-34B showing only slight improvements over StructLM-7B.
This suggests that structured knowledge grounding is still a challenging task
and requires more innovative design to push to a new level.
FuseChat: Knowledge Fusion of Chat Models
Authors: name: Fanqi Wan, name: Ziyi Yang, name: Longguang Zhong, name: Xiaojun Quan, name: Xinting Huang, name: Wei Bi
Published: 2024-02-25 15:11:58

While training large language models (LLMs) from scratch can indeed lead to
models with distinct capabilities and strengths, this approach incurs
substantial costs and may lead to potential redundancy in competencies. An
alternative strategy is to combine existing LLMs into a more robust LLM,
thereby diminishing the necessity for expensive pre-training. However, due to
the diverse architectures of LLMs, direct parameter blending proves to be
unfeasible. Recently, FuseLLM introduced the concept of knowledge
fusion to transfer the collective knowledge of multiple structurally varied
LLMs into a target LLM through lightweight continual training. In this report,
we extend the scalability and flexibility of the FuseLLM framework to
realize the fusion of chat LLMs, resulting in FuseChat.
FuseChat comprises two main stages. Firstly, we undertake knowledge
fusion for structurally and scale-varied source LLMs to derive multiple target
LLMs of identical structure and size via lightweight fine-tuning. Then, these
target LLMs are merged within the parameter space, wherein we propose a novel
method for determining the merging weights based on the variation ratio of
parameter matrices before and after fine-tuning. We validate our approach using
three prominent chat LLMs with diverse architectures and scales, namely
NH2-Mixtral-8x7B, NH2-Solar-10.7B, and
OpenChat-3.5-7B. Experimental results spanning various chat domains
demonstrate the superiority of \textsc{FuseChat-7B} across a broad
spectrum of chat LLMs at 7B and 34B scales, even surpassing GPT-3.5
(March) and approaching Mixtral-8x7B-Instruct. Our code, model
weights, and data are openly accessible at
https://github.com/fanqiwan/FuseLLM.
Do Large Language Models Latently Perform Multi-Hop Reasoning?
Authors: name: Sohee Yang, name: Elena Gribovskaya, name: Nora Kassner, name: Mor Geva, name: Sebastian Riedel
Published: 2024-02-26 18:57:54

We study whether Large Language Models (LLMs) latently perform multi-hop
reasoning with complex prompts such as "The mother of the singer of
'Superstition' is". We look for evidence of a latent reasoning pathway where an
LLM (1) latently identifies "the singer of 'Superstition'" as Stevie Wonder,
the bridge entity, and (2) uses its knowledge of Stevie Wonder's mother to
complete the prompt. We analyze these two hops individually and consider their
co-occurrence as indicative of latent multi-hop reasoning. For the first hop,
we test if changing the prompt to indirectly mention the bridge entity instead
of any other entity increases the LLM's internal recall of the bridge entity.
For the second hop, we test if increasing this recall causes the LLM to better
utilize what it knows about the bridge entity. We find strong evidence of
latent multi-hop reasoning for the prompts of certain relation types, with the
reasoning pathway used in more than 80% of the prompts. However, the
utilization is highly contextual, varying across different types of prompts.
Also, on average, the evidence for the second hop and the full multi-hop
traversal is rather moderate and only substantial for the first hop. Moreover,
we find a clear scaling trend with increasing model size for the first hop of
reasoning but not for the second hop. Our experimental findings suggest
potential challenges and opportunities for future development and applications
of LLMs.
MegaScale: Scaling Large Language Model Training to More Than 10,000
  GPUs
Authors: name: Ziheng Jiang, name: Haibin Lin, name: Yinmin Zhong, name: Qi Huang, name: Yangrui Chen, name: Zhi Zhang, name: Yanghua Peng, name: Xiang Li, name: Cong Xie, name: Shibiao Nong, name: Yulu Jia, name: Sun He, name: Hongmin Chen, name: Zhihao Bai, name: Qi Hou, name: Shipeng Yan, name: Ding Zhou, name: Yiyao Sheng, name: Zhuo Jiang, name: Haohan Xu, name: Haoran Wei, name: Zhang Zhang, name: Pengfei Nie, name: Leqi Zou, name: Sida Zhao, name: Liang Xiang, name: Zherui Liu, name: Zhe Li, name: Xiaoying Jia, name: Jianxi Ye, name: Xin Jin, name: Xin Liu
Published: 2024-02-23 22:10:59

We present the design, implementation and engineering experience in building
and deploying MegaScale, a production system for training large language models
(LLMs) at the scale of more than 10,000 GPUs. Training LLMs at this scale
brings unprecedented challenges to training efficiency and stability. We take a
full-stack approach that co-designs the algorithmic and system components
across model block and optimizer design, computation and communication
overlapping, operator optimization, data pipeline, and network performance
tuning. Maintaining high efficiency throughout the training process (i.e.,
stability) is an important consideration in production given the long extent of
LLM training jobs. Many hard stability issues only emerge at large scale, and
in-depth observability is the key to address them. We develop a set of
diagnosis tools to monitor system components and events deep in the stack,
identify root causes, and derive effective techniques to achieve fault
tolerance and mitigate stragglers. MegaScale achieves 55.2% Model FLOPs
Utilization (MFU) when training a 175B LLM model on 12,288 GPUs, improving the
MFU by 1.34x compared to Megatron-LM. We share our operational experience in
identifying and fixing failures and stragglers. We hope by articulating the
problems and sharing our experience from a systems perspective, this work can
inspire future LLM systems research.