Orionnnn/hub-stats huggingface dataset:
BlockFusion: Expandable 3D Scene Generation using Latent Tri-plane
  Extrapolation
Authors: name: Zhennan Wu, name: Yang Li, name: Han Yan, name: Taizhang Shang, name: Weixuan Sun, name: Senbo Wang, name: Ruikai Cui, name: Weizhe Liu, name: Hiroyuki Sato, name: Hongdong Li, name: Pan Ji
Published: 2024-01-30 14:34:19

We present BlockFusion, a diffusion-based model that generates 3D scenes as
unit blocks and seamlessly incorporates new blocks to extend the scene.
BlockFusion is trained using datasets of 3D blocks that are randomly cropped
from complete 3D scene meshes. Through per-block fitting, all training blocks
are converted into the hybrid neural fields: with a tri-plane containing the
geometry features, followed by a Multi-layer Perceptron (MLP) for decoding the
signed distance values. A variational auto-encoder is employed to compress the
tri-planes into the latent tri-plane space, on which the denoising diffusion
process is performed. Diffusion applied to the latent representations allows
for high-quality and diverse 3D scene generation. To expand a scene during
generation, one needs only to append empty blocks to overlap with the current
scene and extrapolate existing latent tri-planes to populate new blocks. The
extrapolation is done by conditioning the generation process with the feature
samples from the overlapping tri-planes during the denoising iterations. Latent
tri-plane extrapolation produces semantically and geometrically meaningful
transitions that harmoniously blend with the existing scene. A 2D layout
conditioning mechanism is used to control the placement and arrangement of
scene elements. Experimental results indicate that BlockFusion is capable of
generating diverse, geometrically consistent and unbounded large 3D scenes with
unprecedented high-quality shapes in both indoor and outdoor scenarios.
OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on
  E-Branchformer
Authors: name: Yifan Peng, name: Jinchuan Tian, name: William Chen, name: Siddhant Arora, name: Brian Yan, name: Yui Sudo, name: Muhammad Shakeel, name: Kwanghee Choi, name: Jiatong Shi, name: Xuankai Chang, name: Jee-weon Jung, name: Shinji Watanabe
Published: 2024-01-30 01:22:18

Recent studies have advocated for fully open foundation models to promote
transparency and open science. As an initial step, the Open Whisper-style
Speech Model (OWSM) reproduced OpenAI's Whisper using publicly available data
and open-source toolkits. With the aim of reproducing Whisper, the previous
OWSM v1 through v3 models were still based on Transformer, which might lead to
inferior performance compared to other state-of-the-art speech encoders. In
this work, we aim to improve the performance and efficiency of OWSM without
extra training data. We present E-Branchformer based OWSM v3.1 models at two
scales, i.e., 100M and 1B. The 1B model is the largest E-Branchformer based
speech model that has been made publicly available. It outperforms the previous
OWSM v3 in a vast majority of evaluation benchmarks, while demonstrating up to
25% faster inference speed. We publicly release the data preparation scripts,
pre-trained models and training logs.
Repositioning the Subject within Image
Authors: name: Yikai Wang, name: Chenjie Cao, name: Qiaole Dong, name: Yifan Li, name: Yanwei Fu
Published: 2024-01-30 10:04:49

Current image manipulation primarily centers on static manipulation, such as
replacing specific regions within an image or altering its overall style. In
this paper, we introduce an innovative dynamic manipulation task, subject
repositioning. This task involves relocating a user-specified subject to a
desired position while preserving the image's fidelity. Our research reveals
that the fundamental sub-tasks of subject repositioning, which include filling
the void left by the repositioned subject, reconstructing obscured portions of
the subject and blending the subject to be consistent with surrounding areas,
can be effectively reformulated as a unified, prompt-guided inpainting task.
Consequently, we can employ a single diffusion generative model to address
these sub-tasks using various task prompts learned through our proposed task
inversion technique. Additionally, we integrate pre-processing and
post-processing techniques to further enhance the quality of subject
repositioning. These elements together form our SEgment-gEnerate-and-bLEnd
(SEELE) framework. To assess SEELE's effectiveness in subject repositioning, we
assemble a real-world subject repositioning dataset called ReS. Our results on
ReS demonstrate the quality of repositioned image generation.
YOLO-World: Real-Time Open-Vocabulary Object Detection
Authors: name: Tianheng Cheng, name: Lin Song, name: Yixiao Ge, name: Wenyu Liu, name: Xinggang Wang, name: Ying Shan
Published: 2024-01-30 18:59:38

The You Only Look Once (YOLO) series of detectors have established themselves
as efficient and practical tools. However, their reliance on predefined and
trained object categories limits their applicability in open scenarios.
Addressing this limitation, we introduce YOLO-World, an innovative approach
that enhances YOLO with open-vocabulary detection capabilities through
vision-language modeling and pre-training on large-scale datasets.
Specifically, we propose a new Re-parameterizable Vision-Language Path
Aggregation Network (RepVL-PAN) and region-text contrastive loss to facilitate
the interaction between visual and linguistic information. Our method excels in
detecting a wide range of objects in a zero-shot manner with high efficiency.
On the challenging LVIS dataset, YOLO-World achieves 35.4 AP with 52.0 FPS on
V100, which outperforms many state-of-the-art methods in terms of both accuracy
and speed. Furthermore, the fine-tuned YOLO-World achieves remarkable
performance on several downstream tasks, including object detection and
open-vocabulary instance segmentation.
Weak-to-Strong Jailbreaking on Large Language Models
Authors: name: Xuandong Zhao, name: Xianjun Yang, name: Tianyu Pang, name: Chao Du, name: Lei Li, name: Yu-Xiang Wang, name: William Yang Wang
Published: 2024-01-30 18:48:37

Although significant efforts have been dedicated to aligning large language
models (LLMs), red-teaming reports suggest that these carefully aligned LLMs
could still be jailbroken through adversarial prompts, tuning, or decoding.
Upon examining the jailbreaking vulnerability of aligned LLMs, we observe that
the decoding distributions of jailbroken and aligned models differ only in the
initial generations. This observation motivates us to propose the
weak-to-strong jailbreaking attack, where adversaries can utilize smaller
unsafe/aligned LLMs (e.g., 7B) to guide jailbreaking against significantly
larger aligned LLMs (e.g., 70B). To jailbreak, one only needs to additionally
decode two smaller LLMs once, which involves minimal computation and latency
compared to decoding the larger LLMs. The efficacy of this attack is
demonstrated through experiments conducted on five models from three different
organizations. Our study reveals a previously unnoticed yet efficient way of
jailbreaking, exposing an urgent safety issue that needs to be considered when
aligning LLMs. As an initial attempt, we propose a defense strategy to protect
against such attacks, but creating more advanced defenses remains challenging.
The code for replicating the method is available at
https://github.com/XuandongZhao/weak-to-strong
MouSi: Poly-Visual-Expert Vision-Language Models
Authors: name: Xiaoran Fan, name: Tao Ji, name: Changhao Jiang, name: Shuo Li, name: Senjie Jin, name: Sirui Song, name: Junke Wang, name: Boyang Hong, name: Lu Chen, name: Guodong Zheng, name: Ming Zhang, name: Caishuang Huang, name: Rui Zheng, name: Zhiheng Xi, name: Yuhao Zhou, name: Shihan Dou, name: Junjie Ye, name: Hang Yan, name: Tao Gui, name: Qi Zhang, name: Xipeng Qiu, name: Xuanjing Huang, name: Zuxuan Wu, name: Yu-Gang Jiang
Published: 2024-01-30 18:09:11

Current large vision-language models (VLMs) often encounter challenges such
as insufficient capabilities of a single visual component and excessively long
visual tokens. These issues can limit the model's effectiveness in accurately
interpreting complex visual information and over-lengthy contextual
information. Addressing these challenges is crucial for enhancing the
performance and applicability of VLMs. This paper proposes the use of ensemble
experts technique to synergizes the capabilities of individual visual encoders,
including those skilled in image-text matching, OCR, image segmentation, etc.
This technique introduces a fusion network to unify the processing of outputs
from different visual experts, while bridging the gap between image encoders
and pre-trained LLMs. In addition, we explore different positional encoding
schemes to alleviate the waste of positional encoding caused by lengthy image
feature sequences, effectively addressing the issue of position overflow and
length limitations. For instance, in our implementation, this technique
significantly reduces the positional occupancy in models like SAM, from a
substantial 4096 to a more efficient and manageable 64 or even down to 1.
Experimental results demonstrate that VLMs with multiple experts exhibit
consistently superior performance over isolated visual encoders and mark a
significant performance boost as more experts are integrated. We have
open-sourced the training code used in this report. All of these resources can
be found on our project website.
Transfer Learning for Text Diffusion Models
Authors: name: Kehang Han, name: Kathleen Kenealy, name: Aditya Barua, name: Noah Fiedel, name: Noah Constant
Published: 2024-01-30 17:11:56

In this report, we explore the potential for text diffusion to replace
autoregressive (AR) decoding for the training and deployment of large language
models (LLMs). We are particularly interested to see whether pretrained AR
models can be transformed into text diffusion models through a lightweight
adaptation procedure we call ``AR2Diff''. We begin by establishing a strong
baseline setup for training text diffusion models. Comparing across multiple
architectures and pretraining objectives, we find that training a decoder-only
model with a prefix LM objective is best or near-best across several tasks.
Building on this finding, we test various transfer learning setups for text
diffusion models. On machine translation, we find that text diffusion
underperforms the standard AR approach. However, on code synthesis and
extractive QA, we find diffusion models trained from scratch outperform AR
models in many cases. We also observe quality gains from AR2Diff -- adapting AR
models to use diffusion decoding. These results are promising given that text
diffusion is relatively underexplored and can be significantly faster than AR
decoding for long text generation.
H2O-Danube-1.8B Technical Report
Authors: name: Philipp Singer, name: Pascal Pfeiffer, name: Yauhen Babakhin, name: Maximilian Jeblick, name: Nischay Dhankhar, name: Gabor Fodor, name: Sri Satish Ambati
Published: 2024-01-30 08:45:08

We present H2O-Danube-1.8B, a 1.8B language model trained on 1T tokens
following the core principles of LLama 2 and Mistral. We leverage and refine
various techniques for pre-training large language models. Although our model
is trained on significantly fewer total tokens compared to reference models of
similar size, it exhibits highly competitive metrics across a multitude of
benchmarks. We additionally release a chat model trained with supervised
fine-tuning followed by direct preference optimization. We make H2O-Danube-1.8B
openly available under Apache 2.0 license further democratizing LLMs to a wider
audience economically.
T3: Transparent Tracking & Triggering for Fine-grained Overlap of
  Compute & Collectives
Authors: name: Suchita Pati, name: Shaizeen Aga, name: Mahzabeen Islam, name: Nuwan Jayasena, name: Matthew D. Sinclair
Published: 2024-01-30 01:55:34

Large Language Models increasingly rely on distributed techniques for their
training and inference. These techniques require communication across devices
which can reduce scaling efficiency as the number of devices increases. While
some distributed techniques can overlap, and thus, hide this communication with
independent computations, techniques such as Tensor Parallelism (TP) inherently
serialize communication with model execution. One approach to hide this
serialized communication is to interleave it with the producer operation (of
the communicated data) in a fine-grained manner. However, this fine-grained
interleaving of communication and computation in software can be difficult.
Furthermore, as with any concurrent execution, it requires compute and memory
resources to be shared between computation and communication, causing resource
contention that reduces overlapping efficacy.
  To overcome these challenges, we propose T3 which applies hardware-software
co-design to transparently overlap serialized communication while minimizing
resource contention with compute. T3 transparently fuses producer operations
with the subsequent communication via a simple configuration of the producer's
output address space and requires minor software changes. At the hardware
level, T3 adds a lightweight track and trigger mechanism to orchestrate the
producer's compute, and communication. It further uses compute-enhanced
memories for communication's attendant compute. As a result, T3 reduces
resource contention, and efficiently overlaps serialized communication with
computation. For important Transformer models like T-NLG, T3 speeds up
communication-heavy sublayers by 30% geomean (max 47%) and reduces data
movement by 22% geomean (max 36%). Furthermore, T3's benefits persist as models
scale: geomean 29% for sublayers in sim500-billion parameter models, PALM
and MT-NLG.
ReGAL: Refactoring Programs to Discover Generalizable Abstractions
Authors: name: Elias Stengel-Eskin, name: Archiki Prasad, name: Mohit Bansal
Published: 2024-01-29 18:45:30

While large language models (LLMs) are increasingly being used for program
synthesis, they lack the global view needed to develop useful abstractions;
they generally predict programs one at a time, often repeating the same
functionality. Generating redundant code from scratch is both inefficient and
error-prone. To address this, we propose Refactoring for Generalizable
Abstraction Learning (ReGAL), a gradient-free method for learning a library of
reusable functions via code refactorization, i.e. restructuring code without
changing its execution output. ReGAL learns from a small set of existing
programs, iteratively verifying and refining its abstractions via execution. We
find that the shared function libraries discovered by ReGAL make programs
easier to predict across diverse domains. On three datasets (LOGO graphics
generation, Date reasoning, and TextCraft, a Minecraft-based text game), both
open-source and proprietary LLMs improve in accuracy when predicting programs
with ReGAL functions. For CodeLlama-13B, ReGAL results in absolute accuracy
increases of 11.5% on graphics, 26.1% on date understanding, and 8.1% on
TextCraft, outperforming GPT-3.5 in two of three domains. Our analysis reveals
ReGAL's abstractions encapsulate frequently-used subroutines as well as
environment dynamics.
Weaver: Foundation Models for Creative Writing
Authors: name: Tiannan Wang, name: Jiamin Chen, name: Qingrui Jia, name: Shuai Wang, name: Ruoyu Fang, name: Huilin Wang, name: Zhaowei Gao, name: Chunzhao Xie, name: Chuou Xu, name: Jihong Dai, name: Yibin Liu, name: Jialong Wu, name: Shengwei Ding, name: Long Li, name: Zhiwei Huang, name: Xinle Deng, name: Teng Yu, name: Gangan Ma, name: Han Xiao, name: Zixin Chen, name: Danjun Xiang, name: Yunxia Wang, name: Yuanyuan Zhu, name: Yi Xiao, name: Jing Wang, name: Yiru Wang, name: Siran Ding, name: Jiayang Huang, name: Jiayi Xu, name: Yilihamu Tayier, name: Zhenyu Hu, name: Yuan Gao, name: Chengfeng Zheng, name: Yueshu Ye, name: Yihang Li, name: Lei Wan, name: Xinyue Jiang, name: Yujie Wang, name: Siyu Cheng, name: Zhule Song, name: Xiangru Tang, name: Xiaohua Xu, name: Ningyu Zhang, name: Huajun Chen, name: Yuchen Eleanor Jiang, name: Wangchunshu Zhou
Published: 2024-01-30 18:58:43

This work introduces Weaver, our first family of large language models (LLMs)
dedicated to content creation. Weaver is pre-trained on a carefully selected
corpus that focuses on improving the writing capabilities of large language
models. We then fine-tune Weaver for creative and professional writing purposes
and align it to the preference of professional writers using a suit of novel
methods for instruction data synthesis and LLM alignment, making it able to
produce more human-like texts and follow more diverse instructions for content
creation. The Weaver family consists of models of Weaver Mini (1.8B), Weaver
Base (6B), Weaver Pro (14B), and Weaver Ultra (34B) sizes, suitable for
different applications and can be dynamically dispatched by a routing agent
according to query complexity to balance response quality and computation cost.
Evaluation on a carefully curated benchmark for assessing the writing
capabilities of LLMs shows Weaver models of all sizes outperform generalist
LLMs several times larger than them. Notably, our most-capable Weaver Ultra
model surpasses GPT-4, a state-of-the-art generalist LLM, on various writing
scenarios, demonstrating the advantage of training specialized LLMs for writing
purposes. Moreover, Weaver natively supports retrieval-augmented generation
(RAG) and function calling (tool usage). We present various use cases of these
abilities for improving AI-assisted writing systems, including integration of
external knowledge bases, tools, or APIs, and providing personalized writing
assistance. Furthermore, we discuss and summarize a guideline and best
practices for pre-training and fine-tuning domain-specific LLMs.
SERL: A Software Suite for Sample-Efficient Robotic Reinforcement
  Learning
Authors: name: Jianlan Luo, name: Zheyuan Hu, name: Charles Xu, name: You Liang Tan, name: Jacob Berg, name: Archit Sharma, name: Stefan Schaal, name: Chelsea Finn, name: Abhishek Gupta, name: Sergey Levine
Published: 2024-01-29 10:01:10

In recent years, significant progress has been made in the field of robotic
reinforcement learning (RL), enabling methods that handle complex image
observations, train in the real world, and incorporate auxiliary data, such as
demonstrations and prior experience. However, despite these advances, robotic
RL remains hard to use. It is acknowledged among practitioners that the
particular implementation details of these algorithms are often just as
important (if not more so) for performance as the choice of algorithm. We posit
that a significant challenge to widespread adoption of robotic RL, as well as
further development of robotic RL methods, is the comparative inaccessibility
of such methods. To address this challenge, we developed a carefully
implemented library containing a sample efficient off-policy deep RL method,
together with methods for computing rewards and resetting the environment, a
high-quality controller for a widely-adopted robot, and a number of challenging
example tasks. We provide this library as a resource for the community,
describe its design choices, and present experimental results. Perhaps
surprisingly, we find that our implementation can achieve very efficient
learning, acquiring policies for PCB board assembly, cable routing, and object
relocation between 25 to 50 minutes of training per policy on average,
improving over state-of-the-art results reported for similar tasks in the
literature. These policies achieve perfect or near-perfect success rates,
extreme robustness even under perturbations, and exhibit emergent recovery and
correction behaviors. We hope that these promising results and our high-quality
open-source implementation will provide a tool for the robotics community to
facilitate further developments in robotic RL. Our code, documentation, and
videos can be found at https://serl-robot.github.io/
Media2Face: Co-speech Facial Animation Generation With Multi-Modality
  Guidance
Authors: name: Qingcheng Zhao, name: Pengyu Long, name: Qixuan Zhang, name: Dafei Qin, name: Han Liang, name: Longwen Zhang, name: Yingliang Zhang, name: Jingyi Yu, name: Lan Xu
Published: 2024-01-28 16:17:59

The synthesis of 3D facial animations from speech has garnered considerable
attention. Due to the scarcity of high-quality 4D facial data and
well-annotated abundant multi-modality labels, previous methods often suffer
from limited realism and a lack of lexible conditioning. We address this
challenge through a trilogy. We first introduce Generalized Neural Parametric
Facial Asset (GNPFA), an efficient variational auto-encoder mapping facial
geometry and images to a highly generalized expression latent space, decoupling
expressions and identities. Then, we utilize GNPFA to extract high-quality
expressions and accurate head poses from a large array of videos. This presents
the M2F-D dataset, a large, diverse, and scan-level co-speech 3D facial
animation dataset with well-annotated emotional and style labels. Finally, we
propose Media2Face, a diffusion model in GNPFA latent space for co-speech
facial animation generation, accepting rich multi-modality guidances from
audio, text, and image. Extensive experiments demonstrate that our model not
only achieves high fidelity in facial animation synthesis but also broadens the
scope of expressiveness and style adaptability in 3D facial animation.
StableIdentity: Inserting Anybody into Anywhere at First Sight
Authors: name: Qinghe Wang, name: Xu Jia, name: Xiaomin Li, name: Taiqing Li, name: Liqian Ma, name: Yunzhi Zhuge, name: Huchuan Lu
Published: 2024-01-29 09:06:15

Recent advances in large pretrained text-to-image models have shown
unprecedented capabilities for high-quality human-centric generation, however,
customizing face identity is still an intractable problem. Existing methods
cannot ensure stable identity preservation and flexible editability, even with
several images for each subject during training. In this work, we propose
StableIdentity, which allows identity-consistent recontextualization with just
one face image. More specifically, we employ a face encoder with an identity
prior to encode the input face, and then land the face representation into a
space with an editable prior, which is constructed from celeb names. By
incorporating identity prior and editability prior, the learned identity can be
injected anywhere with various contexts. In addition, we design a masked
two-phase diffusion loss to boost the pixel-level perception of the input face
and maintain the diversity of generation. Extensive experiments demonstrate our
method outperforms previous customization methods. In addition, the learned
identity can be flexibly combined with the off-the-shelf modules such as
ControlNet. Notably, to the best knowledge, we are the first to directly inject
the identity learned from a single image into video/3D generation without
finetuning. We believe that the proposed StableIdentity is an important step to
unify image, video, and 3D customized generation models.