Orionnnn/hub-stats huggingface dataset:
CoMoSVC: Consistency Model-based Singing Voice Conversion
Authors: name: Yiwen Lu, name: Zhen Ye, name: Wei Xue, name: Xu Tan, name: Qifeng Liu, name: Yike Guo
Published: 2024-01-03 15:47:17

The diffusion-based Singing Voice Conversion (SVC) methods have achieved
remarkable performances, producing natural audios with high similarity to the
target timbre. However, the iterative sampling process results in slow
inference speed, and acceleration thus becomes crucial. In this paper, we
propose CoMoSVC, a consistency model-based SVC method, which aims to achieve
both high-quality generation and high-speed sampling. A diffusion-based teacher
model is first specially designed for SVC, and a student model is further
distilled under self-consistency properties to achieve one-step sampling.
Experiments on a single NVIDIA GTX4090 GPU reveal that although CoMoSVC has a
significantly faster inference speed than the state-of-the-art (SOTA)
diffusion-based SVC system, it still achieves comparable or superior conversion
performance based on both subjective and objective metrics. Audio samples and
codes are available at https://comosvc.github.io/.
SIGNeRF: Scene Integrated Generation for Neural Radiance Fields
Authors: name: Jan-Niklas Dihlmann, name: Andreas Engelhardt, name: Hendrik Lensch
Published: 2024-01-03 09:46:43

Advances in image diffusion models have recently led to notable improvements
in the generation of high-quality images. In combination with Neural Radiance
Fields (NeRFs), they enabled new opportunities in 3D generation. However, most
generative 3D approaches are object-centric and applying them to editing
existing photorealistic scenes is not trivial. We propose SIGNeRF, a novel
approach for fast and controllable NeRF scene editing and scene-integrated
object generation. A new generative update strategy ensures 3D consistency
across the edited images, without requiring iterative optimization. We find
that depth-conditioned diffusion models inherently possess the capability to
generate 3D consistent views by requesting a grid of images instead of single
views. Based on these insights, we introduce a multi-view reference sheet of
modified images. Our method updates an image collection consistently based on
the reference sheet and refines the original NeRF with the newly generated
image set in one go. By exploiting the depth conditioning mechanism of the
image diffusion model, we gain fine control over the spatial location of the
edit and enforce shape guidance by a selected region or an external mesh.
From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations
Authors: name: Evonne Ng, name: Javier Romero, name: Timur Bagautdinov, name: Shaojie Bai, name: Trevor Darrell, name: Angjoo Kanazawa, name: Alexander Richard
Published: 2024-01-03 18:55:16

We present a framework for generating full-bodied photorealistic avatars that
gesture according to the conversational dynamics of a dyadic interaction. Given
speech audio, we output multiple possibilities of gestural motion for an
individual, including face, body, and hands. The key behind our method is in
combining the benefits of sample diversity from vector quantization with the
high-frequency details obtained through diffusion to generate more dynamic,
expressive motion. We visualize the generated motion using highly
photorealistic avatars that can express crucial nuances in gestures (e.g.
sneers and smirks). To facilitate this line of research, we introduce a
first-of-its-kind multi-view conversational dataset that allows for
photorealistic reconstruction. Experiments show our model generates appropriate
and diverse gestures, outperforming both diffusion- and VQ-only methods.
Furthermore, our perceptual evaluation highlights the importance of
photorealism (vs. meshes) in accurately assessing subtle motion details in
conversational gestures. Code and dataset available online.
aMUSEd: An Open MUSE Reproduction
Authors: name: Suraj Patil, name: William Berman, name: Robin Rombach, name: Patrick von Platen
Published: 2024-01-03 16:10:07

We present aMUSEd, an open-source, lightweight masked image model (MIM) for
text-to-image generation based on MUSE. With 10 percent of MUSE's parameters,
aMUSEd is focused on fast image generation. We believe MIM is under-explored
compared to latent diffusion, the prevailing approach for text-to-image
generation. Compared to latent diffusion, MIM requires fewer inference steps
and is more interpretable. Additionally, MIM can be fine-tuned to learn
additional styles with only a single image. We hope to encourage further
exploration of MIM by demonstrating its effectiveness on large-scale
text-to-image generation and releasing reproducible training code. We also
release checkpoints for two models which directly produce images at 256x256 and
512x512 resolutions.
Image Sculpting: Precise Object Editing with 3D Geometry Control
Authors: name: Jiraphon Yenphraphai, name: Xichen Pan, name: Sainan Liu, name: Daniele Panozzo, name: Saining Xie
Published: 2024-01-02 18:59:35

We present Image Sculpting, a new framework for editing 2D images by
incorporating tools from 3D geometry and graphics. This approach differs
markedly from existing methods, which are confined to 2D spaces and typically
rely on textual instructions, leading to ambiguity and limited control. Image
Sculpting converts 2D objects into 3D, enabling direct interaction with their
3D geometry. Post-editing, these objects are re-rendered into 2D, merging into
the original image to produce high-fidelity results through a coarse-to-fine
enhancement process. The framework supports precise, quantifiable, and
physically-plausible editing options such as pose editing, rotation,
translation, 3D composition, carving, and serial addition. It marks an initial
step towards combining the creative freedom of generative models with the
precision of graphics pipelines.
A Vision Check-up for Language Models
Authors: name: Pratyusha Sharma, name: Tamar Rott Shaham, name: Manel Baradad, name: Stephanie Fu, name: Adrian Rodriguez-Munoz, name: Shivam Duggal, name: Phillip Isola, name: Antonio Torralba
Published: 2024-01-03 18:09:33

What does learning to model relationships between strings teach large
language models (LLMs) about the visual world? We systematically evaluate LLMs'
abilities to generate and recognize an assortment of visual concepts of
increasing complexity and then demonstrate how a preliminary visual
representation learning system can be trained using models of text. As language
models lack the ability to consume or output visual information as pixels, we
use code to represent images in our study. Although LLM-generated images do not
look like natural images, results on image generation and the ability of models
to correct these generated images indicate that precise modeling of strings can
teach language models about numerous aspects of the visual world. Furthermore,
experiments on self-supervised visual representation learning, utilizing images
generated with text models, highlight the potential to train vision models
capable of making semantic assessments of natural images using just LLMs.
Multilingual Instruction Tuning With Just a Pinch of Multilinguality
Authors: name: Uri Shaham, name: Jonathan Herzig, name: Roee Aharoni, name: Idan Szpektor, name: Reut Tsarfaty, name: Matan Eyal
Published: 2024-01-03 17:48:10

As instruction-tuned large language models (LLMs) gain global adoption, their
ability to follow instructions in multiple languages becomes increasingly
crucial. One promising approach is cross-lingual transfer, where a model
acquires specific functionality on some language by finetuning on another
language. In this work, we investigate how multilinguality during instruction
tuning of a multilingual LLM affects instruction-following across languages. We
first show that many languages transfer some instruction-following capabilities
to other languages from even monolingual tuning. Furthermore, we find that only
40 multilingual examples in an English tuning set substantially improve
multilingual instruction-following, both in seen and unseen languages during
tuning. In general, we observe that models tuned on multilingual mixtures
exhibit comparable or superior performance in several languages compared to
monolingually tuned models, despite training on 10x fewer examples in those
languages. Finally, we find that increasing the number of languages in the
instruction tuning set from 1 to only 2, 3, or 4 increases cross-lingual
generalization. Our results suggest that building massively multilingual
instruction-tuned models can be done with only a very small set of multilingual
instruction-responses.
WordArt Designer API: User-Driven Artistic Typography Synthesis with
  Large Language Models on ModelScope
Authors: name: Jun-Yan He, name: Zhi-Qi Cheng, name: Chenyang Li, name: Jingdong Sun, name: Wangmeng Xiang, name: Yusen Hu, name: Xianhui Lin, name: Xiaoyang Kang, name: Zengke Jin, name: Bin Luo, name: Yifeng Geng, name: Xuansong Xie, name: Jingren Zhou
Published: 2024-01-03 12:06:02

This paper introduces the WordArt Designer API, a novel framework for
user-driven artistic typography synthesis utilizing Large Language Models
(LLMs) on ModelScope. We address the challenge of simplifying artistic
typography for non-professionals by offering a dynamic, adaptive, and
computationally efficient alternative to traditional rigid templates. Our
approach leverages the power of LLMs to understand and interpret user input,
facilitating a more intuitive design process. We demonstrate through various
case studies how users can articulate their aesthetic preferences and
functional requirements, which the system then translates into unique and
creative typographic designs. Our evaluations indicate significant improvements
in user satisfaction, design flexibility, and creative expression over existing
systems. The WordArt Designer API not only democratizes the art of typography
but also opens up new possibilities for personalized digital communication and
design.
Moonshot: Towards Controllable Video Generation and Editing with
  Multimodal Conditions
Authors: name: David Junhao Zhang, name: Dongxu Li, name: Hung Le, name: Mike Zheng Shou, name: Caiming Xiong, name: Doyen Sahoo
Published: 2024-01-03 16:43:47

Most existing video diffusion models (VDMs) are limited to mere text
conditions. Thereby, they are usually lacking in control over visual appearance
and geometry structure of the generated videos. This work presents Moonshot, a
new video generation model that conditions simultaneously on multimodal inputs
of image and text. The model builts upon a core module, called multimodal video
block (MVB), which consists of conventional spatialtemporal layers for
representing video features, and a decoupled cross-attention layer to address
image and text inputs for appearance conditioning. In addition, we carefully
design the model architecture such that it can optionally integrate with
pre-trained image ControlNet modules for geometry visual conditions, without
needing of extra training overhead as opposed to prior methods. Experiments
show that with versatile multimodal conditioning mechanisms, Moonshot
demonstrates significant improvement on visual quality and temporal consistency
compared to existing models. In addition, the model can be easily repurposed
for a variety of generative applications, such as personalized video
generation, image animation and video editing, unveiling its potential to serve
as a fundamental architecture for controllable video generation. Models will be
made public on https://github.com/salesforce/LAVIS.
GPT-4V(ision) is a Generalist Web Agent, if Grounded
Authors: name: Boyuan Zheng, name: Boyu Gou, name: Jihyung Kil, name: Huan Sun, name: Yu Su
Published: 2024-01-03 08:33:09

The recent development on large multimodal models (LMMs), especially
GPT-4V(ision) and Gemini, has been quickly expanding the capability boundaries
of multimodal models beyond traditional tasks like image captioning and visual
question answering. In this work, we explore the potential of LMMs like GPT-4V
as a generalist web agent that can follow natural language instructions to
complete tasks on any given website. We propose SEEACT, a generalist web agent
that harnesses the power of LMMs for integrated visual understanding and acting
on the web. We evaluate on the recent MIND2WEB benchmark. In addition to
standard offline evaluation on cached websites, we enable a new online
evaluation setting by developing a tool that allows running web agents on live
websites. We show that GPT-4V presents a great potential for web agents - it
can successfully complete 50% of the tasks on live websites if we manually
ground its textual plans into actions on the websites. This substantially
outperforms text-only LLMs like GPT-4 or smaller models (FLAN-T5 and BLIP-2)
specifically fine-tuned for web agents. However, grounding still remains a
major challenge. Existing LMM grounding strategies like set-of-mark prompting
turns out not effective for web agents, and the best grounding strategy we
develop in this paper leverages both the HTML text and visuals. Yet, there is
still a substantial gap with oracle grounding, leaving ample room for further
improvement.
En3D: An Enhanced Generative Model for Sculpting 3D Humans from 2D
  Synthetic Data
Authors: name: Yifang Men, name: Biwen Lei, name: Yuan Yao, name: Miaomiao Cui, name: Zhouhui Lian, name: Xuansong Xie
Published: 2024-01-02 12:06:31

We present En3D, an enhanced generative scheme for sculpting high-quality 3D
human avatars. Unlike previous works that rely on scarce 3D datasets or limited
2D collections with imbalanced viewing angles and imprecise pose priors, our
approach aims to develop a zero-shot 3D generative scheme capable of producing
visually realistic, geometrically accurate and content-wise diverse 3D humans
without relying on pre-existing 3D or 2D assets. To address this challenge, we
introduce a meticulously crafted workflow that implements accurate physical
modeling to learn the enhanced 3D generative model from synthetic 2D data.
During inference, we integrate optimization modules to bridge the gap between
realistic appearances and coarse 3D shapes. Specifically, En3D comprises three
modules: a 3D generator that accurately models generalizable 3D humans with
realistic appearance from synthesized balanced, diverse, and structured human
images; a geometry sculptor that enhances shape quality using multi-view normal
constraints for intricate human anatomy; and a texturing module that
disentangles explicit texture maps with fidelity and editability, leveraging
semantical UV partitioning and a differentiable rasterizer. Experimental
results show that our approach significantly outperforms prior works in terms
of image quality, geometry accuracy and content diversity. We also showcase the
applicability of our generated avatars for animation and editing, as well as
the scalability of our approach for content-style free adaptation.
Boundary Attention: Learning to Find Faint Boundaries at Any Resolution
Authors: name: Mia Gaia Polansky, name: Charles Herrmann, name: Junhwa Hur, name: Deqing Sun, name: Dor Verbin, name: Todd Zickler
Published: 2024-01-01 19:00:55

We present a differentiable model that explicitly models boundaries --
including contours, corners and junctions -- using a new mechanism that we call
boundary attention. We show that our model provides accurate results even when
the boundary signal is very weak or is swamped by noise. Compared to previous
classical methods for finding faint boundaries, our model has the advantages of
being differentiable; being scalable to larger images; and automatically
adapting to an appropriate level of geometric detail in each part of an image.
Compared to previous deep methods for finding boundaries via end-to-end
training, it has the advantages of providing sub-pixel precision, being more
resilient to noise, and being able to process any image at its native
resolution and aspect ratio.
Taming Mode Collapse in Score Distillation for Text-to-3D Generation
Authors: name: Peihao Wang, name: Dejia Xu, name: Zhiwen Fan, name: Dilin Wang, name: Sreyas Mohan, name: Forrest Iandola, name: Rakesh Ranjan, name: Yilei Li, name: Qiang Liu, name: Zhangyang Wang, name: Vikas Chandra
Published: 2023-12-31 22:47:06

Despite the remarkable performance of score distillation in text-to-3D
generation, such techniques notoriously suffer from view inconsistency issues,
also known as "Janus" artifact, where the generated objects fake each view with
multiple front faces. Although empirically effective methods have approached
this problem via score debiasing or prompt engineering, a more rigorous
perspective to explain and tackle this problem remains elusive. In this paper,
we reveal that the existing score distillation-based text-to-3D generation
frameworks degenerate to maximal likelihood seeking on each view independently
and thus suffer from the mode collapse problem, manifesting as the Janus
artifact in practice. To tame mode collapse, we improve score distillation by
re-establishing in entropy term in the corresponding variational objective,
which is applied to the distribution of rendered images. Maximizing the entropy
encourages diversity among different views in generated 3D assets, thereby
mitigating the Janus problem. Based on this new objective, we derive a new
update rule for 3D score distillation, dubbed Entropic Score Distillation
(ESD). We theoretically reveal that ESD can be simplified and implemented by
just adopting the classifier-free guidance trick upon variational score
distillation. Although embarrassingly straightforward, our extensive
experiments successfully demonstrate that ESD can be an effective treatment for
Janus artifacts in score distillation.
LLaMA Beyond English: An Empirical Study on Language Capability Transfer
Authors: name: Jun Zhao, name: Zhihao Zhang, name: Qi Zhang, name: Tao Gui, name: Xuanjing Huang
Published: 2024-01-02 06:29:02

In recent times, substantial advancements have been witnessed in large
language models (LLMs), exemplified by ChatGPT, showcasing remarkable
proficiency across a range of complex tasks. However, many mainstream LLMs
(e.g. LLaMA) are pretrained on English-dominant corpus, which limits their
performance in other non-English languages. In this paper, we focus on how to
effectively transfer the capabilities of language generation and following
instructions to a non-English language. To answer this question, we conduct an
extensive empirical investigation based on LLaMA, accumulating over 1440 GPU
hours. We analyze the impact of key factors such as vocabulary extension,
further pretraining, and instruction tuning on transfer. To accurately assess
the model's level of knowledge, we employ four widely used standardized testing
benchmarks: C-Eval, MMLU, AGI-Eval, and GAOKAO-Bench. Furthermore, a
comprehensive evaluation of the model's response quality is conducted,
considering aspects such as accuracy, fluency, informativeness, logical
coherence, and harmlessness, based on LLM-Eval, a benchmarks consisting
instruction tasks from 17 diverse categories. Our evaluation results
demonstrate that comparable performance to state-of-the-art transfer models can
be achieved with less than 1% of the pretraining data, both in terms of
knowledge alignment and response quality. Furthermore, the experimental
outcomes across the thirteen low-resource languages also exhibit similar
trends. We anticipate that the conclusions revealed by the experiments will aid
the community in developing non-English LLMs.
Self-Play Fine-Tuning Converts Weak Language Models to Strong Language
  Models
Authors: name: Zixiang Chen, name: Yihe Deng, name: Huizhuo Yuan, name: Kaixuan Ji, name: Quanquan Gu
Published: 2024-01-02 18:53:13

Harnessing the power of human-annotated data through Supervised Fine-Tuning
(SFT) is pivotal for advancing Large Language Models (LLMs). In this paper, we
delve into the prospect of growing a strong LLM out of a weak one without the
need for acquiring additional human-annotated data. We propose a new
fine-tuning method called Self-Play fIne-tuNing (SPIN), which starts from a
supervised fine-tuned model. At the heart of SPIN lies a self-play mechanism,
where the LLM refines its capability by playing against instances of itself.
More specifically, the LLM generates its own training data from its previous
iterations, refining its policy by discerning these self-generated responses
from those obtained from human-annotated data. Our method progressively
elevates the LLM from a nascent model to a formidable one, unlocking the full
potential of human-annotated demonstration data for SFT. Theoretically, we
prove that the global optimum to the training objective function of our method
is achieved only when the LLM policy aligns with the target data distribution.
Empirically, we evaluate our method on several benchmark datasets including the
HuggingFace Open LLM Leaderboard, MT-Bench, and datasets from Big-Bench. Our
results show that SPIN can significantly improve the LLM's performance across a
variety of benchmarks and even outperform models trained through direct
preference optimization (DPO) supplemented with extra GPT-4 preference data.
This sheds light on the promise of self-play, enabling the achievement of
human-level performance in LLMs without the need for expert opponents.
A Comprehensive Study of Knowledge Editing for Large Language Models
Authors: name: Ningyu Zhang, name: Yunzhi Yao, name: Bozhong Tian, name: Peng Wang, name: Shumin Deng, name: Mengru Wang, name: Zekun Xi, name: Shengyu Mao, name: Jintian Zhang, name: Yuansheng Ni, name: Siyuan Cheng, name: Ziwen Xu, name: Xin Xu, name: Jia-Chen Gu, name: Yong Jiang, name: Pengjun Xie, name: Fei Huang, name: Lei Liang, name: Zhiqiang Zhang, name: Xiaowei Zhu, name: Jun Zhou, name: Huajun Chen
Published: 2024-01-02 16:54:58

Large Language Models (LLMs) have shown extraordinary capabilities in
understanding and generating text that closely mirrors human communication.
However, a primary limitation lies in the significant computational demands
during training, arising from their extensive parameterization. This challenge
is further intensified by the dynamic nature of the world, necessitating
frequent updates to LLMs to correct outdated information or integrate new
knowledge, thereby ensuring their continued relevance. Note that many
applications demand continual model adjustments post-training to address
deficiencies or undesirable behaviors. There is an increasing interest in
efficient, lightweight methods for on-the-fly model modifications. To this end,
recent years have seen a burgeoning in the techniques of knowledge editing for
LLMs, which aim to efficiently modify LLMs' behaviors within specific domains
while preserving overall performance across various inputs. In this paper, we
first define the knowledge editing problem and then provide a comprehensive
review of cutting-edge approaches. Drawing inspiration from educational and
cognitive research theories, we propose a unified categorization criterion that
classifies knowledge editing methods into three groups: resorting to external
knowledge, merging knowledge into the model, and editing intrinsic knowledge.
Furthermore, we introduce a new benchmark, KnowEdit, for a comprehensive
empirical evaluation of representative knowledge editing approaches.
Additionally, we provide an in-depth analysis of knowledge location, which can
provide a deeper understanding of the knowledge structures inherent within
LLMs. Finally, we discuss several potential applications of knowledge editing,
outlining its broad and impactful implications.