Orionnnn/hub-stats huggingface dataset:
Toward Joint Language Modeling for Speech Units and Text
Authors: name: Ju-Chieh Chou, name: Chung-Ming Chien, name: Wei-Ning Hsu, name: Karen Livescu, name: Arun Babu, name: Alexis Conneau, name: Alexei Baevski, name: Michael Auli
Published: 2023-10-12 20:53:39

Speech and text are two major forms of human language. The research community
has been focusing on mapping speech to text or vice versa for many years.
However, in the field of language modeling, very little effort has been made to
model them jointly. In light of this, we explore joint language modeling for
speech units and text. Specifically, we compare different speech tokenizers to
transform continuous speech signals into discrete units and use different
methods to construct mixed speech-text data. We introduce automatic metrics to
evaluate how well the joint LM mixes speech and text. We also fine-tune the LM
on downstream spoken language understanding (SLU) tasks with different
modalities (speech or text) and test its performance to assess the model's
learning of shared representations. Our results show that by mixing speech
units and text with our proposed mixing techniques, the joint LM improves over
a speech-only baseline on SLU tasks and shows zero-shot cross-modal
transferability.
LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models
Authors: name: Yixiao Li, name: Yifan Yu, name: Chen Liang, name: Pengcheng He, name: Nikos Karampatziakis, name: Weizhu Chen, name: Tuo Zhao
Published: 2023-10-12 18:34:08

Quantization is an indispensable technique for serving Large Language Models
(LLMs) and has recently found its way into LoRA fine-tuning. In this work we
focus on the scenario where quantization and LoRA fine-tuning are applied
together on a pre-trained model. In such cases it is common to observe a
consistent gap in the performance on downstream tasks between full fine-tuning
and quantization plus LoRA fine-tuning approach. In response, we propose LoftQ
(LoRA-Fine-Tuning-aware Quantization), a novel quantization framework that
simultaneously quantizes an LLM and finds a proper low-rank initialization for
LoRA fine-tuning. Such an initialization alleviates the discrepancy between the
quantized and full-precision model and significantly improves the
generalization in downstream tasks. We evaluate our method on natural language
understanding, question answering, summarization, and natural language
generation tasks. Experiments show that our method is highly effective and
outperforms existing quantization methods, especially in the challenging 2-bit
and 2/4-bit mixed precision regimes. We will release our code.
Table-GPT: Table-tuned GPT for Diverse Table Tasks
Authors: name: Peng Li, name: Yeye He, name: Dror Yashar, name: Weiwei Cui, name: Song Ge, name: Haidong Zhang, name: Danielle Rifinski Fainman, name: Dongmei Zhang, name: Surajit Chaudhuri
Published: 2023-10-13 17:20:56

Language models, such as GPT-3.5 and ChatGPT, demonstrate remarkable
abilities to follow diverse human instructions and perform a wide range of
tasks. However, when probing language models using a range of basic
table-understanding tasks, we observe that today's language models are still
sub-optimal in many table-related tasks, likely because they are pre-trained
predominantly on one-dimensional natural-language texts, whereas
relational tables are two-dimensional objects.
  In this work, we propose a new "table-tuning" paradigm, where we
continue to train/fine-tune language models like GPT-3.5 and ChatGPT, using
diverse table-tasks synthesized from real tables as training data, with the
goal of enhancing language models' ability to understand tables and perform
table tasks. We show that our resulting Table-GPT models demonstrate (1) better
table-understanding capabilities, by consistently outperforming the
vanilla GPT-3.5 and ChatGPT, on a wide-range of table tasks, including holdout
unseen tasks, and (2) strong generalizability, in its ability to respond
to diverse human instructions to perform new table-tasks, in a manner similar
to GPT-3.5 and ChatGPT.
GaussianDreamer: Fast Generation from Text to 3D Gaussian Splatting with
  Point Cloud Priors
Authors: name: Taoran Yi, name: Jiemin Fang, name: Guanjun Wu, name: Lingxi Xie, name: Xiaopeng Zhang, name: Wenyu Liu, name: Qi Tian, name: Xinggang Wang
Published: 2023-10-12 17:22:24

In recent times, the generation of 3D assets from text prompts has shown
impressive results. Both 2D and 3D diffusion models can generate decent 3D
objects based on prompts. 3D diffusion models have good 3D consistency, but
their quality and generalization are limited as trainable 3D data is expensive
and hard to obtain. 2D diffusion models enjoy strong abilities of
generalization and fine generation, but the 3D consistency is hard to
guarantee. This paper attempts to bridge the power from the two types of
diffusion models via the recent explicit and efficient 3D Gaussian splatting
representation. A fast 3D generation framework, named as \name, is proposed,
where the 3D diffusion model provides point cloud priors for initialization and
the 2D diffusion model enriches the geometry and appearance. Operations of
noisy point growing and color perturbation are introduced to enhance the
initialized Gaussians. Our \name can generate a high-quality 3D instance within
25 minutes on one GPU, much faster than previous methods, while the generated
instances can be directly rendered in real time. Demos and code are available
at https://taoranyi.com/gaussiandreamer/.
MotionDirector: Motion Customization of Text-to-Video Diffusion Models
Authors: name: Rui Zhao, name: Yuchao Gu, name: Jay Zhangjie Wu, name: David Junhao Zhang, name: Jiawei Liu, name: Weijia Wu, name: Jussi Keppo, name: Mike Zheng Shou
Published: 2023-10-12 16:26:18

Large-scale pre-trained diffusion models have exhibited remarkable
capabilities in diverse video generations. Given a set of video clips of the
same motion concept, the task of Motion Customization is to adapt existing
text-to-video diffusion models to generate videos with this motion. For
example, generating a video with a car moving in a prescribed manner under
specific camera movements to make a movie, or a video illustrating how a bear
would lift weights to inspire creators. Adaptation methods have been developed
for customizing appearance like subject or style, yet unexplored for motion. It
is straightforward to extend mainstream adaption methods for motion
customization, including full model tuning, parameter-efficient tuning of
additional layers, and Low-Rank Adaptions (LoRAs). However, the motion concept
learned by these methods is often coupled with the limited appearances in the
training videos, making it difficult to generalize the customized motion to
other appearances. To overcome this challenge, we propose MotionDirector, with
a dual-path LoRAs architecture to decouple the learning of appearance and
motion. Further, we design a novel appearance-debiased temporal loss to
mitigate the influence of appearance on the temporal training objective.
Experimental results show the proposed method can generate videos of diverse
appearances for the customized motions. Our method also supports various
downstream applications, such as the mixing of different videos with their
appearance and motion respectively, and animating a single image with
customized motions. Our code and model weights will be released.
HyperHuman: Hyper-Realistic Human Generation with Latent Structural
  Diffusion
Authors: name: Xian Liu, name: Jian Ren, name: Aliaksandr Siarohin, name: Ivan Skorokhodov, name: Yanyu Li, name: Dahua Lin, name: Xihui Liu, name: Ziwei Liu, name: Sergey Tulyakov
Published: 2023-10-12 17:59:34

Despite significant advances in large-scale text-to-image models, achieving
hyper-realistic human image generation remains a desirable yet unsolved task.
Existing models like Stable Diffusion and DALL-E 2 tend to generate human
images with incoherent parts or unnatural poses. To tackle these challenges,
our key insight is that human image is inherently structural over multiple
granularities, from the coarse-level body skeleton to fine-grained spatial
geometry. Therefore, capturing such correlations between the explicit
appearance and latent structure in one model is essential to generate coherent
and natural human images. To this end, we propose a unified framework,
HyperHuman, that generates in-the-wild human images of high realism and diverse
layouts. Specifically, 1) we first build a large-scale human-centric dataset,
named HumanVerse, which consists of 340M images with comprehensive annotations
like human pose, depth, and surface normal. 2) Next, we propose a Latent
Structural Diffusion Model that simultaneously denoises the depth and surface
normal along with the synthesized RGB image. Our model enforces the joint
learning of image appearance, spatial relationship, and geometry in a unified
network, where each branch in the model complements to each other with both
structural awareness and textural richness. 3) Finally, to further boost the
visual quality, we propose a Structure-Guided Refiner to compose the predicted
conditions for more detailed generation of higher resolution. Extensive
experiments demonstrate that our framework yields the state-of-the-art
performance, generating hyper-realistic human images under diverse scenarios.
Project Page: https://snap-research.github.io/HyperHuman/
Octopus: Embodied Vision-Language Programmer from Environmental Feedback
Authors: name: Jingkang Yang, name: Yuhao Dong, name: Shuai Liu, name: Bo Li, name: Ziyue Wang, name: Chencheng Jiang, name: Haoran Tan, name: Jiamu Kang, name: Yuanhan Zhang, name: Kaiyang Zhou, name: Ziwei Liu
Published: 2023-10-12 17:59:58

Large vision-language models (VLMs) have achieved substantial progress in
multimodal perception and reasoning. Furthermore, when seamlessly integrated
into an embodied agent, it signifies a crucial stride towards the creation of
autonomous and context-aware systems capable of formulating plans and executing
commands with precision. In this paper, we introduce Octopus, a novel VLM
designed to proficiently decipher an agent's vision and textual task objectives
and to formulate intricate action sequences and generate executable code. Our
design allows the agent to adeptly handle a wide spectrum of tasks, ranging
from mundane daily chores in simulators to sophisticated interactions in
complex video games. Octopus is trained by leveraging GPT-4 to control an
explorative agent to generate training data, i.e., action blueprints and the
corresponding executable code, within our experimental environment called
OctoVerse. We also collect the feedback that allows the enhanced training
scheme of Reinforcement Learning with Environmental Feedback (RLEF). Through a
series of experiments, we illuminate Octopus's functionality and present
compelling results, and the proposed RLEF turns out to refine the agent's
decision-making. By open-sourcing our model architecture, simulator, and
dataset, we aspire to ignite further innovation and foster collaborative
applications within the broader embodied AI community.
Idea2Img: Iterative Self-Refinement with GPT-4V(ision) for Automatic
  Image Design and Generation
Authors: name: Zhengyuan Yang, name: Jianfeng Wang, name: Linjie Li, name: Kevin Lin, name: Chung-Ching Lin, name: Zicheng Liu, name: Lijuan Wang
Published: 2023-10-12 17:34:20

We introduce ``Idea to Image,'' a system that enables multimodal iterative
self-refinement with GPT-4V(ision) for automatic image design and generation.
Humans can quickly identify the characteristics of different text-to-image
(T2I) models via iterative explorations. This enables them to efficiently
convert their high-level generation ideas into effective T2I prompts that can
produce good images. We investigate if systems based on large multimodal models
(LMMs) can develop analogous multimodal self-refinement abilities that enable
exploring unknown models or environments via self-refining tries. Idea2Img
cyclically generates revised T2I prompts to synthesize draft images, and
provides directional feedback for prompt revision, both conditioned on its
memory of the probed T2I model's characteristics. The iterative self-refinement
brings Idea2Img various advantages over vanilla T2I models. Notably, Idea2Img
can process input ideas with interleaved image-text sequences, follow ideas
with design instructions, and generate images of better semantic and visual
qualities. The user preference study validates the efficacy of multimodal
iterative self-refinement on automatic image design and generation.
Lemur: Harmonizing Natural Language and Code for Language Agents
Authors: name: Yiheng Xu, name: Hongjin Su, name: Chen Xing, name: Boyu Mi, name: Qian Liu, name: Weijia Shi, name: Binyuan Hui, name: Fan Zhou, name: Yitao Liu, name: Tianbao Xie, name: Zhoujun Cheng, name: Siheng Zhao, name: Lingpeng Kong, name: Bailin Wang, name: Caiming Xiong, name: Tao Yu
Published: 2023-10-10 17:57:45

We introduce Lemur and Lemur-Chat, openly accessible language models
optimized for both natural language and coding capabilities to serve as the
backbone of versatile language agents. The evolution from language chat models
to functional language agents demands that models not only master human
interaction, reasoning, and planning but also ensure grounding in the relevant
environments. This calls for a harmonious blend of language and coding
capabilities in the models. Lemur and Lemur-Chat are proposed to address this
necessity, demonstrating balanced proficiencies in both domains, unlike
existing open-source models that tend to specialize in either. Through
meticulous pre-training using a code-intensive corpus and instruction
fine-tuning on text and code data, our models achieve state-of-the-art averaged
performance across diverse text and coding benchmarks among open-source models.
Comprehensive experiments demonstrate Lemur's superiority over existing
open-source models and its proficiency across various agent tasks involving
human communication, tool usage, and interaction under fully- and partially-
observable environments. The harmonization between natural and programming
languages enables Lemur-Chat to significantly narrow the gap with proprietary
models on agent abilities, providing key insights into developing advanced
open-source agents adept at reasoning, planning, and operating seamlessly
across environments. https://github.com/OpenLemur/Lemur
Prometheus: Inducing Fine-grained Evaluation Capability in Language
  Models
Authors: name: Seungone Kim, name: Jamin Shin, name: Yejin Cho, name: Joel Jang, name: Shayne Longpre, name: Hwaran Lee, name: Sangdoo Yun, name: Seongjin Shin, name: Sungdong Kim, name: James Thorne, name: Minjoon Seo
Published: 2023-10-12 16:50:08

Recently, using a powerful proprietary Large Language Model (LLM) (e.g.,
GPT-4) as an evaluator for long-form responses has become the de facto
standard. However, for practitioners with large-scale evaluation tasks and
custom criteria in consideration (e.g., child-readability), using proprietary
LLMs as an evaluator is unreliable due to the closed-source nature,
uncontrolled versioning, and prohibitive costs. In this work, we propose
Prometheus, a fully open-source LLM that is on par with GPT-4's evaluation
capabilities when the appropriate reference materials (reference answer, score
rubric) are accompanied. We first construct the Feedback Collection, a new
dataset that consists of 1K fine-grained score rubrics, 20K instructions, and
100K responses and language feedback generated by GPT-4. Using the Feedback
Collection, we train Prometheus, a 13B evaluator LLM that can assess any given
long-form text based on customized score rubric provided by the user.
Experimental results show that Prometheus scores a Pearson correlation of 0.897
with human evaluators when evaluating with 45 customized score rubrics, which
is on par with GPT-4 (0.882), and greatly outperforms ChatGPT (0.392).
Furthermore, measuring correlation with GPT-4 with 1222 customized score
rubrics across four benchmarks (MT Bench, Vicuna Bench, Feedback Bench, Flask
Eval) shows similar trends, bolstering Prometheus's capability as an evaluator
LLM. Lastly, Prometheus achieves the highest accuracy on two human preference
benchmarks (HHH Alignment & MT Bench Human Judgment) compared to open-sourced
reward models explicitly trained on human preference datasets, highlighting its
potential as an universal reward model. We open-source our code, dataset, and
model at https://github.com/kaistAI/Prometheus.
EIPE-text: Evaluation-Guided Iterative Plan Extraction for Long-Form
  Narrative Text Generation
Authors: name: Wang You, name: Wenshan Wu, name: Yaobo Liang, name: Shaoguang Mao, name: Chenfei Wu, name: Maosong Cao, name: Yuzhe Cai, name: Yiduo Guo, name: Yan Xia, name: Furu Wei, name: Nan Duan
Published: 2023-10-12 10:21:37

Plan-and-Write is a common hierarchical approach in long-form narrative text
generation, which first creates a plan to guide the narrative writing.
Following this approach, several studies rely on simply prompting large
language models for planning, which often yields suboptimal results. In this
paper, we propose a new framework called Evaluation-guided Iterative Plan
Extraction for long-form narrative text generation (EIPE-text), which extracts
plans from the corpus of narratives and utilizes the extracted plans to
construct a better planner. EIPE-text has three stages: plan extraction,
learning, and inference. In the plan extraction stage, it iteratively extracts
and improves plans from the narrative corpus and constructs a plan corpus. We
propose a question answer (QA) based evaluation mechanism to automatically
evaluate the plans and generate detailed plan refinement instructions to guide
the iterative improvement. In the learning stage, we build a better planner by
fine-tuning with the plan corpus or in-context learning with examples in the
plan corpus. Finally, we leverage a hierarchical approach to generate long-form
narratives. We evaluate the effectiveness of EIPE-text in the domains of novels
and storytelling. Both GPT-4-based evaluations and human evaluations
demonstrate that our method can generate more coherent and relevant long-form
narratives. Our code will be released in the future.
LangNav: Language as a Perceptual Representation for Navigation
Authors: name: Bowen Pan, name: Rameswar Panda, name: SouYoung Jin, name: Rogerio Feris, name: Aude Oliva, name: Phillip Isola, name: Yoon Kim
Published: 2023-10-11 20:52:30

We explore the use of language as a perceptual representation for
vision-and-language navigation. Our approach uses off-the-shelf vision systems
(for image captioning and object detection) to convert an agent's egocentric
panoramic view at each time step into natural language descriptions. We then
finetune a pretrained language model to select an action, based on the current
view and the trajectory history, that would best fulfill the navigation
instructions. In contrast to the standard setup which adapts a pretrained
language model to work directly with continuous visual features from pretrained
vision models, our approach instead uses (discrete) language as the perceptual
representation. We explore two use cases of our language-based navigation
(LangNav) approach on the R2R vision-and-language navigation benchmark:
generating synthetic trajectories from a prompted large language model (GPT-4)
with which to finetune a smaller language model; and sim-to-real transfer where
we transfer a policy learned on a simulated environment (ALFRED) to a
real-world environment (R2R). Our approach is found to improve upon strong
baselines that rely on visual features in settings where only a few gold
trajectories (10-100) are available, demonstrating the potential of using
language as a perceptual representation for navigation tasks.
Drag View: Generalizable Novel View Synthesis with Unposed Imagery
Authors: name: Zhiwen Fan, name: Panwang Pan, name: Peihao Wang, name: Yifan Jiang, name: Hanwen Jiang, name: Dejia Xu, name: Zehao Zhu, name: Dilin Wang, name: Zhangyang Wang
Published: 2023-10-05 17:24:36

We introduce DragView, a novel and interactive framework for generating novel
views of unseen scenes. DragView initializes the new view from a single source
image, and the rendering is supported by a sparse set of unposed multi-view
images, all seamlessly executed within a single feed-forward pass. Our approach
begins with users dragging a source view through a local relative coordinate
system. Pixel-aligned features are obtained by projecting the sampled 3D points
along the target ray onto the source view. We then incorporate a view-dependent
modulation layer to effectively handle occlusion during the projection.
Additionally, we broaden the epipolar attention mechanism to encompass all
source pixels, facilitating the aggregation of initialized coordinate-aligned
point features from other unposed views. Finally, we employ another transformer
to decode ray features into final pixel intensities. Crucially, our framework
does not rely on either 2D prior models or the explicit estimation of camera
poses. During testing, DragView showcases the capability to generalize to new
scenes unseen during training, also utilizing only unposed support images,
enabling the generation of photo-realistic new views characterized by flexible
camera trajectories. In our experiments, we conduct a comprehensive comparison
of the performance of DragView with recent scene representation networks
operating under pose-free conditions, as well as with generalizable NeRFs
subject to noisy test camera poses. DragView consistently demonstrates its
superior performance in view synthesis quality, while also being more
user-friendly. Project page: https://zhiwenfan.github.io/DragView/.
UniAudio: An Audio Foundation Model Toward Universal Audio Generation
Authors: name: Dongchao Yang, name: Jinchuan Tian, name: Xu Tan, name: Rongjie Huang, name: Songxiang Liu, name: Xuankai Chang, name: Jiatong Shi, name: Sheng Zhao, name: Jiang Bian, name: Xixin Wu, name: Zhou Zhao, name: Helen Meng
Published: 2023-10-01 15:49:46

Language models (LMs) have demonstrated the capability to handle a variety of
generative tasks. This paper presents the UniAudio system, which, unlike prior
task-specific approaches, leverages LMs techniques to generate multiple types
of audio (including speech, sounds, music, and singing) with given input
conditions. UniAudio 1) first tokenizes all types of target audio along with
other condition modalities, 2) concatenates source-target pair as a single
sequence, and 3) performs next-token prediction using LMs. Also, a multi-scale
Transformer model is proposed to handle the overly long sequences caused by the
residual vector quantization based neural codec in tokenization. Training of
UniAudio is scaled up to 165K hours of audio and 1B parameters, based on all
generative tasks, aiming to obtain sufficient prior knowledge not only in the
intrinsic properties of audio but also the inter-relationship between audio and
other modalities. Therefore, the trained UniAudio model has the potential to
become a foundation model for universal audio generation: it shows strong
capability in all trained tasks and can seamlessly support new audio generation
tasks after simple fine-tuning. Experiments demonstrate that UniAudio achieves
state-of-the-art or at least competitive results on most of the 11 tasks. Demo
and code are released at https://github.com/yangdongchao/UniAudio