Orionnnn/hub-stats huggingface dataset:
Reverse Training to Nurse the Reversal Curse
Authors: name: Olga Golovneva, name: Zeyuan Allen-Zhu, name: Jason Weston, name: Sainbayar Sukhbaatar
Published: 2024-03-20 17:55:35

Large language models (LLMs) have a surprising failure: when trained on "A
has a feature B", they do not generalize to "B is a feature of A", which is
termed the Reversal Curse. Even when training with trillions of tokens this
issue still appears due to Zipf's law - hence even if we train on the entire
internet. This work proposes an alternative training scheme, called reverse
training, whereby all words are used twice, doubling the amount of available
tokens. The LLM is trained in both forward and reverse directions by reversing
the training strings while preserving (i.e., not reversing) chosen substrings,
such as entities. We show that data-matched reverse-trained models provide
superior performance to standard models on standard tasks, and compute-matched
reverse-trained models provide far superior performance on reversal tasks,
helping resolve the reversal curse issue.
IDAdapter: Learning Mixed Features for Tuning-Free Personalization of
  Text-to-Image Models
Authors: name: Siying Cui, name: Jiankang Deng, name: Jia Guo, name: Xiang An, name: Yongle Zhao, name: Xinyu Wei, name: Ziyong Feng
Published: 2024-03-20 12:13:04

Leveraging Stable Diffusion for the generation of personalized portraits has
emerged as a powerful and noteworthy tool, enabling users to create
high-fidelity, custom character avatars based on their specific prompts.
However, existing personalization methods face challenges, including test-time
fine-tuning, the requirement of multiple input images, low preservation of
identity, and limited diversity in generated outcomes. To overcome these
challenges, we introduce IDAdapter, a tuning-free approach that enhances the
diversity and identity preservation in personalized image generation from a
single face image. IDAdapter integrates a personalized concept into the
generation process through a combination of textual and visual injections and a
face identity loss. During the training phase, we incorporate mixed features
from multiple reference images of a specific identity to enrich
identity-related content details, guiding the model to generate images with
more diverse styles, expressions, and angles compared to previous works.
Extensive evaluations demonstrate the effectiveness of our method, achieving
both diversity and identity fidelity in generated images.
Mora: Enabling Generalist Video Generation via A Multi-Agent Framework
Authors: name: Zhengqing Yuan, name: Ruoxi Chen, name: Zhaoxu Li, name: Haolong Jia, name: Lifang He, name: Chi Wang, name: Lichao Sun
Published: 2024-03-20 02:19:21

Sora is the first large-scale generalist video generation model that garnered
significant attention across society. Since its launch by OpenAI in February
2024, no other video generation models have paralleled {Sora}'s performance or
its capacity to support a broad spectrum of video generation tasks.
Additionally, there are only a few fully published video generation models,
with the majority being closed-source. To address this gap, this paper proposes
a new multi-agent framework Mora, which incorporates several advanced visual AI
agents to replicate generalist video generation demonstrated by Sora. In
particular, Mora can utilize multiple visual agents and successfully mimic
Sora's video generation capabilities in various tasks, such as (1)
text-to-video generation, (2) text-conditional image-to-video generation, (3)
extend generated videos, (4) video-to-video editing, (5) connect videos and (6)
simulate digital worlds. Our extensive experimental results show that Mora
achieves performance that is proximate to that of Sora in various tasks.
However, there exists an obvious performance gap between our work and Sora when
assessed holistically. In summary, we hope this project can guide the future
trajectory of video generation through collaborative AI agents.
TexDreamer: Towards Zero-Shot High-Fidelity 3D Human Texture Generation
Authors: name: Yufei Liu, name: Junwei Zhu, name: Junshu Tang, name: Shijie Zhang, name: Jiangning Zhang, name: Weijian Cao, name: Chengjie Wang, name: Yunsheng Wu, name: Dongjin Huang
Published: 2024-03-19 17:02:07

Texturing 3D humans with semantic UV maps remains a challenge due to the
difficulty of acquiring reasonably unfolded UV. Despite recent text-to-3D
advancements in supervising multi-view renderings using large text-to-image
(T2I) models, issues persist with generation speed, text consistency, and
texture quality, resulting in data scarcity among existing datasets. We present
TexDreamer, the first zero-shot multimodal high-fidelity 3D human texture
generation model. Utilizing an efficient texture adaptation finetuning
strategy, we adapt large T2I model to a semantic UV structure while preserving
its original generalization capability. Leveraging a novel feature translator
module, the trained model is capable of generating high-fidelity 3D human
textures from either text or image within seconds. Furthermore, we introduce
ArTicuLated humAn textureS (ATLAS), the largest high-resolution (1024 X 1024)
3D human texture dataset which contains 50k high-fidelity textures with text
descriptions.
FRESCO: Spatial-Temporal Correspondence for Zero-Shot Video Translation
Authors: name: Shuai Yang, name: Yifan Zhou, name: Ziwei Liu, name: Chen Change Loy
Published: 2024-03-19 17:59:18

The remarkable efficacy of text-to-image diffusion models has motivated
extensive exploration of their potential application in video domains.
Zero-shot methods seek to extend image diffusion models to videos without
necessitating model training. Recent methods mainly focus on incorporating
inter-frame correspondence into attention mechanisms. However, the soft
constraint imposed on determining where to attend to valid features can
sometimes be insufficient, resulting in temporal inconsistency. In this paper,
we introduce FRESCO, intra-frame correspondence alongside inter-frame
correspondence to establish a more robust spatial-temporal constraint. This
enhancement ensures a more consistent transformation of semantically similar
content across frames. Beyond mere attention guidance, our approach involves an
explicit update of features to achieve high spatial-temporal consistency with
the input video, significantly improving the visual coherence of the resulting
translated videos. Extensive experiments demonstrate the effectiveness of our
proposed framework in producing high-quality, coherent videos, marking a
notable improvement over existing zero-shot methods.
GaussianFlow: Splatting Gaussian Dynamics for 4D Content Creation
Authors: name: Quankai Gao, name: Qiangeng Xu, name: Zhe Cao, name: Ben Mildenhall, name: Wenchao Ma, name: Le Chen, name: Danhang Tang, name: Ulrich Neumann
Published: 2024-03-19 02:22:21

Creating 4D fields of Gaussian Splatting from images or videos is a
challenging task due to its under-constrained nature. While the optimization
can draw photometric reference from the input videos or be regulated by
generative models, directly supervising Gaussian motions remains underexplored.
In this paper, we introduce a novel concept, Gaussian flow, which connects the
dynamics of 3D Gaussians and pixel velocities between consecutive frames. The
Gaussian flow can be efficiently obtained by splatting Gaussian dynamics into
the image space. This differentiable process enables direct dynamic supervision
from optical flow. Our method significantly benefits 4D dynamic content
generation and 4D novel view synthesis with Gaussian Splatting, especially for
contents with rich motions that are hard to be handled by existing methods. The
common color drifting issue that happens in 4D generation is also resolved with
improved Guassian dynamics. Superior visual quality on extensive experiments
demonstrates our method's effectiveness. Quantitative and qualitative
evaluations show that our method achieves state-of-the-art results on both
tasks of 4D generation and 4D novel view synthesis. Project page:
https://zerg-overmind.github.io/GaussianFlow.github.io/
LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic
  Prompt Compression
Authors: name: Zhuoshi Pan, name: Qianhui Wu, name: Huiqiang Jiang, name: Menglin Xia, name: Xufang Luo, name: Jue Zhang, name: Qingwei Lin, name: Victor Rühle, name: Yuqing Yang, name: Chin-Yew Lin, name: H. Vicky Zhao, name: Lili Qiu, name: Dongmei Zhang
Published: 2024-03-19 17:59:56

This paper focuses on task-agnostic prompt compression for better
generalizability and efficiency. Considering the redundancy in natural
language, existing approaches compress prompts by removing tokens or lexical
units according to their information entropy obtained from a causal language
model such as LLaMa-7B. The challenge is that information entropy may be a
suboptimal compression metric: (i) it only leverages unidirectional context and
may fail to capture all essential information needed for prompt compression;
(ii) it is not aligned with the prompt compression objective.
  To address these issues, we propose a data distillation procedure to derive
knowledge from an LLM to compress prompts without losing crucial information,
and meantime, introduce an extractive text compression dataset. We formulate
prompt compression as a token classification problem to guarantee the
faithfulness of the compressed prompt to the original one, and use a
Transformer encoder as the base architecture to capture all essential
information for prompt compression from the full bidirectional context. Our
approach leads to lower latency by explicitly learning the compression
objective with smaller models such as XLM-RoBERTa-large and mBERT.
  We evaluate our method on both in-domain and out-of-domain datasets,
including MeetingBank, LongBench, ZeroScrolls, GSM8K, and BBH. Despite its
small size, our model shows significant performance gains over strong baselines
and demonstrates robust generalization ability across different LLMs.
Additionally, our model is 3x-6x faster than existing prompt compression
methods, while accelerating the end-to-end latency by 1.6x-2.9x with
compression ratios of 2x-5x.
mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document
  Understanding
Authors: name: Anwen Hu, name: Haiyang Xu, name: Jiabo Ye, name: Ming Yan, name: Liang Zhang, name: Bo Zhang, name: Chen Li, name: Ji Zhang, name: Qin Jin, name: Fei Huang, name: Jingren Zhou
Published: 2024-03-19 16:48:40

Structure information is critical for understanding the semantics of
text-rich images, such as documents, tables, and charts. Existing Multimodal
Large Language Models (MLLMs) for Visual Document Understanding are equipped
with text recognition ability but lack general structure understanding
abilities for text-rich document images. In this work, we emphasize the
importance of structure information in Visual Document Understanding and
propose the Unified Structure Learning to boost the performance of MLLMs. Our
Unified Structure Learning comprises structure-aware parsing tasks and
multi-grained text localization tasks across 5 domains: document, webpage,
table, chart, and natural image. To better encode structure information, we
design a simple and effective vision-to-text module H-Reducer, which can not
only maintain the layout information but also reduce the length of visual
features by merging horizontal adjacent patches through convolution, enabling
the LLM to understand high-resolution images more efficiently. Furthermore, by
constructing structure-aware text sequences and multi-grained pairs of texts
and bounding boxes for publicly available text-rich images, we build a
comprehensive training set DocStruct4M to support structure learning. Finally,
we construct a small but high-quality reasoning tuning dataset DocReason25K to
trigger the detailed explanation ability in the document domain. Our model
DocOwl 1.5 achieves state-of-the-art performance on 10 visual document
understanding benchmarks, improving the SOTA performance of MLLMs with a 7B LLM
by more than 10 points in 5/10 benchmarks. Our codes, models, and datasets are
publicly available at
https://github.com/X-PLUG/mPLUG-DocOwl/tree/main/DocOwl1.5.
Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for
  Large Language Models
Authors: name: Zehui Chen, name: Kuikun Liu, name: Qiuchen Wang, name: Wenwei Zhang, name: Jiangning Liu, name: Dahua Lin, name: Kai Chen, name: Feng Zhao
Published: 2024-03-19 16:26:10

Open-sourced Large Language Models (LLMs) have achieved great success in
various NLP tasks, however, they are still far inferior to API-based models
when acting as agents. How to integrate agent ability into general LLMs becomes
a crucial and urgent problem. This paper first delivers three key observations:
(1) the current agent training corpus is entangled with both formats following
and agent reasoning, which significantly shifts from the distribution of its
pre-training data; (2) LLMs exhibit different learning speeds on the
capabilities required by agent tasks; and (3) current approaches have
side-effects when improving agent abilities by introducing hallucinations.
Based on the above findings, we propose Agent-FLAN to effectively Fine-tune
LANguage models for Agents. Through careful decomposition and redesign of the
training corpus, Agent-FLAN enables Llama2-7B to outperform prior best works by
3.5\% across various agent evaluation datasets. With comprehensively
constructed negative samples, Agent-FLAN greatly alleviates the hallucination
issues based on our established evaluation benchmark. Besides, it consistently
improves the agent capability of LLMs when scaling model sizes while slightly
enhancing the general capability of LLMs. The code will be available at
https://github.com/InternLM/Agent-FLAN.
Vid2Robot: End-to-end Video-conditioned Policy Learning with
  Cross-Attention Transformers
Authors: name: Vidhi Jain, name: Maria Attarian, name: Nikhil J Joshi, name: Ayzaan Wahid, name: Danny Driess, name: Quan Vuong, name: Pannag R Sanketi, name: Pierre Sermanet, name: Stefan Welker, name: Christine Chan, name: Igor Gilitschenski, name: Yonatan Bisk, name: Debidatta Dwibedi
Published: 2024-03-19 17:47:37

While large-scale robotic systems typically rely on textual instructions for
tasks, this work explores a different approach: can robots infer the task
directly from observing humans? This shift necessitates the robot's ability to
decode human intent and translate it into executable actions within its
physical constraints and environment. We introduce Vid2Robot, a novel
end-to-end video-based learning framework for robots. Given a video
demonstration of a manipulation task and current visual observations, Vid2Robot
directly produces robot actions. This is achieved through a unified
representation model trained on a large dataset of human video and robot
trajectory. The model leverages cross-attention mechanisms to fuse prompt video
features to the robot's current state and generate appropriate actions that
mimic the observed task. To further improve policy performance, we propose
auxiliary contrastive losses that enhance the alignment between human and robot
video representations. We evaluate Vid2Robot on real-world robots,
demonstrating a 20% improvement in performance compared to other
video-conditioned policies when using human demonstration videos. Additionally,
our model exhibits emergent capabilities, such as successfully transferring
observed motions from one object to another, and long-horizon composition, thus
showcasing its potential for real-world applications. Project website:
vid2robot.github.io
TnT-LLM: Text Mining at Scale with Large Language Models
Authors: name: Mengting Wan, name: Tara Safavi, name: Sujay Kumar Jauhar, name: Yujin Kim, name: Scott Counts, name: Jennifer Neville, name: Siddharth Suri, name: Chirag Shah, name: Ryen W White, name: Longqi Yang, name: Reid Andersen, name: Georg Buscher, name: Dhruv Joshi, name: Nagu Rangan
Published: 2024-03-18 18:45:28

Transforming unstructured text into structured and meaningful forms,
organized by useful category labels, is a fundamental step in text mining for
downstream analysis and application. However, most existing methods for
producing label taxonomies and building text-based label classifiers still rely
heavily on domain expertise and manual curation, making the process expensive
and time-consuming. This is particularly challenging when the label space is
under-specified and large-scale data annotations are unavailable. In this
paper, we address these challenges with Large Language Models (LLMs), whose
prompt-based interface facilitates the induction and use of large-scale pseudo
labels. We propose TnT-LLM, a two-phase framework that employs LLMs to automate
the process of end-to-end label generation and assignment with minimal human
effort for any given use-case. In the first phase, we introduce a zero-shot,
multi-stage reasoning approach which enables LLMs to produce and refine a label
taxonomy iteratively. In the second phase, LLMs are used as data labelers that
yield training samples so that lightweight supervised classifiers can be
reliably built, deployed, and served at scale. We apply TnT-LLM to the analysis
of user intent and conversational domain for Bing Copilot (formerly Bing Chat),
an open-domain chat-based search engine. Extensive experiments using both human
and automatic evaluation metrics demonstrate that TnT-LLM generates more
accurate and relevant label taxonomies when compared against state-of-the-art
baselines, and achieves a favorable balance between accuracy and efficiency for
classification at scale. We also share our practical experiences and insights
on the challenges and opportunities of using LLMs for large-scale text mining
in real-world applications.
Chart-based Reasoning: Transferring Capabilities from LLMs to VLMs
Authors: name: Victor Carbune, name: Hassan Mansoor, name: Fangyu Liu, name: Rahul Aralikatte, name: Gilles Baechler, name: Jindong Chen, name: Abhanshu Sharma
Published: 2024-03-19 10:03:07

Vision-language models (VLMs) are achieving increasingly strong performance
on multimodal tasks. However, reasoning capabilities remain limited
particularly for smaller VLMs, while those of large-language models (LLMs) have
seen numerous improvements. We propose a technique to transfer capabilities
from LLMs to VLMs. On the recently introduced ChartQA, our method obtains
state-of-the-art performance when applied on the PaLI3-5B VLM by
chen2023pali3, while also enabling much better performance on PlotQA
and FigureQA.
  We first improve the chart representation by continuing the pre-training
stage using an improved version of the chart-to-table translation task by
liu2023deplot. We then propose constructing a 20x larger dataset than
the original training set. To improve general reasoning capabilities and
improve numerical operations, we synthesize reasoning traces using the table
representation of charts. Lastly, our model is fine-tuned using the multitask
loss introduced by hsieh2023distilling.
  Our variant ChartPaLI-5B outperforms even 10x larger models such as PaLIX-55B
without using an upstream OCR system, while keeping inference time constant
compared to the PaLI3-5B baseline. When rationales are further refined with a
simple program-of-thought prompt chen2023program, our model outperforms
the recently introduced Gemini Ultra and GPT-4V.
GVGEN: Text-to-3D Generation with Volumetric Representation
Authors: name: Xianglong He, name: Junyi Chen, name: Sida Peng, name: Di Huang, name: Yangguang Li, name: Xiaoshui Huang, name: Chun Yuan, name: Wanli Ouyang, name: Tong He
Published: 2024-03-19 17:57:52

In recent years, 3D Gaussian splatting has emerged as a powerful technique
for 3D reconstruction and generation, known for its fast and high-quality
rendering capabilities. To address these shortcomings, this paper introduces a
novel diffusion-based framework, GVGEN, designed to efficiently generate 3D
Gaussian representations from text input. We propose two innovative
techniques:(1) Structured Volumetric Representation. We first arrange
disorganized 3D Gaussian points as a structured form GaussianVolume. This
transformation allows the capture of intricate texture details within a volume
composed of a fixed number of Gaussians. To better optimize the representation
of these details, we propose a unique pruning and densifying method named the
Candidate Pool Strategy, enhancing detail fidelity through selective
optimization. (2) Coarse-to-fine Generation Pipeline. To simplify the
generation of GaussianVolume and empower the model to generate instances with
detailed 3D geometry, we propose a coarse-to-fine pipeline. It initially
constructs a basic geometric structure, followed by the prediction of complete
Gaussian attributes. Our framework, GVGEN, demonstrates superior performance in
qualitative and quantitative assessments compared to existing 3D generation
methods. Simultaneously, it maintains a fast generation speed (sim7
seconds), effectively striking a balance between quality and efficiency.
ComboVerse: Compositional 3D Assets Creation Using Spatially-Aware
  Diffusion Guidance
Authors: name: Yongwei Chen, name: Tengfei Wang, name: Tong Wu, name: Xingang Pan, name: Kui Jia, name: Ziwei Liu
Published: 2024-03-19 03:39:43

Generating high-quality 3D assets from a given image is highly desirable in
various applications such as AR/VR. Recent advances in single-image 3D
generation explore feed-forward models that learn to infer the 3D model of an
object without optimization. Though promising results have been achieved in
single object generation, these methods often struggle to model complex 3D
assets that inherently contain multiple objects. In this work, we present
ComboVerse, a 3D generation framework that produces high-quality 3D assets with
complex compositions by learning to combine multiple models. 1) We first
perform an in-depth analysis of this ``multi-object gap'' from both model and
data perspectives. 2) Next, with reconstructed 3D models of different objects,
we seek to adjust their sizes, rotation angles, and locations to create a 3D
asset that matches the given image. 3) To automate this process, we apply
spatially-aware score distillation sampling (SSDS) from pretrained diffusion
models to guide the positioning of objects. Our proposed framework emphasizes
spatial alignment of objects, compared with standard score distillation
sampling, and thus achieves more accurate results. Extensive experiments
validate ComboVerse achieves clear improvements over existing methods in
generating compositional 3D assets.
FouriScale: A Frequency Perspective on Training-Free High-Resolution
  Image Synthesis
Authors: name: Linjiang Huang, name: Rongyao Fang, name: Aiping Zhang, name: Guanglu Song, name: Si Liu, name: Yu Liu, name: Hongsheng Li
Published: 2024-03-19 17:59:33

In this study, we delve into the generation of high-resolution images from
pre-trained diffusion models, addressing persistent challenges, such as
repetitive patterns and structural distortions, that emerge when models are
applied beyond their trained resolutions. To address this issue, we introduce
an innovative, training-free approach FouriScale from the perspective of
frequency domain analysis. We replace the original convolutional layers in
pre-trained diffusion models by incorporating a dilation technique along with a
low-pass operation, intending to achieve structural consistency and scale
consistency across resolutions, respectively. Further enhanced by a
padding-then-crop strategy, our method can flexibly handle text-to-image
generation of various aspect ratios. By using the FouriScale as guidance, our
method successfully balances the structural integrity and fidelity of generated
images, achieving an astonishing capacity of arbitrary-size, high-resolution,
and high-quality generation. With its simplicity and compatibility, our method
can provide valuable insights for future explorations into the synthesis of
ultra-high-resolution images. The code will be released at
https://github.com/LeonHLJ/FouriScale.