Orionnnn/hub-stats huggingface dataset:
Improving fine-grained understanding in image-text pre-training
Authors: name: Ioana Bica, name: Anastasija Ilić, name: Matthias Bauer, name: Goker Erdogan, name: Matko Bošnjak, name: Christos Kaplanis, name: Alexey A. Gritsenko, name: Matthias Minderer, name: Charles Blundell, name: Razvan Pascanu, name: Jovana Mitrović
Published: 2024-01-18 10:28:45

We introduce SPARse Fine-grained Contrastive Alignment (SPARC), a simple
method for pretraining more fine-grained multimodal representations from
image-text pairs. Given that multiple image patches often correspond to single
words, we propose to learn a grouping of image patches for every token in the
caption. To achieve this, we use a sparse similarity metric between image
patches and language tokens and compute for each token a language-grouped
vision embedding as the weighted average of patches. The token and
language-grouped vision embeddings are then contrasted through a fine-grained
sequence-wise loss that only depends on individual samples and does not require
other batch samples as negatives. This enables more detailed information to be
learned in a computationally inexpensive manner. SPARC combines this
fine-grained loss with a contrastive loss between global image and text
embeddings to learn representations that simultaneously encode global and local
information. We thoroughly evaluate our proposed method and show improved
performance over competing approaches both on image-level tasks relying on
coarse-grained information, e.g. classification, as well as region-level tasks
relying on fine-grained information, e.g. retrieval, object detection, and
segmentation. Moreover, SPARC improves model faithfulness and captioning in
foundational vision-language models.
ChatQA: Building GPT-4 Level Conversational QA Models
Authors: name: Zihan Liu, name: Wei Ping, name: Rajarshi Roy, name: Peng Xu, name: Mohammad Shoeybi, name: Bryan Catanzaro
Published: 2024-01-18 18:59:11

In this work, we introduce ChatQA, a family of conversational question
answering (QA) models, that obtain GPT-4 level accuracies. Specifically, we
propose a two-stage instruction tuning method that can significantly improve
the zero-shot conversational QA results from large language models (LLMs). To
handle retrieval in conversational QA, we fine-tune a dense retriever on a
multi-turn QA dataset, which provides comparable results to using the
state-of-the-art query rewriting model while largely reducing deployment cost.
Notably, our ChatQA-70B can outperform GPT-4 in terms of average score on 10
conversational QA datasets (54.14 vs. 53.90), without relying on any synthetic
data from OpenAI GPT models.
Self-Rewarding Language Models
Authors: name: Weizhe Yuan, name: Richard Yuanzhe Pang, name: Kyunghyun Cho, name: Sainbayar Sukhbaatar, name: Jing Xu, name: Jason Weston
Published: 2024-01-18 14:43:47

We posit that to achieve superhuman agents, future models require superhuman
feedback in order to provide an adequate training signal. Current approaches
commonly train reward models from human preferences, which may then be
bottlenecked by human performance level, and secondly these separate frozen
reward models cannot then learn to improve during LLM training. In this work,
we study Self-Rewarding Language Models, where the language model itself is
used via LLM-as-a-Judge prompting to provide its own rewards during training.
We show that during Iterative DPO training that not only does instruction
following ability improve, but also the ability to provide high-quality rewards
to itself. Fine-tuning Llama 2 70B on three iterations of our approach yields a
model that outperforms many existing systems on the AlpacaEval 2.0 leaderboard,
including Claude 2, Gemini Pro, and GPT-4 0613. While only a preliminary study,
this work opens the door to the possibility of models that can continually
improve in both axes.
WorldDreamer: Towards General World Models for Video Generation via
  Predicting Masked Tokens
Authors: name: Xiaofeng Wang, name: Zheng Zhu, name: Guan Huang, name: Boyuan Wang, name: Xinze Chen, name: Jiwen Lu
Published: 2024-01-18 14:01:20

World models play a crucial role in understanding and predicting the dynamics
of the world, which is essential for video generation. However, existing world
models are confined to specific scenarios such as gaming or driving, limiting
their ability to capture the complexity of general world dynamic environments.
Therefore, we introduce WorldDreamer, a pioneering world model to foster a
comprehensive comprehension of general world physics and motions, which
significantly enhances the capabilities of video generation. Drawing
inspiration from the success of large language models, WorldDreamer frames
world modeling as an unsupervised visual sequence modeling challenge. This is
achieved by mapping visual inputs to discrete tokens and predicting the masked
ones. During this process, we incorporate multi-modal prompts to facilitate
interaction within the world model. Our experiments show that WorldDreamer
excels in generating videos across different scenarios, including natural
scenes and driving environments. WorldDreamer showcases versatility in
executing tasks such as text-to-video conversion, image-tovideo synthesis, and
video editing. These results underscore WorldDreamer's effectiveness in
capturing dynamic elements within diverse general world environments.
GARField: Group Anything with Radiance Fields
Authors: name: Chung Min Kim, name: Mingxuan Wu, name: Justin Kerr, name: Ken Goldberg, name: Matthew Tancik, name: Angjoo Kanazawa
Published: 2024-01-17 18:57:53

Grouping is inherently ambiguous due to the multiple levels of granularity in
which one can decompose a scene -- should the wheels of an excavator be
considered separate or part of the whole? We present Group Anything with
Radiance Fields (GARField), an approach for decomposing 3D scenes into a
hierarchy of semantically meaningful groups from posed image inputs. To do this
we embrace group ambiguity through physical scale: by optimizing a
scale-conditioned 3D affinity feature field, a point in the world can belong to
different groups of different sizes. We optimize this field from a set of 2D
masks provided by Segment Anything (SAM) in a way that respects coarse-to-fine
hierarchy, using scale to consistently fuse conflicting masks from different
viewpoints. From this field we can derive a hierarchy of possible groupings via
automatic tree construction or user interaction. We evaluate GARField on a
variety of in-the-wild scenes and find it effectively extracts groups at many
levels: clusters of objects, objects, and various subparts. GARField inherently
represents multi-view consistent groupings and produces higher fidelity groups
than the input SAM masks. GARField's hierarchical grouping could have exciting
downstream applications such as 3D asset extraction or dynamic scene
understanding. See the project website at https://www.garfield.studio/
SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene
  Understanding
Authors: name: Baoxiong Jia, name: Yixin Chen, name: Huangyue Yu, name: Yan Wang, name: Xuesong Niu, name: Tengyu Liu, name: Qing Li, name: Siyuan Huang
Published: 2024-01-17 17:04:35

3D vision-language grounding, which focuses on aligning language with the 3D
physical environment, stands as a cornerstone in the development of embodied
agents. In comparison to recent advancements in the 2D domain, grounding
language in 3D scenes faces several significant challenges: (i) the inherent
complexity of 3D scenes due to the diverse object configurations, their rich
attributes, and intricate relationships; (ii) the scarcity of paired 3D
vision-language data to support grounded learning; and (iii) the absence of a
unified learning framework to distill knowledge from grounded 3D data. In this
work, we aim to address these three major challenges in 3D vision-language by
examining the potential of systematically upscaling 3D vision-language learning
in indoor environments. We introduce the first million-scale 3D vision-language
dataset, SceneVerse, encompassing about 68K 3D indoor scenes and comprising
2.5M vision-language pairs derived from both human annotations and our scalable
scene-graph-based generation approach. We demonstrate that this scaling allows
for a unified pre-training framework, Grounded Pre-training for Scenes (GPS),
for 3D vision-language learning. Through extensive experiments, we showcase the
effectiveness of GPS by achieving state-of-the-art performance on all existing
3D visual grounding benchmarks. The vast potential of SceneVerse and GPS is
unveiled through zero-shot transfer experiments in the challenging 3D
vision-language tasks. Project website: https://scene-verse.github.io .
Compose and Conquer: Diffusion-Based 3D Depth Aware Composable Image
  Synthesis
Authors: name: Jonghyun Lee, name: Hansam Cho, name: Youngjoon Yoo, name: Seoung Bum Kim, name: Yonghyun Jeong
Published: 2024-01-17 08:30:47

Addressing the limitations of text as a source of accurate layout
representation in text-conditional diffusion models, many works incorporate
additional signals to condition certain attributes within a generated image.
Although successful, previous works do not account for the specific
localization of said attributes extended into the three dimensional plane. In
this context, we present a conditional diffusion model that integrates control
over three-dimensional object placement with disentangled representations of
global stylistic semantics from multiple exemplar images. Specifically, we
first introduce depth disentanglement training to leverage the
relative depth of objects as an estimator, allowing the model to identify the
absolute positions of unseen objects through the use of synthetic image
triplets. We also introduce soft guidance, a method for imposing
global semantics onto targeted regions without the use of any additional
localization cues. Our integrated framework, Compose and Conquer
(CnC), unifies these techniques to localize multiple conditions in a
disentangled manner. We demonstrate that our approach allows perception of
objects at varying depths while offering a versatile framework for composing
localized objects with different global semantics. Code:
https://github.com/tomtom1103/compose-and-conquer/
SiT: Exploring Flow and Diffusion-based Generative Models with Scalable
  Interpolant Transformers
Authors: name: Nanye Ma, name: Mark Goldstein, name: Michael S. Albergo, name: Nicholas M. Boffi, name: Eric Vanden-Eijnden, name: Saining Xie
Published: 2024-01-16 18:55:25

We present Scalable Interpolant Transformers (SiT), a family of generative
models built on the backbone of Diffusion Transformers (DiT). The interpolant
framework, which allows for connecting two distributions in a more flexible way
than standard diffusion models, makes possible a modular study of various
design choices impacting generative models built on dynamical transport: using
discrete vs. continuous time learning, deciding the objective for the model to
learn, choosing the interpolant connecting the distributions, and deploying a
deterministic or stochastic sampler. By carefully introducing the above
ingredients, SiT surpasses DiT uniformly across model sizes on the conditional
ImageNet 256x256 benchmark using the exact same backbone, number of parameters,
and GFLOPs. By exploring various diffusion coefficients, which can be tuned
separately from learning, SiT achieves an FID-50K score of 2.06.
ICON: Incremental CONfidence for Joint Pose and Radiance Field
  Optimization
Authors: name: Weiyao Wang, name: Pierre Gleize, name: Hao Tang, name: Xingyu Chen, name: Kevin J Liang, name: Matt Feiszli
Published: 2024-01-17 03:18:02

Neural Radiance Fields (NeRF) exhibit remarkable performance for Novel View
Synthesis (NVS) given a set of 2D images. However, NeRF training requires
accurate camera pose for each input view, typically obtained by
Structure-from-Motion (SfM) pipelines. Recent works have attempted to relax
this constraint, but they still often rely on decent initial poses which they
can refine. Here we aim at removing the requirement for pose initialization. We
present Incremental CONfidence (ICON), an optimization procedure for training
NeRFs from 2D video frames. ICON only assumes smooth camera motion to estimate
initial guess for poses. Further, ICON introduces ``confidence": an adaptive
measure of model quality used to dynamically reweight gradients. ICON relies on
high-confidence poses to learn NeRF, and high-confidence 3D structure (as
encoded by NeRF) to learn poses. We show that ICON, without prior pose
initialization, achieves superior performance in both CO3D and HO3D versus
methods which use SfM pose.
Vision Mamba: Efficient Visual Representation Learning with
  Bidirectional State Space Model
Authors: name: Lianghui Zhu, name: Bencheng Liao, name: Qian Zhang, name: Xinlong Wang, name: Wenyu Liu, name: Xinggang Wang
Published: 2024-01-17 18:56:18

Recently the state space models (SSMs) with efficient hardware-aware designs,
i.e., Mamba, have shown great potential for long sequence modeling. Building
efficient and generic vision backbones purely upon SSMs is an appealing
direction. However, representing visual data is challenging for SSMs due to the
position-sensitivity of visual data and the requirement of global context for
visual understanding. In this paper, we show that the reliance of visual
representation learning on self-attention is not necessary and propose a new
generic vision backbone with bidirectional Mamba blocks (Vim), which marks the
image sequences with position embeddings and compresses the visual
representation with bidirectional state space models. On ImageNet
classification, COCO object detection, and ADE20k semantic segmentation tasks,
Vim achieves higher performance compared to well-established vision
transformers like DeiT, while also demonstrating significantly improved
computation & memory efficiency. For example, Vim is 2.8times faster than
DeiT and saves 86.8% GPU memory when performing batch inference to extract
features on images with a resolution of 1248times1248. The results
demonstrate that Vim is capable of overcoming the computation & memory
constraints on performing Transformer-style understanding for high-resolution
images and it has great potential to become the next-generation backbone for
vision foundation models. Code is available at https://github.com/hustvl/Vim.
TextureDreamer: Image-guided Texture Synthesis through Geometry-aware
  Diffusion
Authors: name: Yu-Ying Yeh, name: Jia-Bin Huang, name: Changil Kim, name: Lei Xiao, name: Thu Nguyen-Phuoc, name: Numair Khan, name: Cheng Zhang, name: Manmohan Chandraker, name: Carl S Marshall, name: Zhao Dong, name: Zhengqin Li
Published: 2024-01-17 18:55:49

We present TextureDreamer, a novel image-guided texture synthesis method to
transfer relightable textures from a small number of input images (3 to 5) to
target 3D shapes across arbitrary categories. Texture creation is a pivotal
challenge in vision and graphics. Industrial companies hire experienced artists
to manually craft textures for 3D assets. Classical methods require densely
sampled views and accurately aligned geometry, while learning-based methods are
confined to category-specific shapes within the dataset. In contrast,
TextureDreamer can transfer highly detailed, intricate textures from real-world
environments to arbitrary objects with only a few casually captured images,
potentially significantly democratizing texture creation. Our core idea,
personalized geometry-aware score distillation (PGSD), draws inspiration from
recent advancements in diffuse models, including personalized modeling for
texture information extraction, variational score distillation for detailed
appearance synthesis, and explicit geometry guidance with ControlNet. Our
integration and several essential modifications substantially improve the
texture quality. Experiments on real images spanning different categories show
that TextureDreamer can successfully transfer highly realistic, semantic
meaningful texture to arbitrary objects, surpassing the visual quality of
previous state-of-the-art.
Asynchronous Local-SGD Training for Language Modeling
Authors: name: Bo Liu, name: Rachita Chhaparia, name: Arthur Douillard, name: Satyen Kale, name: Andrei A. Rusu, name: Jiajun Shen, name: Arthur Szlam, name: Marc'Aurelio Ranzato
Published: 2024-01-17 11:17:04

Local stochastic gradient descent (Local-SGD), also referred to as federated
averaging, is an approach to distributed optimization where each device
performs more than one SGD update per communication. This work presents an
empirical study of {\it asynchronous} Local-SGD for training language models;
that is, each worker updates the global parameters as soon as it has finished
its SGD steps. We conduct a comprehensive investigation by examining how worker
hardware heterogeneity, model size, number of workers, and optimizer could
impact the learning performance. We find that with naive implementations,
asynchronous Local-SGD takes more iterations to converge than its synchronous
counterpart despite updating the (global) model parameters more frequently. We
identify momentum acceleration on the global parameters when worker gradients
are stale as a key challenge. We propose a novel method that utilizes a delayed
Nesterov momentum update and adjusts the workers' local training steps based on
their computation speed. This approach, evaluated with models up to 150M
parameters on the C4 dataset, matches the performance of synchronous Local-SGD
in terms of perplexity per update step, and significantly surpasses it in terms
of wall clock time.
ReFT: Reasoning with Reinforced Fine-Tuning
Authors: name: Trung Quoc Luong, name: Xinbo Zhang, name: Zhanming Jie, name: Peng Sun, name: Xiaoran Jin, name: Hang Li
Published: 2024-01-17 04:43:21

One way to enhance the reasoning capability of Large Language Models (LLMs)
is to conduct Supervised Fine-Tuning (SFT) using Chain-of-Thought (CoT)
annotations. This approach does not show sufficiently strong generalization
ability, however, because the training only relies on the given CoT data. In
math problem-solving, for example, there is usually only one annotated
reasoning path for each question in the training data. Intuitively, it would be
better for the algorithm to learn from multiple annotated reasoning paths given
a question. To address this issue, we propose a simple yet effective approach
called Reinforced Fine-Tuning (ReFT) to enhance the generalizability of
learning LLMs for reasoning, with math problem-solving as an example. ReFT
first warmups the model with SFT, and then employs on-line reinforcement
learning, specifically the PPO algorithm in this paper, to further fine-tune
the model, where an abundance of reasoning paths are automatically sampled
given the question and the rewards are naturally derived from the ground-truth
answers. Extensive experiments on GSM8K, MathQA, and SVAMP datasets show that
ReFT significantly outperforms SFT, and the performance can be potentially
further boosted by combining inference-time strategies such as majority voting
and re-ranking. Note that ReFT obtains the improvement by learning from the
same training questions as SFT, without relying on extra or augmented training
questions. This indicates a superior generalization ability for ReFT.
DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and
  DeepSpeed-Inference
Authors: name: Connor Holmes, name: Masahiro Tanaka, name: Michael Wyatt, name: Ammar Ahmad Awan, name: Jeff Rasley, name: Samyam Rajbhandari, name: Reza Yazdani Aminabadi, name: Heyang Qin, name: Arash Bakhtiari, name: Lev Kurilenko, name: Yuxiong He
Published: 2024-01-09 06:49:40

The deployment and scaling of large language models (LLMs) have become
critical as they permeate various applications, demanding high-throughput and
low-latency serving systems. Existing frameworks struggle to balance these
requirements, especially for workloads with long prompts. This paper introduces
DeepSpeed-FastGen, a system that employs Dynamic SplitFuse, a novel prompt and
generation composition strategy, to deliver up to 2.3x higher effective
throughput, 2x lower latency on average, and up to 3.7x lower (token-level)
tail latency, compared to state-of-the-art systems like vLLM. We leverage a
synergistic combination of DeepSpeed-MII and DeepSpeed-Inference to provide an
efficient and easy-to-use serving system for LLMs. DeepSpeed-FastGen's advanced
implementation supports a range of models and offers both non-persistent and
persistent deployment options, catering to diverse user scenarios from
interactive sessions to long-running applications. We present a detailed
benchmarking methodology, analyze the performance through latency-throughput
curves, and investigate scalability via load balancing. Our evaluations
demonstrate substantial improvements in throughput and latency across various
models and hardware configurations. We discuss our roadmap for future
enhancements, including broader model support and new hardware backends. The
DeepSpeed-FastGen code is readily available for community engagement and
contribution.
UniVG: Towards UNIfied-modal Video Generation
Authors: name: Ludan Ruan, name: Lei Tian, name: Chuanwei Huang, name: Xu Zhang, name: Xinyan Xiao
Published: 2024-01-17 09:46:13

Diffusion based video generation has received extensive attention and
achieved considerable success within both the academic and industrial
communities. However, current efforts are mainly concentrated on
single-objective or single-task video generation, such as generation driven by
text, by image, or by a combination of text and image. This cannot fully meet
the needs of real-world application scenarios, as users are likely to input
images and text conditions in a flexible manner, either individually or in
combination. To address this, we propose a Unified-modal Video Genearation
system that is capable of handling multiple video generation tasks across text
and image modalities. To this end, we revisit the various video generation
tasks within our system from the perspective of generative freedom, and
classify them into high-freedom and low-freedom video generation categories.
For high-freedom video generation, we employ Multi-condition Cross Attention to
generate videos that align with the semantics of the input images or text. For
low-freedom video generation, we introduce Biased Gaussian Noise to replace the
pure random Gaussian Noise, which helps to better preserve the content of the
input conditions. Our method achieves the lowest Fr\'echet Video Distance (FVD)
on the public academic benchmark MSR-VTT, surpasses the current open-source
methods in human evaluations, and is on par with the current close-source
method Gen2. For more samples, visit https://univg-baidu.github.io.