Orionnnn/hub-stats huggingface dataset:
Describing Differences in Image Sets with Natural Language
Authors: name: Lisa Dunlap, name: Yuhui Zhang, name: Xiaohan Wang, name: Ruiqi Zhong, name: Trevor Darrell, name: Jacob Steinhardt, name: Joseph E. Gonzalez, name: Serena Yeung-Levy
Published: 2023-12-05 18:59:16

How do two sets of images differ? Discerning set-level differences is crucial
for understanding model behaviors and analyzing datasets, yet manually sifting
through thousands of images is impractical. To aid in this discovery process,
we explore the task of automatically describing the differences between two
sets of images, which we term Set Difference Captioning. This task
takes in image sets D_A and D_B, and outputs a description that is more
often true on D_A than D_B. We outline a two-stage approach that first
proposes candidate difference descriptions from image sets and then re-ranks
the candidates by checking how well they can differentiate the two sets. We
introduce VisDiff, which first captions the images and prompts a language model
to propose candidate descriptions, then re-ranks these descriptions using CLIP.
To evaluate VisDiff, we collect VisDiffBench, a dataset with 187 paired image
sets with ground truth difference descriptions. We apply VisDiff to various
domains, such as comparing datasets (e.g., ImageNet vs. ImageNetV2), comparing
classification models (e.g., zero-shot CLIP vs. supervised ResNet), summarizing
model failure modes (supervised ResNet), characterizing differences between
generative models (e.g., StableDiffusionV1 and V2), and discovering what makes
images memorable. Using VisDiff, we are able to find interesting and previously
unknown differences in datasets and models, demonstrating its utility in
revealing nuanced insights.
LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Models
Authors: name: Hao Zhang, name: Hongyang Li, name: Feng Li, name: Tianhe Ren, name: Xueyan Zou, name: Shilong Liu, name: Shijia Huang, name: Jianfeng Gao, name: Lei Zhang, name: Chunyuan Li, name: Jianwei Yang
Published: 2023-12-05 18:29:31

With the recent significant advancements in large multi-modal models (LMMs),
the importance of their grounding capability in visual chat is increasingly
recognized. Despite recent efforts to enable LMMs to support grounding, their
capabilities for grounding and chat are usually separate, and their chat
performance drops dramatically when asked to ground. The problem is the lack of
a dataset for grounded visual chat (GVC). Existing grounding datasets only
contain short captions. To address this issue, we have created GVC data that
allows for the combination of grounding and chat capabilities. To better
evaluate the GVC capabilities, we have introduced a benchmark called
Grounding-Bench. Additionally, we have proposed a model design that can support
GVC and various types of visual prompts by connecting segmentation models with
language models. Experimental results demonstrate that our model outperforms
other LMMs on Grounding-Bench. Furthermore, our model achieves competitive
performance on classic grounding benchmarks like RefCOCO/+/g and Flickr30K
Entities. Our code will be released at
https://github.com/UX-Decoder/LLaVA-Grounding .
WhisBERT: Multimodal Text-Audio Language Modeling on 100M Words
Authors: name: Lukas Wolf, name: Klemen Kotar, name: Greta Tuckute, name: Eghbal Hosseini, name: Tamar Regev, name: Ethan Wilcox, name: Alex Warstadt
Published: 2023-12-05 18:03:13

Training on multiple modalities of input can augment the capabilities of a
language model. Here, we ask whether such a training regime can improve the
quality and efficiency of these systems as well. We focus on text--audio and
introduce Whisbert, which is inspired by the text--image approach of FLAVA
singh_flava_2022. In accordance with Babylm warstadt2023papers
guidelines, we pretrain Whisbert on a dataset comprising only 100 million words
plus their corresponding speech from the word-aligned version of the People's
Speech dataset galvez_peoples_2021. To assess the impact of
multimodality, we compare versions of the model that are trained on text only
and on both audio and text simultaneously. We find that while Whisbert is able
to perform well on multimodal masked modeling and surpasses the Babylm
baselines in most benchmark tasks, it struggles to optimize its complex
objective and outperform its text-only Whisbert baseline.
Training Chain-of-Thought via Latent-Variable Inference
Authors: name: Du Phan, name: Matthew D. Hoffman, name: David Dohan, name: Sholto Douglas, name: Tuan Anh Le, name: Aaron Parisi, name: Pavel Sountsov, name: Charles Sutton, name: Sharad Vikram, name: Rif A. Saurous
Published: 2023-11-28 17:47:32

Large language models (LLMs) solve problems more accurately and interpretably
when instructed to work out the answer step by step using a
``chain-of-thought'' (CoT) prompt. One can also improve LLMs' performance on a
specific task by supervised fine-tuning, i.e., by using gradient ascent on some
tunable parameters to maximize the average log-likelihood of correct answers
from a labeled training set. Naively combining CoT with supervised tuning
requires supervision not just of the correct answers, but also of detailed
rationales that lead to those answers; these rationales are expensive to
produce by hand. Instead, we propose a fine-tuning strategy that tries to
maximize the marginal log-likelihood of generating a correct answer
using CoT prompting, approximately averaging over all possible rationales. The
core challenge is sampling from the posterior over rationales conditioned on
the correct answer; we address it using a simple Markov-chain Monte Carlo
(MCMC) expectation-maximization (EM) algorithm inspired by the self-taught
reasoner (STaR), memoized wake-sleep, Markovian score climbing, and persistent
contrastive divergence. This algorithm also admits a novel control-variate
technique that drives the variance of our gradient estimates to zero as the
model improves. Applying our technique to GSM8K and the tasks in BIG-Bench
Hard, we find that this MCMC-EM fine-tuning technique typically improves the
model's accuracy on held-out examples more than STaR or prompt-tuning with or
without CoT.
Rank-without-GPT: Building GPT-Independent Listwise Rerankers on
  Open-Source Large Language Models
Authors: name: Xinyu Zhang, name: Sebastian Hofstätter, name: Patrick Lewis, name: Raphael Tang, name: Jimmy Lin
Published: 2023-12-05 18:57:40

Listwise rerankers based on large language models (LLM) are the zero-shot
state-of-the-art. However, current works in this direction all depend on the
GPT models, making it a single point of failure in scientific reproducibility.
Moreover, it raises the concern that the current research findings only hold
for GPT models but not LLM in general. In this work, we lift this pre-condition
and build for the first time effective listwise rerankers without any form of
dependency on GPT. Our passage retrieval experiments show that our best list se
reranker surpasses the listwise rerankers based on GPT-3.5 by 13% and achieves
97% effectiveness of the ones built on GPT-4. Our results also show that the
existing training datasets, which were expressly constructed for pointwise
ranking, are insufficient for building such listwise rerankers. Instead,
high-quality listwise ranking data is required and crucial, calling for further
work on building human-annotated listwise data resources.
Axiomatic Preference Modeling for Longform Question Answering
Authors: name: Corby Rosset, name: Guoqing Zheng, name: Victor Dibia, name: Ahmed Awadallah, name: Paul Bennett
Published: 2023-12-02 23:11:41

The remarkable abilities of large language models (LLMs) like GPT-4 partially
stem from post-training processes like Reinforcement Learning from Human
Feedback (RLHF) involving human preferences encoded in a reward model. However,
these reward models (RMs) often lack direct knowledge of why, or under what
principles, the preferences annotations were made. In this study, we identify
principles that guide RMs to better align with human preferences, and then
develop an axiomatic framework to generate a rich variety of preference signals
to uphold them. We use these axiomatic signals to train a model for scoring
answers to longform questions. Our approach yields a Preference Model with only
about 220M parameters that agrees with gold human-annotated preference labels
more often than GPT-4. The contributions of this work include: training a
standalone preference model that can score human- and LLM-generated answers on
the same scale; developing an axiomatic framework for generating training data
pairs tailored to certain principles; and showing that a small amount of
axiomatic signals can help small models outperform GPT-4 in preference scoring.
We release our model on huggingface:
https://huggingface.co/corbyrosset/axiomatic_preference_model
VideoSwap: Customized Video Subject Swapping with Interactive Semantic
  Point Correspondence
Authors: name: Yuchao Gu, name: Yipin Zhou, name: Bichen Wu, name: Licheng Yu, name: Jia-Wei Liu, name: Rui Zhao, name: Jay Zhangjie Wu, name: David Junhao Zhang, name: Mike Zheng Shou, name: Kevin Tang
Published: 2023-12-04 17:58:06

Current diffusion-based video editing primarily focuses on
structure-preserved editing by utilizing various dense correspondences to
ensure temporal consistency and motion alignment. However, these approaches are
often ineffective when the target edit involves a shape change. To embark on
video editing with shape change, we explore customized video subject swapping
in this work, where we aim to replace the main subject in a source video with a
target subject having a distinct identity and potentially different shape. In
contrast to previous methods that rely on dense correspondences, we introduce
the VideoSwap framework that exploits semantic point correspondences, inspired
by our observation that only a small number of semantic points are necessary to
align the subject's motion trajectory and modify its shape. We also introduce
various user-point interactions (\eg, removing points and dragging points) to
address various semantic point correspondence. Extensive experiments
demonstrate state-of-the-art video subject swapping results across a variety of
real-world videos.
DeepCache: Accelerating Diffusion Models for Free
Authors: name: Xinyin Ma, name: Gongfan Fang, name: Xinchao Wang
Published: 2023-12-01 17:01:06

Diffusion models have recently gained unprecedented attention in the field of
image synthesis due to their remarkable generative capabilities.
Notwithstanding their prowess, these models often incur substantial
computational costs, primarily attributed to the sequential denoising process
and cumbersome model size. Traditional methods for compressing diffusion models
typically involve extensive retraining, presenting cost and feasibility
challenges. In this paper, we introduce DeepCache, a novel training-free
paradigm that accelerates diffusion models from the perspective of model
architecture. DeepCache capitalizes on the inherent temporal redundancy
observed in the sequential denoising steps of diffusion models, which caches
and retrieves features across adjacent denoising stages, thereby curtailing
redundant computations. Utilizing the property of the U-Net, we reuse the
high-level features while updating the low-level features in a very cheap way.
This innovative strategy, in turn, enables a speedup factor of 2.3times for
Stable Diffusion v1.5 with only a 0.05 decline in CLIP Score, and 4.1times
for LDM-4-G with a slight decrease of 0.22 in FID on ImageNet. Our experiments
also demonstrate DeepCache's superiority over existing pruning and distillation
methods that necessitate retraining and its compatibility with current sampling
techniques. Furthermore, we find that under the same throughput, DeepCache
effectively achieves comparable or even marginally improved results with DDIM
or PLMS. The code is available at https://github.com/horseee/DeepCache
Fast View Synthesis of Casual Videos
Authors: name: Yao-Chih Lee, name: Zhoutong Zhang, name: Kevin Blackburn-Matzen, name: Simon Niklaus, name: Jianming Zhang, name: Jia-Bin Huang, name: Feng Liu
Published: 2023-12-04 18:55:48

Novel view synthesis from an in-the-wild video is difficult due to challenges
like scene dynamics and lack of parallax. While existing methods have shown
promising results with implicit neural radiance fields, they are slow to train
and render. This paper revisits explicit video representations to synthesize
high-quality novel views from a monocular video efficiently. We treat static
and dynamic video content separately. Specifically, we build a global static
scene model using an extended plane-based scene representation to synthesize
temporally coherent novel video. Our plane-based scene representation is
augmented with spherical harmonics and displacement maps to capture
view-dependent effects and model non-planar complex surface geometry. We opt to
represent the dynamic content as per-frame point clouds for efficiency. While
such representations are inconsistency-prone, minor temporal inconsistencies
are perceptually masked due to motion. We develop a method to quickly estimate
such a hybrid video representation and render novel views in real time. Our
experiments show that our method can render high-quality novel views from an
in-the-wild video with comparable quality to state-of-the-art methods while
being 100x faster in training and enabling real-time rendering.
Segment Any 3D Gaussians
Authors: name: Jiazhong Cen, name: Jiemin Fang, name: Chen Yang, name: Lingxi Xie, name: Xiaopeng Zhang, name: Wei Shen, name: Qi Tian
Published: 2023-12-01 17:15:24

Interactive 3D segmentation in radiance fields is an appealing task since its
importance in 3D scene understanding and manipulation. However, existing
methods face challenges in either achieving fine-grained, multi-granularity
segmentation or contending with substantial computational overhead, inhibiting
real-time interaction. In this paper, we introduce Segment Any 3D GAussians
(SAGA), a novel 3D interactive segmentation approach that seamlessly blends a
2D segmentation foundation model with 3D Gaussian Splatting (3DGS), a recent
breakthrough of radiance fields. SAGA efficiently embeds multi-granularity 2D
segmentation results generated by the segmentation foundation model into 3D
Gaussian point features through well-designed contrastive training. Evaluation
on existing benchmarks demonstrates that SAGA can achieve competitive
performance with state-of-the-art methods. Moreover, SAGA achieves
multi-granularity segmentation and accommodates various prompts, including
points, scribbles, and 2D masks. Notably, SAGA can finish the 3D segmentation
within milliseconds, achieving nearly 1000x acceleration compared to previous
SOTA. The project page is at https://jumpat.github.io/SAGA.
Customize your NeRF: Adaptive Source Driven 3D Scene Editing via
  Local-Global Iterative Training
Authors: name: Runze He, name: Shaofei Huang, name: Xuecheng Nie, name: Tianrui Hui, name: Luoqi Liu, name: Jiao Dai, name: Jizhong Han, name: Guanbin Li, name: Si Liu
Published: 2023-12-04 06:25:06

In this paper, we target the adaptive source driven 3D scene editing task by
proposing a CustomNeRF model that unifies a text description or a reference
image as the editing prompt. However, obtaining desired editing results
conformed with the editing prompt is nontrivial since there exist two
significant challenges, including accurate editing of only foreground regions
and multi-view consistency given a single-view reference image. To tackle the
first challenge, we propose a Local-Global Iterative Editing (LGIE) training
scheme that alternates between foreground region editing and full-image
editing, aimed at foreground-only manipulation while preserving the background.
For the second challenge, we also design a class-guided regularization that
exploits class priors within the generation model to alleviate the
inconsistency problem among different views in image-driven editing. Extensive
experiments show that our CustomNeRF produces precise editing results under
various real scenes for both text- and image-driven settings.
SANeRF-HQ: Segment Anything for NeRF in High Quality
Authors: name: Yichen Liu, name: Benran Hu, name: Chi-Keung Tang, name: Yu-Wing Tai
Published: 2023-12-03 23:09:38

Recently, the Segment Anything Model (SAM) has showcased remarkable
capabilities of zero-shot segmentation, while NeRF (Neural Radiance Fields) has
gained popularity as a method for various 3D problems beyond novel view
synthesis. Though there exist initial attempts to incorporate these two methods
into 3D segmentation, they face the challenge of accurately and consistently
segmenting objects in complex scenarios. In this paper, we introduce the
Segment Anything for NeRF in High Quality (SANeRF-HQ) to achieve high quality
3D segmentation of any object in a given scene. SANeRF-HQ utilizes SAM for
open-world object segmentation guided by user-supplied prompts, while
leveraging NeRF to aggregate information from different viewpoints. To overcome
the aforementioned challenges, we employ density field and RGB similarity to
enhance the accuracy of segmentation boundary during the aggregation.
Emphasizing on segmentation accuracy, we evaluate our method quantitatively on
multiple NeRF datasets where high-quality ground-truths are available or
manually annotated. SANeRF-HQ shows a significant quality improvement over
previous state-of-the-art methods in NeRF object segmentation, provides higher
flexibility for object localization, and enables more consistent object
segmentation across multiple views. Additional information can be found at
https://lyclyc52.github.io/SANeRF-HQ/.
VideoRF: Rendering Dynamic Radiance Fields as 2D Feature Video Streams
Authors: name: Liao Wang, name: Kaixin Yao, name: Chengcheng Guo, name: Zhirui Zhang, name: Qiang Hu, name: Jingyi Yu, name: Lan Xu, name: Minye Wu
Published: 2023-12-03 14:14:35

Neural Radiance Fields (NeRFs) excel in photorealistically rendering static
scenes. However, rendering dynamic, long-duration radiance fields on ubiquitous
devices remains challenging, due to data storage and computational constraints.
In this paper, we introduce VideoRF, the first approach to enable real-time
streaming and rendering of dynamic radiance fields on mobile platforms. At the
core is a serialized 2D feature image stream representing the 4D radiance field
all in one. We introduce a tailored training scheme directly applied to this 2D
domain to impose the temporal and spatial redundancy of the feature image
stream. By leveraging the redundancy, we show that the feature image stream can
be efficiently compressed by 2D video codecs, which allows us to exploit video
hardware accelerators to achieve real-time decoding. On the other hand, based
on the feature image stream, we propose a novel rendering pipeline for VideoRF,
which has specialized space mappings to query radiance properties efficiently.
Paired with a deferred shading model, VideoRF has the capability of real-time
rendering on mobile devices thanks to its efficiency. We have developed a
real-time interactive player that enables online streaming and rendering of
dynamic scenes, offering a seamless and immersive free-viewpoint experience
across a range of devices, from desktops to mobile phones.
Generative Powers of Ten
Authors: name: Xiaojuan Wang, name: Janne Kontkanen, name: Brian Curless, name: Steve Seitz, name: Ira Kemelmacher, name: Ben Mildenhall, name: Pratul Srinivasan, name: Dor Verbin, name: Aleksander Holynski
Published: 2023-12-04 18:59:25

We present a method that uses a text-to-image model to generate consistent
content across multiple image scales, enabling extreme semantic zooms into a
scene, e.g., ranging from a wide-angle landscape view of a forest to a macro
shot of an insect sitting on one of the tree branches. We achieve this through
a joint multi-scale diffusion sampling approach that encourages consistency
across different scales while preserving the integrity of each individual
sampling process. Since each generated scale is guided by a different text
prompt, our method enables deeper levels of zoom than traditional
super-resolution methods that may struggle to create new contextual structure
at vastly different scales. We compare our method qualitatively with
alternative techniques in image super-resolution and outpainting, and show that
our method is most effective at generating consistent multi-scale content.
GPS-Gaussian: Generalizable Pixel-wise 3D Gaussian Splatting for
  Real-time Human Novel View Synthesis
Authors: name: Shunyuan Zheng, name: Boyao Zhou, name: Ruizhi Shao, name: Boning Liu, name: Shengping Zhang, name: Liqiang Nie, name: Yebin Liu
Published: 2023-12-04 18:59:55

We present a new approach, termed GPS-Gaussian, for synthesizing novel views
of a character in a real-time manner. The proposed method enables 2K-resolution
rendering under a sparse-view camera setting. Unlike the original Gaussian
Splatting or neural implicit rendering methods that necessitate per-subject
optimizations, we introduce Gaussian parameter maps defined on the source views
and regress directly Gaussian Splatting properties for instant novel view
synthesis without any fine-tuning or optimization. To this end, we train our
Gaussian parameter regression module on a large amount of human scan data,
jointly with a depth estimation module to lift 2D parameter maps to 3D space.
The proposed framework is fully differentiable and experiments on several
datasets demonstrate that our method outperforms state-of-the-art methods while
achieving an exceeding rendering speed.
Rejuvenating image-GPT as Strong Visual Representation Learners
Authors: name: Sucheng Ren, name: Zeyu Wang, name: Hongru Zhu, name: Junfei Xiao, name: Alan Yuille, name: Cihang Xie
Published: 2023-12-04 18:59:20

This paper enhances image-GPT (iGPT), one of the pioneering works that
introduce autoregressive pretraining to predict next pixels for visual
representation learning. Two simple yet essential changes are made. First, we
shift the prediction target from raw pixels to semantic tokens, enabling a
higher-level understanding of visual content. Second, we supplement the
autoregressive modeling by instructing the model to predict not only the next
tokens but also the visible tokens. This pipeline is particularly effective
when semantic tokens are encoded by discriminatively trained models, such as
CLIP. We introduce this novel approach as D-iGPT. Extensive experiments
showcase that D-iGPT excels as a strong learner of visual representations: A
notable achievement of D-iGPT is its compelling performance on the ImageNet-1K
dataset -- by training on publicly available datasets, D-iGPT achieves 89.5\%
top-1 accuracy with a vanilla ViT-Large model. This model also shows strong
generalization on the downstream task and robustness on out-of-distribution
samples. Code is avaiable at
https://github.com/OliverRensu/D-iGPT{https://github.com/OliverRensu/D-iGPT}.