Orionnnn/hub-stats huggingface dataset:
ThemeStation: Generating Theme-Aware 3D Assets from Few Exemplars
Authors: name: Zhenwei Wang, name: Tengfei Wang, name: Gerhard Hancke, name: Ziwei Liu, name: Rynson W. H. Lau
Published: 2024-03-22 17:59:01

Real-world applications often require a large gallery of 3D assets that share
a consistent theme. While remarkable advances have been made in general 3D
content creation from text or image, synthesizing customized 3D assets
following the shared theme of input 3D exemplars remains an open and
challenging problem. In this work, we present ThemeStation, a novel approach
for theme-aware 3D-to-3D generation. ThemeStation synthesizes customized 3D
assets based on given few exemplars with two goals: 1) unity for generating 3D
assets that thematically align with the given exemplars and 2) diversity for
generating 3D assets with a high degree of variations. To this end, we design a
two-stage framework that draws a concept image first, followed by a
reference-informed 3D modeling stage. We propose a novel dual score
distillation (DSD) loss to jointly leverage priors from both the input
exemplars and the synthesized concept image. Extensive experiments and user
studies confirm that ThemeStation surpasses prior works in producing diverse
theme-aware 3D models with impressive quality. ThemeStation also enables
various applications such as controllable 3D-to-3D generation.
InternVideo2: Scaling Video Foundation Models for Multimodal Video
  Understanding
Authors: name: Yi Wang, name: Kunchang Li, name: Xinhao Li, name: Jiashuo Yu, name: Yinan He, name: Guo Chen, name: Baoqi Pei, name: Rongkun Zheng, name: Jilan Xu, name: Zun Wang, name: Yansong Shi, name: Tianxiang Jiang, name: Songze Li, name: Hongjie Zhang, name: Yifei Huang, name: Yu Qiao, name: Yali Wang, name: Limin Wang
Published: 2024-03-22 17:57:42

We introduce InternVideo2, a new video foundation model (ViFM) that achieves
the state-of-the-art performance in action recognition, video-text tasks, and
video-centric dialogue. Our approach employs a progressive training paradigm
that unifies the different self- or weakly-supervised learning frameworks of
masked video token reconstruction, cross-modal contrastive learning, and next
token prediction. Different training stages would guide our model to capture
different levels of structure and semantic information through different
pretext tasks. At the data level, we prioritize the spatiotemporal consistency
by semantically segmenting videos and generating video-audio-speech captions.
This improves the alignment between video and text. We scale both data and
model size for our InternVideo2. Through extensive experiments, we validate our
designs and demonstrate the state-of-the-art performance on over 60 video and
audio tasks. Notably, our model outperforms others on various video-related
captioning, dialogue, and long video understanding benchmarks, highlighting its
ability to reason and comprehend long temporal contexts. Code and models are
available at https://github.com/OpenGVLab/InternVideo2/.
SiMBA: Simplified Mamba-Based Architecture for Vision and Multivariate
  Time series
Authors: name: Badri N. Patro, name: Vijay S. Agneeswaran
Published: 2024-03-22 17:22:56

Transformers have widely adopted attention networks for sequence mixing and
MLPs for channel mixing, playing a pivotal role in achieving breakthroughs
across domains. However, recent literature highlights issues with attention
networks, including low inductive bias and quadratic complexity concerning
input sequence length. State Space Models (SSMs) like S4 and others (Hippo,
Global Convolutions, liquid S4, LRU, Mega, and Mamba), have emerged to address
the above issues to help handle longer sequence lengths. Mamba, while being the
state-of-the-art SSM, has a stability issue when scaled to large networks for
computer vision datasets. We propose SiMBA, a new architecture that introduces
Einstein FFT (EinFFT) for channel modeling by specific eigenvalue computations
and uses the Mamba block for sequence modeling. Extensive performance studies
across image and time-series benchmarks demonstrate that SiMBA outperforms
existing SSMs, bridging the performance gap with state-of-the-art transformers.
Notably, SiMBA establishes itself as the new state-of-the-art SSM on ImageNet
and transfer learning benchmarks such as Stanford Car and Flower as well as
task learning benchmarks as well as seven time series benchmark datasets. The
project page is available on this website
~https://github.com/badripatro/Simba.
Can large language models explore in-context?
Authors: name: Akshay Krishnamurthy, name: Keegan Harris, name: Dylan J. Foster, name: Cyril Zhang, name: Aleksandrs Slivkins
Published: 2024-03-22 17:50:43

We investigate the extent to which contemporary Large Language Models (LLMs)
can engage in exploration, a core capability in reinforcement learning and
decision making. We focus on native performance of existing LLMs, without
training interventions. We deploy LLMs as agents in simple multi-armed bandit
environments, specifying the environment description and interaction history
entirely in-context, i.e., within the LLM prompt. We experiment with GPT-3.5,
GPT-4, and Llama2, using a variety of prompt designs, and find that the models
do not robustly engage in exploration without substantial interventions: i)
Across all of our experiments, only one configuration resulted in satisfactory
exploratory behavior: GPT-4 with chain-of-thought reasoning and an externally
summarized interaction history, presented as sufficient statistics; ii) All
other configurations did not result in robust exploratory behavior, including
those with chain-of-thought reasoning but unsummarized history. Although these
findings can be interpreted positively, they suggest that external
summarization -- which may not be possible in more complex settings -- is
important for obtaining desirable behavior from LLM agents. We conclude that
non-trivial algorithmic interventions, such as fine-tuning or dataset curation,
may be required to empower LLM-based decision making agents in complex
settings.
StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation
  from Text
Authors: name: Roberto Henschel, name: Levon Khachatryan, name: Daniil Hayrapetyan, name: Hayk Poghosyan, name: Vahram Tadevosyan, name: Zhangyang Wang, name: Shant Navasardyan, name: Humphrey Shi
Published: 2024-03-21 18:27:29

Text-to-video diffusion models enable the generation of high-quality videos
that follow text instructions, making it easy to create diverse and individual
content. However, existing approaches mostly focus on high-quality short video
generation (typically 16 or 24 frames), ending up with hard-cuts when naively
extended to the case of long video synthesis. To overcome these limitations, we
introduce StreamingT2V, an autoregressive approach for long video generation of
80, 240, 600, 1200 or more frames with smooth transitions. The key components
are:(i) a short-term memory block called conditional attention module (CAM),
which conditions the current generation on the features extracted from the
previous chunk via an attentional mechanism, leading to consistent chunk
transitions, (ii) a long-term memory block called appearance preservation
module, which extracts high-level scene and object features from the first
video chunk to prevent the model from forgetting the initial scene, and (iii) a
randomized blending approach that enables to apply a video enhancer
autoregressively for infinitely long videos without inconsistencies between
chunks. Experiments show that StreamingT2V generates high motion amount. In
contrast, all competing image-to-video methods are prone to video stagnation
when applied naively in an autoregressive manner. Thus, we propose with
StreamingT2V a high-quality seamless text-to-long video generator that
outperforms competitors with consistency and motion. Our code will be available
at: https://github.com/Picsart-AI-Research/StreamingT2V
GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction
  and Generation
Authors: name: Yinghao Xu, name: Zifan Shi, name: Wang Yifan, name: Hansheng Chen, name: Ceyuan Yang, name: Sida Peng, name: Yujun Shen, name: Gordon Wetzstein
Published: 2024-03-21 17:59:34

We introduce GRM, a large-scale reconstructor capable of recovering a 3D
asset from sparse-view images in around 0.1s. GRM is a feed-forward
transformer-based model that efficiently incorporates multi-view information to
translate the input pixels into pixel-aligned Gaussians, which are unprojected
to create a set of densely distributed 3D Gaussians representing a scene.
Together, our transformer architecture and the use of 3D Gaussians unlock a
scalable and efficient reconstruction framework. Extensive experimental results
demonstrate the superiority of our method over alternatives regarding both
reconstruction quality and efficiency. We also showcase the potential of GRM in
generative tasks, i.e., text-to-3D and image-to-3D, by integrating it with
existing multi-view diffusion models. Our project website is at:
https://justimyhxu.github.io/projects/grm/.
ReNoise: Real Image Inversion Through Iterative Noising
Authors: name: Daniel Garibi, name: Or Patashnik, name: Andrey Voynov, name: Hadar Averbuch-Elor, name: Daniel Cohen-Or
Published: 2024-03-21 17:52:08

Recent advancements in text-guided diffusion models have unlocked powerful
image manipulation capabilities. However, applying these methods to real images
necessitates the inversion of the images into the domain of the pretrained
diffusion model. Achieving faithful inversion remains a challenge, particularly
for more recent models trained to generate images with a small number of
denoising steps. In this work, we introduce an inversion method with a high
quality-to-operation ratio, enhancing reconstruction accuracy without
increasing the number of operations. Building on reversing the diffusion
sampling process, our method employs an iterative renoising mechanism at each
inversion sampling step. This mechanism refines the approximation of a
predicted point along the forward diffusion trajectory, by iteratively applying
the pretrained diffusion model, and averaging these predictions. We evaluate
the performance of our ReNoise technique using various sampling algorithms and
models, including recent accelerated diffusion models. Through comprehensive
evaluations and comparisons, we show its effectiveness in terms of both
accuracy and speed. Furthermore, we confirm that our method preserves
editability by demonstrating text-driven image editing on real images.
MyVLM: Personalizing VLMs for User-Specific Queries
Authors: name: Yuval Alaluf, name: Elad Richardson, name: Sergey Tulyakov, name: Kfir Aberman, name: Daniel Cohen-Or
Published: 2024-03-21 17:51:01

Recent large-scale vision-language models (VLMs) have demonstrated remarkable
capabilities in understanding and generating textual descriptions for visual
content. However, these models lack an understanding of user-specific concepts.
In this work, we take a first step toward the personalization of VLMs, enabling
them to learn and reason over user-provided concepts. For example, we explore
whether these models can learn to recognize you in an image and communicate
what you are doing, tailoring the model to reflect your personal experiences
and relationships. To effectively recognize a variety of user-specific
concepts, we augment the VLM with external concept heads that function as
toggles for the model, enabling the VLM to identify the presence of specific
target concepts in a given image. Having recognized the concept, we learn a new
concept embedding in the intermediate feature space of the VLM. This embedding
is tasked with guiding the language model to naturally integrate the target
concept in its generated response. We apply our technique to BLIP-2 and LLaVA
for personalized image captioning and further show its applicability for
personalized visual question-answering. Our experiments demonstrate our ability
to generalize to unseen images of learned concepts while preserving the model
behavior on unrelated inputs.
Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient
  Inference
Authors: name: Han Zhao, name: Min Zhang, name: Wei Zhao, name: Pengxiang Ding, name: Siteng Huang, name: Donglin Wang
Published: 2024-03-21 16:17:57

In recent years, the application of multimodal large language models (MLLM)
in various fields has achieved remarkable success. However, as the foundation
model for many downstream tasks, current MLLMs are composed of the well-known
Transformer network, which has a less efficient quadratic computation
complexity. To improve the efficiency of such basic models, we propose Cobra, a
linear computational complexity MLLM. Specifically, Cobra integrates the
efficient Mamba language model into the visual modality. Moreover, we explore
and study various modal fusion schemes to create an effective multi-modal
Mamba. Extensive experiments demonstrate that (1) Cobra achieves extremely
competitive performance with current computationally efficient state-of-the-art
methods, e.g., LLaVA-Phi, TinyLLaVA, and MobileVLM v2, and has faster
speed due to Cobra's linear sequential modeling. (2) Interestingly, the results
of closed-set challenging prediction benchmarks show that Cobra performs well
in overcoming visual illusions and spatial relationship judgments. (3) Notably,
Cobra even achieves comparable performance to LLaVA with about 43% of the
number of parameters. We will make all codes of Cobra open-source and hope that
the proposed method can facilitate future research on complexity problems in
MLLM. Our project page is available at: https://sites.google.com/view/cobravlm.
Recourse for reclamation: Chatting with generative language models
Authors: name: Jennifer Chien, name: Kevin R. McKee, name: Jackie Kay, name: William Isaac
Published: 2024-03-21 15:14:25

Researchers and developers increasingly rely on toxicity scoring to moderate
generative language model outputs, in settings such as customer service,
information retrieval, and content generation. However, toxicity scoring may
render pertinent information inaccessible, rigidify or "value-lock" cultural
norms, and prevent language reclamation processes, particularly for
marginalized people. In this work, we extend the concept of algorithmic
recourse to generative language models: we provide users a novel mechanism to
achieve their desired prediction by dynamically setting thresholds for toxicity
filtering. Users thereby exercise increased agency relative to interactions
with the baseline system. A pilot study (n = 30) supports the potential of
our proposed recourse mechanism, indicating improvements in usability compared
to fixed-threshold toxicity-filtering of model outputs. Future work should
explore the intersection of toxicity scoring, model controllability, user
agency, and language reclamation processes -- particularly with regard to the
bias that many communities encounter when interacting with generative language
models.
MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual
  Math Problems?
Authors: name: Renrui Zhang, name: Dongzhi Jiang, name: Yichi Zhang, name: Haokun Lin, name: Ziyu Guo, name: Pengshuo Qiu, name: Aojun Zhou, name: Pan Lu, name: Kai-Wei Chang, name: Peng Gao, name: Hongsheng Li
Published: 2024-03-21 17:59:50

The remarkable progress of Multi-modal Large Language Models (MLLMs) has
garnered unparalleled attention, due to their superior performance in visual
contexts. However, their capabilities in visual math problem-solving remain
insufficiently evaluated and understood. We investigate current benchmarks to
incorporate excessive visual content within textual questions, which
potentially assist MLLMs in deducing answers without truly interpreting the
input diagrams. To this end, we introduce MathVerse, an all-around visual math
benchmark designed for an equitable and in-depth evaluation of MLLMs. We
meticulously collect 2,612 high-quality, multi-subject math problems with
diagrams from publicly available sources. Each problem is then transformed by
human annotators into six distinct versions, each offering varying degrees of
information content in multi-modality, contributing to 15K test samples in
total. This approach allows MathVerse to comprehensively assess whether and how
much MLLMs can truly understand the visual diagrams for mathematical reasoning.
In addition, we propose a Chain-of-Thought (CoT) evaluation strategy for a
fine-grained assessment of the output answers. Rather than naively judging True
or False, we employ GPT-4(V) to adaptively extract crucial reasoning steps, and
then score each step with detailed error analysis, which can reveal the
intermediate CoT reasoning quality by MLLMs. We hope the MathVerse benchmark
may provide unique insights to guide the future development of MLLMs. Project
page: https://mathverse-cuhk.github.io
Gaussian Frosting: Editable Complex Radiance Fields with Real-Time
  Rendering
Authors: name: Antoine Guédon, name: Vincent Lepetit
Published: 2024-03-21 16:53:03

We propose Gaussian Frosting, a novel mesh-based representation for
high-quality rendering and editing of complex 3D effects in real-time. Our
approach builds on the recent 3D Gaussian Splatting framework, which optimizes
a set of 3D Gaussians to approximate a radiance field from images. We propose
first extracting a base mesh from Gaussians during optimization, then building
and refining an adaptive layer of Gaussians with a variable thickness around
the mesh to better capture the fine details and volumetric effects near the
surface, such as hair or grass. We call this layer Gaussian Frosting, as it
resembles a coating of frosting on a cake. The fuzzier the material, the
thicker the frosting. We also introduce a parameterization of the Gaussians to
enforce them to stay inside the frosting layer and automatically adjust their
parameters when deforming, rescaling, editing or animating the mesh. Our
representation allows for efficient rendering using Gaussian splatting, as well
as editing and animation by modifying the base mesh. We demonstrate the
effectiveness of our method on various synthetic and real scenes, and show that
it outperforms existing surface-based approaches. We will release our code and
a web-based viewer as additional contributions. Our project page is the
following: https://anttwo.github.io/frosting/
DreamReward: Text-to-3D Generation with Human Preference
Authors: name: Junliang Ye, name: Fangfu Liu, name: Qixiu Li, name: Zhengyi Wang, name: Yikai Wang, name: Xinzhou Wang, name: Yueqi Duan, name: Jun Zhu
Published: 2024-03-21 17:58:04

3D content creation from text prompts has shown remarkable success recently.
However, current text-to-3D methods often generate 3D results that do not align
well with human preferences. In this paper, we present a comprehensive
framework, coined DreamReward, to learn and improve text-to-3D models from
human preference feedback. To begin with, we collect 25k expert comparisons
based on a systematic annotation pipeline including rating and ranking. Then,
we build Reward3D -- the first general-purpose text-to-3D human preference
reward model to effectively encode human preferences. Building upon the 3D
reward model, we finally perform theoretical analysis and present the Reward3D
Feedback Learning (DreamFL), a direct tuning algorithm to optimize the
multi-view diffusion models with a redefined scorer. Grounded by theoretical
proof and extensive experiment comparisons, our DreamReward successfully
generates high-fidelity and 3D consistent results with significant boosts in
prompt alignment with human intention. Our results demonstrate the great
potential for learning from human feedback to improve text-to-3D models.
AnyV2V: A Plug-and-Play Framework For Any Video-to-Video Editing Tasks
Authors: name: Max Ku, name: Cong Wei, name: Weiming Ren, name: Huan Yang, name: Wenhu Chen
Published: 2024-03-21 15:15:00

Video-to-video editing involves editing a source video along with additional
control (such as text prompts, subjects, or styles) to generate a new video
that aligns with the source video and the provided control. Traditional methods
have been constrained to certain editing types, limiting their ability to meet
the wide range of user demands. In this paper, we introduce AnyV2V, a novel
training-free framework designed to simplify video editing into two primary
steps: (1) employing an off-the-shelf image editing model (e.g.
InstructPix2Pix, InstantID, etc) to modify the first frame, (2) utilizing an
existing image-to-video generation model (e.g. I2VGen-XL) for DDIM inversion
and feature injection. In the first stage, AnyV2V can plug in any existing
image editing tools to support an extensive array of video editing tasks.
Beyond the traditional prompt-based editing methods, AnyV2V also can support
novel video editing tasks, including reference-based style transfer,
subject-driven editing, and identity manipulation, which were unattainable by
previous methods. In the second stage, AnyV2V can plug in any existing
image-to-video models to perform DDIM inversion and intermediate feature
injection to maintain the appearance and motion consistency with the source
video. On the prompt-based editing, we show that AnyV2V can outperform the
previous best approach by 35\% on prompt alignment, and 25\% on human
preference. On the three novel tasks, we show that AnyV2V also achieves a high
success rate. We believe AnyV2V will continue to thrive due to its ability to
seamlessly integrate the fast-evolving image editing methods. Such
compatibility can help AnyV2V to increase its versatility to cater to diverse
user demands.
Explorative Inbetweening of Time and Space
Authors: name: Haiwen Feng, name: Zheng Ding, name: Zhihao Xia, name: Simon Niklaus, name: Victoria Abrevaya, name: Michael J. Black, name: Xuaner Zhang
Published: 2024-03-21 17:57:31

We introduce bounded generation as a generalized task to control video
generation to synthesize arbitrary camera and subject motion based only on a
given start and end frame. Our objective is to fully leverage the inherent
generalization capability of an image-to-video model without additional
training or fine-tuning of the original model. This is achieved through the
proposed new sampling strategy, which we call Time Reversal Fusion, that fuses
the temporally forward and backward denoising paths conditioned on the start
and end frame, respectively. The fused path results in a video that smoothly
connects the two frames, generating inbetweening of faithful subject motion,
novel views of static scenes, and seamless video looping when the two bounding
frames are identical. We curate a diverse evaluation dataset of image pairs and
compare against the closest existing methods. We find that Time Reversal Fusion
outperforms related work on all subtasks, exhibiting the ability to generate
complex motions and 3D-consistent views guided by bounded frames. See project
page at https://time-reversal.github.io.
StyleCineGAN: Landscape Cinemagraph Generation using a Pre-trained
  StyleGAN
Authors: name: Jongwoo Choi, name: Kwanggyoon Seo, name: Amirsaman Ashtari, name: Junyong Noh
Published: 2024-03-21 07:21:51

We propose a method that can generate cinemagraphs automatically from a still
landscape image using a pre-trained StyleGAN. Inspired by the success of recent
unconditional video generation, we leverage a powerful pre-trained image
generator to synthesize high-quality cinemagraphs. Unlike previous approaches
that mainly utilize the latent space of a pre-trained StyleGAN, our approach
utilizes its deep feature space for both GAN inversion and cinemagraph
generation. Specifically, we propose multi-scale deep feature warping (MSDFW),
which warps the intermediate features of a pre-trained StyleGAN at different
resolutions. By using MSDFW, the generated cinemagraphs are of high resolution
and exhibit plausible looping animation. We demonstrate the superiority of our
method through user studies and quantitative comparisons with state-of-the-art
cinemagraph generation methods and a video generation method that uses a
pre-trained StyleGAN.