Orionnnn/hub-stats huggingface dataset:
VideoMamba: State Space Model for Efficient Video Understanding
Authors: name: Kunchang Li, name: Xinhao Li, name: Yi Wang, name: Yinan He, name: Yali Wang, name: Limin Wang, name: Yu Qiao
Published: 2024-03-11 17:59:34

Addressing the dual challenges of local redundancy and global dependencies in
video understanding, this work innovatively adapts the Mamba to the video
domain. The proposed VideoMamba overcomes the limitations of existing 3D
convolution neural networks and video transformers. Its linear-complexity
operator enables efficient long-term modeling, which is crucial for
high-resolution long video understanding. Extensive evaluations reveal
VideoMamba's four core abilities: (1) Scalability in the visual domain without
extensive dataset pretraining, thanks to a novel self-distillation technique;
(2) Sensitivity for recognizing short-term actions even with fine-grained
motion differences; (3) Superiority in long-term video understanding,
showcasing significant advancements over traditional feature-based models; and
(4) Compatibility with other modalities, demonstrating robustness in
multi-modal contexts. Through these distinct advantages, VideoMamba sets a new
benchmark for video understanding, offering a scalable and efficient solution
for comprehensive video understanding. All the code and models are available at
https://github.com/OpenGVLab/VideoMamba.
V3D: Video Diffusion Models are Effective 3D Generators
Authors: name: Zilong Chen, name: Yikai Wang, name: Feng Wang, name: Zhengyi Wang, name: Huaping Liu
Published: 2024-03-11 14:03:36

Automatic 3D generation has recently attracted widespread attention. Recent
methods have greatly accelerated the generation speed, but usually produce
less-detailed objects due to limited model capacity or 3D data. Motivated by
recent advancements in video diffusion models, we introduce V3D, which
leverages the world simulation capacity of pre-trained video diffusion models
to facilitate 3D generation. To fully unleash the potential of video diffusion
to perceive the 3D world, we further introduce geometrical consistency prior
and extend the video diffusion model to a multi-view consistent 3D generator.
Benefiting from this, the state-of-the-art video diffusion model could be
fine-tuned to generate 360degree orbit frames surrounding an object given a
single image. With our tailored reconstruction pipelines, we can generate
high-quality meshes or 3D Gaussians within 3 minutes. Furthermore, our method
can be extended to scene-level novel view synthesis, achieving precise control
over the camera path with sparse input views. Extensive experiments demonstrate
the superior performance of the proposed approach, especially in terms of
generation quality and multi-view consistency. Our code is available at
https://github.com/heheyas/V3D
FaceChain-SuDe: Building Derived Class to Inherit Category Attributes
  for One-shot Subject-Driven Generation
Authors: name: Pengchong Qiao, name: Lei Shang, name: Chang Liu, name: Baigui Sun, name: Xiangyang Ji, name: Jie Chen
Published: 2024-03-11 14:43:40

Subject-driven generation has garnered significant interest recently due to
its ability to personalize text-to-image generation. Typical works focus on
learning the new subject's private attributes. However, an important fact has
not been taken seriously that a subject is not an isolated new concept but
should be a specialization of a certain category in the pre-trained model. This
results in the subject failing to comprehensively inherit the attributes in its
category, causing poor attribute-related generations. In this paper, motivated
by object-oriented programming, we model the subject as a derived class whose
base class is its semantic category. This modeling enables the subject to
inherit public attributes from its category while learning its private
attributes from the user-provided example. Specifically, we propose a
plug-and-play method, Subject-Derived regularization (SuDe). It constructs the
base-derived class modeling by constraining the subject-driven generated images
to semantically belong to the subject's category. Extensive experiments under
three baselines and two backbones on various subjects show that our SuDe
enables imaginative attribute-related generations while maintaining subject
fidelity. Codes will be open sourced soon at FaceChain
(https://github.com/modelscope/facechain).
VidProM: A Million-scale Real Prompt-Gallery Dataset for Text-to-Video
  Diffusion Models
Authors: name: Wenhao Wang, name: Yi Yang
Published: 2024-03-10 05:40:12

The arrival of Sora marks a new era for text-to-video diffusion models,
bringing significant advancements in video generation and potential
applications. However, Sora, as well as other text-to-video diffusion models,
highly relies on the prompts, and there is no publicly available dataset
featuring a study of text-to-video prompts. In this paper, we introduce
VidProM, the first large-scale dataset comprising 1.67 million unique
text-to-video prompts from real users. Additionally, the dataset includes 6.69
million videos generated by four state-of-the-art diffusion models and some
related data. We initially demonstrate the curation of this large-scale
dataset, which is a time-consuming and costly process. Subsequently, we show
how the proposed VidProM differs from DiffusionDB, a large-scale prompt-gallery
dataset for image generation. Based on the analysis of these prompts, we
identify the necessity for a new prompt dataset specifically designed for
text-to-video generation and gain insights into the preferences of real users
when creating videos. Our large-scale and diverse dataset also inspires many
exciting new research areas. For instance, to develop better, more efficient,
and safer text-to-video diffusion models, we suggest exploring text-to-video
prompt engineering, efficient video generation, and video copy detection for
diffusion models. We make the collected dataset VidProM publicly available at
GitHub and Hugging Face under the CC-BY- NC 4.0 License.
Stealing Part of a Production Language Model
Authors: name: Nicholas Carlini, name: Daniel Paleka, name: Krishnamurthy Dj Dvijotham, name: Thomas Steinke, name: Jonathan Hayase, name: A. Feder Cooper, name: Katherine Lee, name: Matthew Jagielski, name: Milad Nasr, name: Arthur Conmy, name: Eric Wallace, name: David Rolnick, name: Florian Tramèr
Published: 2024-03-11 11:46:12

We introduce the first model-stealing attack that extracts precise,
nontrivial information from black-box production language models like OpenAI's
ChatGPT or Google's PaLM-2. Specifically, our attack recovers the embedding
projection layer (up to symmetries) of a transformer model, given typical API
access. For under \20 USD, our attack extracts the entire projection matrix of
OpenAI's Ada and Babbage language models. We thereby confirm, for the first
time, that these black-box models have a hidden dimension of 1024 and 2048,
respectively. We also recover the exact hidden dimension size of the
gpt-3.5-turbo model, and estimate it would cost under 2,000 in queries to
recover the entire projection matrix. We conclude with potential defenses and
mitigations, and discuss the implications of possible future work that could
extend our attack.
Personalized Audiobook Recommendations at Spotify Through Graph Neural
  Networks
Authors: name: Marco De Nadai, name: Francesco Fabbri, name: Paul Gigioli, name: Alice Wang, name: Ang Li, name: Fabrizio Silvestri, name: Laura Kim, name: Shawn Lin, name: Vladan Radosavljevic, name: Sandeep Ghael, name: David Nyhan, name: Hugues Bouchard, name: Mounia Lalmas-Roelleke, name: Andreas Damianou
Published: 2024-03-08 09:53:07

In the ever-evolving digital audio landscape, Spotify, well-known for its
music and talk content, has recently introduced audiobooks to its vast user
base. While promising, this move presents significant challenges for
personalized recommendations. Unlike music and podcasts, audiobooks, initially
available for a fee, cannot be easily skimmed before purchase, posing higher
stakes for the relevance of recommendations. Furthermore, introducing a new
content type into an existing platform confronts extreme data sparsity, as most
users are unfamiliar with this new content type. Lastly, recommending content
to millions of users requires the model to react fast and be scalable. To
address these challenges, we leverage podcast and music user preferences and
introduce 2T-HGNN, a scalable recommendation system comprising Heterogeneous
Graph Neural Networks (HGNNs) and a Two Tower (2T) model. This novel approach
uncovers nuanced item relationships while ensuring low latency and complexity.
We decouple users from the HGNN graph and propose an innovative multi-link
neighbor sampler. These choices, together with the 2T component, significantly
reduce the complexity of the HGNN model. Empirical evaluations involving
millions of users show significant improvement in the quality of personalized
recommendations, resulting in a +46% increase in new audiobooks start rate and
a +23% boost in streaming rates. Intriguingly, our model's impact extends
beyond audiobooks, benefiting established products like podcasts.
CRM: Single Image to 3D Textured Mesh with Convolutional Reconstruction
  Model
Authors: name: Zhengyi Wang, name: Yikai Wang, name: Yifei Chen, name: Chendong Xiang, name: Shuo Chen, name: Dajiang Yu, name: Chongxuan Li, name: Hang Su, name: Jun Zhu
Published: 2024-03-08 04:25:29

Feed-forward 3D generative models like the Large Reconstruction Model (LRM)
have demonstrated exceptional generation speed. However, the transformer-based
methods do not leverage the geometric priors of the triplane component in their
architecture, often leading to sub-optimal quality given the limited size of 3D
data and slow training. In this work, we present the Convolutional
Reconstruction Model (CRM), a high-fidelity feed-forward single image-to-3D
generative model. Recognizing the limitations posed by sparse 3D data, we
highlight the necessity of integrating geometric priors into network design.
CRM builds on the key observation that the visualization of triplane exhibits
spatial correspondence of six orthographic images. First, it generates six
orthographic view images from a single input image, then feeds these images
into a convolutional U-Net, leveraging its strong pixel-level alignment
capabilities and significant bandwidth to create a high-resolution triplane.
CRM further employs Flexicubes as geometric representation, facilitating direct
end-to-end optimization on textured meshes. Overall, our model delivers a
high-fidelity textured mesh from an image in just 10 seconds, without any
test-time optimization.
ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment
Authors: name: Xiwei Hu, name: Rui Wang, name: Yixiao Fang, name: Bin Fu, name: Pei Cheng, name: Gang Yu
Published: 2024-03-08 08:08:10

Diffusion models have demonstrated remarkable performance in the domain of
text-to-image generation. However, most widely used models still employ CLIP as
their text encoder, which constrains their ability to comprehend dense prompts,
encompassing multiple objects, detailed attributes, complex relationships,
long-text alignment, etc. In this paper, we introduce an Efficient Large
Language Model Adapter, termed ELLA, which equips text-to-image diffusion
models with powerful Large Language Models (LLM) to enhance text alignment
without training of either U-Net or LLM. To seamlessly bridge two pre-trained
models, we investigate a range of semantic alignment connector designs and
propose a novel module, the Timestep-Aware Semantic Connector (TSC), which
dynamically extracts timestep-dependent conditions from LLM. Our approach
adapts semantic features at different stages of the denoising process,
assisting diffusion models in interpreting lengthy and intricate prompts over
sampling timesteps. Additionally, ELLA can be readily incorporated with
community models and tools to improve their prompt-following capabilities. To
assess text-to-image models in dense prompt following, we introduce Dense
Prompt Graph Benchmark (DPG-Bench), a challenging benchmark consisting of 1K
dense prompts. Extensive experiments demonstrate the superiority of ELLA in
dense prompt following compared to state-of-the-art methods, particularly in
multiple object compositions involving diverse attributes and relationships.
DeepSeek-VL: Towards Real-World Vision-Language Understanding
Authors: name: Haoyu Lu, name: Wen Liu, name: Bo Zhang, name: Bingxuan Wang, name: Kai Dong, name: Bo Liu, name: Jingxiang Sun, name: Tongzheng Ren, name: Zhuoshu Li, name: Yaofeng Sun, name: Chengqi Deng, name: Hanwei Xu, name: Zhenda Xie, name: Chong Ruan
Published: 2024-03-08 18:46:00

We present DeepSeek-VL, an open-source Vision-Language (VL) Model designed
for real-world vision and language understanding applications. Our approach is
structured around three key dimensions:
  We strive to ensure our data is diverse, scalable, and extensively covers
real-world scenarios including web screenshots, PDFs, OCR, charts, and
knowledge-based content, aiming for a comprehensive representation of practical
contexts. Further, we create a use case taxonomy from real user scenarios and
construct an instruction tuning dataset accordingly. The fine-tuning with this
dataset substantially improves the model's user experience in practical
applications. Considering efficiency and the demands of most real-world
scenarios, DeepSeek-VL incorporates a hybrid vision encoder that efficiently
processes high-resolution images (1024 x 1024), while maintaining a relatively
low computational overhead. This design choice ensures the model's ability to
capture critical semantic and detailed information across various visual tasks.
We posit that a proficient Vision-Language Model should, foremost, possess
strong language abilities. To ensure the preservation of LLM capabilities
during pretraining, we investigate an effective VL pretraining strategy by
integrating LLM training from the beginning and carefully managing the
competitive dynamics observed between vision and language modalities.
  The DeepSeek-VL family (both 1.3B and 7B models) showcases superior user
experiences as a vision-language chatbot in real-world applications, achieving
state-of-the-art or competitive performance across a wide range of
visual-language benchmarks at the same model size while maintaining robust
performance on language-centric benchmarks. We have made both 1.3B and 7B
models publicly accessible to foster innovations based on this foundation
model.
CogView3: Finer and Faster Text-to-Image Generation via Relay Diffusion
Authors: name: Wendi Zheng, name: Jiayan Teng, name: Zhuoyi Yang, name: Weihan Wang, name: Jidong Chen, name: Xiaotao Gu, name: Yuxiao Dong, name: Ming Ding, name: Jie Tang
Published: 2024-03-08 07:32:50

Recent advancements in text-to-image generative systems have been largely
driven by diffusion models. However, single-stage text-to-image diffusion
models still face challenges, in terms of computational efficiency and the
refinement of image details. To tackle the issue, we propose CogView3, an
innovative cascaded framework that enhances the performance of text-to-image
diffusion. CogView3 is the first model implementing relay diffusion in the
realm of text-to-image generation, executing the task by first creating
low-resolution images and subsequently applying relay-based super-resolution.
This methodology not only results in competitive text-to-image outputs but also
greatly reduces both training and inference costs. Our experimental results
demonstrate that CogView3 outperforms SDXL, the current state-of-the-art
open-source text-to-image diffusion model, by 77.0\% in human evaluations, all
while requiring only about 1/2 of the inference time. The distilled variant of
CogView3 achieves comparable performance while only utilizing 1/10 of the
inference time by SDXL.
VideoElevator: Elevating Video Generation Quality with Versatile
  Text-to-Image Diffusion Models
Authors: name: Yabo Zhang, name: Yuxiang Wei, name: Xianhui Lin, name: Zheng Hui, name: Peiran Ren, name: Xuansong Xie, name: Xiangyang Ji, name: Wangmeng Zuo
Published: 2024-03-08 16:44:54

Text-to-image diffusion models (T2I) have demonstrated unprecedented
capabilities in creating realistic and aesthetic images. On the contrary,
text-to-video diffusion models (T2V) still lag far behind in frame quality and
text alignment, owing to insufficient quality and quantity of training videos.
In this paper, we introduce VideoElevator, a training-free and plug-and-play
method, which elevates the performance of T2V using superior capabilities of
T2I. Different from conventional T2V sampling (i.e., temporal and spatial
modeling), VideoElevator explicitly decomposes each sampling step into temporal
motion refining and spatial quality elevating. Specifically, temporal motion
refining uses encapsulated T2V to enhance temporal consistency, followed by
inverting to the noise distribution required by T2I. Then, spatial quality
elevating harnesses inflated T2I to directly predict less noisy latent, adding
more photo-realistic details. We have conducted experiments in extensive
prompts under the combination of various T2V and T2I. The results show that
VideoElevator not only improves the performance of T2V baselines with
foundational T2I, but also facilitates stylistic video synthesis with
personalized T2I. Our code is available at
https://github.com/YBYBZhang/VideoElevator.
Pix2Gif: Motion-Guided Diffusion for GIF Generation
Authors: name: Hitesh Kandala, name: Jianfeng Gao, name: Jianwei Yang
Published: 2024-03-07 16:18:28

We present Pix2Gif, a motion-guided diffusion model for image-to-GIF (video)
generation. We tackle this problem differently by formulating the task as an
image translation problem steered by text and motion magnitude prompts, as
shown in teaser fig. To ensure that the model adheres to motion guidance, we
propose a new motion-guided warping module to spatially transform the features
of the source image conditioned on the two types of prompts. Furthermore, we
introduce a perceptual loss to ensure the transformed feature map remains
within the same space as the target image, ensuring content consistency and
coherence. In preparation for the model training, we meticulously curated data
by extracting coherent image frames from the TGIF video-caption dataset, which
provides rich information about the temporal changes of subjects. After
pretraining, we apply our model in a zero-shot manner to a number of video
datasets. Extensive qualitative and quantitative experiments demonstrate the
effectiveness of our model -- it not only captures the semantic prompt from
text but also the spatial ones from motion guidance. We train all our models
using a single node of 16xV100 GPUs. Code, dataset and models are made public
at: https://hiteshk03.github.io/Pix2Gif/.
StableDrag: Stable Dragging for Point-based Image Editing
Authors: name: Yutao Cui, name: Xiaotong Zhao, name: Guozhen Zhang, name: Shengming Cao, name: Kai Ma, name: Limin Wang
Published: 2024-03-07 12:11:02

Point-based image editing has attracted remarkable attention since the
emergence of DragGAN. Recently, DragDiffusion further pushes forward the
generative quality via adapting this dragging technique to diffusion models.
Despite these great success, this dragging scheme exhibits two major drawbacks,
namely inaccurate point tracking and incomplete motion supervision, which may
result in unsatisfactory dragging outcomes. To tackle these issues, we build a
stable and precise drag-based editing framework, coined as StableDrag, by
designing a discirminative point tracking method and a confidence-based latent
enhancement strategy for motion supervision. The former allows us to precisely
locate the updated handle points, thereby boosting the stability of long-range
manipulation, while the latter is responsible for guaranteeing the optimized
latent as high-quality as possible across all the manipulation steps. Thanks to
these unique designs, we instantiate two types of image editing models
including StableDrag-GAN and StableDrag-Diff, which attains more stable
dragging performance, through extensive qualitative experiments and
quantitative assessment on DragBench.
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
Authors: name: Wei-Lin Chiang, name: Lianmin Zheng, name: Ying Sheng, name: Anastasios Nikolas Angelopoulos, name: Tianle Li, name: Dacheng Li, name: Hao Zhang, name: Banghua Zhu, name: Michael Jordan, name: Joseph E. Gonzalez, name: Ion Stoica
Published: 2024-03-07 01:22:38

Large Language Models (LLMs) have unlocked new capabilities and applications;
however, evaluating the alignment with human preferences still poses
significant challenges. To address this issue, we introduce Chatbot Arena, an
open platform for evaluating LLMs based on human preferences. Our methodology
employs a pairwise comparison approach and leverages input from a diverse user
base through crowdsourcing. The platform has been operational for several
months, amassing over 240K votes. This paper describes the platform, analyzes
the data we have collected so far, and explains the tried-and-true statistical
methods we are using for efficient and accurate evaluation and ranking of
models. We confirm that the crowdsourced questions are sufficiently diverse and
discriminating and that the crowdsourced human votes are in good agreement with
those of expert raters. These analyses collectively establish a robust
foundation for the credibility of Chatbot Arena. Because of its unique value
and openness, Chatbot Arena has emerged as one of the most referenced LLM
leaderboards, widely cited by leading LLM developers and companies. Our demo is
publicly available at https://chat.lmsys.org.
Common 7B Language Models Already Possess Strong Math Capabilities
Authors: name: Chen Li, name: Weiqi Wang, name: Jingcheng Hu, name: Yixuan Wei, name: Nanning Zheng, name: Han Hu, name: Zheng Zhang, name: Houwen Peng
Published: 2024-03-07 18:00:40

Mathematical capabilities were previously believed to emerge in common
language models only at a very large scale or require extensive math-related
pre-training. This paper shows that the LLaMA-2 7B model with common
pre-training already exhibits strong mathematical abilities, as evidenced by
its impressive accuracy of 97.7% and 72.0% on the GSM8K and MATH benchmarks,
respectively, when selecting the best response from 256 random generations. The
primary issue with the current base model is the difficulty in consistently
eliciting its inherent mathematical capabilities. Notably, the accuracy for the
first answer drops to 49.5% and 7.9% on the GSM8K and MATH benchmarks,
respectively. We find that simply scaling up the SFT data can significantly
enhance the reliability of generating correct answers. However, the potential
for extensive scaling is constrained by the scarcity of publicly available math
questions. To overcome this limitation, we employ synthetic data, which proves
to be nearly as effective as real data and shows no clear saturation when
scaled up to approximately one million samples. This straightforward approach
achieves an accuracy of 82.6% on GSM8K and 40.6% on MATH using LLaMA-2 7B
models, surpassing previous models by 14.2% and 20.8%, respectively. We also
provide insights into scaling behaviors across different reasoning complexities
and error types.