Orionnnn/hub-stats huggingface dataset:
Text-Conditioned Resampler For Long Form Video Understanding
Authors: name: Bruno Korbar, name: Yongqin Xian, name: Alessio Tonioni, name: Andrew Zisserman, name: Federico Tombari
Published: 2023-12-19 06:42:47

Videos are highly redundant data source and it is often enough to identify a
few key moments to solve any given task. In this paper, we present a
text-conditioned video resampler (TCR) module that uses a pre-trained and
frozen visual encoder and large language model (LLM) to process long video
sequences for a task. TCR localises relevant visual features from the video
given a text condition and provides them to a LLM to generate a text response.
Due to its lightweight design and use of cross-attention, TCR can process more
than 100 frames at a time allowing the model to use much longer chunks of video
than earlier works. We make the following contributions: (i) we design a
transformer-based sampling architecture that can process long videos
conditioned on a task, together with a training method that enables it to
bridge pre-trained visual and language models; (ii) we empirically validate its
efficacy on a wide variety of evaluation tasks, and set a new state-of-the-art
on NextQA, EgoSchema, and the EGO4D-LTA challenge; and (iii) we determine tasks
which require longer video contexts and that can thus be used effectively for
further evaluation of long-range video models.
StarVector: Generating Scalable Vector Graphics Code from Images
Authors: name: Juan A. Rodriguez, name: Shubham Agarwal, name: Issam H. Laradji, name: Pau Rodriguez, name: David Vazquez, name: Christopher Pal, name: Marco Pedersoli
Published: 2023-12-17 08:07:32

Scalable Vector Graphics (SVGs) have become integral in modern image
rendering applications due to their infinite scalability in resolution,
versatile usability, and editing capabilities. SVGs are particularly popular in
the fields of web development and graphic design. Existing approaches for SVG
modeling using deep learning often struggle with generating complex SVGs and
are restricted to simpler ones that require extensive processing and
simplification. This paper introduces StarVector, a multimodal SVG generation
model that effectively integrates Code Generation Large Language Models
(CodeLLMs) and vision models. Our approach utilizes a CLIP image encoder to
extract visual representations from pixel-based images, which are then
transformed into visual tokens via an adapter module. These visual tokens are
pre-pended to the SVG token embeddings, and the sequence is modeled by the
StarCoder model using next-token prediction, effectively learning to align the
visual and code tokens. This enables StarVector to generate unrestricted SVGs
that accurately represent pixel images. To evaluate StarVector's performance,
we present SVG-Bench, a comprehensive benchmark for evaluating SVG methods
across multiple datasets and relevant metrics. Within this benchmark, we
introduce novel datasets including SVG-Stack, a large-scale dataset of
real-world SVG examples, and use it to pre-train StarVector as a large
foundation model for SVGs. Our results demonstrate significant enhancements in
visual quality and complexity handling over current methods, marking a notable
advancement in SVG generation technology. Code and models:
https://github.com/joanrod/star-vector
Topic-VQ-VAE: Leveraging Latent Codebooks for Flexible Topic-Guided
  Document Generation
Authors: name: YoungJoon Yoo, name: Jongwon Choi
Published: 2023-12-15 15:01:10

This paper introduces a novel approach for topic modeling utilizing latent
codebooks from Vector-Quantized Variational Auto-Encoder~(VQ-VAE), discretely
encapsulating the rich information of the pre-trained embeddings such as the
pre-trained language model. From the novel interpretation of the latent
codebooks and embeddings as conceptual bag-of-words, we propose a new
generative topic model called Topic-VQ-VAE~(TVQ-VAE) which inversely generates
the original documents related to the respective latent codebook. The TVQ-VAE
can visualize the topics with various generative distributions including the
traditional BoW distribution and the autoregressive image generation. Our
experimental results on document analysis and image generation demonstrate that
TVQ-VAE effectively captures the topic context which reveals the underlying
structures of the dataset and supports flexible forms of document generation.
Official implementation of the proposed TVQ-VAE is available at
https://github.com/clovaai/TVQ-VAE.
LLM in a flash: Efficient Large Language Model Inference with Limited
  Memory
Authors: name: Keivan Alizadeh, name: Iman Mirzadeh, name: Dmitry Belenko, name: Karen Khatamifard, name: Minsik Cho, name: Carlo C Del Mundo, name: Mohammad Rastegari, name: Mehrdad Farajtabar
Published: 2023-12-12 18:57:08

Large language models (LLMs) are central to modern natural language
processing, delivering exceptional performance in various tasks. However, their
intensive computational and memory requirements present challenges, especially
for devices with limited DRAM capacity. This paper tackles the challenge of
efficiently running LLMs that exceed the available DRAM capacity by storing the
model parameters on flash memory but bringing them on demand to DRAM. Our
method involves constructing an inference cost model that harmonizes with the
flash memory behavior, guiding us to optimize in two critical areas: reducing
the volume of data transferred from flash and reading data in larger, more
contiguous chunks. Within this flash memory-informed framework, we introduce
two principal techniques. First, "windowing'" strategically reduces data
transfer by reusing previously activated neurons, and second, "row-column
bundling", tailored to the sequential data access strengths of flash memory,
increases the size of data chunks read from flash memory. These methods
collectively enable running models up to twice the size of the available DRAM,
with a 4-5x and 20-25x increase in inference speed compared to naive loading
approaches in CPU and GPU, respectively. Our integration of sparsity awareness,
context-adaptive loading, and a hardware-oriented design paves the way for
effective inference of LLMs on devices with limited memory.
A Challenger to GPT-4V? Early Explorations of Gemini in Visual Expertise
Authors: name: Chaoyou Fu, name: Renrui Zhang, name: Haojia Lin, name: Zihan Wang, name: Timin Gao, name: Yongdong Luo, name: Yubo Huang, name: Zhengye Zhang, name: Longtian Qiu, name: Gaoxiang Ye, name: Yunhang Shen, name: Mengdan Zhang, name: Peixian Chen, name: Sirui Zhao, name: Xiawu Zheng, name: Shaohui Lin, name: Deqiang Jiang, name: Di Yin, name: Peng Gao, name: Ke Li, name: Xing Sun, name: Rongrong Ji
Published: 2023-12-19 18:59:22

The surge of interest towards Multi-modal Large Language Models (MLLMs),
e.g., GPT-4V(ision) from OpenAI, has marked a significant trend in both
academia and industry. They endow Large Language Models (LLMs) with powerful
capabilities in visual understanding, enabling them to tackle diverse
multi-modal tasks. Very recently, Google released Gemini, its newest and most
capable MLLM built from the ground up for multi-modality. In light of the
superior reasoning capabilities, can Gemini challenge GPT-4V's leading position
in multi-modal learning? In this paper, we present a preliminary exploration of
Gemini Pro's visual understanding proficiency, which comprehensively covers
four domains: fundamental perception, advanced cognition, challenging vision
tasks, and various expert capacities. We compare Gemini Pro with the
state-of-the-art GPT-4V to evaluate its upper limits, along with the latest
open-sourced MLLM, Sphinx, which reveals the gap between manual efforts and
black-box systems. The qualitative samples indicate that, while GPT-4V and
Gemini showcase different answering styles and preferences, they can exhibit
comparable visual reasoning capabilities, and Sphinx still trails behind them
concerning domain generalizability. Specifically, GPT-4V tends to elaborate
detailed explanations and intermediate steps, and Gemini prefers to output a
direct and concise answer. The quantitative evaluation on the popular MME
benchmark also demonstrates the potential of Gemini to be a strong challenger
to GPT-4V. Our early investigation of Gemini also observes some common issues
of MLLMs, indicating that there still remains a considerable distance towards
artificial general intelligence. Our project for tracking the progress of MLLM
is released at
https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models.
Towards Accurate Guided Diffusion Sampling through Symplectic Adjoint
  Method
Authors: name: Jiachun Pan, name: Hanshu Yan, name: Jun Hao Liew, name: Jiashi Feng, name: Vincent Y. F. Tan
Published: 2023-12-19 10:30:31

Training-free guided sampling in diffusion models leverages off-the-shelf
pre-trained networks, such as an aesthetic evaluation model, to guide the
generation process. Current training-free guided sampling algorithms obtain the
guidance energy function based on a one-step estimate of the clean image.
However, since the off-the-shelf pre-trained networks are trained on clean
images, the one-step estimation procedure of the clean image may be inaccurate,
especially in the early stages of the generation process in diffusion models.
This causes the guidance in the early time steps to be inaccurate. To overcome
this problem, we propose Symplectic Adjoint Guidance (SAG), which calculates
the gradient guidance in two inner stages. Firstly, SAG estimates the clean
image via n function calls, where n serves as a flexible hyperparameter
that can be tailored to meet specific image quality requirements. Secondly, SAG
uses the symplectic adjoint method to obtain the gradients accurately and
efficiently in terms of the memory requirements. Extensive experiments
demonstrate that SAG generates images with higher qualities compared to the
baselines in both guided image and video generation tasks.
MAG-Edit: Localized Image Editing in Complex Scenarios via
  Mask-Based Attention-Adjusted
  Guidance
Authors: name: Qi Mao, name: Lan Chen, name: Yuchao Gu, name: Zhen Fang, name: Mike Zheng Shou
Published: 2023-12-18 17:55:44

Recent diffusion-based image editing approaches have exhibited impressive
editing capabilities in images with simple compositions. However, localized
editing in complex scenarios has not been well-studied in the literature,
despite its growing real-world demands. Existing mask-based inpainting methods
fall short of retaining the underlying structure within the edit region.
Meanwhile, mask-free attention-based methods often exhibit editing leakage and
misalignment in more complex compositions. In this work, we develop
MAG-Edit, a training-free, inference-stage optimization method,
which enables localized image editing in complex scenarios. In particular,
MAG-Edit optimizes the noise latent feature in diffusion models by maximizing
two mask-based cross-attention constraints of the edit token, which in turn
gradually enhances the local alignment with the desired prompt. Extensive
quantitative and qualitative experiments demonstrate the effectiveness of our
method in achieving both text alignment and structure preservation for
localized editing within complex scenarios.
Your Student is Better Than Expected: Adaptive Teacher-Student
  Collaboration for Text-Conditional Diffusion Models
Authors: name: Nikita Starodubcev, name: Artem Fedorov, name: Artem Babenko, name: Dmitry Baranchuk
Published: 2023-12-17 22:40:38

Knowledge distillation methods have recently shown to be a promising
direction to speedup the synthesis of large-scale diffusion models by requiring
only a few inference steps. While several powerful distillation methods were
recently proposed, the overall quality of student samples is typically lower
compared to the teacher ones, which hinders their practical usage. In this
work, we investigate the relative quality of samples produced by the teacher
text-to-image diffusion model and its distilled student version. As our main
empirical finding, we discover that a noticeable portion of student samples
exhibit superior fidelity compared to the teacher ones, despite the
``approximate'' nature of the student. Based on this finding, we propose an
adaptive collaboration between student and teacher diffusion models for
effective text-to-image synthesis. Specifically, the distilled model produces
the initial sample, and then an oracle decides whether it needs further
improvements with a slow teacher model. Extensive experiments demonstrate that
the designed pipeline surpasses state-of-the-art text-to-image alternatives for
various inference budgets in terms of human preference. Furthermore, the
proposed approach can be naturally used in popular applications such as
text-guided image editing and controllable generation.
VecFusion: Vector Font Generation with Diffusion
Authors: name: Vikas Thamizharasan, name: Difan Liu, name: Shantanu Agarwal, name: Matthew Fisher, name: Michael Gharbi, name: Oliver Wang, name: Alec Jacobson, name: Evangelos Kalogerakis
Published: 2023-12-16 20:49:00

We present VecFusion, a new neural architecture that can generate vector
fonts with varying topological structures and precise control point positions.
Our approach is a cascaded diffusion model which consists of a raster diffusion
model followed by a vector diffusion model. The raster model generates
low-resolution, rasterized fonts with auxiliary control point information,
capturing the global style and shape of the font, while the vector model
synthesizes vector fonts conditioned on the low-resolution raster fonts from
the first stage. To synthesize long and complex curves, our vector diffusion
model uses a transformer architecture and a novel vector representation that
enables the modeling of diverse vector geometry and the precise prediction of
control points. Our experiments show that, in contrast to previous generative
models for vector graphics, our new cascaded vector diffusion model generates
higher quality vector fonts, with complex structures and diverse styles.
GauFRe: Gaussian Deformation Fields for Real-time Dynamic Novel View
  Synthesis
Authors: name: Yiqing Liang, name: Numair Khan, name: Zhengqin Li, name: Thu Nguyen-Phuoc, name: Douglas Lanman, name: James Tompkin, name: Lei Xiao
Published: 2023-12-18 18:59:03

We propose a method for dynamic scene reconstruction using deformable 3D
Gaussians that is tailored for monocular video. Building upon the efficiency of
Gaussian splatting, our approach extends the representation to accommodate
dynamic elements via a deformable set of Gaussians residing in a canonical
space, and a time-dependent deformation field defined by a multi-layer
perceptron (MLP). Moreover, under the assumption that most natural scenes have
large regions that remain static, we allow the MLP to focus its
representational power by additionally including a static Gaussian point cloud.
The concatenated dynamic and static point clouds form the input for the
Gaussian Splatting rasterizer, enabling real-time rendering. The differentiable
pipeline is optimized end-to-end with a self-supervised rendering loss. Our
method achieves results that are comparable to state-of-the-art dynamic neural
radiance field methods while allowing much faster optimization and rendering.
Project website: https://lynl7130.github.io/gaufre/index.html
GAvatar: Animatable 3D Gaussian Avatars with Implicit Mesh Learning
Authors: name: Ye Yuan, name: Xueting Li, name: Yangyi Huang, name: Shalini De Mello, name: Koki Nagano, name: Jan Kautz, name: Umar Iqbal
Published: 2023-12-18 18:59:12

Gaussian splatting has emerged as a powerful 3D representation that harnesses
the advantages of both explicit (mesh) and implicit (NeRF) 3D representations.
In this paper, we seek to leverage Gaussian splatting to generate realistic
animatable avatars from textual descriptions, addressing the limitations (e.g.,
flexibility and efficiency) imposed by mesh or NeRF-based representations.
However, a naive application of Gaussian splatting cannot generate high-quality
animatable avatars and suffers from learning instability; it also cannot
capture fine avatar geometries and often leads to degenerate body parts. To
tackle these problems, we first propose a primitive-based 3D Gaussian
representation where Gaussians are defined inside pose-driven primitives to
facilitate animation. Second, to stabilize and amortize the learning of
millions of Gaussians, we propose to use neural implicit fields to predict the
Gaussian attributes (e.g., colors). Finally, to capture fine avatar geometries
and extract detailed meshes, we propose a novel SDF-based implicit mesh
learning approach for 3D Gaussians that regularizes the underlying geometries
and extracts highly detailed textured meshes. Our proposed method, GAvatar,
enables the large-scale generation of diverse animatable avatars using only
text prompts. GAvatar significantly surpasses existing methods in terms of both
appearance and geometry quality, and achieves extremely fast rendering (100
fps) at 1K resolution.
VolumeDiffusion: Flexible Text-to-3D Generation with Efficient
  Volumetric Encoder
Authors: name: Zhicong Tang, name: Shuyang Gu, name: Chunyu Wang, name: Ting Zhang, name: Jianmin Bao, name: Dong Chen, name: Baining Guo
Published: 2023-12-18 18:59:05

This paper introduces a pioneering 3D volumetric encoder designed for
text-to-3D generation. To scale up the training data for the diffusion model, a
lightweight network is developed to efficiently acquire feature volumes from
multi-view images. The 3D volumes are then trained on a diffusion model for
text-to-3D generation using a 3D U-Net. This research further addresses the
challenges of inaccurate object captions and high-dimensional feature volumes.
The proposed model, trained on the public Objaverse dataset, demonstrates
promising outcomes in producing diverse and recognizable samples from text
prompts. Notably, it empowers finer control over object part characteristics
through textual cues, fostering model creativity by seamlessly combining
multiple concepts within a single object. This research significantly
contributes to the progress of 3D generation by introducing an efficient,
flexible, and scalable representation methodology. Code is available at
https://github.com/tzco/VolumeDiffusion.
SCEdit: Efficient and Controllable Image Diffusion Generation via Skip
  Connection Editing
Authors: name: Zeyinzi Jiang, name: Chaojie Mao, name: Yulin Pan, name: Zhen Han, name: Jingfeng Zhang
Published: 2023-12-18 17:54:14

Image diffusion models have been utilized in various tasks, such as
text-to-image generation and controllable image synthesis. Recent research has
introduced tuning methods that make subtle adjustments to the original models,
yielding promising results in specific adaptations of foundational generative
diffusion models. Rather than modifying the main backbone of the diffusion
model, we delve into the role of skip connection in U-Net and reveal that
hierarchical features aggregating long-distance information across encoder and
decoder make a significant impact on the content and quality of image
generation. Based on the observation, we propose an efficient generative tuning
framework, dubbed SCEdit, which integrates and edits Skip Connection using a
lightweight tuning module named SC-Tuner. Furthermore, the proposed framework
allows for straightforward extension to controllable image synthesis by
injecting different conditions with Controllable SC-Tuner, simplifying and
unifying the network design for multi-condition inputs. Our SCEdit
substantially reduces training parameters, memory usage, and computational
expense due to its lightweight tuners, with backward propagation only passing
to the decoder blocks. Extensive experiments conducted on text-to-image
generation and controllable image synthesis tasks demonstrate the superiority
of our method in terms of efficiency and performance. Project page:
https://scedit.github.io/
MagicScroll: Nontypical Aspect-Ratio Image Generation for Visual
  Storytelling via Multi-Layered Semantic-Aware Denoising
Authors: name: Bingyuan Wang, name: Hengyu Meng, name: Zeyu Cai, name: Lanjiong Li, name: Yue Ma, name: Qifeng Chen, name: Zeyu Wang
Published: 2023-12-18 03:09:05

Visual storytelling often uses nontypical aspect-ratio images like scroll
paintings, comic strips, and panoramas to create an expressive and compelling
narrative. While generative AI has achieved great success and shown the
potential to reshape the creative industry, it remains a challenge to generate
coherent and engaging content with arbitrary size and controllable style,
concept, and layout, all of which are essential for visual storytelling. To
overcome the shortcomings of previous methods including repetitive content,
style inconsistency, and lack of controllability, we propose MagicScroll, a
multi-layered, progressive diffusion-based image generation framework with a
novel semantic-aware denoising process. The model enables fine-grained control
over the generated image on object, scene, and background levels with text,
image, and layout conditions. We also establish the first benchmark for
nontypical aspect-ratio image generation for visual storytelling including
mediums like paintings, comics, and cinematic panoramas, with customized
metrics for systematic evaluation. Through comparative and ablation studies,
MagicScroll showcases promising results in aligning with the narrative text,
improving visual coherence, and engaging the audience. We plan to release the
code and benchmark in the hope of a better collaboration between AI researchers
and creative practitioners involving visual storytelling.
Cascade Speculative Drafting for Even Faster LLM Inference
Authors: name: Ziyi Chen, name: Xiaocong Yang, name: Jiacheng Lin, name: Chenkai Sun, name: Jie Huang, name: Kevin Chen-Chuan Chang
Published: 2023-12-18 18:59:46

Speculative decoding enhances the efficiency of large language models (LLMs)
by leveraging a draft model to draft for a larger target model to review.
However, drafting in speculative decoding involves slow autoregressive
generation and generating tokens of different importance with the same time
allocation. These two inefficiencies lead to its suboptimal performance. To
address this issue, we introduce Cascade Speculative Drafting (CS. Drafting), a
novel approach that employs two types of cascades. The Vertical Cascade
eliminates autoregressive generation from neural models. The Horizontal Cascade
constitutes efficient time allocation in drafting with its optimality supported
by our theoretical analysis. Combining both cascades, our CS. Drafting
algorithm has achieved up to 72 percent additional speedup over speculative
decoding in our experiments while keeping the same output distribution.