Orionnnn/hub-stats huggingface dataset:
Does Gaussian Splatting need SFM Initialization?
Authors: name: Yalda Foroutan, name: Daniel Rebain, name: Kwang Moo Yi, name: Andrea Tagliasacchi
Published: 2024-04-18 23:52:42

3D Gaussian Splatting has recently been embraced as a versatile and effective
method for scene reconstruction and novel view synthesis, owing to its
high-quality results and compatibility with hardware rasterization. Despite its
advantages, Gaussian Splatting's reliance on high-quality point cloud
initialization by Structure-from-Motion (SFM) algorithms is a significant
limitation to be overcome. To this end, we investigate various initialization
strategies for Gaussian Splatting and delve into how volumetric reconstructions
from Neural Radiance Fields (NeRF) can be utilized to bypass the dependency on
SFM data. Our findings demonstrate that random initialization can perform much
better if carefully designed and that by employing a combination of improved
initialization strategies and structure distillation from low-cost NeRF models,
it is possible to achieve equivalent results, or at times even superior, to
those obtained from SFM initialization.
PhysDreamer: Physics-Based Interaction with 3D Objects via Video
  Generation
Authors: name: Tianyuan Zhang, name: Hong-Xing Yu, name: Rundi Wu, name: Brandon Y. Feng, name: Changxi Zheng, name: Noah Snavely, name: Jiajun Wu, name: William T. Freeman
Published: 2024-04-19 17:41:05

Realistic object interactions are crucial for creating immersive virtual
experiences, yet synthesizing realistic 3D object dynamics in response to novel
interactions remains a significant challenge. Unlike unconditional or
text-conditioned dynamics generation, action-conditioned dynamics requires
perceiving the physical material properties of objects and grounding the 3D
motion prediction on these properties, such as object stiffness. However,
estimating physical material properties is an open problem due to the lack of
material ground-truth data, as measuring these properties for real objects is
highly difficult. We present PhysDreamer, a physics-based approach that endows
static 3D objects with interactive dynamics by leveraging the object dynamics
priors learned by video generation models. By distilling these priors,
PhysDreamer enables the synthesis of realistic object responses to novel
interactions, such as external forces or agent manipulations. We demonstrate
our approach on diverse examples of elastic objects and evaluate the realism of
the synthesized interactions through a user study. PhysDreamer takes a step
towards more engaging and realistic virtual experiences by enabling static 3D
objects to dynamically respond to interactive stimuli in a physically plausible
manner. See our project page at https://physdreamer.github.io/.
MeshLRM: Large Reconstruction Model for High-Quality Mesh
Authors: name: Xinyue Wei, name: Kai Zhang, name: Sai Bi, name: Hao Tan, name: Fujun Luan, name: Valentin Deschaintre, name: Kalyan Sunkavalli, name: Hao Su, name: Zexiang Xu
Published: 2024-04-18 17:59:41

We propose MeshLRM, a novel LRM-based approach that can reconstruct a
high-quality mesh from merely four input images in less than one second.
Different from previous large reconstruction models (LRMs) that focus on
NeRF-based reconstruction, MeshLRM incorporates differentiable mesh extraction
and rendering within the LRM framework. This allows for end-to-end mesh
reconstruction by fine-tuning a pre-trained NeRF LRM with mesh rendering.
Moreover, we improve the LRM architecture by simplifying several complex
designs in previous LRMs. MeshLRM's NeRF initialization is sequentially trained
with low- and high-resolution images; this new LRM training strategy enables
significantly faster convergence and thereby leads to better quality with less
compute. Our approach achieves state-of-the-art mesh reconstruction from
sparse-view inputs and also allows for many downstream applications, including
text-to-3D and single-image-to-3D generation. Project page:
https://sarahweiii.github.io/meshlrm/
EdgeFusion: On-Device Text-to-Image Generation
Authors: name: Thibault Castells, name: Hyoung-Kyu Song, name: Tairen Piao, name: Shinkook Choi, name: Bo-Kyeong Kim, name: Hanyoung Yim, name: Changgwun Lee, name: Jae Gon Kim, name: Tae-Ho Kim
Published: 2024-04-18 06:02:54

The intensive computational burden of Stable Diffusion (SD) for text-to-image
generation poses a significant hurdle for its practical application. To tackle
this challenge, recent research focuses on methods to reduce sampling steps,
such as Latent Consistency Model (LCM), and on employing architectural
optimizations, including pruning and knowledge distillation. Diverging from
existing approaches, we uniquely start with a compact SD variant, BK-SDM. We
observe that directly applying LCM to BK-SDM with commonly used crawled
datasets yields unsatisfactory results. It leads us to develop two strategies:
(1) leveraging high-quality image-text pairs from leading generative models and
(2) designing an advanced distillation process tailored for LCM. Through our
thorough exploration of quantization, profiling, and on-device deployment, we
achieve rapid generation of photo-realistic, text-aligned images in just two
steps, with latency under one second on resource-limited edge devices.
Dynamic Typography: Bringing Words to Life
Authors: name: Zichen Liu, name: Yihao Meng, name: Hao Ouyang, name: Yue Yu, name: Bolin Zhao, name: Daniel Cohen-Or, name: Huamin Qu
Published: 2024-04-17 17:59:55

Text animation serves as an expressive medium, transforming static
communication into dynamic experiences by infusing words with motion to evoke
emotions, emphasize meanings, and construct compelling narratives. Crafting
animations that are semantically aware poses significant challenges, demanding
expertise in graphic design and animation. We present an automated text
animation scheme, termed "Dynamic Typography", which combines two challenging
tasks. It deforms letters to convey semantic meaning and infuses them with
vibrant movements based on user prompts. Our technique harnesses vector
graphics representations and an end-to-end optimization-based framework. This
framework employs neural displacement fields to convert letters into base
shapes and applies per-frame motion, encouraging coherence with the intended
textual concept. Shape preservation techniques and perceptual loss
regularization are employed to maintain legibility and structural integrity
throughout the animation process. We demonstrate the generalizability of our
approach across various text-to-video models and highlight the superiority of
our end-to-end methodology over baseline methods, which might comprise separate
tasks. Through quantitative and qualitative evaluations, we demonstrate the
effectiveness of our framework in generating coherent text animations that
faithfully interpret user prompts while maintaining readability. Our code is
available at: https://animate-your-word.github.io/demo/.
Reuse Your Rewards: Reward Model Transfer for Zero-Shot Cross-Lingual
  Alignment
Authors: name: Zhaofeng Wu, name: Ananth Balashankar, name: Yoon Kim, name: Jacob Eisenstein, name: Ahmad Beirami
Published: 2024-04-18 16:52:36

Aligning language models (LMs) based on human-annotated preference data is a
crucial step in obtaining practical and performant LM-based systems. However,
multilingual human preference data are difficult to obtain at scale, making it
challenging to extend this framework to diverse languages. In this work, we
evaluate a simple approach for zero-shot cross-lingual alignment, where a
reward model is trained on preference data in one source language and directly
applied to other target languages. On summarization and open-ended dialog
generation, we show that this method is consistently successful under
comprehensive evaluation settings, including human evaluation: cross-lingually
aligned models are preferred by humans over unaligned models on up to >70% of
evaluation instances. We moreover find that a different-language reward model
sometimes yields better aligned models than a same-language reward model. We
also identify best practices when there is no language-specific data for even
supervised finetuning, another component in alignment.
MoA: Mixture-of-Attention for Subject-Context Disentanglement in
  Personalized Image Generation
Authors: name: Kuan-Chieh, name: Wang, name: Daniil Ostashev, name: Yuwei Fang, name: Sergey Tulyakov, name: Kfir Aberman
Published: 2024-04-17 17:08:05

We introduce a new architecture for personalization of text-to-image
diffusion models, coined Mixture-of-Attention (MoA). Inspired by the
Mixture-of-Experts mechanism utilized in large language models (LLMs), MoA
distributes the generation workload between two attention pathways: a
personalized branch and a non-personalized prior branch. MoA is designed to
retain the original model's prior by fixing its attention layers in the prior
branch, while minimally intervening in the generation process with the
personalized branch that learns to embed subjects in the layout and context
generated by the prior branch. A novel routing mechanism manages the
distribution of pixels in each layer across these branches to optimize the
blend of personalized and generic content creation. Once trained, MoA
facilitates the creation of high-quality, personalized images featuring
multiple subjects with compositions and interactions as diverse as those
generated by the original model. Crucially, MoA enhances the distinction
between the model's pre-existing capability and the newly augmented
personalized intervention, thereby offering a more disentangled subject-context
control that was previously unattainable. Project page:
https://snap-research.github.io/mixture-of-attention
Toward Self-Improvement of LLMs via Imagination, Searching, and
  Criticizing
Authors: name: Ye Tian, name: Baolin Peng, name: Linfeng Song, name: Lifeng Jin, name: Dian Yu, name: Haitao Mi, name: Dong Yu
Published: 2024-04-18 15:21:34

Despite the impressive capabilities of Large Language Models (LLMs) on
various tasks, they still struggle with scenarios that involves complex
reasoning and planning. Recent work proposed advanced prompting techniques and
the necessity of fine-tuning with high-quality data to augment LLMs' reasoning
abilities. However, these approaches are inherently constrained by data
availability and quality. In light of this, self-correction and self-learning
emerge as viable solutions, employing strategies that allow LLMs to refine
their outputs and learn from self-assessed rewards. Yet, the efficacy of LLMs
in self-refining its response, particularly in complex reasoning and planning
task, remains dubious. In this paper, we introduce AlphaLLM for the
self-improvements of LLMs, which integrates Monte Carlo Tree Search (MCTS) with
LLMs to establish a self-improving loop, thereby enhancing the capabilities of
LLMs without additional annotations. Drawing inspiration from the success of
AlphaGo, AlphaLLM addresses the unique challenges of combining MCTS with LLM
for self-improvement, including data scarcity, the vastness search spaces of
language tasks, and the subjective nature of feedback in language tasks.
AlphaLLM is comprised of prompt synthesis component, an efficient MCTS approach
tailored for language tasks, and a trio of critic models for precise feedback.
Our experimental results in mathematical reasoning tasks demonstrate that
AlphaLLM significantly enhances the performance of LLMs without additional
annotations, showing the potential for self-improvement in LLMs.
TriForce: Lossless Acceleration of Long Sequence Generation with
  Hierarchical Speculative Decoding
Authors: name: Hanshi Sun, name: Zhuoming Chen, name: Xinyu Yang, name: Yuandong Tian, name: Beidi Chen
Published: 2024-04-18 05:25:54

With large language models (LLMs) widely deployed in long content generation
recently, there has emerged an increasing demand for efficient long-sequence
inference support. However, key-value (KV) cache, which is stored to avoid
re-computation, has emerged as a critical bottleneck by growing linearly in
size with the sequence length. Due to the auto-regressive nature of LLMs, the
entire KV cache will be loaded for every generated token, resulting in low
utilization of computational cores and high latency. While various compression
methods for KV cache have been proposed to alleviate this issue, they suffer
from degradation in generation quality. We introduce TriForce, a hierarchical
speculative decoding system that is scalable to long sequence generation. This
approach leverages the original model weights and dynamic sparse KV cache via
retrieval as a draft model, which serves as an intermediate layer in the
hierarchy and is further speculated by a smaller model to reduce its drafting
latency. TriForce not only facilitates impressive speedups for Llama2-7B-128K,
achieving up to 2.31times on an A100 GPU but also showcases scalability in
handling even longer contexts. For the offloading setting on two RTX 4090 GPUs,
TriForce achieves 0.108s/tokenx2014only half as slow as the
auto-regressive baseline on an A100, which attains 7.78times on our
optimized offloading system. Additionally, TriForce performs 4.86times than
DeepSpeed-Zero-Inference on a single RTX 4090 GPU. TriForce's robustness is
highlighted by its consistently outstanding performance across various
temperatures. The code is available at
https://github.com/Infini-AI-Lab/TriForce.
OpenBezoar: Small, Cost-Effective and Open Models Trained on Mixes of
  Instruction Data
Authors: name: Chandeepa Dissanayake, name: Lahiru Lowe, name: Sachith Gunasekara, name: Yasiru Ratnayake
Published: 2024-04-18 13:57:18

Instruction fine-tuning pretrained LLMs for diverse downstream tasks has
demonstrated remarkable success and has captured the interest of both academics
and practitioners. To ensure such fine-tuned LLMs align with human preferences,
techniques such as RLHF and DPO have emerged. At the same time, there is
increasing interest in smaller parameter counts for models. In this work, using
OpenLLaMA 3Bv2 as a base model, we describe the recipe used to fine-tune the
OpenBezoar family of models. In this recipe: We first generate synthetic
instruction fine-tuning data using an open and commercially non-restrictive
instruction fine-tuned variant of the Falcon-40B model under three schemes
based on: LaMini-LM, WizardLM/Evol-Instruct (with databricks-dolly-15k as a
seed dataset) and Orca (with the Flan Collection as a seed dataset), then
filter these generations using GPT-4 as a human proxy. We then perform
cost-effective QLoRA-based supervised fine-tuning sequentially with each
scheme. The resulting checkpoint is further fine-tuned with a subset of the
HH-RLHF dataset to minimize distribution shift prior to using the DPO loss to
obtain the final checkpoint. Evaluation is done with the LM Eval Harness
tasks/metrics as well as on MT-Bench using the "LLM-as-a-judge" framework with
Claude 2.1, with the finding that the final checkpoint,
"OpenBezoar-HH-RLHF-DPO", demonstrates superior performance over many models at
the 3B parameter scale, even outperforming the top model in one of the
categories on the Huggingface Open LLM Leaderboard. We release
"OpenBezoar-SFT", "OpenBezoar-HH-RLHF-SFT", "OpenBezoar-HH-RLHF-DPO"
checkpoints, alongside our generated datasets on HuggingFace at
https://huggingface.co/collections/SurgeGlobal/open-bezoar-6620a24923e12127e9e2b9cc
and our codebase at
https://bitbucket.org/paladinanalytics/workspace/projects/OP.
BLINK: Multimodal Large Language Models Can See but Not Perceive
Authors: name: Xingyu Fu, name: Yushi Hu, name: Bangzheng Li, name: Yu Feng, name: Haoyu Wang, name: Xudong Lin, name: Dan Roth, name: Noah A. Smith, name: Wei-Chiu Ma, name: Ranjay Krishna
Published: 2024-04-18 17:59:54

We introduce Blink, a new benchmark for multimodal language models (LLMs)
that focuses on core visual perception abilities not found in other
evaluations. Most of the Blink tasks can be solved by humans "within a blink"
(e.g., relative depth estimation, visual correspondence, forensics detection,
and multi-view reasoning). However, we find these perception-demanding tasks
cast significant challenges for current multimodal LLMs because they resist
mediation through natural language. Blink reformats 14 classic computer vision
tasks into 3,807 multiple-choice questions, paired with single or multiple
images and visual prompting. While humans get 95.70% accuracy on average, Blink
is surprisingly challenging for existing multimodal LLMs: even the
best-performing GPT-4V and Gemini achieve accuracies of 51.26% and 45.72%, only
13.17% and 7.63% higher than random guessing, indicating that such perception
abilities have not "emerged" yet in recent multimodal LLMs. Our analysis also
highlights that specialist CV models could solve these problems much better,
suggesting potential pathways for future improvements. We believe Blink will
stimulate the community to help multimodal LLMs catch up with human-level
visual perception.
Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language
  Models
Authors: name: Aitor Ormazabal, name: Che Zheng, name: Cyprien de Masson d'Autume, name: Dani Yogatama, name: Deyu Fu, name: Donovan Ong, name: Eric Chen, name: Eugenie Lamprecht, name: Hai Pham, name: Isaac Ong, name: Kaloyan Aleksiev, name: Lei Li, name: Matthew Henderson, name: Max Bain, name: Mikel Artetxe, name: Nishant Relan, name: Piotr Padlewski, name: Qi Liu, name: Ren Chen, name: Samuel Phua, name: Yazheng Yang, name: Yi Tay, name: Yuqi Wang, name: Zhongkai Zhu, name: Zhihui Xie
Published: 2024-04-18 17:59:48

We introduce Reka Core, Flash, and Edge, a series of powerful multimodal
language models trained from scratch by Reka. Reka models are able to process
and reason with text, images, video, and audio inputs. This technical report
discusses details of training some of these models and provides comprehensive
evaluation results. We show that Reka Edge and Reka Flash are not only
state-of-the-art but also outperform many much larger models, delivering
outsized values for their respective compute class. Meanwhile, our most capable
and largest model, Reka Core, approaches the best frontier models on both
automatic evaluations and blind human evaluations. On image question answering
benchmarks (e.g. MMMU, VQAv2), Core performs competitively to GPT4-V.
Meanwhile, on multimodal chat, Core ranks as the second most preferred model
under a blind third-party human evaluation setup, outperforming other models
such as Claude 3 Opus. On text benchmarks, Core not only performs competitively
to other frontier models on a set of well-established benchmarks (e.g. MMLU,
GSM8K) but also outperforms GPT4-0613 on human evaluation. On video question
answering (Perception-Test), Core outperforms Gemini Ultra. Models are shipped
in production at http://chat.reka.ai . A showcase of non cherry picked
qualitative examples can also be found at http://showcase.reka.ai .
AniClipart: Clipart Animation with Text-to-Video Priors
Authors: name: Ronghuan Wu, name: Wanchao Su, name: Kede Ma, name: Jing Liao
Published: 2024-04-18 17:24:28

Clipart, a pre-made graphic art form, offers a convenient and efficient way
of illustrating visual content. Traditional workflows to convert static clipart
images into motion sequences are laborious and time-consuming, involving
numerous intricate steps like rigging, key animation and in-betweening. Recent
advancements in text-to-video generation hold great potential in resolving this
problem. Nevertheless, direct application of text-to-video generation models
often struggles to retain the visual identity of clipart images or generate
cartoon-style motions, resulting in unsatisfactory animation outcomes. In this
paper, we introduce AniClipart, a system that transforms static clipart images
into high-quality motion sequences guided by text-to-video priors. To generate
cartoon-style and smooth motion, we first define B\'{e}zier curves over
keypoints of the clipart image as a form of motion regularization. We then
align the motion trajectories of the keypoints with the provided text prompt by
optimizing the Video Score Distillation Sampling (VSDS) loss, which encodes
adequate knowledge of natural motion within a pretrained text-to-video
diffusion model. With a differentiable As-Rigid-As-Possible shape deformation
algorithm, our method can be end-to-end optimized while maintaining deformation
rigidity. Experimental results show that the proposed AniClipart consistently
outperforms existing image-to-video generation models, in terms of text-video
alignment, visual identity preservation, and motion consistency. Furthermore,
we showcase the versatility of AniClipart by adapting it to generate a broader
array of animation formats, such as layered animation, which allows topological
changes.
Long-form music generation with latent diffusion
Authors: name: Zach Evans, name: Julian D. Parker, name: CJ Carr, name: Zack Zukowski, name: Josiah Taylor, name: Jordi Pons
Published: 2024-04-16 06:09:33

Audio-based generative models for music have seen great strides recently, but
so far have not managed to produce full-length music tracks with coherent
musical structure. We show that by training a generative model on long temporal
contexts it is possible to produce long-form music of up to 4m45s. Our model
consists of a diffusion-transformer operating on a highly downsampled
continuous latent representation (latent rate of 21.5Hz). It obtains
state-of-the-art generations according to metrics on audio quality and prompt
alignment, and subjective tests reveal that it produces full-length music with
coherent structure.
TextHawk: Exploring Efficient Fine-Grained Perception of Multimodal
  Large Language Models
Authors: name: Ya-Qi Yu, name: Minghui Liao, name: Jihao Wu, name: Yongxin Liao, name: Xiaoyu Zheng, name: Wei Zeng
Published: 2024-04-14 09:48:37

Multimodal Large Language Models (MLLMs) have shown impressive results on
various multimodal tasks. However, most existing MLLMs are not well suited for
document-oriented tasks, which require fine-grained image perception and
information compression. In this paper, we present TextHawk, a MLLM that is
specifically designed for document-oriented tasks, while preserving the general
capabilities of MLLMs. TextHawk is aimed to explore efficient fine-grained
perception by designing four dedicated components. Firstly, a ReSampling and
ReArrangement (ReSA) module is proposed to reduce the redundancy in the
document texts and lower the computational cost of the MLLM. We explore
encoding the positions of each local feature by presenting Scalable Positional
Embeddings (SPEs), which can preserve the scalability of various image sizes. A
Query Proposal Network (QPN) is then adopted to initialize the queries
dynamically among different sub-images. To further enhance the fine-grained
visual perceptual ability of the MLLM, we design a Multi-Level Cross-Attention
(MLCA) mechanism that captures the hierarchical structure and semantic
relations of document images. Furthermore, we create a new instruction-tuning
dataset for document-oriented tasks by enriching the multimodal document data
with Gemini Pro. We conduct extensive experiments on both general and
document-oriented MLLM benchmarks, and show that TextHawk outperforms the
state-of-the-art methods, demonstrating its effectiveness and superiority in
fine-grained document perception and general abilities.