Orionnnn/hub-stats huggingface dataset:
Uni-SMART: Universal Science Multimodal Analysis and Research
  Transformer
Authors: name: Hengxing Cai, name: Xiaochen Cai, name: Shuwen Yang, name: Jiankun Wang, name: Lin Yao, name: Zhifeng Gao, name: Junhan Chang, name: Sihang Li, name: Mingjun Xu, name: Changxin Wang, name: Hongshuai Wang, name: Yongge Li, name: Mujie Lin, name: Yaqi Li, name: Yuqi Yin, name: Linfeng Zhang, name: Guolin Ke
Published: 2024-03-15 13:43:47

In scientific research and its application, scientific literature analysis is
crucial as it allows researchers to build on the work of others. However, the
fast growth of scientific knowledge has led to a massive increase in scholarly
articles, making in-depth literature analysis increasingly challenging and
time-consuming. The emergence of Large Language Models (LLMs) has offered a new
way to address this challenge. Known for their strong abilities in summarizing
texts, LLMs are seen as a potential tool to improve the analysis of scientific
literature. However, existing LLMs have their own limits. Scientific literature
often includes a wide range of multimodal elements, such as molecular
structure, tables, and charts, which are hard for text-focused LLMs to
understand and analyze. This issue points to the urgent need for new solutions
that can fully understand and analyze multimodal content in scientific
literature. To answer this demand, we present Uni-SMART (Universal Science
Multimodal Analysis and Research Transformer), an innovative model designed for
in-depth understanding of multimodal scientific literature. Through rigorous
quantitative evaluation across several domains, Uni-SMART demonstrates superior
performance over leading text-focused LLMs. Furthermore, our exploration
extends to practical applications, including patent infringement detection and
nuanced analysis of charts. These applications not only highlight Uni-SMART's
adaptability but also its potential to revolutionize how we interact with
scientific literature.
RAFT: Adapting Language Model to Domain Specific RAG
Authors: name: Tianjun Zhang, name: Shishir G. Patil, name: Naman Jain, name: Sheng Shen, name: Matei Zaharia, name: Ion Stoica, name: Joseph E. Gonzalez
Published: 2024-03-15 09:26:02

Pretraining Large Language Models (LLMs) on large corpora of textual data is
now a standard paradigm. When using these LLMs for many downstream
applications, it is common to additionally bake in new knowledge (e.g.,
time-critical news, or private domain knowledge) into the pretrained model
either through RAG-based-prompting, or fine-tuning. However, the optimal
methodology for the model to gain such new knowledge remains an open question.
In this paper, we present Retrieval Augmented FineTuning (RAFT), a training
recipe that improves the model's ability to answer questions in a "open-book"
in-domain settings. In RAFT, given a question, and a set of retrieved
documents, we train the model to ignore those documents that don't help in
answering the question, which we call, distractor documents. RAFT accomplishes
this by citing verbatim the right sequence from the relevant document that
would help answer the question. This coupled with RAFT's chain-of-thought-style
response helps improve the model's ability to reason. In domain-specific RAG,
RAFT consistently improves the model's performance across PubMed, HotpotQA, and
Gorilla datasets, presenting a post-training recipe to improve pre-trained LLMs
to in-domain RAG. RAFT's code and demo are open-sourced at
github.com/ShishirPatil/gorilla.
EfficientVMamba: Atrous Selective Scan for Light Weight Visual Mamba
Authors: name: Xiaohuan Pei, name: Tao Huang, name: Chang Xu
Published: 2024-03-15 02:48:47

Prior efforts in light-weight model development mainly centered on CNN and
Transformer-based designs yet faced persistent challenges. CNNs adept at local
feature extraction compromise resolution while Transformers offer global reach
but escalate computational demands O(N^2). This ongoing trade-off
between accuracy and efficiency remains a significant hurdle. Recently, state
space models (SSMs), such as Mamba, have shown outstanding performance and
competitiveness in various tasks such as language modeling and computer vision,
while reducing the time complexity of global information extraction to
O(N). Inspired by this, this work proposes to explore the potential
of visual state space models in light-weight model design and introduce a novel
efficient model variant dubbed EfficientVMamba. Concretely, our EfficientVMamba
integrates a atrous-based selective scan approach by efficient skip sampling,
constituting building blocks designed to harness both global and local
representational features. Additionally, we investigate the integration between
SSM blocks and convolutions, and introduce an efficient visual state space
block combined with an additional convolution branch, which further elevate the
model performance. Experimental results show that, EfficientVMamba scales down
the computational complexity while yields competitive results across a variety
of vision tasks. For example, our EfficientVMamba-S with 1.3G FLOPs improves
Vim-Ti with 1.5G FLOPs by a large margin of 5.6% accuracy on ImageNet.
Code is available at: https://github.com/TerryPei/EfficientVMamba.
Recurrent Drafter for Fast Speculative Decoding in Large Language Models
Authors: name: Aonan Zhang, name: Chong Wang, name: Yi Wang, name: Xuanyu Zhang, name: Yunfei Cheng
Published: 2024-03-14 23:40:56

In this paper, we introduce an improved approach of speculative decoding
aimed at enhancing the efficiency of serving large language models. Our method
capitalizes on the strengths of two established techniques: the classic
two-model speculative decoding approach, and the more recent single-model
approach, Medusa. Drawing inspiration from Medusa, our approach adopts a
single-model strategy for speculative decoding. However, our method
distinguishes itself by employing a single, lightweight draft head with a
recurrent dependency design, akin in essence to the small, draft model uses in
classic speculative decoding, but without the complexities of the full
transformer architecture. And because of the recurrent dependency, we can use
beam search to swiftly filter out undesired candidates with the draft head. The
outcome is a method that combines the simplicity of single-model design and
avoids the need to create a data-dependent tree attention structure only for
inference in Medusa. We empirically demonstrate the effectiveness of the
proposed method on several popular open source language models, along with a
comprehensive analysis of the trade-offs involved in adopting this approach.
Alignment Studio: Aligning Large Language Models to Particular
  Contextual Regulations
Authors: name: Swapnaja Achintalwar, name: Ioana Baldini, name: Djallel Bouneffouf, name: Joan Byamugisha, name: Maria Chang, name: Pierre Dognin, name: Eitan Farchi, name: Ndivhuwo Makondo, name: Aleksandra Mojsilovic, name: Manish Nagireddy, name: Karthikeyan Natesan Ramamurthy, name: Inkit Padhi, name: Orna Raz, name: Jesus Rios, name: Prasanna Sattigeri, name: Moninder Singh, name: Siphiwe Thwala, name: Rosario A. Uceda-Sosa, name: Kush R. Varshney
Published: 2024-03-08 21:26:49

The alignment of large language models is usually done by model providers to
add or control behaviors that are common or universally understood across use
cases and contexts. In contrast, in this article, we present an approach and
architecture that empowers application developers to tune a model to their
particular values, social norms, laws and other regulations, and orchestrate
between potentially conflicting requirements in context. We lay out three main
components of such an Alignment Studio architecture: Framers, Instructors, and
Auditors that work in concert to control the behavior of a language model. We
illustrate this approach with a running example of aligning a company's
internal-facing enterprise chatbot to its business conduct guidelines.
MusicHiFi: Fast High-Fidelity Stereo Vocoding
Authors: name: Ge Zhu, name: Juan-Pablo Caceres, name: Zhiyao Duan, name: Nicholas J. Bryan
Published: 2024-03-15 17:27:42

Diffusion-based audio and music generation models commonly generate music by
constructing an image representation of audio (e.g., a mel-spectrogram) and
then converting it to audio using a phase reconstruction model or vocoder.
Typical vocoders, however, produce monophonic audio at lower resolutions (e.g.,
16-24 kHz), which limits their effectiveness. We propose MusicHiFi -- an
efficient high-fidelity stereophonic vocoder. Our method employs a cascade of
three generative adversarial networks (GANs) that convert low-resolution
mel-spectrograms to audio, upsamples to high-resolution audio via bandwidth
expansion, and upmixes to stereophonic audio. Compared to previous work, we
propose 1) a unified GAN-based generator and discriminator architecture and
training procedure for each stage of our cascade, 2) a new fast, near
downsampling-compatible bandwidth extension module, and 3) a new fast
downmix-compatible mono-to-stereo upmixer that ensures the preservation of
monophonic content in the output. We evaluate our approach using both objective
and subjective listening tests and find our approach yields comparable or
better audio quality, better spatialization control, and significantly faster
inference speed compared to past work. Sound examples are at
https://MusicHiFi.github.io/web/.
Isotropic3D: Image-to-3D Generation Based on a Single CLIP Embedding
Authors: name: Pengkun Liu, name: Yikai Wang, name: Fuchun Sun, name: Jiafang Li, name: Hang Xiao, name: Hongxiang Xue, name: Xinzhou Wang
Published: 2024-03-15 15:27:58

Encouraged by the growing availability of pre-trained 2D diffusion models,
image-to-3D generation by leveraging Score Distillation Sampling (SDS) is
making remarkable progress. Most existing methods combine novel-view lifting
from 2D diffusion models which usually take the reference image as a condition
while applying hard L2 image supervision at the reference view. Yet heavily
adhering to the image is prone to corrupting the inductive knowledge of the 2D
diffusion model leading to flat or distorted 3D generation frequently. In this
work, we reexamine image-to-3D in a novel perspective and present Isotropic3D,
an image-to-3D generation pipeline that takes only an image CLIP embedding as
input. Isotropic3D allows the optimization to be isotropic w.r.t. the azimuth
angle by solely resting on the SDS loss. The core of our framework lies in a
two-stage diffusion model fine-tuning. Firstly, we fine-tune a text-to-3D
diffusion model by substituting its text encoder with an image encoder, by
which the model preliminarily acquires image-to-image capabilities. Secondly,
we perform fine-tuning using our Explicit Multi-view Attention (EMA) which
combines noisy multi-view images with the noise-free reference image as an
explicit condition. CLIP embedding is sent to the diffusion model throughout
the whole process while reference images are discarded once after fine-tuning.
As a result, with a single image CLIP embedding, Isotropic3D is capable of
generating multi-view mutually consistent images and also a 3D model with more
symmetrical and neat content, well-proportioned geometry, rich colored texture,
and less distortion compared with existing image-to-3D methods while still
preserving the similarity to the reference image to a large extent. The project
page is available at https://isotropic3d.github.io/. The code and models are
available at https://github.com/pkunliu/Isotropic3D.
Controllable Text-to-3D Generation via Surface-Aligned Gaussian
  Splatting
Authors: name: Zhiqi Li, name: Yiming Chen, name: Lingzhe Zhao, name: Peidong Liu
Published: 2024-03-15 02:57:20

While text-to-3D and image-to-3D generation tasks have received considerable
attention, one important but under-explored field between them is controllable
text-to-3D generation, which we mainly focus on in this work. To address this
task, 1) we introduce Multi-view ControlNet (MVControl), a novel neural network
architecture designed to enhance existing pre-trained multi-view diffusion
models by integrating additional input conditions, such as edge, depth, normal,
and scribble maps. Our innovation lies in the introduction of a conditioning
module that controls the base diffusion model using both local and global
embeddings, which are computed from the input condition images and camera
poses. Once trained, MVControl is able to offer 3D diffusion guidance for
optimization-based 3D generation. And, 2) we propose an efficient multi-stage
3D generation pipeline that leverages the benefits of recent large
reconstruction models and score distillation algorithm. Building upon our
MVControl architecture, we employ a unique hybrid diffusion guidance method to
direct the optimization process. In pursuit of efficiency, we adopt 3D
Gaussians as our representation instead of the commonly used implicit
representations. We also pioneer the use of SuGaR, a hybrid representation that
binds Gaussians to mesh triangle faces. This approach alleviates the issue of
poor geometry in 3D Gaussians and enables the direct sculpting of fine-grained
geometry on the mesh. Extensive experiments demonstrate that our method
achieves robust generalization and enables the controllable generation of
high-quality 3D content.
StreamMultiDiffusion: Real-Time Interactive Generation with Region-Based
  Semantic Control
Authors: name: Jaerin Lee, name: Daniel Sungho Jung, name: Kanggeon Lee, name: Kyoung Mu Lee
Published: 2024-03-14 02:51:01

The enormous success of diffusion models in text-to-image synthesis has made
them promising candidates for the next generation of end-user applications for
image generation and editing. Previous works have focused on improving the
usability of diffusion models by reducing the inference time or increasing user
interactivity by allowing new, fine-grained controls such as region-based text
prompts. However, we empirically find that integrating both branches of works
is nontrivial, limiting the potential of diffusion models. To solve this
incompatibility, we present StreamMultiDiffusion, the first real-time
region-based text-to-image generation framework. By stabilizing fast inference
techniques and restructuring the model into a newly proposed multi-prompt
stream batch architecture, we achieve times 10 faster panorama generation
than existing solutions, and the generation speed of 1.57 FPS in region-based
text-to-image synthesis on a single RTX 2080 Ti GPU. Our solution opens up a
new paradigm for interactive image generation named semantic palette, where
high-quality images are generated in real-time from given multiple hand-drawn
regions, encoding prescribed semantic meanings (e.g., eagle, girl). Our code
and demo application are available at
https://github.com/ironjr/StreamMultiDiffusion.
3D-VLA: A 3D Vision-Language-Action Generative World Model
Authors: name: Haoyu Zhen, name: Xiaowen Qiu, name: Peihao Chen, name: Jincheng Yang, name: Xin Yan, name: Yilun Du, name: Yining Hong, name: Chuang Gan
Published: 2024-03-14 17:58:41

Recent vision-language-action (VLA) models rely on 2D inputs, lacking
integration with the broader realm of the 3D physical world. Furthermore, they
perform action prediction by learning a direct mapping from perception to
action, neglecting the vast dynamics of the world and the relations between
actions and dynamics. In contrast, human beings are endowed with world models
that depict imagination about future scenarios to plan actions accordingly. To
this end, we propose 3D-VLA by introducing a new family of embodied foundation
models that seamlessly link 3D perception, reasoning, and action through a
generative world model. Specifically, 3D-VLA is built on top of a 3D-based
large language model (LLM), and a set of interaction tokens is introduced to
engage with the embodied environment. Furthermore, to inject generation
abilities into the model, we train a series of embodied diffusion models and
align them into the LLM for predicting the goal images and point clouds. To
train our 3D-VLA, we curate a large-scale 3D embodied instruction dataset by
extracting vast 3D-related information from existing robotics datasets. Our
experiments on held-in datasets demonstrate that 3D-VLA significantly improves
the reasoning, multimodal generation, and planning capabilities in embodied
environments, showcasing its potential in real-world applications.
Quiet-STaR: Language Models Can Teach Themselves to Think Before
  Speaking
Authors: name: Eric Zelikman, name: Georges Harik, name: Yijia Shao, name: Varuna Jayasiri, name: Nick Haber, name: Noah D. Goodman
Published: 2024-03-14 17:58:16

When writing and talking, people sometimes pause to think. Although
reasoning-focused works have often framed reasoning as a method of answering
questions or completing agentic tasks, reasoning is implicit in almost all
written text. For example, this applies to the steps not stated between the
lines of a proof or to the theory of mind underlying a conversation. In the
Self-Taught Reasoner (STaR, Zelikman et al. 2022), useful thinking is learned
by inferring rationales from few-shot examples in question-answering and
learning from those that lead to a correct answer. This is a highly constrained
setting -- ideally, a language model could instead learn to infer unstated
rationales in arbitrary text. We present Quiet-STaR, a generalization of STaR
in which LMs learn to generate rationales at each token to explain future text,
improving their predictions. We address key challenges, including 1) the
computational cost of generating continuations, 2) the fact that the LM does
not initially know how to generate or use internal thoughts, and 3) the need to
predict beyond individual next tokens. To resolve these, we propose a tokenwise
parallel sampling algorithm, using learnable tokens indicating a thought's
start and end, and an extended teacher-forcing technique. Encouragingly,
generated rationales disproportionately help model difficult-to-predict tokens
and improve the LM's ability to directly answer difficult questions. In
particular, after continued pretraining of an LM on a corpus of internet text
with Quiet-STaR, we find zero-shot improvements on GSM8K
(5.9%rightarrow10.9%) and CommonsenseQA (36.3%rightarrow47.2%) and
observe a perplexity improvement of difficult tokens in natural text.
Crucially, these improvements require no fine-tuning on these tasks. Quiet-STaR
marks a step towards LMs that can learn to reason in a more general and
scalable way.
GiT: Towards Generalist Vision Transformer through Universal Language
  Interface
Authors: name: Haiyang Wang, name: Hao Tang, name: Li Jiang, name: Shaoshuai Shi, name: Muhammad Ferjad Naeem, name: Hongsheng Li, name: Bernt Schiele, name: Liwei Wang
Published: 2024-03-14 13:47:41

This paper proposes a simple, yet effective framework, called GiT,
simultaneously applicable for various vision tasks only with a vanilla ViT.
Motivated by the universality of the Multi-layer Transformer architecture (e.g,
GPT) widely used in large language models (LLMs), we seek to broaden its scope
to serve as a powerful vision foundation model (VFM). However, unlike language
modeling, visual tasks typically require specific modules, such as bounding box
heads for detection and pixel decoders for segmentation, greatly hindering the
application of powerful multi-layer transformers in the vision domain. To solve
this, we design a universal language interface that empowers the successful
auto-regressive decoding to adeptly unify various visual tasks, from
image-level understanding (e.g., captioning), over sparse perception (e.g.,
detection), to dense prediction (e.g., segmentation). Based on the above
designs, the entire model is composed solely of a ViT, without any specific
additions, offering a remarkable architectural simplification. GiT is a
multi-task visual model, jointly trained across five representative benchmarks
without task-specific fine-tuning. Interestingly, our GiT builds a new
benchmark in generalist performance, and fosters mutual enhancement across
tasks, leading to significant improvements compared to isolated training. This
reflects a similar impact observed in LLMs. Further enriching training with 27
datasets, GiT achieves strong zero-shot results over various tasks. Due to its
simple design, this paradigm holds promise for narrowing the architectural gap
between vision and language. Code and models will be available at
https://github.com/Haiyang-W/GiT.
BurstAttention: An Efficient Distributed Attention Framework for
  Extremely Long Sequences
Authors: name: Sun Ao, name: Weilin Zhao, name: Xu Han, name: Cheng Yang, name: Zhiyuan Liu, name: Chuan Shi, name: Maosong Sun, name: Shengnan Wang, name: Teng Su
Published: 2024-03-14 12:51:58

Effective attention modules have played a crucial role in the success of
Transformer-based large language models (LLMs), but the quadratic time and
memory complexities of these attention modules also pose a challenge when
processing long sequences. One potential solution for the long sequence problem
is to utilize distributed clusters to parallelize the computation of attention
modules across multiple devices (e.g., GPUs). However, adopting a distributed
approach inevitably introduces extra memory overheads to store local attention
results and incurs additional communication costs to aggregate local results
into global ones. In this paper, we propose a distributed attention framework
named ``BurstAttention'' to optimize memory access and communication operations
at both the global cluster and local device levels. In our experiments, we
compare BurstAttention with other competitive distributed attention solutions
for long sequence processing. The experimental results under different length
settings demonstrate that BurstAttention offers significant advantages for
processing long sequences compared with these competitive baselines, reducing
40% communication overheads and achieving 2 X speedup during training 32K
sequence length on 8 X A100.
Griffon v2: Advancing Multimodal Perception with High-Resolution Scaling
  and Visual-Language Co-Referring
Authors: name: Yufei Zhan, name: Yousong Zhu, name: Hongyin Zhao, name: Fan Yang, name: Ming Tang, name: Jinqiao Wang
Published: 2024-03-14 12:21:37

Large Vision Language Models have achieved fine-grained object perception,
but the limitation of image resolution remains a significant obstacle to
surpass the performance of task-specific experts in complex and dense
scenarios. Such limitation further restricts the model's potential to achieve
nuanced visual and language referring in domains such as GUI Agents, Counting
and \etc. To address this issue, we introduce a unified high-resolution
generalist model, Griffon v2, enabling flexible object referring with visual
and textual prompts. To efficiently scaling up image resolution, we design a
simple and lightweight down-sampling projector to overcome the input tokens
constraint in Large Language Models. This design inherently preserves the
complete contexts and fine details, and significantly improves multimodal
perception ability especially for small objects. Building upon this, we further
equip the model with visual-language co-referring capabilities through a
plug-and-play visual tokenizer. It enables user-friendly interaction with
flexible target images, free-form texts and even coordinates. Experiments
demonstrate that Griffon v2 can localize any objects of interest with visual
and textual referring, achieve state-of-the-art performance on REC, phrase
grounding, and REG tasks, and outperform expert models in object detection and
object counting. Data, codes and models will be released at
https://github.com/jefferyZhan/Griffon.