Orionnnn/hub-stats huggingface dataset:
VisFocus: Prompt-Guided Vision Encoders for OCR-Free Dense Document
  Understanding
Authors: name: Ofir Abramovich, name: Niv Nayman, name: Sharon Fogel, name: Inbal Lavi, name: Ron Litman, name: Shahar Tsiper, name: Royee Tichauer, name: Srikar Appalaraju, name: Shai Mazor, name: R. Manmatha
Published: 2024-07-17 14:16:46

In recent years, notable advancements have been made in the domain of visual
document understanding, with the prevailing architecture comprising a cascade
of vision and language models. The text component can either be extracted
explicitly with the use of external OCR models in OCR-based approaches, or
alternatively, the vision model can be endowed with reading capabilities in
OCR-free approaches. Typically, the queries to the model are input exclusively
to the language component, necessitating the visual features to encompass the
entire document. In this paper, we present VisFocus, an OCR-free method
designed to better exploit the vision encoder's capacity by coupling it
directly with the language prompt. To do so, we replace the down-sampling
layers with layers that receive the input prompt and allow highlighting
relevant parts of the document, while disregarding others. We pair the
architecture enhancements with a novel pre-training task, using language
masking on a snippet of the document text fed to the visual encoder in place of
the prompt, to empower the model with focusing capabilities. Consequently,
VisFocus learns to allocate its attention to text patches pertinent to the
provided prompt. Our experiments demonstrate that this prompt-guided visual
encoding approach significantly improves performance, achieving
state-of-the-art results on various benchmarks.
SciCode: A Research Coding Benchmark Curated by Scientists
Authors: name: Minyang Tian, name: Luyu Gao, name: Shizhuo Dylan Zhang, name: Xinan Chen, name: Cunwei Fan, name: Xuefei Guo, name: Roland Haas, name: Pan Ji, name: Kittithat Krongchon, name: Yao Li, name: Shengyan Liu, name: Di Luo, name: Yutao Ma, name: Hao Tong, name: Kha Trinh, name: Chenyu Tian, name: Zihan Wang, name: Bohao Wu, name: Yanyu Xiong, name: Shengzhu Yin, name: Minhui Zhu, name: Kilian Lieret, name: Yanxin Lu, name: Genglin Liu, name: Yufeng Du, name: Tianhua Tao, name: Ofir Press, name: Jamie Callan, name: Eliu Huerta, name: Hao Peng
Published: 2024-07-18 05:15:24

Since language models (LMs) now outperform average humans on many challenging
tasks, it has become increasingly difficult to develop challenging,
high-quality, and realistic evaluations. We address this issue by examining
LMs' capabilities to generate code for solving real scientific research
problems. Incorporating input from scientists and AI researchers in 16 diverse
natural science sub-fields, including mathematics, physics, chemistry, biology,
and materials science, we created a scientist-curated coding benchmark,
SciCode. The problems in SciCode naturally factorize into multiple subproblems,
each involving knowledge recall, reasoning, and code synthesis. In total,
SciCode contains 338 subproblems decomposed from 80 challenging main problems.
It offers optional descriptions specifying useful scientific background
information and scientist-annotated gold-standard solutions and test cases for
evaluation. Claude3.5-Sonnet, the best-performing model among those tested, can
solve only 4.6% of the problems in the most realistic setting. We believe that
SciCode demonstrates both contemporary LMs' progress towards becoming helpful
scientific assistants and sheds light on the development and evaluation of
scientific AI in the future.
Fast Matrix Multiplications for Lookup Table-Quantized LLMs
Authors: name: Han Guo, name: William Brandon, name: Radostin Cholakov, name: Jonathan Ragan-Kelley, name: Eric P. Xing, name: Yoon Kim
Published: 2024-07-15 17:55:42

The deployment of large language models (LLMs) is often constrained by memory
bandwidth, where the primary bottleneck is the cost of transferring model
parameters from the GPU's global memory to its registers. When coupled with
custom kernels that fuse the dequantization and matmul operations, weight-only
quantization can thus enable faster inference by reducing the amount of memory
movement. However, developing high-performance kernels for weight-quantized
LLMs presents substantial challenges, especially when the weights are
compressed to non-evenly-divisible bit widths (e.g., 3 bits) with non-uniform,
lookup table (LUT) quantization. This paper describes FLUTE, a flexible lookup
table engine for LUT-quantized LLMs, which uses offline restructuring of the
quantized weight matrix to minimize bit manipulations associated with
unpacking, and vectorization and duplication of the lookup table to mitigate
shared memory bandwidth constraints. At batch sizes < 32 and quantization group
size of 128 (typical in LLM inference), the FLUTE kernel can be 2-4x faster
than existing GEMM kernels. As an application of FLUTE, we explore a simple
extension to lookup table-based NormalFloat quantization and apply it to
quantize LLaMA3 to various configurations, obtaining competitive quantization
performance against strong baselines while obtaining an end-to-end throughput
increase of 1.5 to 2 times.
ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG
  Capabilities
Authors: name: Peng Xu, name: Wei Ping, name: Xianchao Wu, name: Zihan Liu, name: Mohammad Shoeybi, name: Bryan Catanzaro
Published: 2024-07-19 17:35:47

In this work, we introduce ChatQA 2, a Llama3-based model designed to bridge
the gap between open-access LLMs and leading proprietary models (e.g.,
GPT-4-Turbo) in long-context understanding and retrieval-augmented generation
(RAG) capabilities. These two capabilities are essential for LLMs to process
large volumes of information that cannot fit into a single prompt and are
complementary to each other, depending on the downstream tasks and
computational budgets. We present a detailed continued training recipe to
extend the context window of Llama3-70B-base from 8K to 128K tokens, along with
a three-stage instruction tuning process to enhance the model's
instruction-following, RAG performance, and long-context understanding
capabilities. Our results demonstrate that the Llama3-ChatQA-2-70B model
achieves accuracy comparable to GPT-4-Turbo-2024-0409 on many long-context
understanding tasks and surpasses it on the RAG benchmark. Interestingly, we
find that the state-of-the-art long-context retriever can alleviate the top-k
context fragmentation issue in RAG, further improving RAG-based results for
long-context understanding tasks. We also provide extensive comparisons between
RAG and long-context solutions using state-of-the-art long-context LLMs.
EVLM: An Efficient Vision-Language Model for Visual Understanding
Authors: name: Kaibing Chen, name: Dong Shen, name: Hanwen Zhong, name: Huasong Zhong, name: Kui Xia, name: Di Xu, name: Wei Yuan, name: Yifei Hu, name: Bin Wen, name: Tianke Zhang, name: Changyi Liu, name: Dewen Fan, name: Huihui Xiao, name: Jiahong Wu, name: Fan Yang, name: Size Li, name: Di Zhang
Published: 2024-07-19 10:09:51

In the field of multi-modal language models, the majority of methods are
built on an architecture similar to LLaVA. These models use a single-layer ViT
feature as a visual prompt, directly feeding it into the language models
alongside textual tokens. However, when dealing with long sequences of visual
signals or inputs such as videos, the self-attention mechanism of language
models can lead to significant computational overhead. Additionally, using
single-layer ViT features makes it challenging for large language models to
perceive visual signals fully. This paper proposes an efficient multi-modal
language model to minimize computational costs while enabling the model to
perceive visual signals as comprehensively as possible. Our method primarily
includes: (1) employing cross-attention to image-text interaction similar to
Flamingo. (2) utilize hierarchical ViT features. (3) introduce the Mixture of
Experts (MoE) mechanism to enhance model effectiveness. Our model achieves
competitive scores on public multi-modal benchmarks and performs well in tasks
such as image captioning and video captioning.
Visual Text Generation in the Wild
Authors: name: Yuanzhi Zhu, name: Jiawei Liu, name: Feiyu Gao, name: Wenyu Liu, name: Xinggang Wang, name: Peng Wang, name: Fei Huang, name: Cong Yao, name: Zhibo Yang
Published: 2024-07-19 09:08:20

Recently, with the rapid advancements of generative models, the field of
visual text generation has witnessed significant progress. However, it is still
challenging to render high-quality text images in real-world scenarios, as
three critical criteria should be satisfied: (1) Fidelity: the generated text
images should be photo-realistic and the contents are expected to be the same
as specified in the given conditions; (2) Reasonability: the regions and
contents of the generated text should cohere with the scene; (3) Utility: the
generated text images can facilitate related tasks (e.g., text detection and
recognition). Upon investigation, we find that existing methods, either
rendering-based or diffusion-based, can hardly meet all these aspects
simultaneously, limiting their application range. Therefore, we propose in this
paper a visual text generator (termed SceneVTG), which can produce high-quality
text images in the wild. Following a two-stage paradigm, SceneVTG leverages a
Multimodal Large Language Model to recommend reasonable text regions and
contents across multiple scales and levels, which are used by a conditional
diffusion model as conditions to generate text images. Extensive experiments
demonstrate that the proposed SceneVTG significantly outperforms traditional
rendering-based methods and recent diffusion-based methods in terms of fidelity
and reasonability. Besides, the generated images provide superior utility for
tasks involving text detection and text recognition. Code and datasets are
available at AdvancedLiterateMachinery.
LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inference
Authors: name: Qichen Fu, name: Minsik Cho, name: Thomas Merth, name: Sachin Mehta, name: Mohammad Rastegari, name: Mahyar Najibi
Published: 2024-07-19 06:34:45

The inference of transformer-based large language models consists of two
sequential stages: 1) a prefilling stage to compute the KV cache of prompts and
generate the first token, and 2) a decoding stage to generate subsequent
tokens. For long prompts, the KV cache must be computed for all tokens during
the prefilling stage, which can significantly increase the time needed to
generate the first token. Consequently, the prefilling stage may become a
bottleneck in the generation process. An open question remains whether all
prompt tokens are essential for generating the first token. To answer this, we
introduce a novel method, LazyLLM, that selectively computes the KV for tokens
important for the next token prediction in both the prefilling and decoding
stages. Contrary to static pruning approaches that prune the prompt at once,
LazyLLM allows language models to dynamically select different subsets of
tokens from the context in different generation steps, even though they might
be pruned in previous steps. Extensive experiments on standard datasets across
various tasks demonstrate that LazyLLM is a generic method that can be
seamlessly integrated with existing language models to significantly accelerate
the generation without fine-tuning. For instance, in the multi-document
question-answering task, LazyLLM accelerates the prefilling stage of the LLama
2 7B model by 2.34x while maintaining accuracy.
Phi-3 Safety Post-Training: Aligning Language Models with a "Break-Fix"
  Cycle
Authors: name: Emman Haider, name: Daniel Perez-Becker, name: Thomas Portet, name: Piyush Madan, name: Amit Garg, name: David Majercak, name: Wen Wen, name: Dongwoo Kim, name: Ziyi Yang, name: Jianwen Zhang, name: Hiteshi Sharma, name: Blake Bullwinkel, name: Martin Pouliot, name: Amanda Minnich, name: Shiven Chawla, name: Solianna Herrera, name: Shahed Warreth, name: Maggie Engler, name: Gary Lopez, name: Nina Chikanov, name: Raja Sekhar Rao Dheekonda, name: Bolor-Erdene Jagdagdorj, name: Roman Lutz, name: Richard Lundeen, name: Tori Westerhoff, name: Pete Bryan, name: Christian Seifert, name: Ram Shankar Siva Kumar, name: Andrew Berkley, name: Alex Kessler
Published: 2024-07-18 18:06:59

Recent innovations in language model training have demonstrated that it is
possible to create highly performant models that are small enough to run on a
smartphone. As these models are deployed in an increasing number of domains, it
is critical to ensure that they are aligned with human preferences and safety
considerations. In this report, we present our methodology for safety aligning
the Phi-3 series of language models. We utilized a "break-fix" cycle,
performing multiple rounds of dataset curation, safety post-training,
benchmarking, red teaming, and vulnerability identification to cover a variety
of harm areas in both single and multi-turn scenarios. Our results indicate
that this approach iteratively improved the performance of the Phi-3 models
across a wide range of responsible AI benchmarks.
Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse
  Autoencoders
Authors: name: Senthooran Rajamanoharan, name: Tom Lieberum, name: Nicolas Sonnerat, name: Arthur Conmy, name: Vikrant Varma, name: János Kramár, name: Neel Nanda
Published: 2024-07-19 16:07:19

Sparse autoencoders (SAEs) are a promising unsupervised approach for
identifying causally relevant and interpretable linear features in a language
model's (LM) activations. To be useful for downstream tasks, SAEs need to
decompose LM activations faithfully; yet to be interpretable the decomposition
must be sparse -- two objectives that are in tension. In this paper, we
introduce JumpReLU SAEs, which achieve state-of-the-art reconstruction fidelity
at a given sparsity level on Gemma 2 9B activations, compared to other recent
advances such as Gated and TopK SAEs. We also show that this improvement does
not come at the cost of interpretability through manual and automated
interpretability studies. JumpReLU SAEs are a simple modification of vanilla
(ReLU) SAEs -- where we replace the ReLU with a discontinuous JumpReLU
activation function -- and are similarly efficient to train and run. By
utilising straight-through-estimators (STEs) in a principled manner, we show
how it is possible to train JumpReLU SAEs effectively despite the discontinuous
JumpReLU function introduced in the SAE's forward pass. Similarly, we use STEs
to directly train L0 to be sparse, instead of training on proxies such as L1,
avoiding problems like shrinkage.
The Vision of Autonomic Computing: Can LLMs Make It a Reality?
Authors: name: Zhiyang Zhang, name: Fangkai Yang, name: Xiaoting Qin, name: Jue Zhang, name: Qingwei Lin, name: Gong Cheng, name: Dongmei Zhang, name: Saravan Rajmohan, name: Qi Zhang
Published: 2024-07-19 15:30:32

The Vision of Autonomic Computing (ACV), proposed over two decades ago,
envisions computing systems that self-manage akin to biological organisms,
adapting seamlessly to changing environments. Despite decades of research,
achieving ACV remains challenging due to the dynamic and complex nature of
modern computing systems. Recent advancements in Large Language Models (LLMs)
offer promising solutions to these challenges by leveraging their extensive
knowledge, language understanding, and task automation capabilities. This paper
explores the feasibility of realizing ACV through an LLM-based multi-agent
framework for microservice management. We introduce a five-level taxonomy for
autonomous service maintenance and present an online evaluation benchmark based
on the Sock Shop microservice demo project to assess our framework's
performance. Our findings demonstrate significant progress towards achieving
Level 3 autonomy, highlighting the effectiveness of LLMs in detecting and
resolving issues within microservice architectures. This study contributes to
advancing autonomic computing by pioneering the integration of LLMs into
microservice management frameworks, paving the way for more adaptive and
self-managing computing systems. The code will be made available at
https://aka.ms/ACV-LLM.
Stable Audio Open
Authors: name: Zach Evans, name: Julian D. Parker, name: CJ Carr, name: Zack Zukowski, name: Josiah Taylor, name: Jordi Pons
Published: 2024-07-19 14:40:23

Open generative models are vitally important for the community, allowing for
fine-tunes and serving as baselines when presenting new models. However, most
current text-to-audio models are private and not accessible for artists and
researchers to build upon. Here we describe the architecture and training
process of a new open-weights text-to-audio model trained with Creative Commons
data. Our evaluation shows that the model's performance is competitive with the
state-of-the-art across various metrics. Notably, the reported FDopenl3 results
(measuring the realism of the generations) showcase its potential for
high-quality stereo sound synthesis at 44.1kHz.
Efficient Audio Captioning with Encoder-Level Knowledge Distillation
Authors: name: Xuenan Xu, name: Haohe Liu, name: Mengyue Wu, name: Wenwu Wang, name: Mark D. Plumbley
Published: 2024-07-19 14:09:50

Significant improvement has been achieved in automated audio captioning (AAC)
with recent models. However, these models have become increasingly large as
their performance is enhanced. In this work, we propose a knowledge
distillation (KD) framework for AAC. Our analysis shows that in the
encoder-decoder based AAC models, it is more effective to distill knowledge
into the encoder as compared with the decoder. To this end, we incorporate
encoder-level KD loss into training, in addition to the standard supervised
loss and sequence-level KD loss. We investigate two encoder-level KD methods,
based on mean squared error (MSE) loss and contrastive loss, respectively.
Experimental results demonstrate that contrastive KD is more robust than MSE
KD, exhibiting superior performance in data-scarce situations. By leveraging
audio-only data into training in the KD framework, our student model achieves
competitive performance, with an inference speed that is 19 times
fasterAn online demo is available at
\url{https://huggingface.co/spaces/wsntxxn/efficient_audio_captioning}.
SparseCraft: Few-Shot Neural Reconstruction through Stereopsis Guided
  Geometric Linearization
Authors: name: Mae Younes, name: Amine Ouasfi, name: Adnane Boukhayma
Published: 2024-07-19 12:36:36

We present a novel approach for recovering 3D shape and view dependent
appearance from a few colored images, enabling efficient 3D reconstruction and
novel view synthesis. Our method learns an implicit neural representation in
the form of a Signed Distance Function (SDF) and a radiance field. The model is
trained progressively through ray marching enabled volumetric rendering, and
regularized with learning-free multi-view stereo (MVS) cues. Key to our
contribution is a novel implicit neural shape function learning strategy that
encourages our SDF field to be as linear as possible near the level-set, hence
robustifying the training against noise emanating from the supervision and
regularization signals. Without using any pretrained priors, our method, called
SparseCraft, achieves state-of-the-art performances both in novel-view
synthesis and reconstruction from sparse views in standard benchmarks, while
requiring less than 10 minutes for training.
PlacidDreamer: Advancing Harmony in Text-to-3D Generation
Authors: name: Shuo Huang, name: Shikun Sun, name: Zixuan Wang, name: Xiaoyu Qin, name: Yanmin Xiong, name: Yuan Zhang, name: Pengfei Wan, name: Di Zhang, name: Jia Jia
Published: 2024-07-19 02:00:04

Recently, text-to-3D generation has attracted significant attention,
resulting in notable performance enhancements. Previous methods utilize
end-to-end 3D generation models to initialize 3D Gaussians, multi-view
diffusion models to enforce multi-view consistency, and text-to-image diffusion
models to refine details with score distillation algorithms. However, these
methods exhibit two limitations. Firstly, they encounter conflicts in
generation directions since different models aim to produce diverse 3D assets.
Secondly, the issue of over-saturation in score distillation has not been
thoroughly investigated and solved. To address these limitations, we propose
PlacidDreamer, a text-to-3D framework that harmonizes initialization,
multi-view generation, and text-conditioned generation with a single multi-view
diffusion model, while simultaneously employing a novel score distillation
algorithm to achieve balanced saturation. To unify the generation direction, we
introduce the Latent-Plane module, a training-friendly plug-in extension that
enables multi-view diffusion models to provide fast geometry reconstruction for
initialization and enhanced multi-view images to personalize the text-to-image
diffusion model. To address the over-saturation problem, we propose to view
score distillation as a multi-objective optimization problem and introduce the
Balanced Score Distillation algorithm, which offers a Pareto Optimal solution
that achieves both rich details and balanced saturation. Extensive experiments
validate the outstanding capabilities of our PlacidDreamer. The code is
available at https://github.com/HansenHuang0823/PlacidDreamer.
CLAY: A Controllable Large-scale Generative Model for Creating
  High-quality 3D Assets
Authors: name: Longwen Zhang, name: Ziyu Wang, name: Qixuan Zhang, name: Qiwei Qiu, name: Anqi Pang, name: Haoran Jiang, name: Wei Yang, name: Lan Xu, name: Jingyi Yu
Published: 2024-05-30 05:57:36

In the realm of digital creativity, our potential to craft intricate 3D
worlds from imagination is often hampered by the limitations of existing
digital tools, which demand extensive expertise and efforts. To narrow this
disparity, we introduce CLAY, a 3D geometry and material generator designed to
effortlessly transform human imagination into intricate 3D digital structures.
CLAY supports classic text or image inputs as well as 3D-aware controls from
diverse primitives (multi-view images, voxels, bounding boxes, point clouds,
implicit representations, etc). At its core is a large-scale generative model
composed of a multi-resolution Variational Autoencoder (VAE) and a minimalistic
latent Diffusion Transformer (DiT), to extract rich 3D priors directly from a
diverse range of 3D geometries. Specifically, it adopts neural fields to
represent continuous and complete surfaces and uses a geometry generative
module with pure transformer blocks in latent space. We present a progressive
training scheme to train CLAY on an ultra large 3D model dataset obtained
through a carefully designed processing pipeline, resulting in a 3D native
geometry generator with 1.5 billion parameters. For appearance generation, CLAY
sets out to produce physically-based rendering (PBR) textures by employing a
multi-view material diffusion model that can generate 2K resolution textures
with diffuse, roughness, and metallic modalities. We demonstrate using CLAY for
a range of controllable 3D asset creations, from sketchy conceptual designs to
production ready assets with intricate details. Even first time users can
easily use CLAY to bring their vivid 3D imaginations to life, unleashing
unlimited creativity.