Orionnnn/hub-stats huggingface dataset:
AsyncDiff: Parallelizing Diffusion Models by Asynchronous Denoising
Authors: name: Zigeng Chen, name: Xinyin Ma, name: Gongfan Fang, name: Zhenxiong Tan, name: Xinchao Wang
Published: 2024-06-11 03:09:37

Diffusion models have garnered significant interest from the community for
their great generative ability across various applications. However, their
typical multi-step sequential-denoising nature gives rise to high cumulative
latency, thereby precluding the possibilities of parallel computation. To
address this, we introduce AsyncDiff, a universal and plug-and-play
acceleration scheme that enables model parallelism across multiple devices. Our
approach divides the cumbersome noise prediction model into multiple
components, assigning each to a different device. To break the dependency chain
between these components, it transforms the conventional sequential denoising
into an asynchronous process by exploiting the high similarity between hidden
states in consecutive diffusion steps. Consequently, each component is
facilitated to compute in parallel on separate devices. The proposed strategy
significantly reduces inference latency while minimally impacting the
generative quality. Specifically, for the Stable Diffusion v2.1, AsyncDiff
achieves a 2.7x speedup with negligible degradation and a 4.0x speedup with
only a slight reduction of 0.38 in CLIP Score, on four NVIDIA A5000 GPUs. Our
experiments also demonstrate that AsyncDiff can be readily applied to video
diffusion models with encouraging performances. The code is available at
https://github.com/czg1225/AsyncDiff.
TextGrad: Automatic "Differentiation" via Text
Authors: name: Mert Yuksekgonul, name: Federico Bianchi, name: Joseph Boen, name: Sheng Liu, name: Zhi Huang, name: Carlos Guestrin, name: James Zou
Published: 2024-06-11 17:32:21

AI is undergoing a paradigm shift, with breakthroughs achieved by systems
orchestrating multiple large language models (LLMs) and other complex
components. As a result, developing principled and automated optimization
methods for compound AI systems is one of the most important new challenges.
Neural networks faced a similar challenge in its early days until
backpropagation and automatic differentiation transformed the field by making
optimization turn-key. Inspired by this, we introduce TextGrad, a powerful
framework performing automatic ``differentiation'' via text. TextGrad
backpropagates textual feedback provided by LLMs to improve individual
components of a compound AI system. In our framework, LLMs provide rich,
general, natural language suggestions to optimize variables in computation
graphs, ranging from code snippets to molecular structures. TextGrad follows
PyTorch's syntax and abstraction and is flexible and easy-to-use. It works
out-of-the-box for a variety of tasks, where the users only provide the
objective function without tuning components or prompts of the framework. We
showcase TextGrad's effectiveness and generality across a diverse range of
applications, from question answering and molecule optimization to radiotherapy
treatment planning. Without modifying the framework, TextGrad improves the
zero-shot accuracy of GPT-4o in Google-Proof Question Answering from 51% to
55%, yields 20% relative performance gain in optimizing LeetCode-Hard
coding problem solutions, improves prompts for reasoning, designs new druglike
small molecules with desirable in silico binding, and designs radiation
oncology treatment plans with high specificity. TextGrad lays a foundation to
accelerate the development of the next-generation of AI systems.
Zero-shot Image Editing with Reference Imitation
Authors: name: Xi Chen, name: Yutong Feng, name: Mengting Chen, name: Yiyang Wang, name: Shilong Zhang, name: Yu Liu, name: Yujun Shen, name: Hengshuang Zhao
Published: 2024-06-11 17:59:51

Image editing serves as a practical yet challenging task considering the
diverse demands from users, where one of the hardest parts is to precisely
describe how the edited image should look like. In this work, we present a new
form of editing, termed imitative editing, to help users exercise their
creativity more conveniently. Concretely, to edit an image region of interest,
users are free to directly draw inspiration from some in-the-wild references
(e.g., some relative pictures come across online), without having to cope with
the fit between the reference and the source. Such a design requires the system
to automatically figure out what to expect from the reference to perform the
editing. For this purpose, we propose a generative training framework, dubbed
MimicBrush, which randomly selects two frames from a video clip, masks some
regions of one frame, and learns to recover the masked regions using the
information from the other frame. That way, our model, developed from a
diffusion prior, is able to capture the semantic correspondence between
separate images in a self-supervised manner. We experimentally show the
effectiveness of our method under various test cases as well as its superiority
over existing alternatives. We also construct a benchmark to facilitate further
research.
Simple and Effective Masked Diffusion Language Models
Authors: name: Subham Sekhar Sahoo, name: Marianne Arriola, name: Yair Schiff, name: Aaron Gokaslan, name: Edgar Marroquin, name: Justin T Chiu, name: Alexander Rush, name: Volodymyr Kuleshov
Published: 2024-06-11 17:51:40

While diffusion models excel at generating high-quality images, prior work
reports a significant performance gap between diffusion and autoregressive (AR)
methods in language modeling. In this work, we show that simple masked discrete
diffusion is more performant than previously thought. We apply an effective
training recipe that improves the performance of masked diffusion models and
derive a simplified, Rao-Blackwellized objective that results in additional
improvements. Our objective has a simple form -- it is a mixture of classical
masked language modeling losses -- and can be used to train encoder-only
language models that admit efficient samplers, including ones that can generate
arbitrary lengths of text semi-autoregressively like a traditional language
model. On language modeling benchmarks, a range of masked diffusion models
trained with modern engineering practices achieves a new state-of-the-art among
diffusion models, and approaches AR perplexity. We release our code at:
https://github.com/kuleshov-group/mdlm
Improve Mathematical Reasoning in Language Models by Automated Process
  Supervision
Authors: name: Liangchen Luo, name: Yinxiao Liu, name: Rosanne Liu, name: Samrat Phatale, name: Harsh Lara, name: Yunxuan Li, name: Lei Shu, name: Yun Zhu, name: Lei Meng, name: Jiao Sun, name: Abhinav Rastogi
Published: 2024-06-05 19:25:40

Complex multi-step reasoning tasks, such as solving mathematical problems or
generating code, remain a significant hurdle for even the most advanced large
language models (LLMs). Verifying LLM outputs with an Outcome Reward Model
(ORM) is a standard inference-time technique aimed at enhancing the reasoning
performance of LLMs. However, this still proves insufficient for reasoning
tasks with a lengthy or multi-hop reasoning chain, where the intermediate
outcomes are neither properly rewarded nor penalized. Process supervision
addresses this limitation by assigning intermediate rewards during the
reasoning process. To date, the methods used to collect process supervision
data have relied on either human annotation or per-step Monte Carlo estimation,
both prohibitively expensive to scale, thus hindering the broad application of
this technique. In response to this challenge, we propose a novel
divide-and-conquer style Monte Carlo Tree Search (MCTS) algorithm named
OmegaPRM for the efficient collection of high-quality process
supervision data. This algorithm swiftly identifies the first error in the
Chain of Thought (CoT) with binary search and balances the positive and
negative examples, thereby ensuring both efficiency and quality. As a result,
we are able to collect over 1.5 million process supervision annotations to
train a Process Reward Model (PRM). Utilizing this fully automated process
supervision alongside the weighted self-consistency algorithm, we have enhanced
the instruction tuned Gemini Pro model's math reasoning performance, achieving
a 69.4\% success rate on the MATH benchmark, a 36\% relative improvement from
the 51\% base model performance. Additionally, the entire process operates
without any human intervention, making our method both financially and
computationally cost-effective compared to existing methods.
4Real: Towards Photorealistic 4D Scene Generation via Video Diffusion
  Models
Authors: name: Heng Yu, name: Chaoyang Wang, name: Peiye Zhuang, name: Willi Menapace, name: Aliaksandr Siarohin, name: Junli Cao, name: Laszlo A Jeni, name: Sergey Tulyakov, name: Hsin-Ying Lee
Published: 2024-06-11 17:19:26

Existing dynamic scene generation methods mostly rely on distilling knowledge
from pre-trained 3D generative models, which are typically fine-tuned on
synthetic object datasets. As a result, the generated scenes are often
object-centric and lack photorealism. To address these limitations, we
introduce a novel pipeline designed for photorealistic text-to-4D scene
generation, discarding the dependency on multi-view generative models and
instead fully utilizing video generative models trained on diverse real-world
datasets. Our method begins by generating a reference video using the video
generation model. We then learn the canonical 3D representation of the video
using a freeze-time video, delicately generated from the reference video. To
handle inconsistencies in the freeze-time video, we jointly learn a per-frame
deformation to model these imperfections. We then learn the temporal
deformation based on the canonical representation to capture dynamic
interactions in the reference video. The pipeline facilitates the generation of
dynamic scenes with enhanced photorealism and structural integrity, viewable
from multiple perspectives, thereby setting a new standard in 4D scene
generation.
Margin-aware Preference Optimization for Aligning Diffusion Models
  without Reference
Authors: name: Jiwoo Hong, name: Sayak Paul, name: Noah Lee, name: Kashif Rasul, name: James Thorne, name: Jongheon Jeong
Published: 2024-06-10 16:14:45

Modern alignment techniques based on human preferences, such as RLHF and DPO,
typically employ divergence regularization relative to the reference model to
ensure training stability. However, this often limits the flexibility of models
during alignment, especially when there is a clear distributional discrepancy
between the preference data and the reference model. In this paper, we focus on
the alignment of recent text-to-image diffusion models, such as Stable
Diffusion XL (SDXL), and find that this "reference mismatch" is indeed a
significant problem in aligning these models due to the unstructured nature of
visual modalities: e.g., a preference for a particular stylistic aspect can
easily induce such a discrepancy. Motivated by this observation, we propose a
novel and memory-friendly preference alignment method for diffusion models that
does not depend on any reference model, coined margin-aware preference
optimization (MaPO). MaPO jointly maximizes the likelihood margin between the
preferred and dispreferred image sets and the likelihood of the preferred sets,
simultaneously learning general stylistic features and preferences. For
evaluation, we introduce two new pairwise preference datasets, which comprise
self-generated image pairs from SDXL, Pick-Style and Pick-Safety, simulating
diverse scenarios of reference mismatch. Our experiments validate that MaPO can
significantly improve alignment on Pick-Style and Pick-Safety and general
preference alignment when used with Pick-a-Pic v2, surpassing the base SDXL and
other existing methods. Our code, models, and datasets are publicly available
via https://mapo-t2i.github.io
Lighting Every Darkness with 3DGS: Fast Training and Real-Time Rendering
  for HDR View Synthesis
Authors: name: Xin Jin, name: Pengyi Jiao, name: Zheng-Peng Duan, name: Xingchao Yang, name: Chun-Le Guo, name: Bo Ren, name: Chongyi Li
Published: 2024-06-10 12:33:08

Volumetric rendering based methods, like NeRF, excel in HDR view synthesis
from RAWimages, especially for nighttime scenes. While, they suffer from long
training times and cannot perform real-time rendering due to dense sampling
requirements. The advent of 3D Gaussian Splatting (3DGS) enables real-time
rendering and faster training. However, implementing RAW image-based view
synthesis directly using 3DGS is challenging due to its inherent drawbacks: 1)
in nighttime scenes, extremely low SNR leads to poor structure-from-motion
(SfM) estimation in distant views; 2) the limited representation capacity of
spherical harmonics (SH) function is unsuitable for RAW linear color space; and
3) inaccurate scene structure hampers downstream tasks such as refocusing. To
address these issues, we propose LE3D (Lighting Every darkness with 3DGS). Our
method proposes Cone Scatter Initialization to enrich the estimation of SfM,
and replaces SH with a Color MLP to represent the RAW linear color space.
Additionally, we introduce depth distortion and near-far regularizations to
improve the accuracy of scene structure for downstream tasks. These designs
enable LE3D to perform real-time novel view synthesis, HDR rendering,
refocusing, and tone-mapping changes. Compared to previous volumetric rendering
based methods, LE3D reduces training time to 1% and improves rendering speed by
up to 4,000 times for 2K resolution images in terms of FPS. Code and viewer can
be found in https://github.com/Srameo/LE3D .
ExtraNeRF: Visibility-Aware View Extrapolation of Neural Radiance Fields
  with Diffusion Models
Authors: name: Meng-Li Shih, name: Wei-Chiu Ma, name: Aleksander Holynski, name: Forrester Cole, name: Brian L. Curless, name: Janne Kontkanen
Published: 2024-06-10 09:44:06

We propose ExtraNeRF, a novel method for extrapolating the range of views
handled by a Neural Radiance Field (NeRF). Our main idea is to leverage NeRFs
to model scene-specific, fine-grained details, while capitalizing on diffusion
models to extrapolate beyond our observed data. A key ingredient is to track
visibility to determine what portions of the scene have not been observed, and
focus on reconstructing those regions consistently with diffusion models. Our
primary contributions include a visibility-aware diffusion-based inpainting
module that is fine-tuned on the input imagery, yielding an initial NeRF with
moderate quality (often blurry) inpainted regions, followed by a second
diffusion model trained on the input imagery to consistently enhance, notably
sharpen, the inpainted imagery from the first pass. We demonstrate high-quality
results, extrapolating beyond a small number of (typically six or fewer) input
views, effectively outpainting the NeRF as well as inpainting newly disoccluded
regions inside the original viewing volume. We compare with related work both
quantitatively and qualitatively and show significant gains over prior art.
GTR: Improving Large 3D Reconstruction Models through Geometry and
  Texture Refinement
Authors: name: Peiye Zhuang, name: Songfang Han, name: Chaoyang Wang, name: Aliaksandr Siarohin, name: Jiaxu Zou, name: Michael Vasilkovsky, name: Vladislav Shakhrai, name: Sergey Korolev, name: Sergey Tulyakov, name: Hsin-Ying Lee
Published: 2024-06-09 05:19:24

We propose a novel approach for 3D mesh reconstruction from multi-view
images. Our method takes inspiration from large reconstruction models like LRM
that use a transformer-based triplane generator and a Neural Radiance Field
(NeRF) model trained on multi-view images. However, in our method, we introduce
several important modifications that allow us to significantly enhance 3D
reconstruction quality. First of all, we examine the original LRM architecture
and find several shortcomings. Subsequently, we introduce respective
modifications to the LRM architecture, which lead to improved multi-view image
representation and more computationally efficient training. Second, in order to
improve geometry reconstruction and enable supervision at full image
resolution, we extract meshes from the NeRF field in a differentiable manner
and fine-tune the NeRF model through mesh rendering. These modifications allow
us to achieve state-of-the-art performance on both 2D and 3D evaluation
metrics, such as a PSNR of 28.67 on Google Scanned Objects (GSO) dataset.
Despite these superior results, our feed-forward model still struggles to
reconstruct complex textures, such as text and portraits on assets. To address
this, we introduce a lightweight per-instance texture refinement procedure.
This procedure fine-tunes the triplane representation and the NeRF color
estimation model on the mesh surface using the input multi-view images in just
4 seconds. This refinement improves the PSNR to 29.79 and achieves faithful
reconstruction of complex textures, such as text. Additionally, our approach
enables various downstream applications, including text- or image-to-3D
generation.
Husky: A Unified, Open-Source Language Agent for Multi-Step Reasoning
Authors: name: Joongwon Kim, name: Bhargavi Paranjape, name: Tushar Khot, name: Hannaneh Hajishirzi
Published: 2024-06-10 17:07:25

Language agents perform complex tasks by using tools to execute each step
precisely. However, most existing agents are based on proprietary models or
designed to target specific tasks, such as mathematics or multi-hop question
answering. We introduce Husky, a holistic, open-source language agent that
learns to reason over a unified action space to address a diverse set of
complex tasks involving numerical, tabular, and knowledge-based reasoning.
Husky iterates between two stages: 1) generating the next action to take
towards solving a given task and 2) executing the action using expert models
and updating the current solution state. We identify a thorough ontology of
actions for addressing complex tasks and curate high-quality data to train
expert models for executing these actions. Our experiments show that Husky
outperforms prior language agents across 14 evaluation datasets. Moreover, we
introduce HuskyQA, a new evaluation set which stress tests language agents for
mixed-tool reasoning, with a focus on retrieving missing knowledge and
performing numerical reasoning. Despite using 7B models, Husky matches or even
exceeds frontier LMs such as GPT-4 on these tasks, showcasing the efficacy of
our holistic approach in addressing complex reasoning problems. Our code and
models are available at https://github.com/agent-husky/Husky-v1.
ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training
  Multiplication-Less Reparameterization
Authors: name: Haoran You, name: Yipin Guo, name: Yichao Fu, name: Wei Zhou, name: Huihong Shi, name: Xiaofan Zhang, name: Souvik Kundu, name: Amir Yazdanbakhsh, name: Yingyan Lin
Published: 2024-06-10 02:47:55

Large language models (LLMs) have shown impressive performance on language
tasks but face challenges when deployed on resource-constrained devices due to
their extensive parameters and reliance on dense multiplications, resulting in
high memory demands and latency bottlenecks. Shift-and-add reparameterization
offers a promising solution by replacing costly multiplications with
hardware-friendly primitives in both the attention and multi-layer perceptron
(MLP) layers of an LLM. However, current reparameterization techniques require
training from scratch or full parameter fine-tuning to restore accuracy, which
is resource-intensive for LLMs. To address this, we propose accelerating
pretrained LLMs through post-training shift-and-add reparameterization,
creating efficient multiplication-free models, dubbed ShiftAddLLM.
Specifically, we quantize each weight matrix into binary matrices paired with
group-wise scaling factors. The associated multiplications are reparameterized
into (1) shifts between activations and scaling factors and (2) queries and
adds according to the binary matrices. To reduce accuracy loss, we present a
multi-objective optimization method to minimize both weight and output
activation reparameterization errors. Additionally, based on varying
sensitivity across layers to reparameterization, we develop an automated bit
allocation strategy to further reduce memory usage and latency. Experiments on
five LLM families and eight tasks consistently validate the effectiveness of
ShiftAddLLM, achieving average perplexity improvements of 5.6 and 22.7 points
at comparable or lower latency compared to the most competitive quantized LLMs
at 3 and 2 bits, respectively, and more than 80% memory and energy reductions
over the original LLMs. Codes and models are available at
https://github.com/GATECH-EIC/ShiftAddLLM.
Tx-LLM: A Large Language Model for Therapeutics
Authors: name: Juan Manuel Zambrano Chaves, name: Eric Wang, name: Tao Tu, name: Eeshit Dhaval Vaishnav, name: Byron Lee, name: S. Sara Mahdavi, name: Christopher Semturs, name: David Fleet, name: Vivek Natarajan, name: Shekoofeh Azizi
Published: 2024-06-10 14:33:02

Developing therapeutics is a lengthy and expensive process that requires the
satisfaction of many different criteria, and AI models capable of expediting
the process would be invaluable. However, the majority of current AI approaches
address only a narrowly defined set of tasks, often circumscribed within a
particular domain. To bridge this gap, we introduce Tx-LLM, a generalist large
language model (LLM) fine-tuned from PaLM-2 which encodes knowledge about
diverse therapeutic modalities. Tx-LLM is trained using a collection of 709
datasets that target 66 tasks spanning various stages of the drug discovery
pipeline. Using a single set of weights, Tx-LLM simultaneously processes a wide
variety of chemical or biological entities(small molecules, proteins, nucleic
acids, cell lines, diseases) interleaved with free-text, allowing it to predict
a broad range of associated properties, achieving competitive with
state-of-the-art (SOTA) performance on 43 out of 66 tasks and exceeding SOTA on
22. Among these, Tx-LLM is particularly powerful and exceeds best-in-class
performance on average for tasks combining molecular SMILES representations
with text such as cell line names or disease names, likely due to context
learned during pretraining. We observe evidence of positive transfer between
tasks with diverse drug types (e.g.,tasks involving small molecules and tasks
involving proteins), and we study the impact of model size, domain finetuning,
and prompting strategies on performance. We believe Tx-LLM represents an
important step towards LLMs encoding biochemical knowledge and could have a
future role as an end-to-end tool across the drug discovery development
pipeline.