Orionnnn/hub-stats huggingface dataset:
Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient
  LLMs Under Compression
Authors: name: Junyuan Hong, name: Jinhao Duan, name: Chenhui Zhang, name: Zhangheng Li, name: Chulin Xie, name: Kelsey Lieberman, name: James Diffenderfer, name: Brian Bartoldson, name: Ajay Jaiswal, name: Kaidi Xu, name: Bhavya Kailkhura, name: Dan Hendrycks, name: Dawn Song, name: Zhangyang Wang, name: Bo Li
Published: 2024-03-18 01:38:19

Compressing high-capability Large Language Models (LLMs) has emerged as a
favored strategy for resource-efficient inferences. While state-of-the-art
(SoTA) compression methods boast impressive advancements in preserving benign
task performance, the potential risks of compression in terms of safety and
trustworthiness have been largely neglected. This study conducts the first,
thorough evaluation of three (3) leading LLMs using five (5) SoTA compression
techniques across eight (8) trustworthiness dimensions. Our experiments
highlight the intricate interplay between compression and trustworthiness,
revealing some interesting patterns. We find that quantization is currently a
more effective approach than pruning in achieving efficiency and
trustworthiness simultaneously. For instance, a 4-bit quantized model retains
the trustworthiness of its original counterpart, but model pruning
significantly degrades trustworthiness, even at 50% sparsity. Moreover,
employing quantization within a moderate bit range could unexpectedly improve
certain trustworthiness dimensions such as ethics and fairness. Conversely,
extreme quantization to very low bit levels (3 bits) tends to significantly
reduce trustworthiness. This increased risk cannot be uncovered by looking at
benign performance alone, in turn, mandating comprehensive trustworthiness
evaluation in practice. These findings culminate in practical recommendations
for simultaneously achieving high utility, efficiency, and trustworthiness in
LLMs. Models and code are available at https://decoding-comp-trust.github.io/.
VP3D: Unleashing 2D Visual Prompt for Text-to-3D Generation
Authors: name: Yang Chen, name: Yingwei Pan, name: Haibo Yang, name: Ting Yao, name: Tao Mei
Published: 2024-03-25 17:59:31

Recent innovations on text-to-3D generation have featured Score Distillation
Sampling (SDS), which enables the zero-shot learning of implicit 3D models
(NeRF) by directly distilling prior knowledge from 2D diffusion models.
However, current SDS-based models still struggle with intricate text prompts
and commonly result in distorted 3D models with unrealistic textures or
cross-view inconsistency issues. In this work, we introduce a novel Visual
Prompt-guided text-to-3D diffusion model (VP3D) that explicitly unleashes the
visual appearance knowledge in 2D visual prompt to boost text-to-3D generation.
Instead of solely supervising SDS with text prompt, VP3D first capitalizes on
2D diffusion model to generate a high-quality image from input text, which
subsequently acts as visual prompt to strengthen SDS optimization with explicit
visual appearance. Meanwhile, we couple the SDS optimization with additional
differentiable reward function that encourages rendering images of 3D models to
better visually align with 2D visual prompt and semantically match with text
prompt. Through extensive experiments, we show that the 2D Visual Prompt in our
VP3D significantly eases the learning of visual appearance of 3D models and
thus leads to higher visual fidelity with more detailed textures. It is also
appealing in view that when replacing the self-generating visual prompt with a
given reference image, VP3D is able to trigger a new task of stylized
text-to-3D generation. Our project page is available at
https://vp3d-cvpr24.github.io.
Be Yourself: Bounded Attention for Multi-Subject Text-to-Image
  Generation
Authors: name: Omer Dahary, name: Or Patashnik, name: Kfir Aberman, name: Daniel Cohen-Or
Published: 2024-03-25 17:52:07

Text-to-image diffusion models have an unprecedented ability to generate
diverse and high-quality images. However, they often struggle to faithfully
capture the intended semantics of complex input prompts that include multiple
subjects. Recently, numerous layout-to-image extensions have been introduced to
improve user control, aiming to localize subjects represented by specific
tokens. Yet, these methods often produce semantically inaccurate images,
especially when dealing with multiple semantically or visually similar
subjects. In this work, we study and analyze the causes of these limitations.
Our exploration reveals that the primary issue stems from inadvertent semantic
leakage between subjects in the denoising process. This leakage is attributed
to the diffusion model's attention layers, which tend to blend the visual
features of different subjects. To address these issues, we introduce Bounded
Attention, a training-free method for bounding the information flow in the
sampling process. Bounded Attention prevents detrimental leakage among subjects
and enables guiding the generation to promote each subject's individuality,
even with complex multi-subject conditioning. Through extensive
experimentation, we demonstrate that our method empowers the generation of
multiple subjects that better align with given prompts and layouts.
FlashFace: Human Image Personalization with High-fidelity Identity
  Preservation
Authors: name: Shilong Zhang, name: Lianghua Huang, name: Xi Chen, name: Yifei Zhang, name: Zhi-Fan Wu, name: Yutong Feng, name: Wei Wang, name: Yujun Shen, name: Yu Liu, name: Ping Luo
Published: 2024-03-25 17:59:57

This work presents FlashFace, a practical tool with which users can easily
personalize their own photos on the fly by providing one or a few reference
face images and a text prompt. Our approach is distinguishable from existing
human photo customization methods by higher-fidelity identity preservation and
better instruction following, benefiting from two subtle designs. First, we
encode the face identity into a series of feature maps instead of one image
token as in prior arts, allowing the model to retain more details of the
reference faces (e.g., scars, tattoos, and face shape ). Second, we introduce a
disentangled integration strategy to balance the text and image guidance during
the text-to-image generation process, alleviating the conflict between the
reference faces and the text prompts (e.g., personalizing an adult into a
"child" or an "elder"). Extensive experimental results demonstrate the
effectiveness of our method on various applications, including human image
personalization, face swapping under language prompts, making virtual
characters into real people, etc. Project Page:
https://jshilong.github.io/flashface-page.
LLM Agent Operating System
Authors: name: Kai Mei, name: Zelong Li, name: Shuyuan Xu, name: Ruosong Ye, name: Yingqiang Ge, name: Yongfeng Zhang
Published: 2024-03-25 17:32:23

The integration and deployment of large language model (LLM)-based
intelligent agents have been fraught with challenges that compromise their
efficiency and efficacy. Among these issues are sub-optimal scheduling and
resource allocation of agent requests over the LLM, the difficulties in
maintaining context during interactions between agent and LLM, and the
complexities inherent in integrating heterogeneous agents with different
capabilities and specializations. The rapid increase of agent quantity and
complexity further exacerbates these issues, often leading to bottlenecks and
sub-optimal utilization of resources. Inspired by these challenges, this paper
presents AIOS, an LLM agent operating system, which embeds large language model
into operating systems (OS). Specifically, AIOS is designed to optimize
resource allocation, facilitate context switch across agents, enable concurrent
execution of agents, provide tool service for agents, and maintain access
control for agents. We present the architecture of such an operating system,
outline the core challenges it aims to resolve, and provide the basic design
and implementation of the AIOS. Our experiments on concurrent execution of
multiple agents demonstrate the reliability and efficiency of our AIOS modules.
Through this, we aim to not only improve the performance and efficiency of LLM
agents but also to pioneer for better development and deployment of the AIOS
ecosystem in the future. The project is open-source at
https://github.com/agiresearch/AIOS.
SDXS: Real-Time One-Step Latent Diffusion Models with Image Conditions
Authors: name: Yuda Song, name: Zehao Sun, name: Xuanwu Yin
Published: 2024-03-25 11:16:23

Recent advancements in diffusion models have positioned them at the forefront
of image generation. Despite their superior performance, diffusion models are
not without drawbacks; they are characterized by complex architectures and
substantial computational demands, resulting in significant latency due to
their iterative sampling process. To mitigate these limitations, we introduce a
dual approach involving model miniaturization and a reduction in sampling
steps, aimed at significantly decreasing model latency. Our methodology
leverages knowledge distillation to streamline the U-Net and image decoder
architectures, and introduces an innovative one-step DM training technique that
utilizes feature matching and score distillation. We present two models,
SDXS-512 and SDXS-1024, achieving inference speeds of approximately 100 FPS
(30x faster than SD v1.5) and 30 FP (60x faster than SDXL) on a single GPU,
respectively. Moreover, our training approach offers promising applications in
image-conditioned control, facilitating efficient image-to-image translation.
TRIP: Temporal Residual Learning with Image Noise Prior for
  Image-to-Video Diffusion Models
Authors: name: Zhongwei Zhang, name: Fuchen Long, name: Yingwei Pan, name: Zhaofan Qiu, name: Ting Yao, name: Yang Cao, name: Tao Mei
Published: 2024-03-25 17:59:40

Recent advances in text-to-video generation have demonstrated the utility of
powerful diffusion models. Nevertheless, the problem is not trivial when
shaping diffusion models to animate static image (i.e., image-to-video
generation). The difficulty originates from the aspect that the diffusion
process of subsequent animated frames should not only preserve the faithful
alignment with the given image but also pursue temporal coherence among
adjacent frames. To alleviate this, we present TRIP, a new recipe of
image-to-video diffusion paradigm that pivots on image noise prior derived from
static image to jointly trigger inter-frame relational reasoning and ease the
coherent temporal modeling via temporal residual learning. Technically, the
image noise prior is first attained through one-step backward diffusion process
based on both static image and noised video latent codes. Next, TRIP executes a
residual-like dual-path scheme for noise prediction: 1) a shortcut path that
directly takes image noise prior as the reference noise of each frame to
amplify the alignment between the first frame and subsequent frames; 2) a
residual path that employs 3D-UNet over noised video and static image latent
codes to enable inter-frame relational reasoning, thereby easing the learning
of the residual noise for each frame. Furthermore, both reference and residual
noise of each frame are dynamically merged via attention mechanism for final
video generation. Extensive experiments on WebVid-10M, DTDB and MSR-VTT
datasets demonstrate the effectiveness of our TRIP for image-to-video
generation. Please see our project page at https://trip-i2v.github.io/TRIP/.
DragAPart: Learning a Part-Level Motion Prior for Articulated Objects
Authors: name: Ruining Li, name: Chuanxia Zheng, name: Christian Rupprecht, name: Andrea Vedaldi
Published: 2024-03-22 17:58:59

We introduce DragAPart, a method that, given an image and a set of drags as
input, can generate a new image of the same object in a new state, compatible
with the action of the drags. Differently from prior works that focused on
repositioning objects, DragAPart predicts part-level interactions, such as
opening and closing a drawer. We study this problem as a proxy for learning a
generalist motion model, not restricted to a specific kinematic structure or
object category. To this end, we start from a pre-trained image generator and
fine-tune it on a new synthetic dataset, Drag-a-Move, which we introduce.
Combined with a new encoding for the drags and dataset randomization, the new
model generalizes well to real images and different categories. Compared to
prior motion-controlled generators, we demonstrate much better part-level
motion understanding.
Champ: Controllable and Consistent Human Image Animation with 3D
  Parametric Guidance
Authors: name: Shenhao Zhu, name: Junming Leo Chen, name: Zuozhuo Dai, name: Yinghui Xu, name: Xun Cao, name: Yao Yao, name: Hao Zhu, name: Siyu Zhu
Published: 2024-03-21 18:52:58

In this study, we introduce a methodology for human image animation by
leveraging a 3D human parametric model within a latent diffusion framework to
enhance shape alignment and motion guidance in curernt human generative
techniques. The methodology utilizes the SMPL(Skinned Multi-Person Linear)
model as the 3D human parametric model to establish a unified representation of
body shape and pose. This facilitates the accurate capture of intricate human
geometry and motion characteristics from source videos. Specifically, we
incorporate rendered depth images, normal maps, and semantic maps obtained from
SMPL sequences, alongside skeleton-based motion guidance, to enrich the
conditions to the latent diffusion model with comprehensive 3D shape and
detailed pose attributes. A multi-layer motion fusion module, integrating
self-attention mechanisms, is employed to fuse the shape and motion latent
representations in the spatial domain. By representing the 3D human parametric
model as the motion guidance, we can perform parametric shape alignment of the
human body between the reference image and the source video motion.
Experimental evaluations conducted on benchmark datasets demonstrate the
methodology's superior ability to generate high-quality human animations that
accurately capture both pose and shape variations. Furthermore, our approach
also exhibits superior generalization capabilities on the proposed wild
dataset. Project page: https://fudan-generative-vision.github.io/champ.
AllHands: Ask Me Anything on Large-scale Verbatim Feedback via Large
  Language Models
Authors: name: Chaoyun Zhang, name: Zicheng Ma, name: Yuhao Wu, name: Shilin He, name: Si Qin, name: Minghua Ma, name: Xiaoting Qin, name: Yu Kang, name: Yuyi Liang, name: Xiaoyu Gou, name: Yajie Xue, name: Qingwei Lin, name: Saravan Rajmohan, name: Dongmei Zhang, name: Qi Zhang
Published: 2024-03-22 12:13:16

Verbatim feedback constitutes a valuable repository of user experiences,
opinions, and requirements essential for software development. Effectively and
efficiently extracting valuable insights from such data poses a challenging
task. This paper introduces Allhands , an innovative analytic framework
designed for large-scale feedback analysis through a natural language
interface, leveraging large language models (LLMs). Allhands adheres to a
conventional feedback analytic workflow, initially conducting classification
and topic modeling on the feedback to convert them into a structurally
augmented format, incorporating LLMs to enhance accuracy, robustness,
generalization, and user-friendliness. Subsequently, an LLM agent is employed
to interpret users' diverse questions in natural language on feedback,
translating them into Python code for execution, and delivering comprehensive
multi-modal responses, including text, code, tables, and images.
  We evaluate Allhands across three diverse feedback datasets. The experiments
demonstrate that Allhands achieves superior efficacy at all stages of analysis,
including classification and topic modeling, eventually providing users with an
``ask me anything'' experience with comprehensive, correct and human-readable
response. To the best of our knowledge, Allhands stands as the first
comprehensive feedback analysis framework that supports diverse and customized
requirements for insight extraction through a natural language interface.
VidLA: Video-Language Alignment at Scale
Authors: name: Mamshad Nayeem Rizve, name: Fan Fei, name: Jayakrishnan Unnikrishnan, name: Son Tran, name: Benjamin Z. Yao, name: Belinda Zeng, name: Mubarak Shah, name: Trishul Chilimbi
Published: 2024-03-21 22:36:24

In this paper, we propose VidLA, an approach for video-language alignment at
scale. There are two major limitations of previous video-language alignment
approaches. First, they do not capture both short-range and long-range temporal
dependencies and typically employ complex hierarchical deep network
architectures that are hard to integrate with existing pretrained image-text
foundation models. To effectively address this limitation, we instead keep the
network architecture simple and use a set of data tokens that operate at
different temporal resolutions in a hierarchical manner, accounting for the
temporally hierarchical nature of videos. By employing a simple two-tower
architecture, we are able to initialize our video-language model with
pretrained image-text foundation models, thereby boosting the final
performance. Second, existing video-language alignment works struggle due to
the lack of semantically aligned large-scale training data. To overcome it, we
leverage recent LLMs to curate the largest video-language dataset to date with
better visual grounding. Furthermore, unlike existing video-text datasets which
only contain short clips, our dataset is enriched with video clips of varying
durations to aid our temporally hierarchical data tokens in extracting better
representations at varying temporal scales. Overall, empirical results show
that our proposed approach surpasses state-of-the-art methods on multiple
retrieval benchmarks, especially on longer videos, and performs competitively
on classification benchmarks.
Compiler generated feedback for Large Language Models
Authors: name: Dejan Grubisic, name: Chris Cummins, name: Volker Seeker, name: Hugh Leather
Published: 2024-03-18 23:25:13

We introduce a novel paradigm in compiler optimization powered by Large
Language Models with compiler feedback to optimize the code size of LLVM
assembly. The model takes unoptimized LLVM IR as input and produces optimized
IR, the best optimization passes, and instruction counts of both unoptimized
and optimized IRs. Then we compile the input with generated optimization passes
and evaluate if the predicted instruction count is correct, generated IR is
compilable, and corresponds to compiled code. We provide this feedback back to
LLM and give it another chance to optimize code. This approach adds an extra
0.53% improvement over -Oz to the original model. Even though, adding more
information with feedback seems intuitive, simple sampling techniques achieve
much higher performance given 10 or more samples.
FollowIR: Evaluating and Teaching Information Retrieval Models to Follow
  Instructions
Authors: name: Orion Weller, name: Benjamin Chang, name: Sean MacAvaney, name: Kyle Lo, name: Arman Cohan, name: Benjamin Van Durme, name: Dawn Lawrie, name: Luca Soldaini
Published: 2024-03-22 14:42:29

Modern Large Language Models (LLMs) are capable of following long and complex
instructions that enable a diverse amount of user tasks. However, despite
Information Retrieval (IR) models using LLMs as the backbone of their
architectures, nearly all of them still only take queries as input, with no
instructions. For the handful of recent models that do take instructions, it's
unclear how they use them. We introduce our dataset FollowIR, which contains a
rigorous instruction evaluation benchmark as well as a training set for helping
IR models learn to better follow real-world instructions. FollowIR builds off
the long history of the TREC conferences: as TREC provides human annotators
with instructions (also known as narratives) to determine document relevance,
so should IR models be able to understand and decide relevance based on these
detailed instructions. Our evaluation benchmark starts with three deeply judged
TREC collections and alters the annotator instructions, re-annotating relevant
documents. Through this process, we can measure how well IR models follow
instructions, through a new pairwise evaluation framework. Our results indicate
that existing retrieval models fail to correctly use instructions, using them
for basic keywords and struggling to understand long-form information. However,
we show that it is possible for IR models to learn to follow complex
instructions: our new FollowIR-7B model has significant improvements (over 13%)
after fine-tuning on our training set.
LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancement
Authors: name: Nicholas Lee, name: Thanakul Wattanawong, name: Sehoon Kim, name: Karttikeya Mangalam, name: Sheng Shen, name: Gopala Anumanchipali, name: Michael W. Mahoney, name: Kurt Keutzer, name: Amir Gholami
Published: 2024-03-22 08:57:07

Pretrained large language models (LLMs) are currently state-of-the-art for
solving the vast majority of natural language processing tasks. While many
real-world applications still require fine-tuning to reach satisfactory levels
of performance, many of them are in the low-data regime, making fine-tuning
challenging. To address this, we propose LLM2LLM, a targeted and iterative data
augmentation strategy that uses a teacher LLM to enhance a small seed dataset
by augmenting additional data that can be used for fine-tuning on a specific
task. LLM2LLM (1) fine-tunes a baseline student LLM on the initial seed data,
(2) evaluates and extracts data points that the model gets wrong, and (3) uses
a teacher LLM to generate synthetic data based on these incorrect data points,
which are then added back into the training data. This approach amplifies the
signal from incorrectly predicted data points by the LLM during training and
reintegrates them into the dataset to focus on more challenging examples for
the LLM. Our results show that LLM2LLM significantly enhances the performance
of LLMs in the low-data regime, outperforming both traditional fine-tuning and
other data augmentation baselines. LLM2LLM reduces the dependence on
labor-intensive data curation and paves the way for more scalable and
performant LLM solutions, allowing us to tackle data-constrained domains and
tasks. We achieve improvements up to 24.2% on the GSM8K dataset, 32.6% on
CaseHOLD, 32.0% on SNIPS, 52.6% on TREC and 39.8% on SST-2 over regular
fine-tuning in the low-data regime using a LLaMA2-7B student model.
LATTE3D: Large-scale Amortized Text-To-Enhanced3D Synthesis
Authors: name: Kevin Xie, name: Jonathan Lorraine, name: Tianshi Cao, name: Jun Gao, name: James Lucas, name: Antonio Torralba, name: Sanja Fidler, name: Xiaohui Zeng
Published: 2024-03-22 17:59:37

Recent text-to-3D generation approaches produce impressive 3D results but
require time-consuming optimization that can take up to an hour per prompt.
Amortized methods like ATT3D optimize multiple prompts simultaneously to
improve efficiency, enabling fast text-to-3D synthesis. However, they cannot
capture high-frequency geometry and texture details and struggle to scale to
large prompt sets, so they generalize poorly. We introduce LATTE3D, addressing
these limitations to achieve fast, high-quality generation on a significantly
larger prompt set. Key to our method is 1) building a scalable architecture and
2) leveraging 3D data during optimization through 3D-aware diffusion priors,
shape regularization, and model initialization to achieve robustness to diverse
and complex training prompts. LATTE3D amortizes both neural field and textured
surface generation to produce highly detailed textured meshes in a single
forward pass. LATTE3D generates 3D objects in 400ms, and can be further
enhanced with fast test-time optimization.