Orionnnn/hub-stats huggingface dataset:
TokenPacker: Efficient Visual Projector for Multimodal LLM
Authors: name: Wentong Li, name: Yuqian Yuan, name: Jian Liu, name: Dongqi Tang, name: Song Wang, name: Jianke Zhu, name: Lei Zhang
Published: 2024-07-02 16:10:55

The visual projector serves as an essential bridge between the visual encoder
and the Large Language Model (LLM) in a Multimodal LLM (MLLM). Typically, MLLMs
adopt a simple MLP to preserve all visual contexts via one-to-one
transformation. However, the visual tokens are redundant and can be
considerably increased when dealing with high-resolution images, impairing the
efficiency of MLLMs significantly. Some recent works have introduced resampler
or abstractor to reduce the number of resulting visual tokens. Unfortunately,
they fail to capture finer details and undermine the visual reasoning
capabilities of MLLMs. In this work, we propose a novel visual projector, which
adopts a coarse-to-fine scheme to inject the enriched characteristics to
generate the condensed visual tokens. In specific, we first interpolate the
visual features as a low-resolution point query, providing the overall visual
representation as the foundation. Then, we introduce a region-to-point
injection module that utilizes high-resolution, multi-level region-based cues
as fine-grained reference keys and values, allowing them to be fully absorbed
within the corresponding local context region. This step effectively updates
the coarse point query, transforming it into an enriched one for the subsequent
LLM reasoning. Extensive experiments demonstrate that our approach compresses
the visual tokens by 75%~89%, while achieves comparable or even better
performance across diverse benchmarks with significantly higher efficiency. The
source codes can be found at https://github.com/CircleRadon/TokenPacker.
μ-Bench: A Vision-Language Benchmark for Microscopy Understanding
Authors: name: Alejandro Lozano, name: Jeffrey Nirschl, name: James Burgess, name: Sanket Rajan Gupte, name: Yuhui Zhang, name: Alyssa Unell, name: Serena Yeung-Levy
Published: 2024-07-01 20:30:26

Recent advances in microscopy have enabled the rapid generation of terabytes
of image data in cell biology and biomedical research. Vision-language models
(VLMs) offer a promising solution for large-scale biological image analysis,
enhancing researchers' efficiency, identifying new image biomarkers, and
accelerating hypothesis generation and scientific discovery. However, there is
a lack of standardized, diverse, and large-scale vision-language benchmarks to
evaluate VLMs' perception and cognition capabilities in biological image
understanding. To address this gap, we introduce {\mu}-Bench, an expert-curated
benchmark encompassing 22 biomedical tasks across various scientific
disciplines (biology, pathology), microscopy modalities (electron,
fluorescence, light), scales (subcellular, cellular, tissue), and organisms in
both normal and abnormal states. We evaluate state-of-the-art biomedical,
pathology, and general VLMs on {\mu}-Bench and find that: i) current models
struggle on all categories, even for basic tasks such as distinguishing
microscopy modalities; ii) current specialist models fine-tuned on biomedical
data often perform worse than generalist models; iii) fine-tuning in specific
microscopy domains can cause catastrophic forgetting, eroding prior biomedical
knowledge encoded in their base model. iv) weight interpolation between
fine-tuned and pre-trained models offers one solution to forgetting and
improves general performance across biomedical tasks. We release {\mu}-Bench
under a permissive license to accelerate the research and development of
microscopy foundation models.
Magic Insert: Style-Aware Drag-and-Drop
Authors: name: Nataniel Ruiz, name: Yuanzhen Li, name: Neal Wadhwa, name: Yael Pritch, name: Michael Rubinstein, name: David E. Jacobs, name: Shlomi Fruchter
Published: 2024-07-02 17:59:50

We present Magic Insert, a method for dragging-and-dropping subjects from a
user-provided image into a target image of a different style in a physically
plausible manner while matching the style of the target image. This work
formalizes the problem of style-aware drag-and-drop and presents a method for
tackling it by addressing two sub-problems: style-aware personalization and
realistic object insertion in stylized images. For style-aware personalization,
our method first fine-tunes a pretrained text-to-image diffusion model using
LoRA and learned text tokens on the subject image, and then infuses it with a
CLIP representation of the target style. For object insertion, we use
Bootstrapped Domain Adaption to adapt a domain-specific photorealistic object
insertion model to the domain of diverse artistic styles. Overall, the method
significantly outperforms traditional approaches such as inpainting. Finally,
we present a dataset, SubjectPlop, to facilitate evaluation and future progress
in this area. Project page: https://magicinsert.github.io/
Understanding Alignment in Multimodal LLMs: A Comprehensive Study
Authors: name: Elmira Amirloo, name: Jean-Philippe Fauconnier, name: Christoph Roesmann, name: Christian Kerl, name: Rinu Boney, name: Yusu Qian, name: Zirui Wang, name: Afshin Dehghan, name: Yinfei Yang, name: Zhe Gan, name: Peter Grasch
Published: 2024-07-02 17:55:03

Preference alignment has become a crucial component in enhancing the
performance of Large Language Models (LLMs), yet its impact in Multimodal Large
Language Models (MLLMs) remains comparatively underexplored. Similar to
language models, MLLMs for image understanding tasks encounter challenges like
hallucination. In MLLMs, hallucination can occur not only by stating incorrect
facts but also by producing responses that are inconsistent with the image
content. A primary objective of alignment for MLLMs is to encourage these
models to align responses more closely with image information. Recently,
multiple works have introduced preference datasets for MLLMs and examined
different alignment methods, including Direct Preference Optimization (DPO) and
Proximal Policy Optimization (PPO). However, due to variations in datasets,
base model types, and alignment methods, it remains unclear which specific
elements contribute most significantly to the reported improvements in these
works. In this paper, we independently analyze each aspect of preference
alignment in MLLMs. We start by categorizing the alignment algorithms into two
groups, offline (such as DPO), and online (such as online-DPO), and show that
combining offline and online methods can improve the performance of the model
in certain scenarios. We review a variety of published multimodal preference
datasets and discuss how the details of their construction impact model
performance. Based on these insights, we introduce a novel way of creating
multimodal preference data called Bias-Driven Hallucination Sampling (BDHS)
that needs neither additional annotation nor external models, and show that it
can achieve competitive performance to previously published alignment work for
multimodal models across a range of benchmarks.
Revealing Fine-Grained Values and Opinions in Large Language Models
Authors: name: Dustin Wright, name: Arnav Arora, name: Nadav Borenstein, name: Srishti Yadav, name: Serge Belongie, name: Isabelle Augenstein
Published: 2024-06-27 15:01:53

Uncovering latent values and opinions in large language models (LLMs) can
help identify biases and mitigate potential harm. Recently, this has been
approached by presenting LLMs with survey questions and quantifying their
stances towards morally and politically charged statements. However, the
stances generated by LLMs can vary greatly depending on how they are prompted,
and there are many ways to argue for or against a given position. In this work,
we propose to address this by analysing a large and robust dataset of 156k LLM
responses to the 62 propositions of the Political Compass Test (PCT) generated
by 6 LLMs using 420 prompt variations. We perform coarse-grained analysis of
their generated stances and fine-grained analysis of the plain text
justifications for those stances. For fine-grained analysis, we propose to
identify tropes in the responses: semantically similar phrases that are
recurrent and consistent across different prompts, revealing patterns in the
text that a given LLM is prone to produce. We find that demographic features
added to prompts significantly affect outcomes on the PCT, reflecting bias, as
well as disparities between the results of tests when eliciting closed-form vs.
open domain responses. Additionally, patterns in the plain text rationales via
tropes show that similar justifications are repeatedly generated across models
and prompts even with disparate stances.
To Forget or Not? Towards Practical Knowledge Unlearning for Large
  Language Models
Authors: name: Bozhong Tian, name: Xiaozhuan Liang, name: Siyuan Cheng, name: Qingbin Liu, name: Mengru Wang, name: Dianbo Sui, name: Xi Chen, name: Huajun Chen, name: Ningyu Zhang
Published: 2024-07-02 03:34:16

Large Language Models (LLMs) trained on extensive corpora inevitably retain
sensitive data, such as personal privacy information and copyrighted material.
Recent advancements in knowledge unlearning involve updating LLM parameters to
erase specific knowledge. However, current unlearning paradigms are mired in
vague forgetting boundaries, often erasing knowledge indiscriminately. In this
work, we introduce KnowUnDo, a benchmark containing copyrighted content and
user privacy domains to evaluate if the unlearning process inadvertently erases
essential knowledge. Our findings indicate that existing unlearning methods
often suffer from excessive unlearning. To address this, we propose a simple
yet effective method, MemFlex, which utilizes gradient information to precisely
target and unlearn sensitive parameters. Experimental results show that MemFlex
is superior to existing methods in both precise knowledge unlearning and
general knowledge retaining of LLMs. Code and dataset will be released at
https://github.com/zjunlp/KnowUnDo.
OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video
  Generation
Authors: name: Kepan Nan, name: Rui Xie, name: Penghao Zhou, name: Tiehan Fan, name: Zhenheng Yang, name: Zhijie Chen, name: Xiang Li, name: Jian Yang, name: Ying Tai
Published: 2024-07-02 15:40:29

Text-to-video (T2V) generation has recently garnered significant attention
thanks to the large multi-modality model Sora. However, T2V generation still
faces two important challenges: 1) Lacking a precise open sourced high-quality
dataset. The previous popular video datasets, e.g. WebVid-10M and Panda-70M,
are either with low quality or too large for most research institutions.
Therefore, it is challenging but crucial to collect a precise high-quality
text-video pairs for T2V generation. 2) Ignoring to fully utilize textual
information. Recent T2V methods have focused on vision transformers, using a
simple cross attention module for video generation, which falls short of
thoroughly extracting semantic information from text prompt. To address these
issues, we introduce OpenVid-1M, a precise high-quality dataset with expressive
captions. This open-scenario dataset contains over 1 million text-video pairs,
facilitating research on T2V generation. Furthermore, we curate 433K 1080p
videos from OpenVid-1M to create OpenVidHD-0.4M, advancing high-definition
video generation. Additionally, we propose a novel Multi-modal Video Diffusion
Transformer (MVDiT) capable of mining both structure information from visual
tokens and semantic information from text tokens. Extensive experiments and
ablation studies verify the superiority of OpenVid-1M over previous datasets
and the effectiveness of our MVDiT.
What Matters in Detecting AI-Generated Videos like Sora?
Authors: name: Chirui Chang, name: Zhengzhe Liu, name: Xiaoyang Lyu, name: Xiaojuan Qi
Published: 2024-06-27 23:03:58

Recent advancements in diffusion-based video generation have showcased
remarkable results, yet the gap between synthetic and real-world videos remains
under-explored. In this study, we examine this gap from three fundamental
perspectives: appearance, motion, and geometry, comparing real-world videos
with those generated by a state-of-the-art AI model, Stable Video Diffusion. To
achieve this, we train three classifiers using 3D convolutional networks, each
targeting distinct aspects: vision foundation model features for appearance,
optical flow for motion, and monocular depth for geometry. Each classifier
exhibits strong performance in fake video detection, both qualitatively and
quantitatively. This indicates that AI-generated videos are still easily
detectable, and a significant gap between real and fake videos persists.
Furthermore, utilizing the Grad-CAM, we pinpoint systematic failures of
AI-generated videos in appearance, motion, and geometry. Finally, we propose an
Ensemble-of-Experts model that integrates appearance, optical flow, and depth
information for fake video detection, resulting in enhanced robustness and
generalization ability. Our model is capable of detecting videos generated by
Sora with high accuracy, even without exposure to any Sora videos during
training. This suggests that the gap between real and fake videos can be
generalized across various video generative models. Project page:
https://justin-crchang.github.io/3DCNNDetection.github.io/
Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systems
Authors: name: Philippe Laban, name: Alexander R. Fabbri, name: Caiming Xiong, name: Chien-Sheng Wu
Published: 2024-07-01 15:23:42

LLMs and RAG systems are now capable of handling millions of input tokens or
more. However, evaluating the output quality of such systems on long-context
tasks remains challenging, as tasks like Needle-in-a-Haystack lack complexity.
In this work, we argue that summarization can play a central role in such
evaluation. We design a procedure to synthesize Haystacks of documents,
ensuring that specific insights repeat across documents. The "Summary
of a Haystack" (SummHay) task then requires a system to process the Haystack
and generate, given a query, a summary that identifies the relevant insights
and precisely cites the source documents. Since we have precise knowledge of
what insights should appear in a haystack summary and what documents should be
cited, we implement a highly reproducible automatic evaluation that can score
summaries on two aspects - Coverage and Citation. We generate Haystacks in two
domains (conversation, news), and perform a large-scale evaluation of 10 LLMs
and corresponding 50 RAG systems. Our findings indicate that SummHay is an open
challenge for current systems, as even systems provided with an Oracle signal
of document relevance lag our estimate of human performance (56\%) by 10+
points on a Joint Score. Without a retriever, long-context LLMs like GPT-4o and
Claude 3 Opus score below 20% on SummHay. We show SummHay can also be used to
study enterprise RAG systems and position bias in long-context models. We hope
future systems can equal and surpass human performance on SummHay.
Agentless: Demystifying LLM-based Software Engineering Agents
Authors: name: Chunqiu Steven Xia, name: Yinlin Deng, name: Soren Dunn, name: Lingming Zhang
Published: 2024-07-01 17:24:45

Recent advancements in large language models (LLMs) have significantly
advanced the automation of software development tasks, including code
synthesis, program repair, and test generation. More recently, researchers and
industry practitioners have developed various autonomous LLM agents to perform
end-to-end software development tasks. These agents are equipped with the
ability to use tools, run commands, observe feedback from the environment, and
plan for future actions. However, the complexity of these agent-based
approaches, together with the limited abilities of current LLMs, raises the
following question: Do we really have to employ complex autonomous software
agents? To attempt to answer this question, we build Agentless -- an agentless
approach to automatically solve software development problems. Compared to the
verbose and complex setup of agent-based approaches, Agentless employs a
simplistic two-phase process of localization followed by repair, without
letting the LLM decide future actions or operate with complex tools. Our
results on the popular SWE-bench Lite benchmark show that surprisingly the
simplistic Agentless is able to achieve both the highest performance (27.33%)
and lowest cost (\$0.34) compared with all existing open-source software
agents! Furthermore, we manually classified the problems in SWE-bench Lite and
found problems with exact ground truth patch or insufficient/misleading issue
descriptions. As such, we construct SWE-bench Lite-S by excluding such
problematic issues to perform more rigorous evaluation and comparison. Our work
highlights the current overlooked potential of a simple, interpretable
technique in autonomous software development. We hope Agentless will help reset
the baseline, starting point, and horizon for autonomous software agents, and
inspire future work along this crucial direction.
MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via
  Dynamic Sparse Attention
Authors: name: Huiqiang Jiang, name: Yucheng Li, name: Chengruidong Zhang, name: Qianhui Wu, name: Xufang Luo, name: Surin Ahn, name: Zhenhua Han, name: Amir H. Abdi, name: Dongsheng Li, name: Chin-Yew Lin, name: Yuqing Yang, name: Lili Qiu
Published: 2024-07-02 17:59:56

The computational challenges of Large Language Model (LLM) inference remain a
significant barrier to their widespread deployment, especially as prompt
lengths continue to increase. Due to the quadratic complexity of the attention
computation, it takes 30 minutes for an 8B LLM to process a prompt of 1M tokens
(i.e., the pre-filling stage) on a single A100 GPU. Existing methods for
speeding up prefilling often fail to maintain acceptable accuracy or efficiency
when applied to long-context LLMs. To address this gap, we introduce MInference
(Milliontokens Inference), a sparse calculation method designed to accelerate
pre-filling of long-sequence processing. Specifically, we identify three unique
patterns in long-context attention matrices-the A-shape, Vertical-Slash, and
Block-Sparsethat can be leveraged for efficient sparse computation on GPUs. We
determine the optimal pattern for each attention head offline and dynamically
build sparse indices based on the assigned pattern during inference. With the
pattern and sparse indices, we perform efficient sparse attention calculations
via our optimized GPU kernels to significantly reduce the latency in the
pre-filling stage of long-context LLMs. Our proposed technique can be directly
applied to existing LLMs without any modifications to the pre-training setup or
additional fine-tuning. By evaluating on a wide range of downstream tasks,
including InfiniteBench, RULER, PG-19, and Needle In A Haystack, and models
including LLaMA-3-1M, GLM4-1M, Yi-200K, Phi-3-128K, and Qwen2-128K, we
demonstrate that MInference effectively reduces inference latency by up to 10x
for pre-filling on an A100, while maintaining accuracy. Our code is available
at https://aka.ms/MInference.
Consistency Flow Matching: Defining Straight Flows with Velocity
  Consistency
Authors: name: Ling Yang, name: Zixiang Zhang, name: Zhilong Zhang, name: Xingchao Liu, name: Minkai Xu, name: Wentao Zhang, name: Chenlin Meng, name: Stefano Ermon, name: Bin Cui
Published: 2024-07-02 16:15:37

Flow matching (FM) is a general framework for defining probability paths via
Ordinary Differential Equations (ODEs) to transform between noise and data
samples. Recent approaches attempt to straighten these flow trajectories to
generate high-quality samples with fewer function evaluations, typically
through iterative rectification methods or optimal transport solutions. In this
paper, we introduce Consistency Flow Matching (Consistency-FM), a novel FM
method that explicitly enforces self-consistency in the velocity field.
Consistency-FM directly defines straight flows starting from different times to
the same endpoint, imposing constraints on their velocity values. Additionally,
we propose a multi-segment training approach for Consistency-FM to enhance
expressiveness, achieving a better trade-off between sampling quality and
speed. Preliminary experiments demonstrate that our Consistency-FM
significantly improves training efficiency by converging 4.4x faster than
consistency models and 1.7x faster than rectified flow models while achieving
better generation quality. Our code is available at:
https://github.com/YangLing0818/consistency_flow_matching
FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized
  Sounds
Authors: name: Yiming Zhang, name: Yicheng Gu, name: Yanhong Zeng, name: Zhening Xing, name: Yuancheng Wang, name: Zhizheng Wu, name: Kai Chen
Published: 2024-07-01 17:35:56

We study Neural Foley, the automatic generation of high-quality sound effects
synchronizing with videos, enabling an immersive audio-visual experience.
Despite its wide range of applications, existing approaches encounter
limitations when it comes to simultaneously synthesizing high-quality and
video-aligned (i.e.,, semantic relevant and temporal synchronized) sounds. To
overcome these limitations, we propose FoleyCrafter, a novel framework that
leverages a pre-trained text-to-audio model to ensure high-quality audio
generation. FoleyCrafter comprises two key components: the semantic adapter for
semantic alignment and the temporal controller for precise audio-video
synchronization. The semantic adapter utilizes parallel cross-attention layers
to condition audio generation on video features, producing realistic sound
effects that are semantically relevant to the visual content. Meanwhile, the
temporal controller incorporates an onset detector and a timestampbased adapter
to achieve precise audio-video alignment. One notable advantage of FoleyCrafter
is its compatibility with text prompts, enabling the use of text descriptions
to achieve controllable and diverse video-to-audio generation according to user
intents. We conduct extensive quantitative and qualitative experiments on
standard benchmarks to verify the effectiveness of FoleyCrafter. Models and
codes are available at https://github.com/open-mmlab/FoleyCrafter.
The SIFo Benchmark: Investigating the Sequential Instruction Following
  Ability of Large Language Models
Authors: name: Xinyi Chen, name: Baohao Liao, name: Jirui Qi, name: Panagiotis Eustratiadis, name: Christof Monz, name: Arianna Bisazza, name: Maarten de Rijke
Published: 2024-06-28 15:34:26

Following multiple instructions is a crucial ability for large language
models (LLMs). Evaluating this ability comes with significant challenges: (i)
limited coherence between multiple instructions, (ii) positional bias where the
order of instructions affects model performance, and (iii) a lack of
objectively verifiable tasks. To address these issues, we introduce a benchmark
designed to evaluate models' abilities to follow multiple instructions through
sequential instruction following (SIFo) tasks. In SIFo, the successful
completion of multiple instructions is verifiable by examining only the final
instruction. Our benchmark evaluates instruction following using four tasks
(text modification, question answering, mathematics, and security rule
following), each assessing different aspects of sequential instruction
following. Our evaluation of popular LLMs, both closed-source and open-source,
shows that more recent and larger models significantly outperform their older
and smaller counterparts on the SIFo tasks, validating the benchmark's
effectiveness. All models struggle with following sequences of instructions,
hinting at an important lack of robustness of today's language models.