Orionnnn/hub-stats huggingface dataset:
Prompt2Perturb (P2P): Text-Guided Diffusion-Based Adversarial Attacks on
  Breast Ultrasound Images
Authors: name: Yasamin Medghalchi, name: Moein Heidari, name: Clayton Allard, name: Leonid Sigal, name: Ilker Hacihaliloglu
Published: 2024-12-13 06:56:12

Deep neural networks (DNNs) offer significant promise for improving breast
cancer diagnosis in medical imaging. However, these models are highly
susceptible to adversarial attacks--small, imperceptible changes that can
mislead classifiers--raising critical concerns about their reliability and
security. Traditional attacks rely on fixed-norm perturbations, misaligning
with human perception. In contrast, diffusion-based attacks require pre-trained
models, demanding substantial data when these models are unavailable, limiting
practical use in data-scarce scenarios. In medical imaging, however, this is
often unfeasible due to the limited availability of datasets. Building on
recent advancements in learnable prompts, we propose Prompt2Perturb (P2P), a
novel language-guided attack method capable of generating meaningful attack
examples driven by text instructions. During the prompt learning phase, our
approach leverages learnable prompts within the text encoder to create subtle,
yet impactful, perturbations that remain imperceptible while guiding the model
towards targeted outcomes. In contrast to current prompt learning-based
approaches, our P2P stands out by directly updating text embeddings, avoiding
the need for retraining diffusion models. Further, we leverage the finding that
optimizing only the early reverse diffusion steps boosts efficiency while
ensuring that the generated adversarial examples incorporate subtle noise, thus
preserving ultrasound image quality without introducing noticeable artifacts.
We show that our method outperforms state-of-the-art attack techniques across
three breast ultrasound datasets in FID and LPIPS. Moreover, the generated
images are both more natural in appearance and more effective compared to
existing adversarial attacks. Our code will be publicly available
https://github.com/yasamin-med/P2P.
BiMediX2: Bio-Medical EXpert LMM for Diverse Medical Modalities
Authors: name: Sahal Shaji Mullappilly, name: Mohammed Irfan Kurpath, name: Sara Pieri, name: Saeed Yahya Alseiari, name: Shanavas Cholakkal, name: Khaled Aldahmani, name: Fahad Khan, name: Rao Anwer, name: Salman Khan, name: Timothy Baldwin, name: Hisham Cholakkal
Published: 2024-12-10 18:59:35

This paper introduces BiMediX2, a bilingual (Arabic-English) Bio-Medical
EXpert Large Multimodal Model (LMM) with a unified architecture that integrates
text and visual modalities, enabling advanced image understanding and medical
applications. BiMediX2 leverages the Llama3.1 architecture and integrates text
and visual capabilities to facilitate seamless interactions in both English and
Arabic, supporting text-based inputs and multi-turn conversations involving
medical images. The model is trained on an extensive bilingual healthcare
dataset consisting of 1.6M samples of diverse medical interactions for both
text and image modalities, mixed in Arabic and English. We also propose the
first bilingual GPT-4o based medical LMM benchmark named BiMed-MBench. BiMediX2
is benchmarked on both text-based and image-based tasks, achieving
state-of-the-art performance across several medical benchmarks. It outperforms
recent state-of-the-art models in medical LLM evaluation benchmarks. Our model
also sets a new benchmark in multimodal medical evaluations with over 9%
improvement in English and over 20% in Arabic evaluations. Additionally, it
surpasses GPT-4 by around 9% in UPHILL factual accuracy evaluations and excels
in various medical Visual Question Answering, Report Generation, and Report
Summarization tasks. The project page including source code and the trained
model, is available at https://github.com/mbzuai-oryx/BiMediX2.
Multimodal Music Generation with Explicit Bridges and Retrieval
  Augmentation
Authors: name: Baisen Wang, name: Le Zhuo, name: Zhaokai Wang, name: Chenxi Bao, name: Wu Chengjing, name: Xuecheng Nie, name: Jiao Dai, name: Jizhong Han, name: Yue Liao, name: Si Liu
Published: 2024-12-12 16:33:21

Multimodal music generation aims to produce music from diverse input
modalities, including text, videos, and images. Existing methods use a common
embedding space for multimodal fusion. Despite their effectiveness in other
modalities, their application in multimodal music generation faces challenges
of data scarcity, weak cross-modal alignment, and limited controllability. This
paper addresses these issues by using explicit bridges of text and music for
multimodal alignment. We introduce a novel method named Visuals Music Bridge
(VMB). Specifically, a Multimodal Music Description Model converts visual
inputs into detailed textual descriptions to provide the text bridge; a
Dual-track Music Retrieval module that combines broad and targeted retrieval
strategies to provide the music bridge and enable user control. Finally, we
design an Explicitly Conditioned Music Generation framework to generate music
based on the two bridges. We conduct experiments on video-to-music,
image-to-music, text-to-music, and controllable music generation tasks, along
with experiments on controllability. The results demonstrate that VMB
significantly enhances music quality, modality, and customization alignment
compared to previous methods. VMB sets a new standard for interpretable and
expressive multimodal music generation with applications in various multimedia
fields. Demos and code are available at https://github.com/wbs2788/VMB.
Apollo: An Exploration of Video Understanding in Large Multimodal Models
Authors: name: Orr Zohar, name: Xiaohan Wang, name: Yann Dubois, name: Nikhil Mehta, name: Tong Xiao, name: Philippe Hansen-Estruch, name: Licheng Yu, name: Xiaofang Wang, name: Felix Juefei-Xu, name: Ning Zhang, name: Serena Yeung-Levy, name: Xide Xia
Published: 2024-12-13 18:53:24

Despite the rapid integration of video perception capabilities into Large
Multimodal Models (LMMs), the underlying mechanisms driving their video
understanding remain poorly understood. Consequently, many design decisions in
this domain are made without proper justification or analysis. The high
computational cost of training and evaluating such models, coupled with limited
open research, hinders the development of video-LMMs. To address this, we
present a comprehensive study that helps uncover what effectively drives video
understanding in LMMs.
  We begin by critically examining the primary contributors to the high
computational requirements associated with video-LMM research and discover
Scaling Consistency, wherein design and training decisions made on smaller
models and datasets (up to a critical size) effectively transfer to larger
models. Leveraging these insights, we explored many video-specific aspects of
video-LMMs, including video sampling, architectures, data composition, training
schedules, and more. For example, we demonstrated that fps sampling during
training is vastly preferable to uniform frame sampling and which vision
encoders are the best for video representation.
  Guided by these findings, we introduce Apollo, a state-of-the-art family of
LMMs that achieve superior performance across different model sizes. Our models
can perceive hour-long videos efficiently, with Apollo-3B outperforming most
existing 7B models with an impressive 55.1 on LongVideoBench. Apollo-7B is
state-of-the-art compared to 7B LMMs with a 70.9 on MLVU, and 63.3 on
Video-MME.
InstanceCap: Improving Text-to-Video Generation via Instance-aware
  Structured Caption
Authors: name: Tiehan Fan, name: Kepan Nan, name: Rui Xie, name: Penghao Zhou, name: Zhenheng Yang, name: Chaoyou Fu, name: Xiang Li, name: Jian Yang, name: Ying Tai
Published: 2024-12-12 13:48:40

Text-to-video generation has evolved rapidly in recent years, delivering
remarkable results. Training typically relies on video-caption paired data,
which plays a crucial role in enhancing generation performance. However,
current video captions often suffer from insufficient details, hallucinations
and imprecise motion depiction, affecting the fidelity and consistency of
generated videos. In this work, we propose a novel instance-aware structured
caption framework, termed InstanceCap, to achieve instance-level and
fine-grained video caption for the first time. Based on this scheme, we design
an auxiliary models cluster to convert original video into instances to enhance
instance fidelity. Video instances are further used to refine dense prompts
into structured phrases, achieving concise yet precise descriptions.
Furthermore, a 22K InstanceVid dataset is curated for training, and an
enhancement pipeline that tailored to InstanceCap structure is proposed for
inference. Experimental results demonstrate that our proposed InstanceCap
significantly outperform previous models, ensuring high fidelity between
captions and videos while reducing hallucinations.
Large Action Models: From Inception to Implementation
Authors: name: Lu Wang, name: Fangkai Yang, name: Chaoyun Zhang, name: Junting Lu, name: Jiaxu Qian, name: Shilin He, name: Pu Zhao, name: Bo Qiao, name: Ray Huang, name: Si Qin, name: Qisheng Su, name: Jiayi Ye, name: Yudi Zhang, name: Jian-Guang Lou, name: Qingwei Lin, name: Saravan Rajmohan, name: Dongmei Zhang, name: Qi Zhang
Published: 2024-12-13 11:19:56

As AI continues to advance, there is a growing demand for systems that go
beyond language-based assistance and move toward intelligent agents capable of
performing real-world actions. This evolution requires the transition from
traditional Large Language Models (LLMs), which excel at generating textual
responses, to Large Action Models (LAMs), designed for action generation and
execution within dynamic environments. Enabled by agent systems, LAMs hold the
potential to transform AI from passive language understanding to active task
completion, marking a significant milestone in the progression toward
artificial general intelligence.
  In this paper, we present a comprehensive framework for developing LAMs,
offering a systematic approach to their creation, from inception to deployment.
We begin with an overview of LAMs, highlighting their unique characteristics
and delineating their differences from LLMs. Using a Windows OS-based agent as
a case study, we provide a detailed, step-by-step guide on the key stages of
LAM development, including data collection, model training, environment
integration, grounding, and evaluation. This generalizable workflow can serve
as a blueprint for creating functional LAMs in various application domains. We
conclude by identifying the current limitations of LAMs and discussing
directions for future research and industrial deployment, emphasizing the
challenges and opportunities that lie ahead in realizing the full potential of
LAMs in real-world applications.
  The code for the data collection process utilized in this paper is publicly
available at: https://github.com/microsoft/UFO/tree/main/dataflow, and
comprehensive documentation can be found at
https://microsoft.github.io/UFO/dataflow/overview/.
GenEx: Generating an Explorable World
Authors: name: Taiming Lu, name: Tianmin Shu, name: Junfei Xiao, name: Luoxin Ye, name: Jiahao Wang, name: Cheng Peng, name: Chen Wei, name: Daniel Khashabi, name: Rama Chellappa, name: Alan Yuille, name: Jieneng Chen
Published: 2024-12-12 18:59:57

Understanding, navigating, and exploring the 3D physical real world has long
been a central challenge in the development of artificial intelligence. In this
work, we take a step toward this goal by introducing GenEx, a system capable of
planning complex embodied world exploration, guided by its generative
imagination that forms priors (expectations) about the surrounding
environments. GenEx generates an entire 3D-consistent imaginative environment
from as little as a single RGB image, bringing it to life through panoramic
video streams. Leveraging scalable 3D world data curated from Unreal Engine,
our generative model is rounded in the physical world. It captures a continuous
360-degree environment with little effort, offering a boundless landscape for
AI agents to explore and interact with. GenEx achieves high-quality world
generation, robust loop consistency over long trajectories, and demonstrates
strong 3D capabilities such as consistency and active 3D mapping. Powered by
generative imagination of the world, GPT-assisted agents are equipped to
perform complex embodied tasks, including both goal-agnostic exploration and
goal-driven navigation. These agents utilize predictive expectation regarding
unseen parts of the physical world to refine their beliefs, simulate different
outcomes based on potential decisions, and make more informed choices. In
summary, we demonstrate that GenEx provides a transformative platform for
advancing embodied AI in imaginative spaces and brings potential for extending
these capabilities to real-world exploration.
FreeScale: Unleashing the Resolution of Diffusion Models via Tuning-Free
  Scale Fusion
Authors: name: Haonan Qiu, name: Shiwei Zhang, name: Yujie Wei, name: Ruihang Chu, name: Hangjie Yuan, name: Xiang Wang, name: Yingya Zhang, name: Ziwei Liu
Published: 2024-12-12 18:59:59

Visual diffusion models achieve remarkable progress, yet they are typically
trained at limited resolutions due to the lack of high-resolution data and
constrained computation resources, hampering their ability to generate
high-fidelity images or videos at higher resolutions. Recent efforts have
explored tuning-free strategies to exhibit the untapped potential
higher-resolution visual generation of pre-trained models. However, these
methods are still prone to producing low-quality visual content with repetitive
patterns. The key obstacle lies in the inevitable increase in high-frequency
information when the model generates visual content exceeding its training
resolution, leading to undesirable repetitive patterns deriving from the
accumulated errors. To tackle this challenge, we propose FreeScale, a
tuning-free inference paradigm to enable higher-resolution visual generation
via scale fusion. Specifically, FreeScale processes information from different
receptive scales and then fuses it by extracting desired frequency components.
Extensive experiments validate the superiority of our paradigm in extending the
capabilities of higher-resolution visual generation for both image and video
models. Notably, compared with the previous best-performing method, FreeScale
unlocks the generation of 8k-resolution images for the first time.
SCBench: A KV Cache-Centric Analysis of Long-Context Methods
Authors: name: Yucheng Li, name: Huiqiang Jiang, name: Qianhui Wu, name: Xufang Luo, name: Surin Ahn, name: Chengruidong Zhang, name: Amir H. Abdi, name: Dongsheng Li, name: Jianfeng Gao, name: Yuqing Yang, name: Lili Qiu
Published: 2024-12-13 17:59:52

Long-context LLMs have enabled numerous downstream applications but also
introduced significant challenges related to computational and memory
efficiency. To address these challenges, optimizations for long-context
inference have been developed, centered around the KV cache. However, existing
benchmarks often evaluate in single-request, neglecting the full lifecycle of
the KV cache in real-world use. This oversight is particularly critical, as KV
cache reuse has become widely adopted in LLMs inference frameworks, such as
vLLM and SGLang, as well as by LLM providers, including OpenAI, Microsoft,
Google, and Anthropic. To address this gap, we introduce
SCBench(SharedContextBench), a comprehensive benchmark for evaluating
long-context methods from a KV cachecentric perspective: 1) KV cache
generation, 2) KV cache compression, 3) KV cache retrieval, 4) KV cache
loading. Specifically, SCBench uses test examples with shared context, ranging
12 tasks with two shared context modes, covering four categories of
long-context capabilities: string retrieval, semantic retrieval, global
information, and multi-task. With it, we provide an extensive KV cache-centric
analysis of eight categories long-context solutions, including Gated Linear
RNNs, Mamba-Attention hybrids, and efficient methods such as sparse attention,
KV cache dropping, quantization, retrieval, loading, and prompt compression.
The evaluation is conducted on 8 long-context LLMs. Our findings show that
sub-O(n) memory methods suffer in multi-turn scenarios, while sparse encoding
with O(n) memory and sub-O(n^2) pre-filling computation perform robustly.
Dynamic sparsity yields more expressive KV caches than static patterns, and
layer-level sparsity in hybrid architectures reduces memory usage with strong
performance. Additionally, we identify attention distribution shift issues in
long-generation scenarios. https://aka.ms/SCBench.
FireFlow: Fast Inversion of Rectified Flow for Image Semantic Editing
Authors: name: Yingying Deng, name: Xiangyu He, name: Changwang Mei, name: Peisong Wang, name: Fan Tang
Published: 2024-12-10 13:56:26

Though Rectified Flows (ReFlows) with distillation offers a promising way for
fast sampling, its fast inversion transforms images back to structured noise
for recovery and following editing remains unsolved. This paper introduces
FireFlow, a simple yet effective zero-shot approach that inherits the startling
capacity of ReFlow-based models (such as FLUX) in generation while extending
its capabilities to accurate inversion and editing in 8 steps. We first
demonstrate that a carefully designed numerical solver is pivotal for ReFlow
inversion, enabling accurate inversion and reconstruction with the precision of
a second-order solver while maintaining the practical efficiency of a
first-order Euler method. This solver achieves a 3times runtime speedup
compared to state-of-the-art ReFlow inversion and editing techniques, while
delivering smaller reconstruction errors and superior editing results in a
training-free mode. The code is available at
https://github.com/HolmesShuan/FireFlow{this URL}.
Normalizing Flows are Capable Generative Models
Authors: name: Shuangfei Zhai, name: Ruixiang Zhang, name: Preetum Nakkiran, name: David Berthelot, name: Jiatao Gu, name: Huangjie Zheng, name: Tianrong Chen, name: Miguel Angel Bautista, name: Navdeep Jaitly, name: Josh Susskind
Published: 2024-12-09 09:28:06

Normalizing Flows (NFs) are likelihood-based models for continuous inputs.
They have demonstrated promising results on both density estimation and
generative modeling tasks, but have received relatively little attention in
recent years. In this work, we demonstrate that NFs are more powerful than
previously believed. We present TarFlow: a simple and scalable architecture
that enables highly performant NF models. TarFlow can be thought of as a
Transformer-based variant of Masked Autoregressive Flows (MAFs): it consists of
a stack of autoregressive Transformer blocks on image patches, alternating the
autoregression direction between layers. TarFlow is straightforward to train
end-to-end, and capable of directly modeling and generating pixels. We also
propose three key techniques to improve sample quality: Gaussian noise
augmentation during training, a post training denoising procedure, and an
effective guidance method for both class-conditional and unconditional
settings. Putting these together, TarFlow sets new state-of-the-art results on
likelihood estimation for images, beating the previous best methods by a large
margin, and generates samples with quality and diversity comparable to
diffusion models, for the first time with a stand-alone NF model. We make our
code available at https://github.com/apple/ml-tarflow.
VisionArena: 230K Real World User-VLM Conversations with Preference
  Labels
Authors: name: Christopher Chou, name: Lisa Dunlap, name: Koki Mashita, name: Krishna Mandal, name: Trevor Darrell, name: Ion Stoica, name: Joseph E. Gonzalez, name: Wei-Lin Chiang
Published: 2024-12-11 18:59:46

With the growing adoption and capabilities of vision-language models (VLMs)
comes the need for benchmarks that capture authentic user-VLM interactions. In
response, we create VisionArena, a dataset of 230K real-world conversations
between users and VLMs. Collected from Chatbot Arena - an open-source platform
where users interact with VLMs and submit preference votes - VisionArena spans
73K unique users, 45 VLMs, and 138 languages. Our dataset contains three
subsets: VisionArena-Chat, 200k single and multi-turn conversations between a
user and a VLM; VisionArena-Battle, 30K conversations comparing two anonymous
VLMs with user preference votes; and VisionArena-Bench, an automatic benchmark
of 500 diverse user prompts that efficiently approximate the live Chatbot Arena
model rankings. Additionally, we highlight the types of question asked by
users, the influence of response style on preference, and areas where models
often fail. We find open-ended tasks like captioning and humor are highly
style-dependent, and current VLMs struggle with spatial reasoning and planning
tasks. Lastly, we show finetuning the same base model on VisionArena-Chat
outperforms Llava-Instruct-158K, with a 17-point gain on MMMU and a 46-point
gain on the WildVision benchmark. Dataset at https://huggingface.co/lmarena-ai
ONEBench to Test Them All: Sample-Level Benchmarking Over Open-Ended
  Capabilities
Authors: name: Adhiraj Ghosh, name: Sebastian Dziadzio, name: Ameya Prabhu, name: Vishaal Udandarao, name: Samuel Albanie, name: Matthias Bethge
Published: 2024-12-09 18:37:14

Traditional fixed test sets fall short in evaluating open-ended capabilities
of foundation models. To address this, we propose ONEBench(OpeN-Ended
Benchmarking), a new testing paradigm that consolidates individual evaluation
datasets into a unified, ever-expanding sample pool. ONEBench allows users to
generate custom, open-ended evaluation benchmarks from this pool, corresponding
to specific capabilities of interest. By aggregating samples across test sets,
ONEBench enables the assessment of diverse capabilities beyond those covered by
the original test sets, while mitigating overfitting and dataset bias. Most
importantly, it frames model evaluation as a collective process of selecting
and aggregating sample-level tests.
  The shift from task-specific benchmarks to ONEBench introduces two
challenges: (1)heterogeneity and (2)incompleteness. Heterogeneity refers to the
aggregation over diverse metrics, while incompleteness describes comparing
models evaluated on different data subsets. To address these challenges, we
explore algorithms to aggregate sparse measurements into reliable model scores.
Our aggregation algorithm ensures identifiability(asymptotically recovering
ground-truth scores) and rapid convergence, enabling accurate model ranking
with less data. On homogenous datasets, we show our aggregation algorithm
provides rankings that highly correlate with those produced by average scores.
We also demonstrate robustness to ~95% of measurements missing, reducing
evaluation cost by up to 20x with little-to-no change in model rankings. We
introduce ONEBench-LLM for language models and ONEBench-LMM for vision-language
models, unifying evaluations across these domains. Overall, we present a
technique for open-ended evaluation, which can aggregate over incomplete,
heterogeneous sample-level measurements to continually grow a benchmark
alongside the rapidly developing foundation models.