Orionnnn/hub-stats huggingface dataset:
SynFinTabs: A Dataset of Synthetic Financial Tables for Information and
  Table Extraction
Authors: name: Ethan Bradley, name: Muhammad Roman, name: Karen Rafferty, name: Barry Devereux
Published: 2024-12-05 15:42:59

Table extraction from document images is a challenging AI problem, and
labelled data for many content domains is difficult to come by. Existing table
extraction datasets often focus on scientific tables due to the vast amount of
academic articles that are readily available, along with their source code.
However, there are significant layout and typographical differences between
tables found across scientific, financial, and other domains. Current datasets
often lack the words, and their positions, contained within the tables, instead
relying on unreliable OCR to extract these features for training modern machine
learning models on natural language processing tasks. Therefore, there is a
need for a more general method of obtaining labelled data. We present
SynFinTabs, a large-scale, labelled dataset of synthetic financial tables. Our
hope is that our method of generating these synthetic tables is transferable to
other domains. To demonstrate the effectiveness of our dataset in training
models to extract information from table images, we create FinTabQA, a layout
large language model trained on an extractive question-answering task. We test
our model using real-world financial tables and compare it to a
state-of-the-art generative model and discuss the results. We make the dataset,
model, and dataset generation code publicly available.
Scaling Inference-Time Search with Vision Value Model for Improved
  Visual Comprehension
Authors: name: Wang Xiyao, name: Yang Zhengyuan, name: Li Linjie, name: Lu Hongjin, name: Xu Yuancheng, name: Lin Chung-Ching Lin, name: Lin Kevin, name: Huang Furong, name: Wang Lijuan
Published: 2024-12-04 20:35:07

Despite significant advancements in vision-language models (VLMs), there
lacks effective approaches to enhance response quality by scaling
inference-time computation. This capability is known to be a core step towards
the self-improving models in recent large language model studies. In this
paper, we present Vision Value Model (VisVM) that can guide VLM inference-time
search to generate responses with better visual comprehension. Specifically,
VisVM not only evaluates the generated sentence quality in the current search
step, but also anticipates the quality of subsequent sentences that may result
from the current step, thus providing a long-term value. In this way, VisVM
steers VLMs away from generating sentences prone to hallucinations or
insufficient detail, thereby producing higher quality responses. Experimental
results demonstrate that VisVM-guided search significantly enhances VLMs'
ability to generate descriptive captions with richer visual details and fewer
hallucinations, compared with greedy decoding and search methods with other
visual reward signals. Furthermore, we find that self-training the model with
the VisVM-guided captions improve VLM's performance across a wide range of
multimodal benchmarks, indicating the potential for developing self-improving
VLMs. Our value model and code are available at
https://github.com/si0wang/VisVM.
MEMO: Memory-Guided Diffusion for Expressive Talking Video Generation
Authors: name: Longtao Zheng, name: Yifan Zhang, name: Hanzhong Guo, name: Jiachun Pan, name: Zhenxiong Tan, name: Jiahao Lu, name: Chuanxin Tang, name: Bo An, name: Shuicheng Yan
Published: 2024-12-05 18:57:26

Recent advances in video diffusion models have unlocked new potential for
realistic audio-driven talking video generation. However, achieving seamless
audio-lip synchronization, maintaining long-term identity consistency, and
producing natural, audio-aligned expressions in generated talking videos remain
significant challenges. To address these challenges, we propose Memory-guided
EMOtion-aware diffusion (MEMO), an end-to-end audio-driven portrait animation
approach to generate identity-consistent and expressive talking videos. Our
approach is built around two key modules: (1) a memory-guided temporal module,
which enhances long-term identity consistency and motion smoothness by
developing memory states to store information from a longer past context to
guide temporal modeling via linear attention; and (2) an emotion-aware audio
module, which replaces traditional cross attention with multi-modal attention
to enhance audio-video interaction, while detecting emotions from audio to
refine facial expressions via emotion adaptive layer norm. Extensive
quantitative and qualitative results demonstrate that MEMO generates more
realistic talking videos across diverse image and audio types, outperforming
state-of-the-art methods in overall quality, audio-lip synchronization,
identity consistency, and expression-emotion alignment.
Challenges in Trustworthy Human Evaluation of Chatbots
Authors: name: Wenting Zhao, name: Alexander M. Rush, name: Tanya Goyal
Published: 2024-12-05 17:22:04

Open community-driven platforms like Chatbot Arena that collect user
preference data from site visitors have gained a reputation as one of the most
trustworthy publicly available benchmarks for LLM performance. While now
standard, it is tricky to implement effective guardrails to collect
high-quality annotations from humans. In this paper, we demonstrate that three
sources of bad annotations, both malicious and otherwise, can corrupt the
reliability of open leaderboard rankings. In particular, we show that only 10\%
of poor quality votes by apathetic (site visitors not appropriately
incentivized to give correct votes) or adversarial (bad actors seeking to
inflate the ranking of a target model) annotators can change the rankings of
models by up to 5 places on the leaderboard. Finally, we discuss open
challenges in ensuring high-quality human annotations.
Discriminative Fine-tuning of LVLMs
Authors: name: Yassine Ouali, name: Adrian Bulat, name: Alexandros Xenos, name: Anestis Zaganidis, name: Ioannis Maniadis Metaxas, name: Georgios Tzimiropoulos, name: Brais Martinez
Published: 2024-12-05 17:54:27

Contrastively-trained Vision-Language Models (VLMs) like CLIP have become the
de facto approach for discriminative vision-language representation learning.
However, these models have limited language understanding, often exhibiting a
"bag of words" behavior. At the same time, Large Vision-Language Models
(LVLMs), which combine vision encoders with LLMs, have been shown capable of
detailed vision-language reasoning, yet their autoregressive nature renders
them less suitable for discriminative tasks.
  In this work, we propose to combine "the best of both worlds": a new training
approach for discriminative fine-tuning of LVLMs that results in strong
discriminative and compositional capabilities. Essentially, our approach
converts a generative LVLM into a discriminative one, unlocking its capability
for powerful image-text discrimination combined with enhanced language
understanding.
  Our contributions include: (1) A carefully designed training/optimization
framework that utilizes image-text pairs of variable length and granularity for
training the model with both contrastive and next-token prediction losses. This
is accompanied by ablation studies that justify the necessity of our
framework's components. (2) A parameter-efficient adaptation method using a
combination of soft prompting and LoRA adapters. (3) Significant improvements
over state-of-the-art CLIP-like models of similar size, including standard
image-text retrieval benchmarks and notable gains in compositionality.
AnyDressing: Customizable Multi-Garment Virtual Dressing via Latent
  Diffusion Models
Authors: name: Xinghui Li, name: Qichao Sun, name: Pengze Zhang, name: Fulong Ye, name: Zhichao Liao, name: Wanquan Feng, name: Songtao Zhao, name: Qian He
Published: 2024-12-05 13:16:47

Recent advances in garment-centric image generation from text and image
prompts based on diffusion models are impressive. However, existing methods
lack support for various combinations of attire, and struggle to preserve the
garment details while maintaining faithfulness to the text prompts, limiting
their performance across diverse scenarios. In this paper, we focus on a new
task, i.e., Multi-Garment Virtual Dressing, and we propose a novel AnyDressing
method for customizing characters conditioned on any combination of garments
and any personalized text prompts. AnyDressing comprises two primary networks
named GarmentsNet and DressingNet, which are respectively dedicated to
extracting detailed clothing features and generating customized images.
Specifically, we propose an efficient and scalable module called
Garment-Specific Feature Extractor in GarmentsNet to individually encode
garment textures in parallel. This design prevents garment confusion while
ensuring network efficiency. Meanwhile, we design an adaptive
Dressing-Attention mechanism and a novel Instance-Level Garment Localization
Learning strategy in DressingNet to accurately inject multi-garment features
into their corresponding regions. This approach efficiently integrates
multi-garment texture cues into generated images and further enhances
text-image consistency. Additionally, we introduce a Garment-Enhanced Texture
Learning strategy to improve the fine-grained texture details of garments.
Thanks to our well-craft design, AnyDressing can serve as a plug-in module to
easily integrate with any community control extensions for diffusion models,
improving the diversity and controllability of synthesized images. Extensive
experiments show that AnyDressing achieves state-of-the-art results.
Global MMLU: Understanding and Addressing Cultural and Linguistic Biases
  in Multilingual Evaluation
Authors: name: Shivalika Singh, name: Angelika Romanou, name: Clémentine Fourrier, name: David I. Adelani, name: Jian Gang Ngui, name: Daniel Vila-Suero, name: Peerat Limkonchotiwat, name: Kelly Marchisio, name: Wei Qi Leong, name: Yosephine Susanto, name: Raymond Ng, name: Shayne Longpre, name: Wei-Yin Ko, name: Madeline Smith, name: Antoine Bosselut, name: Alice Oh, name: Andre F. T. Martins, name: Leshem Choshen, name: Daphne Ippolito, name: Enzo Ferrante, name: Marzieh Fadaee, name: Beyza Ermis, name: Sara Hooker
Published: 2024-12-04 13:27:09

Cultural biases in multilingual datasets pose significant challenges for
their effectiveness as global benchmarks. These biases stem not only from
language but also from the cultural knowledge required to interpret questions,
reducing the practical utility of translated datasets like MMLU. Furthermore,
translation often introduces artifacts that can distort the meaning or clarity
of questions in the target language. A common practice in multilingual
evaluation is to rely on machine-translated evaluation sets, but simply
translating a dataset is insufficient to address these challenges. In this
work, we trace the impact of both of these issues on multilingual evaluations
and ensuing model performances. Our large-scale evaluation of state-of-the-art
open and proprietary models illustrates that progress on MMLU depends heavily
on learning Western-centric concepts, with 28% of all questions requiring
culturally sensitive knowledge. Moreover, for questions requiring geographic
knowledge, an astounding 84.9% focus on either North American or European
regions. Rankings of model evaluations change depending on whether they are
evaluated on the full portion or the subset of questions annotated as
culturally sensitive, showing the distortion to model rankings when blindly
relying on translated MMLU. We release Global-MMLU, an improved MMLU with
evaluation coverage across 42 languages -- with improved overall quality by
engaging with compensated professional and community annotators to verify
translation quality while also rigorously evaluating cultural biases present in
the original dataset. This comprehensive Global-MMLU set also includes
designated subsets labeled as culturally sensitive and culturally agnostic to
allow for more holistic, complete evaluation.
KV Shifting Attention Enhances Language Modeling
Authors: name: Mingyu Xu, name: Wei Cheng, name: Bingning Wang, name: Weipeng Chen
Published: 2024-11-29 09:42:38

The current large language models are mainly based on decode-only structure
transformers, which have great in-context learning (ICL) capabilities. It is
generally believed that the important foundation of its ICL capability is the
induction heads mechanism, which requires at least two layers attention. In
order to more efficiently implement the ability of the model's induction, we
revisit the induction heads mechanism and proposed a KV shifting attention. We
theoretically prove that the KV shifting attention reducing the model's
requirements for the depth and width of the induction heads mechanism. Our
experimental results demonstrate that KV shifting attention is beneficial to
learning induction heads and language modeling, which lead to better
performance or faster convergence from toy models to the pre-training models
with more than 10 B parameters.
A Noise is Worth Diffusion Guidance
Authors: name: Donghoon Ahn, name: Jiwon Kang, name: Sanghyun Lee, name: Jaewon Min, name: Minjae Kim, name: Wooseok Jang, name: Hyoungwon Cho, name: Sayak Paul, name: SeonHwa Kim, name: Eunju Cha, name: Kyong Hwan Jin, name: Seungryong Kim
Published: 2024-12-05 06:09:56

Diffusion models excel in generating high-quality images. However, current
diffusion models struggle to produce reliable images without guidance methods,
such as classifier-free guidance (CFG). Are guidance methods truly necessary?
Observing that noise obtained via diffusion inversion can reconstruct
high-quality images without guidance, we focus on the initial noise of the
denoising pipeline. By mapping Gaussian noise to `guidance-free noise', we
uncover that small low-magnitude low-frequency components significantly enhance
the denoising process, removing the need for guidance and thus improving both
inference throughput and memory. Expanding on this, we propose \ours, a novel
method that replaces guidance methods with a single refinement of the initial
noise. This refined noise enables high-quality image generation without
guidance, within the same diffusion pipeline. Our noise-refining model
leverages efficient noise-space learning, achieving rapid convergence and
strong performance with just 50K text-image pairs. We validate its
effectiveness across diverse metrics and analyze how refined noise can
eliminate the need for guidance. See our project page:
https://cvlab-kaist.github.io/NoiseRefine/.
Marco-LLM: Bridging Languages via Massive Multilingual Training for
  Cross-Lingual Enhancement
Authors: name: Lingfeng Ming, name: Bo Zeng, name: Chenyang Lyu, name: Tianqi Shi, name: Yu Zhao, name: Xue Yang, name: Yefeng Liu, name: Yiyu Wang, name: Linlong Xu, name: Yangyang Liu, name: Xiaohu Zhao, name: Hao Wang, name: Heng Liu, name: Hao Zhou, name: Huifeng Yin, name: Zifu Shang, name: Haijun Li, name: Longyue Wang, name: Weihua Luo, name: Kaifu Zhang
Published: 2024-12-05 09:26:58

Large Language Models (LLMs) have achieved remarkable progress in recent
years; however, their excellent performance is still largely limited to major
world languages, primarily English. Many LLMs continue to face challenges with
multilingual tasks, especially when it comes to low-resource languages. To
address this issue, we introduced Marco-LLM: Massive multilingual training for
cross-lingual enhancement LLM. We have collected a substantial amount of
multilingual data for several low-resource languages and conducted extensive
continual pre-training using the Qwen2 models. This effort has resulted in a
multilingual LLM named Marco-LLM. Through comprehensive evaluations on various
multilingual benchmarks, including MMMLU, AGIEval, Belebele, Flores-200, XCOPA
and many others, Marco-LLM has demonstrated substantial improvements over
state-of-the-art LLMs. Furthermore, Marco-LLM achieved substantial enhancements
in any-to-any machine translation tasks, showing the effectiveness of our
multilingual LLM. Marco-LLM is a pioneering multilingual LLM designed to not
only perform exceptionally well in multilingual tasks, including low-resource
languages, but also maintain strong performance in English and other major
languages, closing the performance gap between high- and low-resource language
capabilities. By bridging languages, this effort demonstrates our dedication to
ensuring LLMs work accurately across various languages.
Personalized Multimodal Large Language Models: A Survey
Authors: name: Junda Wu, name: Hanjia Lyu, name: Yu Xia, name: Zhehao Zhang, name: Joe Barrow, name: Ishita Kumar, name: Mehrnoosh Mirtaheri, name: Hongjie Chen, name: Ryan A. Rossi, name: Franck Dernoncourt, name: Tong Yu, name: Ruiyi Zhang, name: Jiuxiang Gu, name: Nesreen K. Ahmed, name: Yu Wang, name: Xiang Chen, name: Hanieh Deilamsalehy, name: Namyong Park, name: Sungchul Kim, name: Huanrui Yang, name: Subrata Mitra, name: Zhengmian Hu, name: Nedim Lipka, name: Dang Nguyen, name: Yue Zhao, name: Jiebo Luo, name: Julian McAuley
Published: 2024-12-03 03:59:03

Multimodal Large Language Models (MLLMs) have become increasingly important
due to their state-of-the-art performance and ability to integrate multiple
data modalities, such as text, images, and audio, to perform complex tasks with
high accuracy. This paper presents a comprehensive survey on personalized
multimodal large language models, focusing on their architecture, training
methods, and applications. We propose an intuitive taxonomy for categorizing
the techniques used to personalize MLLMs to individual users, and discuss the
techniques accordingly. Furthermore, we discuss how such techniques can be
combined or adapted when appropriate, highlighting their advantages and
underlying rationale. We also provide a succinct summary of personalization
tasks investigated in existing research, along with the evaluation metrics
commonly used. Additionally, we summarize the datasets that are useful for
benchmarking personalized MLLMs. Finally, we outline critical open challenges.
This survey aims to serve as a valuable resource for researchers and
practitioners seeking to understand and advance the development of personalized
multimodal large language models.
Monet: Mixture of Monosemantic Experts for Transformers
Authors: name: Jungwoo Park, name: Young Jin Ahn, name: Kee-Eung Kim, name: Jaewoo Kang
Published: 2024-12-05 13:06:03

Understanding the internal computations of large language models (LLMs) is
crucial for aligning them with human values and preventing undesirable
behaviors like toxic content generation. However, mechanistic interpretability
is hindered by polysemanticity -- where individual neurons respond to multiple,
unrelated concepts. While Sparse Autoencoders (SAEs) have attempted to
disentangle these features through sparse dictionary learning, they have
compromised LLM performance due to reliance on post-hoc reconstruction loss. To
address this issue, we introduce Mixture of Monosemantic Experts for
Transformers (Monet) architecture, which incorporates sparse dictionary
learning directly into end-to-end Mixture-of-Experts pretraining. Our novel
expert decomposition method enables scaling the expert count to 262,144 per
layer while total parameters scale proportionally to the square root of the
number of experts. Our analyses demonstrate mutual exclusivity of knowledge
across experts and showcase the parametric knowledge encapsulated within
individual experts. Moreover, Monet allows knowledge manipulation over domains,
languages, and toxicity mitigation without degrading general performance. Our
pursuit of transparent LLMs highlights the potential of scaling expert counts
to enhance} mechanistic interpretability and directly resect the internal
knowledge to fundamentally adjust} model behavior. The source code and
pretrained checkpoints are available at https://github.com/dmis-lab/Monet.
Negative Token Merging: Image-based Adversarial Feature Guidance
Authors: name: Jaskirat Singh, name: Lindsey Li, name: Weijia Shi, name: Ranjay Krishna, name: Yejin Choi, name: Pang Wei Koh, name: Michael F. Cohen, name: Stephen Gould, name: Liang Zheng, name: Luke Zettlemoyer
Published: 2024-12-02 10:06:57

Text-based adversarial guidance using a negative prompt has emerged as a
widely adopted approach to push the output features away from undesired
concepts. While useful, performing adversarial guidance using text alone can be
insufficient to capture complex visual concepts and avoid undesired visual
elements like copyrighted characters. In this paper, for the first time we
explore an alternate modality in this direction by performing adversarial
guidance directly using visual features from a reference image or other images
in a batch. In particular, we introduce negative token merging (NegToMe), a
simple but effective training-free approach which performs adversarial guidance
by selectively pushing apart matching semantic features (between reference and
output generation) during the reverse diffusion process. When used w.r.t. other
images in the same batch, we observe that NegToMe significantly increases
output diversity (racial, gender, visual) without sacrificing output image
quality. Similarly, when used w.r.t. a reference copyrighted asset, NegToMe
helps reduce visual similarity with copyrighted content by 34.57%. NegToMe is
simple to implement using just few-lines of code, uses only marginally higher
(<4%) inference times and generalizes to different diffusion architectures like
Flux, which do not natively support the use of a separate negative prompt. Code
is available at https://negtome.github.io
Code-as-Monitor: Constraint-aware Visual Programming for Reactive and
  Proactive Robotic Failure Detection
Authors: name: Enshen Zhou, name: Qi Su, name: Cheng Chi, name: Zhizheng Zhang, name: Zhongyuan Wang, name: Tiejun Huang, name: Lu Sheng, name: He Wang
Published: 2024-12-05 18:58:27

Automatic detection and prevention of open-set failures are crucial in
closed-loop robotic systems. Recent studies often struggle to simultaneously
identify unexpected failures reactively after they occur and prevent
foreseeable ones proactively. To this end, we propose Code-as-Monitor (CaM), a
novel paradigm leveraging the vision-language model (VLM) for both open-set
reactive and proactive failure detection. The core of our method is to
formulate both tasks as a unified set of spatio-temporal constraint
satisfaction problems and use VLM-generated code to evaluate them for real-time
monitoring. To enhance the accuracy and efficiency of monitoring, we further
introduce constraint elements that abstract constraint-related entities or
their parts into compact geometric elements. This approach offers greater
generality, simplifies tracking, and facilitates constraint-aware visual
programming by leveraging these elements as visual prompts. Experiments show
that CaM achieves a 28.7% higher success rate and reduces execution time by
31.8% under severe disturbances compared to baselines across three simulators
and a real-world setting. Moreover, CaM can be integrated with open-loop
control policies to form closed-loop systems, enabling long-horizon tasks in
cluttered scenes with dynamic environments.