Orionnnn/hub-stats huggingface dataset:
MedINST: Meta Dataset of Biomedical Instructions
Authors: name: Wenhan Han, name: Meng Fang, name: Zihan Zhang, name: Yu Yin, name: Zirui Song, name: Ling Chen, name: Mykola Pechenizkiy, name: Qingyu Chen
Published: 2024-10-17 11:38:54

The integration of large language model (LLM) techniques in the field of
medical analysis has brought about significant advancements, yet the scarcity
of large, diverse, and well-annotated datasets remains a major challenge.
Medical data and tasks, which vary in format, size, and other parameters,
require extensive preprocessing and standardization for effective use in
training LLMs. To address these challenges, we introduce MedINST, the Meta
Dataset of Biomedical Instructions, a novel multi-domain, multi-task
instructional meta-dataset. MedINST comprises 133 biomedical NLP tasks and over
7 million training samples, making it the most comprehensive biomedical
instruction dataset to date. Using MedINST as the meta dataset, we curate
MedINST32, a challenging benchmark with different task difficulties aiming to
evaluate LLMs' generalization ability. We fine-tune several LLMs on MedINST and
evaluate on MedINST32, showcasing enhanced cross-task generalization.
ARKit LabelMaker: A New Scale for Indoor 3D Scene Understanding
Authors: name: Guangda Ji, name: Silvan Weder, name: Francis Engelmann, name: Marc Pollefeys, name: Hermann Blum
Published: 2024-10-17 14:44:35

The performance of neural networks scales with both their size and the amount
of data they have been trained on. This is shown in both language and image
generation. However, this requires scaling-friendly network architectures as
well as large-scale datasets. Even though scaling-friendly architectures like
transformers have emerged for 3D vision tasks, the GPT-moment of 3D vision
remains distant due to the lack of training data. In this paper, we introduce
ARKit LabelMaker, the first large-scale, real-world 3D dataset with dense
semantic annotations. Specifically, we complement ARKitScenes dataset with
dense semantic annotations that are automatically generated at scale. To this
end, we extend LabelMaker, a recent automatic annotation pipeline, to serve the
needs of large-scale pre-training. This involves extending the pipeline with
cutting-edge segmentation models as well as making it robust to the challenges
of large-scale processing. Further, we push forward the state-of-the-art
performance on ScanNet and ScanNet200 dataset with prevalent 3D semantic
segmentation models, demonstrating the efficacy of our generated dataset.
M-RewardBench: Evaluating Reward Models in Multilingual Settings
Authors: name: Srishti Gureja, name: Lester James V. Miranda, name: Shayekh Bin Islam, name: Rishabh Maheshwary, name: Drishti Sharma, name: Gusti Winata, name: Nathan Lambert, name: Sebastian Ruder, name: Sara Hooker, name: Marzieh Fadaee
Published: 2024-10-20 22:09:44

Reward models (RMs) have driven the state-of-the-art performance of LLMs
today by enabling the integration of human feedback into the language modeling
process. However, RMs are primarily trained and evaluated in English, and their
capabilities in multilingual settings remain largely understudied. In this
work, we conduct a systematic evaluation of several reward models in
multilingual settings. We first construct the first-of-its-kind multilingual RM
evaluation benchmark, M-RewardBench, consisting of 2.87k preference instances
for 23 typologically diverse languages, that tests the chat, safety, reasoning,
and translation capabilities of RMs. We then rigorously evaluate a wide range
of reward models on M-RewardBench, offering fresh insights into their
performance across diverse languages. We identify a significant gap in RMs'
performances between English and non-English languages and show that RM
preferences can change substantially from one language to another. We also
present several findings on how different multilingual aspects impact RM
performance. Specifically, we show that the performance of RMs is improved with
improved translation quality. Similarly, we demonstrate that the models exhibit
better performance for high-resource languages. We release M-RewardBench
dataset and the codebase in this study to facilitate a better understanding of
RM evaluation in multilingual settings.
Scalable Ranked Preference Optimization for Text-to-Image Generation
Authors: name: Shyamgopal Karthik, name: Huseyin Coskun, name: Zeynep Akata, name: Sergey Tulyakov, name: Jian Ren, name: Anil Kag
Published: 2024-10-23 16:42:56

Direct Preference Optimization (DPO) has emerged as a powerful approach to
align text-to-image (T2I) models with human feedback. Unfortunately, successful
application of DPO to T2I models requires a huge amount of resources to collect
and label large-scale datasets, e.g., millions of generated paired images
annotated with human preferences. In addition, these human preference datasets
can get outdated quickly as the rapid improvements of T2I models lead to higher
quality images. In this work, we investigate a scalable approach for collecting
large-scale and fully synthetic datasets for DPO training. Specifically, the
preferences for paired images are generated using a pre-trained reward
function, eliminating the need for involving humans in the annotation process,
greatly improving the dataset collection efficiency. Moreover, we demonstrate
that such datasets allow averaging predictions across multiple models and
collecting ranked preferences as opposed to pairwise preferences. Furthermore,
we introduce RankDPO to enhance DPO-based methods using the ranking feedback.
Applying RankDPO on SDXL and SD3-Medium models with our synthetically generated
preference dataset ``Syn-Pic'' improves both prompt-following (on benchmarks
like T2I-Compbench, GenEval, and DPG-Bench) and visual quality (through user
studies). This pipeline presents a practical and scalable solution to develop
better preference datasets to enhance the performance of text-to-image models.
TP-Eval: Tap Multimodal LLMs' Potential in Evaluation by Customizing
  Prompts
Authors: name: Yuxuan Xie, name: Tianhua Li, name: Wenqi Shao, name: Kaipeng Zhang
Published: 2024-10-23 17:54:43

Recently, multimodal large language models (MLLMs) have received much
attention for their impressive capabilities. The evaluation of MLLMs is
becoming critical to analyzing attributes of MLLMs and providing valuable
insights. However, current benchmarks overlook the problem of prompt
sensitivity - minor prompt variations may lead to significant performance
fluctuations. Thus, inappropriate prompts may obscure the models' capabilities,
underestimating the models' performance. Moreover, different models have
different preferences for different prompts, and thus, using the same prompt
for all models will cause evaluation bias. This paper analyzes this deficiency
in existing benchmarks and further introduces a new evaluation framework named
TP-Eval, which introduces a prompt customization method to reduce evaluation
biases and tap models' potential. TP-Eval will rewrite the original prompts to
different customized prompts for different models. In particular, we propose
some well-designed modules for prompt customization tailored to the scenario of
MLLM evaluation. Extensive experiments demonstrate the effectiveness of our
approach to uncovering models' capabilities, and TP-Eval should benefit the
community in developing more comprehensive and convincing MLLM evaluation
benchmarks.
Lightweight Neural App Control
Authors: name: Filippos Christianos, name: Georgios Papoudakis, name: Thomas Coste, name: Jianye Hao, name: Jun Wang, name: Kun Shao
Published: 2024-10-23 13:57:00

This paper introduces a novel mobile phone control architecture, termed ``app
agents", for efficient interactions and controls across various Android apps.
The proposed Lightweight Multi-modal App Control (LiMAC) takes as input a
textual goal and a sequence of past mobile observations, such as screenshots
and corresponding UI trees, to generate precise actions. To address the
computational constraints inherent to smartphones, within LiMAC, we introduce a
small Action Transformer (AcT) integrated with a fine-tuned vision-language
model (VLM) for real-time decision-making and task execution. We evaluate LiMAC
on two open-source mobile control datasets, demonstrating the superior
performance of our small-form-factor approach against fine-tuned versions of
open-source VLMs, such as Florence2 and Qwen2-VL. It also significantly
outperforms prompt engineering baselines utilising closed-source foundation
models like GPT-4o. More specifically, LiMAC increases the overall action
accuracy by up to 19% compared to fine-tuned VLMs, and up to 42% compared to
prompt-engineering baselines.
MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large
  Vision-Language Models
Authors: name: Ziyu Liu, name: Yuhang Zang, name: Xiaoyi Dong, name: Pan Zhang, name: Yuhang Cao, name: Haodong Duan, name: Conghui He, name: Yuanjun Xiong, name: Dahua Lin, name: Jiaqi Wang
Published: 2024-10-23 07:56:48

Visual preference alignment involves training Large Vision-Language Models
(LVLMs) to predict human preferences between visual inputs. This is typically
achieved by using labeled datasets of chosen/rejected pairs and employing
optimization algorithms like direct preference optimization (DPO). Existing
visual alignment methods, primarily designed for single-image scenarios,
struggle to effectively handle the complexity of multi-image tasks due to the
scarcity of diverse training data and the high cost of annotating
chosen/rejected pairs. We present Multi-Image Augmented Direct Preference
Optimization (MIA-DPO), a visual preference alignment approach that effectively
handles multi-image inputs. MIA-DPO mitigates the scarcity of diverse
multi-image training data by extending single-image data with unrelated images
arranged in grid collages or pic-in-pic formats, significantly reducing the
costs associated with multi-image data annotations. Our observation reveals
that attention values of LVLMs vary considerably across different images. We
use attention values to identify and filter out rejected responses the model
may have mistakenly focused on. Our attention-aware selection for constructing
the chosen/rejected pairs without relying on (i) human annotation, (ii) extra
data, and (iii) external models or APIs. MIA-DPO is compatible with various
architectures and outperforms existing methods on five multi-image benchmarks,
achieving an average performance boost of 3.0% on LLaVA-v1.5 and 4.3% on the
recent InternLM-XC2.5. Moreover, MIA-DPO has a minimal effect on the model's
ability to understand single images.
Scaling Diffusion Language Models via Adaptation from Autoregressive
  Models
Authors: name: Shansan Gong, name: Shivam Agarwal, name: Yizhe Zhang, name: Jiacheng Ye, name: Lin Zheng, name: Mukai Li, name: Chenxin An, name: Peilin Zhao, name: Wei Bi, name: Jiawei Han, name: Hao Peng, name: Lingpeng Kong
Published: 2024-10-23 14:04:22

Diffusion Language Models (DLMs) have emerged as a promising new paradigm for
text generative modeling, potentially addressing limitations of autoregressive
(AR) models. However, current DLMs have been studied at a smaller scale
compared to their AR counterparts and lack fair comparison on language modeling
benchmarks. Additionally, training diffusion models from scratch at scale
remains challenging. Given the prevalence of open-source AR language models, we
propose adapting these models to build text diffusion models. We demonstrate
connections between AR and diffusion modeling objectives and introduce a simple
continual pre-training approach for training diffusion models. Through
systematic evaluation on language modeling, reasoning, and commonsense
benchmarks, we show that we can convert AR models ranging from 127M to 7B
parameters (GPT2 and LLaMA) into diffusion models DiffuGPT and DiffuLLaMA,
using less than 200B tokens for training. Our experimental results reveal that
these models outperform earlier DLMs and are competitive with their AR
counterparts. We release a suite of DLMs (with 127M, 355M, and 7B parameters)
capable of generating fluent text, performing in-context learning, filling in
the middle without prompt re-ordering, and following instructions
https://github.com/HKUNLP/DiffuLLaMA.
WorldSimBench: Towards Video Generation Models as World Simulators
Authors: name: Yiran Qin, name: Zhelun Shi, name: Jiwen Yu, name: Xijun Wang, name: Enshen Zhou, name: Lijun Li, name: Zhenfei Yin, name: Xihui Liu, name: Lu Sheng, name: Jing Shao, name: Lei Bai, name: Wanli Ouyang, name: Ruimao Zhang
Published: 2024-10-23 17:56:11

Recent advancements in predictive models have demonstrated exceptional
capabilities in predicting the future state of objects and scenes. However, the
lack of categorization based on inherent characteristics continues to hinder
the progress of predictive model development. Additionally, existing benchmarks
are unable to effectively evaluate higher-capability, highly embodied
predictive models from an embodied perspective. In this work, we classify the
functionalities of predictive models into a hierarchy and take the first step
in evaluating World Simulators by proposing a dual evaluation framework called
WorldSimBench. WorldSimBench includes Explicit Perceptual Evaluation and
Implicit Manipulative Evaluation, encompassing human preference assessments
from the visual perspective and action-level evaluations in embodied tasks,
covering three representative embodied scenarios: Open-Ended Embodied
Environment, Autonomous, Driving, and Robot Manipulation. In the Explicit
Perceptual Evaluation, we introduce the HF-Embodied Dataset, a video assessment
dataset based on fine-grained human feedback, which we use to train a Human
Preference Evaluator that aligns with human perception and explicitly assesses
the visual fidelity of World Simulators. In the Implicit Manipulative
Evaluation, we assess the video-action consistency of World Simulators by
evaluating whether the generated situation-aware video can be accurately
translated into the correct control signals in dynamic environments. Our
comprehensive evaluation offers key insights that can drive further innovation
in video generation models, positioning World Simulators as a pivotal
advancement toward embodied artificial intelligence.
LLM-based Optimization of Compound AI Systems: A Survey
Authors: name: Matthieu Lin, name: Jenny Sheng, name: Andrew Zhao, name: Shenzhi Wang, name: Yang Yue, name: Yiran Wu, name: Huan Liu, name: Jun Liu, name: Gao Huang, name: Yong-Jin Liu
Published: 2024-10-21 18:06:25

In a compound AI system, components such as an LLM call, a retriever, a code
interpreter, or tools are interconnected. The system's behavior is primarily
driven by parameters such as instructions or tool definitions. Recent
advancements enable end-to-end optimization of these parameters using an LLM.
Notably, leveraging an LLM as an optimizer is particularly efficient because it
avoids gradient computation and can generate complex code and instructions.
This paper presents a survey of the principles and emerging trends in LLM-based
optimization of compound AI systems. It covers archetypes of compound AI
systems, approaches to LLM-based end-to-end optimization, and insights into
future directions and broader impacts. Importantly, this survey uses concepts
from program analysis to provide a unified view of how an LLM optimizer is
prompted to optimize a compound AI system. The exhaustive list of paper is
provided at
https://github.com/linyuhongg/LLM-based-Optimization-of-Compound-AI-Systems.
Frontiers in Intelligent Colonoscopy
Authors: name: Ge-Peng Ji, name: Jingyi Liu, name: Peng Xu, name: Nick Barnes, name: Fahad Shahbaz Khan, name: Salman Khan, name: Deng-Ping Fan
Published: 2024-10-22 17:57:12

Colonoscopy is currently one of the most sensitive screening methods for
colorectal cancer. This study investigates the frontiers of intelligent
colonoscopy techniques and their prospective implications for multimodal
medical applications. With this goal, we begin by assessing the current
data-centric and model-centric landscapes through four tasks for colonoscopic
scene perception, including classification, detection, segmentation, and
vision-language understanding. This assessment enables us to identify
domain-specific challenges and reveals that multimodal research in colonoscopy
remains open for further exploration. To embrace the coming multimodal era, we
establish three foundational initiatives: a large-scale multimodal instruction
tuning dataset ColonINST, a colonoscopy-designed multimodal language model
ColonGPT, and a multimodal benchmark. To facilitate ongoing monitoring of this
rapidly evolving field, we provide a public website for the latest updates:
https://github.com/ai4colonoscopy/IntelliScope.
3DGS-Enhancer: Enhancing Unbounded 3D Gaussian Splatting with
  View-consistent 2D Diffusion Priors
Authors: name: Xi Liu, name: Chaoyi Zhou, name: Siyu Huang
Published: 2024-10-21 17:59:09

Novel-view synthesis aims to generate novel views of a scene from multiple
input images or videos, and recent advancements like 3D Gaussian splatting
(3DGS) have achieved notable success in producing photorealistic renderings
with efficient pipelines. However, generating high-quality novel views under
challenging settings, such as sparse input views, remains difficult due to
insufficient information in under-sampled areas, often resulting in noticeable
artifacts. This paper presents 3DGS-Enhancer, a novel pipeline for enhancing
the representation quality of 3DGS representations. We leverage 2D video
diffusion priors to address the challenging 3D view consistency problem,
reformulating it as achieving temporal consistency within a video generation
process. 3DGS-Enhancer restores view-consistent latent features of rendered
novel views and integrates them with the input views through a spatial-temporal
decoder. The enhanced views are then used to fine-tune the initial 3DGS model,
significantly improving its rendering performance. Extensive experiments on
large-scale datasets of unbounded scenes demonstrate that 3DGS-Enhancer yields
superior reconstruction performance and high-fidelity rendering results
compared to state-of-the-art methods. The project webpage is
https://xiliu8006.github.io/3DGS-Enhancer-project .
xGen-MM-Vid (BLIP-3-Video): You Only Need 32 Tokens to Represent a Video
  Even in VLMs
Authors: name: Michael S. Ryoo, name: Honglu Zhou, name: Shrikant Kendre, name: Can Qin, name: Le Xue, name: Manli Shu, name: Silvio Savarese, name: Ran Xu, name: Caiming Xiong, name: Juan Carlos Niebles
Published: 2024-10-21 17:59:11

We present xGen-MM-Vid (BLIP-3-Video): a multimodal language model for
videos, particularly designed to efficiently capture temporal information over
multiple frames. BLIP-3-Video takes advantage of the 'temporal encoder' in
addition to the conventional visual tokenizer, which maps a sequence of tokens
over multiple frames into a compact set of visual tokens. This enables
BLIP3-Video to use much fewer visual tokens than its competing models (e.g., 32
vs. 4608 tokens). We explore different types of temporal encoders, including
learnable spatio-temporal pooling as well as sequential models like Token
Turing Machines. We experimentally confirm that BLIP-3-Video obtains video
question-answering accuracies comparable to much larger state-of-the-art models
(e.g., 34B), while being much smaller (i.e., 4B) and more efficient by using
fewer visual tokens. The project website is at
https://www.salesforceairesearch.com/opensource/xGen-MM-Vid/index.html
Improve Vision Language Model Chain-of-thought Reasoning
Authors: name: Ruohong Zhang, name: Bowen Zhang, name: Yanghao Li, name: Haotian Zhang, name: Zhiqing Sun, name: Zhe Gan, name: Yinfei Yang, name: Ruoming Pang, name: Yiming Yang
Published: 2024-10-21 17:00:06

Chain-of-thought (CoT) reasoning in vision language models (VLMs) is crucial
for improving interpretability and trustworthiness. However, current training
recipes lack robust CoT reasoning data, relying on datasets dominated by short
annotations with minimal rationales. In this work, we show that training VLM on
short answers does not generalize well to reasoning tasks that require more
detailed responses. To address this, we propose a two-fold approach. First, we
distill rationales from GPT-4o model to enrich the training data and fine-tune
VLMs, boosting their CoT performance. Second, we apply reinforcement learning
to further calibrate reasoning quality. Specifically, we construct positive
(correct) and negative (incorrect) pairs of model-generated reasoning chains,
by comparing their predictions with annotated short answers. Using this
pairwise data, we apply the Direct Preference Optimization algorithm to refine
the model's reasoning abilities. Our experiments demonstrate significant
improvements in CoT reasoning on benchmark datasets and better generalization
to direct answer prediction as well. This work emphasizes the importance of
incorporating detailed rationales in training and leveraging reinforcement
learning to strengthen the reasoning capabilities of VLMs.
Math Neurosurgery: Isolating Language Models' Math Reasoning Abilities
  Using Only Forward Passes
Authors: name: Bryan R. Christ, name: Zack Gottesman, name: Jonathan Kropko, name: Thomas Hartvigsen
Published: 2024-10-22 12:00:58

Math reasoning is a highly active area of Large Language Model (LLM) research
because it is a hallmark of artificial intelligence. However, few works have
explored how math reasoning is encoded within LLM parameters and if it is a
skill that can be isolated within a model. Doing so could allow targeted
intervention to improve math performance without altering non-math behavior and
foster understanding of how models encode math reasoning. We introduce Math
Neurosurgery (MathNeuro), a method for isolating math-specific parameters in
LLMs using only forward passes. MathNeuro builds on existing work by using
weights and activations to calculate parameter importance, but isolates
math-specific parameters by removing those important for general language
tasks. Pruning parameters MathNeuro identifies deletes a LLM's math reasoning
ability without destroying its general language ability. Scaling these
parameters by a small constant improves a pretrained or instruction-tuned LLM's
performance by 4-17% on GSM8K while leaving non-math behavior unaltered.
MathNeuro is also data efficient: most of its effectiveness holds when
identifying math-specific parameters using a single sample. MathNeuro
highlights the potential for future work to intervene on math-specific
parameters.