Orionnnn/hub-stats huggingface dataset:
SegBook: A Simple Baseline and Cookbook for Volumetric Medical Image
  Segmentation
Authors: name: Jin Ye, name: Ying Chen, name: Yanjun Li, name: Haoyu Wang, name: Zhongying Deng, name: Ziyan Huang, name: Yanzhou Su, name: Chenglong Ma, name: Yuanfeng Ji, name: Junjun He
Published: 2024-11-21 19:00:01

Computed Tomography (CT) is one of the most popular modalities for medical
imaging. By far, CT images have contributed to the largest publicly available
datasets for volumetric medical segmentation tasks, covering full-body
anatomical structures. Large amounts of full-body CT images provide the
opportunity to pre-train powerful models, e.g., STU-Net pre-trained in a
supervised fashion, to segment numerous anatomical structures. However, it
remains unclear in which conditions these pre-trained models can be transferred
to various downstream medical segmentation tasks, particularly segmenting the
other modalities and diverse targets. To address this problem, a large-scale
benchmark for comprehensive evaluation is crucial for finding these conditions.
Thus, we collected 87 public datasets varying in modality, target, and sample
size to evaluate the transfer ability of full-body CT pre-trained models. We
then employed a representative model, STU-Net with multiple model scales, to
conduct transfer learning across modalities and targets. Our experimental
results show that (1) there may be a bottleneck effect concerning the dataset
size in fine-tuning, with more improvement on both small- and large-scale
datasets than medium-size ones. (2) Models pre-trained on full-body CT
demonstrate effective modality transfer, adapting well to other modalities such
as MRI. (3) Pre-training on the full-body CT not only supports strong
performance in structure detection but also shows efficacy in lesion detection,
showcasing adaptability across target tasks. We hope that this large-scale open
evaluation of transfer learning can direct future research in volumetric
medical image segmentation.
From Generation to Judgment: Opportunities and Challenges of
  LLM-as-a-judge
Authors: name: Dawei Li, name: Bohan Jiang, name: Liangjie Huang, name: Alimohammad Beigi, name: Chengshuai Zhao, name: Zhen Tan, name: Amrita Bhattacharjee, name: Yuxuan Jiang, name: Canyu Chen, name: Tianhao Wu, name: Kai Shu, name: Lu Cheng, name: Huan Liu
Published: 2024-11-25 17:28:44

Assessment and evaluation have long been critical challenges in artificial
intelligence (AI) and natural language processing (NLP). However, traditional
methods, whether matching-based or embedding-based, often fall short of judging
subtle attributes and delivering satisfactory results. Recent advancements in
Large Language Models (LLMs) inspire the "LLM-as-a-judge" paradigm, where LLMs
are leveraged to perform scoring, ranking, or selection across various tasks
and applications. This paper provides a comprehensive survey of LLM-based
judgment and assessment, offering an in-depth overview to advance this emerging
field. We begin by giving detailed definitions from both input and output
perspectives. Then we introduce a comprehensive taxonomy to explore
LLM-as-a-judge from three dimensions: what to judge, how to judge and where to
judge. Finally, we compile benchmarks for evaluating LLM-as-a-judge and
highlight key challenges and promising directions, aiming to provide valuable
insights and inspire future research in this promising research area. Paper
list and more resources about LLM-as-a-judge can be found at
https://github.com/llm-as-a-judge/Awesome-LLM-as-a-judge and
https://llm-as-a-judge.github.io.
MH-MoE:Multi-Head Mixture-of-Experts
Authors: name: Shaohan Huang, name: Xun Wu, name: Shuming Ma, name: Furu Wei
Published: 2024-11-25 09:05:36

Multi-Head Mixture-of-Experts (MH-MoE) demonstrates superior performance by
using the multi-head mechanism to collectively attend to information from
various representation spaces within different experts. In this paper, we
present a novel implementation of MH-MoE that maintains both FLOPs and
parameter parity with sparse Mixture of Experts models. Experimental results on
language models show that the new implementation yields quality improvements
over both vanilla MoE and fine-grained MoE models. Additionally, our
experiments demonstrate that MH-MoE is compatible with 1-bit Large Language
Models (LLMs) such as BitNet.
GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and A
  Comprehensive Multimodal Dataset Towards General Medical AI
Authors: name: Tianbin Li, name: Yanzhou Su, name: Wei Li, name: Bin Fu, name: Zhe Chen, name: Ziyan Huang, name: Guoan Wang, name: Chenglong Ma, name: Ying Chen, name: Ming Hu, name: Yanjun Li, name: Pengcheng Chen, name: Xiaowei Hu, name: Zhongying Deng, name: Yuanfeng Ji, name: Jin Ye, name: Yu Qiao, name: Junjun He
Published: 2024-11-21 18:59:36

Despite significant advancements in general artificial intelligence, such as
GPT-4, their effectiveness in the medical domain (general medical AI, GMAI)
remains constrained due to the absence of specialized medical knowledge. To
address this challenge, we present GMAI-VL-5.5M, a comprehensive multimodal
medical dataset created by converting hundreds of specialized medical datasets
into meticulously constructed image-text pairs. This dataset features
comprehensive task coverage, diverse modalities, and high-quality image-text
data. Building upon this multimodal dataset, we propose GMAI-VL, a general
medical vision-language model with a progressively three-stage training
strategy. This approach significantly enhances the model's ability by
integrating visual and textual information, thereby improving its ability to
process multimodal data and support accurate diagnosis and clinical
decision-making. Experimental evaluations demonstrate that GMAI-VL achieves
state-of-the-art results across a wide range of multimodal medical tasks, such
as visual question answering and medical image diagnosis. Our contributions
include the development of the GMAI-VL-5.5M dataset, the introduction of the
GMAI-VL model, and the establishment of new benchmarks in multiple medical
domains. Code and dataset will be released at
https://github.com/uni-medical/GMAI-VL.
From CISC to RISC: language-model guided assembly transpilation
Authors: name: Ahmed Heakl, name: Chaimaa Abi, name: Rania Hossam, name: Abdulrahman Mahmoud
Published: 2024-11-25 12:37:07

The transition from x86 to ARM architecture is becoming increasingly common
across various domains, primarily driven by ARM's energy efficiency and
improved performance across traditional sectors. However, this ISA shift poses
significant challenges, mainly due to the extensive legacy ecosystem of x86
software and lack of portability across proprietary ecosystems and software
stacks. This paper introduces CRT, a lightweight LLM-based transpiler that
automatically converts x86 assembly to ARM assembly. Our approach bridges the
fundamental architectural gap between x86's CISC-based and ARM's RISC-based
computing paradigms while preserving program semantics and optimizing
performance. We evaluate CRT on diverse real-world applications, achieving
79.25% translation accuracy from x86 to ARMv5 on our comprehensive test suite,
and an 88.68% accuracy from x86 to RISC-V. In practical deployments on Apple M2
hardware (ARMv8), our transpiled code achieves 1.73times speedup compared to
Apple's Rosetta 2 virtualization engine, while delivering 2.41times memory
efficiency and 1.47times better energy consumption. Through testing and
analysis, we show that CRT successfully navigates the CISC/RISC divide and
generates correctly executable RISC code despite machine ``language'' barriers.
We release our code, models, training datasets, and benchmarks at:
https://ahmedheakl.github.io/asm2asm/.
Cautious Optimizers: Improving Training with One Line of Code
Authors: name: Kaizhao Liang, name: Lizhang Chen, name: Bo Liu, name: Qiang Liu
Published: 2024-11-25 04:36:01

AdamW has been the default optimizer for transformer pretraining. For many
years, our community searches for faster and more stable optimizers with only
constraint positive outcomes. In this work, we propose a single-line
modification in Pytorch to any momentum-based optimizer, which we rename
Cautious Optimizer, e.g. C-AdamW and C-Lion. Our theoretical result shows that
this modification preserves Adam's Hamiltonian function and it does not break
the convergence guarantee under the Lyapunov analysis. In addition, a whole new
family of optimizers is revealed by our theoretical insight. Among them, we
pick the simplest one for empirical experiments, showing speed-up on Llama and
MAE pretraining up to 1.47times. Code is available at
https://github.com/kyleliang919/C-Optim
DreamRunner: Fine-Grained Storytelling Video Generation with
  Retrieval-Augmented Motion Adaptation
Authors: name: Zun Wang, name: Jialu Li, name: Han Lin, name: Jaehong Yoon, name: Mohit Bansal
Published: 2024-11-25 18:41:56

Storytelling video generation (SVG) has recently emerged as a task to create
long, multi-motion, multi-scene videos that consistently represent the story
described in the input text script. SVG holds great potential for diverse
content creation in media and entertainment; however, it also presents
significant challenges: (1) objects must exhibit a range of fine-grained,
complex motions, (2) multiple objects need to appear consistently across
scenes, and (3) subjects may require multiple motions with seamless transitions
within a single scene. To address these challenges, we propose DreamRunner, a
novel story-to-video generation method: First, we structure the input script
using a large language model (LLM) to facilitate both coarse-grained scene
planning as well as fine-grained object-level layout and motion planning. Next,
DreamRunner presents retrieval-augmented test-time adaptation to capture target
motion priors for objects in each scene, supporting diverse motion
customization based on retrieved videos, thus facilitating the generation of
new videos with complex, scripted motions. Lastly, we propose a novel
spatial-temporal region-based 3D attention and prior injection module SR3AI for
fine-grained object-motion binding and frame-by-frame semantic control. We
compare DreamRunner with various SVG baselines, demonstrating state-of-the-art
performance in character consistency, text alignment, and smooth transitions.
Additionally, DreamRunner exhibits strong fine-grained condition-following
ability in compositional text-to-video generation, significantly outperforming
baselines on T2V-ComBench. Finally, we validate DreamRunner's robust ability to
generate multi-object interactions with qualitative examples.
Large-Scale Text-to-Image Model with Inpainting is a Zero-Shot
  Subject-Driven Image Generator
Authors: name: Chaehun Shin, name: Jooyoung Choi, name: Heeseung Kim, name: Sungroh Yoon
Published: 2024-11-23 06:17:43

Subject-driven text-to-image generation aims to produce images of a new
subject within a desired context by accurately capturing both the visual
characteristics of the subject and the semantic content of a text prompt.
Traditional methods rely on time- and resource-intensive fine-tuning for
subject alignment, while recent zero-shot approaches leverage on-the-fly image
prompting, often sacrificing subject alignment. In this paper, we introduce
Diptych Prompting, a novel zero-shot approach that reinterprets as an
inpainting task with precise subject alignment by leveraging the emergent
property of diptych generation in large-scale text-to-image models. Diptych
Prompting arranges an incomplete diptych with the reference image in the left
panel, and performs text-conditioned inpainting on the right panel. We further
prevent unwanted content leakage by removing the background in the reference
image and improve fine-grained details in the generated subject by enhancing
attention weights between the panels during inpainting. Experimental results
confirm that our approach significantly outperforms zero-shot image prompting
methods, resulting in images that are visually preferred by users.
Additionally, our method supports not only subject-driven generation but also
stylized image generation and subject-driven image editing, demonstrating
versatility across diverse image generation applications. Project page:
https://diptychprompting.github.io/
The Impossible Test: A 2024 Unsolvable Dataset and A Chance for an AGI
  Quiz
Authors: name: David Noever, name: Forrest McKee
Published: 2024-11-20 04:12:29

This research introduces a novel evaluation framework designed to assess
large language models' (LLMs) ability to acknowledge uncertainty on 675
fundamentally unsolvable problems. Using a curated dataset of graduate-level
grand challenge questions with intentionally unknowable answers, we evaluated
twelve state-of-the-art LLMs, including both open and closed-source models, on
their propensity to admit ignorance rather than generate plausible but
incorrect responses. The best models scored in 62-68% accuracy ranges for
admitting the problem solution was unknown in fields ranging from biology to
philosophy and mathematics. We observed an inverse relationship between problem
difficulty and model accuracy, with GPT-4 demonstrating higher rates of
uncertainty acknowledgment on more challenging problems (35.8%) compared to
simpler ones (20.0%). This pattern indicates that models may be more prone to
generate speculative answers when problems appear more tractable. The study
also revealed significant variations across problem categories, with models
showing difficulty in acknowledging uncertainty in invention and NP-hard
problems while performing relatively better on philosophical and psychological
challenges. These results contribute to the growing body of research on
artificial general intelligence (AGI) assessment by highlighting the importance
of uncertainty recognition as a critical component of future machine
intelligence evaluation. This impossibility test thus extends previous
theoretical frameworks for universal intelligence testing by providing
empirical evidence of current limitations in LLMs' ability to recognize their
own knowledge boundaries, suggesting new directions for improving model
training architectures and evaluation approaches.
Material Anything: Generating Materials for Any 3D Object via Diffusion
Authors: name: Xin Huang, name: Tengfei Wang, name: Ziwei Liu, name: Qing Wang
Published: 2024-11-22 18:59:39

We present Material Anything, a fully-automated, unified diffusion framework
designed to generate physically-based materials for 3D objects. Unlike existing
methods that rely on complex pipelines or case-specific optimizations, Material
Anything offers a robust, end-to-end solution adaptable to objects under
diverse lighting conditions. Our approach leverages a pre-trained image
diffusion model, enhanced with a triple-head architecture and rendering loss to
improve stability and material quality. Additionally, we introduce confidence
masks as a dynamic switcher within the diffusion model, enabling it to
effectively handle both textured and texture-less objects across varying
lighting conditions. By employing a progressive material generation strategy
guided by these confidence masks, along with a UV-space material refiner, our
method ensures consistent, UV-ready material outputs. Extensive experiments
demonstrate our approach outperforms existing methods across a wide range of
object categories and lighting conditions.
Adapting Vision Foundation Models for Robust Cloud Segmentation in
  Remote Sensing Images
Authors: name: Xuechao Zou, name: Shun Zhang, name: Kai Li, name: Shiying Wang, name: Junliang Xing, name: Lei Jin, name: Congyan Lang, name: Pin Tao
Published: 2024-11-20 08:37:39

Cloud segmentation is a critical challenge in remote sensing image
interpretation, as its accuracy directly impacts the effectiveness of
subsequent data processing and analysis. Recently, vision foundation models
(VFM) have demonstrated powerful generalization capabilities across various
visual tasks. In this paper, we present a parameter-efficient adaptive
approach, termed Cloud-Adapter, designed to enhance the accuracy and robustness
of cloud segmentation. Our method leverages a VFM pretrained on general domain
data, which remains frozen, eliminating the need for additional training.
Cloud-Adapter incorporates a lightweight spatial perception module that
initially utilizes a convolutional neural network (ConvNet) to extract dense
spatial representations. These multi-scale features are then aggregated and
serve as contextual inputs to an adapting module, which modulates the frozen
transformer layers within the VFM. Experimental results demonstrate that the
Cloud-Adapter approach, utilizing only 0.6% of the trainable parameters of the
frozen backbone, achieves substantial performance gains. Cloud-Adapter
consistently attains state-of-the-art (SOTA) performance across a wide variety
of cloud segmentation datasets from multiple satellite sources, sensor series,
data processing levels, land cover scenarios, and annotation granularities. We
have released the source code and pretrained models at
https://github.com/XavierJiezou/Cloud-Adapter to support further research.
VideoRepair: Improving Text-to-Video Generation via Misalignment
  Evaluation and Localized Refinement
Authors: name: Daeun Lee, name: Jaehong Yoon, name: Jaemin Cho, name: Mohit Bansal
Published: 2024-11-22 18:31:47

Recent text-to-video (T2V) diffusion models have demonstrated impressive
generation capabilities across various domains. However, these models often
generate videos that have misalignments with text prompts, especially when the
prompts describe complex scenes with multiple objects and attributes. To
address this, we introduce VideoRepair, a novel model-agnostic, training-free
video refinement framework that automatically identifies fine-grained
text-video misalignments and generates explicit spatial and textual feedback,
enabling a T2V diffusion model to perform targeted, localized refinements.
VideoRepair consists of four stages: In (1) video evaluation, we detect
misalignments by generating fine-grained evaluation questions and answering
those questions with MLLM. In (2) refinement planning, we identify accurately
generated objects and then create localized prompts to refine other areas in
the video. Next, in (3) region decomposition, we segment the correctly
generated area using a combined grounding module. We regenerate the video by
adjusting the misaligned regions while preserving the correct regions in (4)
localized refinement. On two popular video generation benchmarks (EvalCrafter
and T2V-CompBench), VideoRepair substantially outperforms recent baselines
across various text-video alignment metrics. We provide a comprehensive
analysis of VideoRepair components and qualitative examples.
BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games
Authors: name: Davide Paglieri, name: Bartłomiej Cupiał, name: Samuel Coward, name: Ulyana Piterbarg, name: Maciej Wolczyk, name: Akbir Khan, name: Eduardo Pignatelli, name: Łukasz Kuciński, name: Lerrel Pinto, name: Rob Fergus, name: Jakob Nicolaus Foerster, name: Jack Parker-Holder, name: Tim Rocktäschel
Published: 2024-11-20 18:54:32

Large Language Models (LLMs) and Vision Language Models (VLMs) possess
extensive knowledge and exhibit promising reasoning abilities; however, they
still struggle to perform well in complex, dynamic environments. Real-world
tasks require handling intricate interactions, advanced spatial reasoning,
long-term planning, and continuous exploration of new strategies-areas in which
we lack effective methodologies for comprehensively evaluating these
capabilities. To address this gap, we introduce BALROG, a novel benchmark
designed to assess the agentic capabilities of LLMs and VLMs through a diverse
set of challenging games. Our benchmark incorporates a range of existing
reinforcement learning environments with varying levels of difficulty,
including tasks that are solvable by non-expert humans in seconds to extremely
challenging ones that may take years to master (e.g., the NetHack Learning
Environment). We devise fine-grained metrics to measure performance and conduct
an extensive evaluation of several popular open-source and closed-source LLMs
and VLMs. Our findings indicate that while current models achieve partial
success in the easier games, they struggle significantly with more challenging
tasks. Notably, we observe severe deficiencies in vision-based decision-making,
as models perform worse when visual representations of the environments are
provided. We release BALROG as an open and user-friendly benchmark to
facilitate future research and development in the agentic community.
Large Multi-modal Models Can Interpret Features in Large Multi-modal
  Models
Authors: name: Kaichen Zhang, name: Yifei Shen, name: Bo Li, name: Ziwei Liu
Published: 2024-11-22 14:41:36

Recent advances in Large Multimodal Models (LMMs) lead to significant
breakthroughs in both academia and industry. One question that arises is how
we, as humans, can understand their internal neural representations. This paper
takes an initial step towards addressing this question by presenting a
versatile framework to identify and interpret the semantics within LMMs.
Specifically, 1) we first apply a Sparse Autoencoder(SAE) to disentangle the
representations into human understandable features. 2) We then present an
automatic interpretation framework to interpreted the open-semantic features
learned in SAE by the LMMs themselves. We employ this framework to analyze the
LLaVA-NeXT-8B model using the LLaVA-OV-72B model, demonstrating that these
features can effectively steer the model's behavior. Our results contribute to
a deeper understanding of why LMMs excel in specific tasks, including EQ tests,
and illuminate the nature of their mistakes along with potential strategies for
their rectification. These findings offer new insights into the internal
mechanisms of LMMs and suggest parallels with the cognitive processes of the
human brain.
One to rule them all: natural language to bind communication, perception
  and action
Authors: name: Simone Colombani, name: Dimitri Ognibene, name: Giuseppe Boccignone
Published: 2024-11-22 16:05:54

In recent years, research in the area of human-robot interaction has focused
on developing robots capable of understanding complex human instructions and
performing tasks in dynamic and diverse environments. These systems have a wide
range of applications, from personal assistance to industrial robotics,
emphasizing the importance of robots interacting flexibly, naturally and safely
with humans. This paper presents an advanced architecture for robotic action
planning that integrates communication, perception, and planning with Large
Language Models (LLMs). Our system is designed to translate commands expressed
in natural language into executable robot actions, incorporating environmental
information and dynamically updating plans based on real-time feedback. The
Planner Module is the core of the system where LLMs embedded in a modified
ReAct framework are employed to interpret and carry out user commands. By
leveraging their extensive pre-trained knowledge, LLMs can effectively process
user requests without the need to introduce new knowledge on the changing
environment. The modified ReAct framework further enhances the execution space
by providing real-time environmental perception and the outcomes of physical
actions. By combining robust and dynamic semantic map representations as graphs
with control components and failure explanations, this architecture enhances a
robot adaptability, task execution, and seamless collaboration with human users
in shared and dynamic environments. Through the integration of continuous
feedback loops with the environment the system can dynamically adjusts the plan
to accommodate unexpected changes, optimizing the robot ability to perform
tasks. Using a dataset of previous experience is possible to provide detailed
feedback about the failure. Updating the LLMs context of the next iteration
with suggestion on how to overcame the issue.