Commander-GPT: Dividing and Routing for Multimodal Sarcasm Detection
Authors: Yazhou Zhang
Chunwang Zou
Bo Wang
Jing Qin
Prayag Tiwari
Venue: Accepted by TMLR ()

Abstract: Multimodal sarcasm understanding is a high-order cognitive task. Although large language models (LLMs) have shown impressive performance on many downstream NLP tasks, growing evidence suggests that they struggle with sarcasm understanding. 
In this paper, we propose Commander-GPT, a modular decision routing framework inspired by military command theory. Rather than relying on a single LLM's capability, Commander-GPT orchestrates a team of specialized LLM agents where each agent will be selectively assigned to a focused sub-task such as context modeling, sentiment analysis, etc. Their outputs are then routed back to the commander, which integrates the information and performs the final sarcasm judgment.
To coordinate these agents, we introduce three types of centralized commanders: 
(1) a trained lightweight encoder-based commander (e.g., multi-modal BERT); (2) four small autoregressive language models, serving as moderately capable commanders (e.g., DeepSeek-VL); (3) two large LLM-based commander (Gemini Pro and GPT-4o) that performs task routing, output aggregation, and sarcasm decision-making in a zero-shot fashion.
We evaluate Commander-GPT on the MMSD and MMSD 2.0 benchmarks, comparing five prompting strategies. Experimental results show that our framework achieves 4.4% and 8.5% improvement in F1 score over state-of-the-art (SoTA) baselines on average, demonstrating its effectiveness.

Published in Transactions on Machine Learning Research (MM/YYY)
Gommander-GPT: Dividing and Routing for Multimodal Sarcasm Detection
Yazhou Zhang
yahou_ahhng Wfy.nggUju.e.ccom
School of Computer Science and Technology, Tianjón Universily
Chunwang Zou
1876132667@q9.oo
Software Engineering College, Zhengzhou University of Light Industry
Bo Wang
bo wwng@tju.edu.cr
School of Computer Science and Technology, Tianjón University
Jing Qin
harry. Spolyu.
Center for Smart Health, School of Nursing, The Hong Kong Pobylechaic Universily
Prayag Tiwari"
preyja..äiaarilieeee..m
School of Information Technology, Halmstad University
Reviewed on OpenReview: http:://peereerree..ee//oorrrrre
Abstract
Multimodal sarcasm understanding is a high-order cognitive task. Although large language models (LLMs) have shown impressive performance many downstream NLP tasks, grow- ing evidence suggests that they struggle with sarcasm understanding. Tn this paper, we propose Commander-GIT,, a modular decision routing framework inspired by military com- mand theory. Rather than relying on single LLM's capability, Commander-GPP orches- trates team of specialized LLM agents where each agent will be selectively assigned focused sub-task such context modeling, sentiment analysis, etc. Their outputs are then routed back to the commander, which integrates the information and performs the final sarcasm judgment. To coordinate these agents, we introduce three types of centralized com- manders: (1) trained lightweight encoder- based commander (e.g., multi-modal BERT); (2) four small autoregressive language models, serving as moderately capable commandors (c.g., DeepSeek-VT, (3) two large LLM-based commander (Gemini Pro and GPT-40) that per- forms task routing, output aggregation, and sarcasm catisioonmaking in zero-shot fashion. We evaluate Gommander-GPP prompting strategies. Experimental results show that our framework achieves4. and 8.5% improvement in F1 score over state-of-the-art. (SoTA) baselines on average, demonstrating its effectiveness.
the MMSD and MMSD 2.0 benchmarks, comparing five
1 Introduction
The era of large language models (LLMs) has been propelled by the scaling laws of language models and the emergence of capabilities with increasing model scale. SoTA LLMs. such as GPT-4o (Achiam et al., 2023). Claude 4 DeepSeek R1 (DeepSeek-AI et al. 2025), Qwen 3 (Yang et al., 2025). et., have demonstrated
"Corresponding authors. https://wwwarnnnpiccomm/////////

---

Published in Transactions on Machine Learning Research (M//YYYY)
"Love when my mans on cleaning spree. Saves me doing it e @ sarcastic not?
Query: the input:
90
GPT-4o: No, this sentence is not sarcastic. The added emojis significantly shift the tone. They convey affection, approval, and playfulness, suggesting the speaker genuinely appreciates their partner is cleaning.
70
50
Yes, this sentence is likely sarcastic. The addition of the thumbs up and multiple hearts emojis makes it even clearer that the tone is sarcastic. The speaker is not actually happy about their partner's cleaning enthusiasm.
Llama 3-8B:
30
GPT-4o
Claude
Llama
Qwen
Rondom
3.5
8B
7B
(a) An example of using LLMs
(b) Comparative results across LLMs and Random
Figure 1: LLM Performance on three sarcasm datasets in prior work.
remarkable performance across wide range of downstream natural language processing (NLP) tasks, includ- ing question answering, machine translation, commonsense reasoning, and code generation. These models exhibit impressive zero-shot and few-shot generalization abilities, leading to the belief that LLMs may have reached a critical threshold general intelligence (Zhang et al,, 2025).
Despite these advances, sarcasm understanding remains significant and unresolved challenge. Sarcasm is nuanced linguistic phenomenon that often employs rhetorical devices such irony, hyperbole, and contradiction to express sentiments that diverge sharply from the literal meanings of words (Liu et al, 2023b). For example, the sentence "Oh great, another meeting that could have been an email." appears to express enthusiasm, but in reality, conveys frustration or annoyance, its sarcastic tone hinging on pragmatic context and emotional subtext. Accurate sarcasm detection requires a combination of contextual reasoning, emotional inference, and figurative language interpretation, multi-modal interaction, etc., all of which demand high- order cognitive capabilities.
The growing evidence proves that even state-of-the-art models such as GPT-4, Claude 3.5 often perform poorly sarcasm detection, sometimes close to random guessing, as shown in Fig, (Yao et al. 2025). This observation challenges the prevailing assumption that general-purpose LLMs, by virtue of their scale, can seamlessly handle high-level pragmatic tasks. This discrepancy reveals a fundamental limitation: current methods treat sarcasm detection as an undivided, end-to-end task, relying on a single model to implicitly execute multiple layers of reasoning. These include sentiment recognition, rhetorical structure identification, contextual interpretation, and visual-textual alignment. Without explicit modeling of these sub-tasks, even powerful LLMs often fail to capture the complex and composite nature of sarcasm, particularly in multimodal scenarios.
To this end, we propose Gomm a structured multi-agent framework that decomposes sarcasin detection into six cognitively meaningful sub-tasks. They are: contect modeling, sentiment analysis, rhetorical device recognition, facial expression recognition, image summarization, and scene text recognition. Each sub- task is handled by an expert LLM or MLLM agent. Rather than invoking all agents for every input, the commander first analyzes the input and then activates only the agents that are most suitable for handling the relevant sub-tasks by introducing the routing scorer. In addition, BLIP-2 (Li et al, 2022). Vision Transformera, and OCR-2.0 (Wei et al,, 2024) are selected as vision agents for image summarization facial expression recognition, and scene text recognition, respectively, while Llama 3-8BB, Qwen 2.5-1.5B and RoBERT serve as linguistic specialists. Their outputs are then routed back to the commander, which integrates the information and performs the final sarcasm judgment.
http:///snttttooooooococoooocsos httpss//aaggnnaeeeoooo////////e2 3-8B
emotions

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Input: <text, image?
Task: Does this input require the <Sublask
Module> analyais step?
Answer Yes or No.
Integrated Analysis:
Oh, working from home...
context is remote work night. The woman's relaxed, amused smile contrasts the mildly negative, sarcastic tone of the statement. Even without explicit rhetorical analysis, the language and setting clearly suggest sarcasm.
Subtask Routing
Making
Subtask Execution
Result Integration
Decision:
Commander
Text Image
The input sarcastic.
Context Modeling: The context here is the everyday reality for many remote workers...
A Image Summarization:
young woman is sitting at home casuel clothes, smiling softly
Sentiment Analysis: The sentiment is mildly negative with While the person is smilling...
Rhetorical Device Recognition: Not activated...
Facial Expression
Scene Text Recognition: Not activated...
Recognition: The is smiling alightly. with a relaxed and somewhat amused look...
Figure The overall architecture of Commander-GPT.
coordinate these agents, we introduce three types centralized commanders: (1) a trained light encoder- based commander (e.g., multi-modal BERT); (2) four small autoregressive language models, serving moderately capable commanders (e.g., DeepSeck-VT--BB)) (3) two large I.LM-based commander (Gemini Pro and GPTT44) that performs task routing, output aggregation, and sarcasm decision-making. This diverse commander configuration enables comprehensive exploration of the trade-offs between performance and scalability.
Finally, we conduct empirical evaluations of Commander-GPT on two benchmark datasets: MMSD and MMSD 2.0. We compare our framework against five state-of-the-arl (SoTA) prompting strategies (e.g., Chain-of-Thougee Plan-and-Solve, 8 Agent, etc.). Experimental results highlight three key observations: (1) Commander- achieves4.4% and 8.5% improvement in F1 over strong basclines, demonstrating the effectiveness; (2) despite advances in large multimodal language models (MLLMs), fine-tuned small models (such as BERT) still exhibit stronger sarcasm detection capabilities; (3) the Commander-GPT model demonstrates robust generalization across diverse backbone LLMs and domains.
Our contributions are summarized as follows:
We propose Commander-GPT, modular multi-agent framework for multimodal sarcasm detec- tion.
We introduce set of centralized commanders with varying model capacities and conduct a system- atic comparison of their orchestration ability in sarcasm detection.
We present extensive experiments on MMSD and MMSD 2.0, showing that Commander GPT 4chieves4.4% and 8.5% F1 improvement over SoTA basclines on average.
3

---

Published in Transactions on Machine Learning Research (MM/YYN
2 The Proposed Approach
2.1 Problem Formulation
Consider a multimodal sarcasm detection task where we are given a dataset = (0,, Yi y:) where cach instance x; = (4, 13) consists a textual component 4 E T and a visual component vs E V, and yi O, represents the binary sarcasm label. The objective is to learn function predicts sarcasm labels.
x 0, that accurately
Traditional approaches employ single monolithic model M to directly map the multimodal input to the prediction: v(,, = M(t, v). However, as established in prior work, this paradigm suffers from the inherent complexity of sarcasm understanding, which requires simultancous processing of multiple linguistic and visual cues that may exhibit semantic contradictions. In this work, we propose a fundamentally different approach inspired by military command structures. We decompose the complex sarcasm detection task into K specialized subtasks, where each subtask Tы for k 1,2, K focuses on extracting specific types of information.
We assume access to a collection of expert models {A= A2....
where each agent Aj has varying capabilities across different subtasks. The capability of agent A, for subtask is characterized by a perfor- mance score Jj.k E [0, 1], where higher values indicate belter suitability. The core challenge lies in learning an optimal assignment function {a 1,, K + 1,2,.... M } that maps each subtask т to the most appro- priate agent Aa(k)- This assignment should maximize the overall system performance while considcring the complementary strengths of different agents.
Given the assignment function a, each subtask тę produces intermediate output 2 dx(z,) through the assigned agent Aa(k)- The final prediction is made by commander model C that aggregates all intermediate representations:
f(1,, 0x) = ((= K = (a:), (2))), (s(x.) (s(xx),,..,xxxxx))))
where x; E (44, vx).
The commander model serves as the central cerision-making unit, analogous to a military command center that processes intelligence reports from specialized units and makes strategic decisions. This hierarchical architecture enables specialized processing while maintaining global coordination through the commander's integrative reasoning capabilities.
It is important to note that while the overall objective can be expressed as a joint minimization over the routing assignment a and the commander parameters 0c, in practice we adopt a staged training procedure rather than full end-to-end optimization. Specifically, the routing classifier is first trained with GPT-4o distilled supervision to learn a, and then the commander trained on top of the routed outputs. This process can be formulated as bilevel optimization:
ax arg min Ero.ts)lroouo pa (z;a))),
(2)
0 arg min E ((n.)))emm(()) fx; (x;aa (e))),)))
(3)
where Croute denotes the binary cross-cntropy loss for routing decisions and Lend denotes the binary sarcasm detection loss. In this work is optimized first and held fixed when training the commander. Exploring fully differentiable end-to-end optimization of router and commander is left for future research.
2.2 Subtask Division
Sarcasm understanding often involves: (1) understanding the broader context and pragmatic implications, (2) recognizing the surfaco-level emotional expression, (3) detecting linguistic markers such as irony and hyperbole, (4) interpreting visual emotional cues that may contradict textual sentiment, (5) understanding the yisual scene context, and (6) processing any textual information embedded in images, as shown in Fig. Basod on this theoretical foundation, we decompose the complex sarcasm detection task into K
cognitively meaningful subtasks, where each subtask Tę focuses on extracting specific types of information that contribute to sarcasm understanding:
4

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Ti: Context Modeling: Analyze broader conversational context and pragmatic implications from text I: (1) + P, where P denotes the pragmatic interpretation space.
T2: Sentiment Analysis:
Extract fine-grained emotional polarity from text t: Φ2(() + S, where S E R2a represents multi-dirnensioaal emotion categories (e.g,, joy, anger, surprise). This identifies the surface-level emotional expression.
T3: Rhetorical Device Recognition: Identify linguistic patterns and rhctorical structures in text
@s(t) R, where represents rhetorical markers such as irony, hyperbole, and understatement.
T4: Facial Exprcssion Recognition: Extract facial emotions from image u: Φs(v) E, where & E I? represents basic emotion categories (happy, sad, angry, lear, surprise, disgust, neutral).
Generate comprehensive scene description from image Φs(u) - DD where D represents natural language descriptions of visual content and scene context.
Ts: Image Summarization:
Tg: Scene Text Recognition: Extract textual content embedded in image e: Φ6(1)) - Tscene, where T Scene represents text sequences found in the visual scene.
This decomposition is motivated by the observation that sarcasm often emerges from contradictions between these different information layers. For instance, a text expressing positive sentiment (72) accompanied by a person's s annoyed facial expression (ra) a chaotic scene (rs) may indicate sarcastic intent. By explicitly modeling these components, our framework can capture the nuanced interplay that previous approaches often miss.
2.3 Routing Approach
The routing scorer is core component in Commander-GPIT enabling the commander to dynamically select relevant sub-task agents for given input z = (t, t). We provide two representative instantiations: learnable classifier (c.g., BERT-based commander) and prompt-based approach for LLMs (e.g., GPT-4o, DeepSock VL).
Learnable Routing Classificr (Multimodal BERT based Commander). Let K denote the total number of sub-tasks (T For each input (t,v), we construct paired samples (R,T)), where Te is the descriptor for sub-task Tk (e.g., "sentiment analysis"). The routing scorer outputs activation probability for each sub-task:
(x(x)= = Po(rk = 2,, (w] hons(r, Tx) + bb)
(4)
where ( Tạ) Fusion (BERT((), VIT(v), Ty) denotes the fused multimodal representation obtained by combining the BERT-encoded text 6, the ViT-encoded image v, and the sub-task descriptor Th- Here, Fusion(-) can be implemented feature concatenation. The parameters Wa, by are trainable for each sub- task, and is the sigmoid function.
To train the routing classifier, we require labeled routing dataset Droute {(x, Ta, r.)). We distill agent activation decisions from a powerful instruction-tuned vision-language model, GPT-4o. Specifically, for each training sample I; = (t,, v.) and each sub-task descriptor Tk, we prompt GPT-4o with a question such as:
Input: <t i, v v
Task: Does this input require the "<T k>" analysis step?
Answer "Yes" or "No".
The binary response is recorded the activation label ra E (0, In this work, we distill approximately 5,000 routing supervision instances, enabling scalable and reliable training of the routing classifier.
The routing scorer is trained by minimizing the binary cross-entropy loss:
N K
Croute = NK
Fik log (xg))( + (1 - rik) log(1 Px (xx))
(5)
5

---

Published in Transactions on Machine Learning Research (MM/YYYY)
During inference, agent Aş is activated for if (x(x)) ok; where aş E [0,1] is tunable threshold.
Prompt-based Routing (LLM Commander). For instruction-tuned LLMs (e.g,, GPT-4o, DeepSeek VL), agent routing is formulated as a natural language inference task. For each (x, Te), we construct the prompt (see App. A3).
The LLM MLLM is queried with I(x, Te) and returns a textual response, which is mapped rx(2)) E (0, (1 if "Yes", 0 if "No"). This approach requires no explicit training and leverages the zero-shot or few-shot abilities of modern LLMss
GPT-4o Distillation. To obtain supervision for the routing classifier, we distilled activation decisions from GPT-4o on approximately 5000 (z, Tk) pairs sampled from the training split across all six subtasks. Each query was designed as a zero-shot, template-base based prompt with strict JSON-formatted output (please refer App. A.5), using temperature 0.0 and top-p = 1.0 to minimize randomness. If GPT-4o's response did not conform to the schema, aulomatically re-issued the prompl up to two times; invalid cases (< 1.2%) were discarded. Prompts were stratified to balance text-only, image-only, and multinodal inputs, well cases with and without facial expressions or scene text, ensuring broad coverage and preventing train test leakage.
Each instance was queried three times independently, and inter-run agreement reached a Cohen's a of 0.82, indicating stroug consistency. For sentiment, majority voting across the three runs was adopted; for routing, items with disagreement were marked low-confidence and woin-weighted during training. In addition, two human annotators reviewed 200 randomly sampled cases, achieving inter-annotator and 87% agrecment between GPT-4o and human consensus. These results suggest that the distilled labels are of sulficiently high qualily and stability to provide reliable supervision for training the routing classifier.
0.79
2.4 Subtask Execution
Once the routing scorer identifies the relevant sub-task agents for a given input x; = (44, v.), each activated agent A& E Ar, independently processes its assigned sub-task These specialized agents operate in parallel, leveraging their unique capabilities to extract specific types of information from the textual and visual components of z,.
Linguistic Specialists. For the textual component i; of 24, three linguistic specialist agents are employod:
Context Modeling Agent (Llama 3-8B): Analyzes broader conversational context and prag- matic implications via: 21 (4+4) Llama 3(" Analyze the contextual implications: " * tz), where 21 captures deeper meaning and situational context.
Sentiment Analysis Agent (RoBERTa):
Extracts fine- fine-grained sentiment polarity using fine- (2(4t) RoBERTagmationss (t,) E R3, ; 3D vector representing positive, neutral, and negative sentiments.
tuned RoBERTa: 22
Rhetorical Device Agent (Qwen 2.5-1.5B): Identifies linguistic patterns and rhetorical struc- tures (irony, hyperbole, understatement) using: = (s(t,) = Qwen- 1.50 (tetaric highlighting key linguistic markers of sarcasm.
Visual Specialists. For the visual component un three visual specialist. agents are employed:
Facial Expression Agent (VIT-FER): Recognizes facial emotions with a Vision Transformer:
24 = (1())) (TT-FER((:) E R7, outputting 7D vector (e.g., happy, sad, angry).
Image Summarization Agent (BLIP-2): Generates natural language scene descriptions: 2s (5(1;) = BI.IP-((1;) + natural language description, providing high-level understanding.
Scene Text Agent (OCR-2.0): Extracts embedded text in the image: 26 = db(())
OCR-2.0(x:) extracted text sequence, which may be crucial for sarcasm.

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Each activated agent produces structured output 2 (information and confidence scores). Parallel execution ensures computational efficiency and precise analysis for complex sarcasm understanding.
2.5 Result Integration
After all activated agents have processed their respective sub-tasks, the Commander module C' is responsible for integrating the intermediate outputs {z keA and making the final sarcasin prediction.
Lightweight Encoder-Based Commander. For encoder-based commanders (e.g., multi-moda BERT), we concatenate the outputs all activated agents and project them through a learned fusson and classification head:
hused = concat([2x for i E Aactive))
(6)
econtext BERT([CLS; hfused SEP])
(7)
Ú softmax W out hcontext bout))
where Wot E p2xduidden and bout E IR? are trainable parameters for binary classification.
LLM Commander. For moderately-sized language model commanders (e.g. DeepSeek-VI-7D) and large- scale LLM-based commanders (e.g., GPT-4o and Gemini Pro), all agent outputs are presented as compre- hensive structured prompt, and the LLM predicts both the sarcasm label and explanation:
Pimput Template )
(9)
ú CLM (Piput)) + {Sarcastic, Non-sarcastic]
(10)
This three-stage architecture ensures that our Commander- GPT framework maintains both the specialized expertise of individual agents and the strategic coordination capabilities of military command structures, lcading to more accurate and interprotable sarcasm detection results.
3 Experiments
In this section, we conduct comprehensive experiments on two widely-used multimodal sarcasm detection benchmarks, MMSD (Cai et al., 2019) and MMSD 2.0 (Qin et al., 2023)
3.1 Datasets
The MMSD dataset is a benchmark for multimodal sarcasm detection, comprising paired tex- tual and visual data collected from Twitter. Each example consists of a tweet and its associated image, with ground-truth sarcasm annotations. The textual content frequently features subtle or implicit sarcasm, while the accompanying images provide additional context, making the task especially challenging for both unimodal and multimodal models.
MMSD:
MMSD 2.0: MMSD 2.0 is enhanced and extended version of MMSD, designed to support more robust evaluation of multimodal sarcasm detection systems. Compared to its predecessor, MMSD 2.0 significantly increasos both the diversity visual content and the quality of text-image alignment. It introduces more challenging cases where understanding sarcasm requires reasoning over complex interactions between modal- ities. Key statistics and properties of the two datasets are summarized in Table
3.2 Experimental Settings
Implementation Details. All experiments were conducted on a server equipped with two NVIDIA RTX 4090 GPUs and 256GB RAM. Gom was implemented using PyTorch, HuggingFace Transform- ers, and the OpenMMLab toolkit. For supervised components (e.g., the BERT-base commander and the routing classifier), we fine-tuned for 10 epochs (approximately 12 hours in total). We used the AdamW

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Table Statistics the MMSD and MMSD 2.0 datasets.
Dataset
Train
Validation
Test
Sarcastic
Non-sarcastic
Source
MMSD
19,816
2,410 2,410
2,409
10,560
14,075
Twitter
MMSD 2.0
19,816
2,409
11,651
12,980
Twitter
optimizer with initial learning rate of 2=10-5, batch size of 64, maximum sequence length of 512, and weight decay of 0.01. A linear warm-up and decay scheduler was applied. Early stopping was triggered if the validation Fl score did not improve for 3 consecutive epochs. Unless otherwise stated, all hyperparameters were tuned on the validation set. For LLM commanders (e.g.. GPT-4o, Gemini Pro, DeepSeek -VL), we performed inference only without parameter fine-tuning.
Baselines and Commander Configurations. We compare Commander- against five representative prompting and reasoning baselines:
Plan-and-Solve (Wang et al. 2023): : pipeline prompting strategy that first plans the solution steps and then solves each subproblem soquentially.
Zero-shot CoT [Kojima et al. 2022):: : A zero-shot chain- chain-of-thouglee thought prompting method that enables
step-by-step reasoning without requiring labeled demonstrations.
Generated Knowledge Prompting (Liu et al. 2021): A method that augments the input with external knowledge generated by language model to enhance reasoning.
Automatic Prompt Engineer (Zhou et ale 2022): A technique that automatically searches for and optimizes prompt templates to maximize downstream task performance.
2 Agent (Wang et al, 2024a): multi-agent coordination framework for complex task decompo-
sition and solution synthesis.
DMSD-CL (Jia et al., 2024): A framework with contrastive learning that mitigates biased textual factors for robust OOD generalization, using counterfactual data augmentation to generate positive samples with dissimilar word biases and negative samples with similar word biases.
Cross-ToT Ranaldi et al, 2024): A cross-lingual Tree-of-Thoughe method that aligns reasoning across languages through a self coff-consittent cross-lingual prompting mechanism, enabling multi-step reasoning in different. languages and achieving state-of-the-art performance with fewer interactions.
Graph of Thoughts (Besta et al 2024): A prompting framework that models LLM outputs graph with thoughts as nodes and dependencies as edges, enabling combination, distillation and feedback across thought networks, improving reasoning and supporting extensible thought transfor- mations.
We further evaluate Commander-GPP under seven commander configurations, ranging from light encoder-basee commanders (BERT), to four medium- sized autoregressive LLMs (Yi-VL (6B) (Young et al. 2024), DeepSeck- VL-Chat (7B) (Lu et al., 2024]), Qwen-VI-CCat (9B) (Bai et al. 2023), and MiniCPM-V-2
(2.8B) (Hu al., 2024)) ), and two large custruction-folloiing LLMs (Gemini Pro and GPT-4o).
3.3 Inference Efficiency
As shown in Fig. S On an RTX 4090 GPU with 256GB CPU memory, the BERT commander achieved an average latency of 0.12s per sample with a peak GPU memory usage of approximately 2.8 GB. For Yi-VL, VoppScc--VL-Chat Qwen-VI. VL-Chat, and MiniCPM-V2 the average latencies were 0.45s, 0.68s, 0.52s, and 0.30s, with memory consumption of 13.8 GB, 32 GB, 10.5 GB, and 3.2 GB, respectively. Tn contrast, Gemini Pro and GPT-4o
we further evaluated the inference time and resource consumption of different commanders.
8

---

Published in Transactions on Machine Learning Research
35
18
CPII Memory
30
1.6
Inference Fimed cosmppppp
25
GPU MemusyyGn (GB)
20
15
6.8
0.6
10
0.4
6.2
BERT
Yi-VI. DeepSeck- VL--see--V-Chatt VL-Chat
Owen- VL-Chat
MiniCPM-12
Gemini Pro
GPT-40
Figure 3: Inference efficiency
showed average latencies of 1.65s and 1.85s. Since Gemini Pro and GPT-4o are closed-source commercial models, our experiments were conducted via API calls, making infeasible report their GPU memory usage. Overall, while large LLM commanders deliver the best performance, their inference costs increase substantially; in contrast, small and medium models provide more favorable trade-off between efficiency and accuracy.
3.4 Main results
Table g summarizes the performance of different commander architectures at three scales: lightweight encoder-based, small open-source LLMs, and SoTA proprietary LLMs. For all experiments, routing- based method achieves the highest Fl scores and provides significant improvements over corresponding baselines.
Lightweight Encoder-Based Commander. The lightweight solution with minimal model size (110M parameters). Compared to direct fine-tuning, our routing method achieves substantial absolute gains: on MMSD, F1 improves from 80.8 86.7 (+5.9), and on MMSD 2.0 from 77.2 to 85.8 (+8.6). This corresponds to relative improvements of 7.3% and 11.1%, respectively. The results indicate that even for parameter-efficien models with limited multimodal reason- ing ability, our approach enables more effective use of learned features, significantly boosting both overall accuracy and robustness.
commander represents the
Small Autoregressive LLMs Commander. For the small open-source LLMs group (Yi-VL, DeepSeek- VT-Chat, Qwen-VI-Chat, MiniCPM-V2), our method consistently outperforms all competitive prompting and multi-agent basclincs. On MMSD 2.0, F1 improvements are espocially pronounced: +23.6 (Yi-VL), +8.5 CeepSeee--V--Chat) +10.3 (Qwen- (wvnn-VL-Cha),, and +22.8 (MiniCPM-V2). For instance, Qwen-VI. Chat reaches 68.9 F1 versus 58.6 for the best prior method, and MiniCPM-V2 rises from 49.2 to 72.0 F1. These results suggest that our approach is highly effective at extracting complementary information from multiple sub-agents in smaller models, compensating for their individual limitations.
SoTA LLMs Commander. For the strongest closed-source models (Gemini Pro, GPT-4o), the routing approach continues to yield improvements, though the margins are reduced compared to smaller models due the high baseline performance. On MMSD 2.0, Gemini Pro improves from 67.4 to 71.9 F1 (+4.5), and GPT-4o increases from 73.2 to 76.5 F1 (+3.3). This demonstrates that even state-of-the-art models,

---

Published in Transactions on Machine Learning Research (MM/YYYY)
which already possess advanced multi-modal capabilities, benefit from structured subtask decomposition and information routing.
These results highlight that Commander-GPTTs effectiveness stems not only from numerical improvements but also from its novel design of structured task decomposition and routing. Unlike end-to-end models that implicitly handle sarcasm detection as a monolithic task, our framework explicitly separates the process into six cognitively meaningful subtasks. This design distributes complex reasoning to specialized expert agents, reducing the burden any single model. The router then selectively activates only those agents most relevant for given input, filtering out unnecessary noise. This "division of labor" enables agents to contribute complementary signals, which the commander integrates to detect subtle contradictions between text and vision. Consequently, the approach yields more robust and interpretable improvements over prior methods.
Further Discussion.
3.5 Comparison of Commander Models
Fig. 10 (see App. A4) presents the F1 scores of various commander models on the MMSD and MMSD 2.0 datascts. The results reveal significant performance differences among LLM backbones. BERT achieves the highest Fl scores (86.7 on MMSD, 85.8 on MMSD 2.0), primarily due to its fine-tuned nature on the sarcasm detection task. In contrast, mainstream vision-language models, such as Yi-VI. (59.1/55.3) and DeepSeek-VL (61.1/60.5), perform substantially worse, likely due to their weaker language modeling capability suboptimal adaptation to sarcasm-rich domains. Larger multimodal models like Qwen-Viz MiniCPM-V2, Gemini Pro, and GPT-4o show a clear performance hierarchy: GPT-4o delivers the strongest results among MLLMs (81.4/76.5), benefiting from both superior language modeling and robust multi-modal reasoning. Overall, these results highlight the importance of both model architecture and scale for effective sarcasm detection, and demonstrate that Commander-GPP GPT can flexibly leverage a wide range of LLMs, with significant performance gain observed as model quality improves.
Further Discussion Notably, different commander models demonstrate distinct advantages within the Commander-GPPT framework. The lightweight BERT commander, once fine-tuned, can effectively exploit structured subtask representations, allowing it to outperform some larger multimodal models. This under- scores that scale alone does not determine success structured input and task specialization are equally crucial. For large models such as GPT-4o and Gemini Pro, while the relative gains are smaller, they remain significant compared to direct reasoning baselines. This indicates that even powerful general-purpose models benefit from explicit task decomposition and routing. Such findings reinforce the universality of Comman- der-GPT: it enhances task adaptability for lightweight. models while simultaneously strengthening robustness and interpretability for large models.
3.6 Ablation Study
Table g and Table ! present the ablation results on the MMSD and MMSD 2.0 datasets, respectively. Across all models, removing any single sub-task module leads to a noticcable drop in both F1 and accuracy (%), confirming the necessity of each component. For all three models, the largest relative performance degradation occurs when either Rhetorical Device Recognition or Contert Modeling is ablated. For example, on MMSD, removing Rhetorical Device Recognition results in a 24.7% F1 decrease for DeepSeek-V VL (from 61.1% to 27.9%), a 7.2% drop for MiniCPM-V2, and a 5.2% decrease for GPT-4o. Removing Context Modeling leads to a 6.9% Fl drop for DeepSeek-VT, 0.1% for MiniCPM-V2, and 2.5% for GPT-4o. On MMSD 2.0, similar trends are observed, with the ablation of Rhetorical Device Recognition causing up to 29.1% relative Fl drop for DeepSeek-VI, and 3.5% for GPT-4o.
For other sub-tasks such as Sentiment Analysis, Facial Expression Recognition, Image Summarization, and Scene Text Recognition, the impact is present but less pronouneed, with typical F1 reductions ranging from 0.2% to 1.1% for GPT-4o, and 0.5% to 1.3% for MiniCPM-V2. All models achieve their best F1 and accuracy when all six modules are integrated, validating the effectiveness and complementarity of the full routing strategy. Overall, these results demonstrate that multi-dimensional sub-task collaboration is critical for robust multi-modal sarcasm detection, with context and rhetorical information providing the most substantial performance gains.
10

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Table 2: Main Experimental Results. The bolded numbers indicate the best performance. Additionally, we calculated the increase in the F1 score.
MMSD
MMSD 2.0
Model
Parameters
Method
Acc.
Pre.
Rec.
F1
Acc.
Pre.
Rec.
Lightweight Encoder- Based Commander
BERT (Fine-tuned)
51.2
80.1
77.2
77.4
VIT
110M
Ours
86.7 (5.9f)
90.1
87.4
86.1
85.8 (8.6f)
87.4
86.1
85.3
Small Autoregressive LLMs Commander
Zero-soot CoT
45.8
59.2
51.4
41.4
6.1
16:2
6.0
6.3
Automatic Prompt Engineer
45.0
Plan-and- Sollvve
51.3
62.6
56.2
47.2
4.4
9.2
4.0
4.8
Generated Knowledge Prompting
52.7
55.7
47.5
59.2
4.3
11.2
4.0
4.6
8 Agent
52.3
11.1
39.6
77.0
31.7
51.3
41.4
21.6
DMSD-CI.
YE-L
6B
Cross-ToT
51.7
62.0
44.9
37.1
47.8
46.7
30.8
GoT
28.3
55.1
12.1
21.3
27.4
41.6
11.6
20.4
Ours
59.1 (6.41)
70.8
71.2
50.5
56.3
(18.21)
67.9
69.0
46.1
Zero-soot CoT
54.4
65.3
DO.1
49.8
48.8
62.8
59.8
41.3
Automatic Prompt Engineer
66.4
59.7
Plan-and--Solve
54.9
61.9
54.3
55.5
48.3
63.0
60.5
40.2
Generated Knowledge Prompting
55.6
56.8
48.7
65.0
27.5
58.1
53.8
18.5
8 Agent
59.7
15.3
13.1
59.7
52.0
64.9
63.3
41.1
DeepSeek-VV PeepSeek-VLL-hhat
7B
DMSD-CL
20.7
44.5
76.3
12.0
Croes: ToT
52.1
GoT
50.0
50.1
13.0
59.8
50.3
49.5
13.6
59.3
Ours
61.1 (1.4f)
69.4
64.9
57.7
60.5 (6.21)
46.7
44.4
94.77
Zero CoT
66.4
86.0
56.5
80.7
33.6
40.0
32.0
35.3
Automatic Prompt Engineer
51.6
33.3
Plan-and-Solvee
59.4
64.8
57.2
61.8
34.8
38.7
32.1
38.0
Generated Knowledge Prompting
50.9
67.1
04.77
40.9
28.0
38.9
38.4
37.8
8 Agent
68.1
67.5
57.6
83.3
58.6
63.3
57.0
60.4
Qwen-VI.en VL-Chat
9B
DMSD-CL
32.7
44.5
31.4
31.2
Croes: ToT
62.5
62.7
GoT
58.3
57.7
57.1
59.5
59.5
60.5
53.3
67.4
Ours
69.3 (1.21)
68.2
58.0
86.2
68.9 (6.21)
67.7
58.8
83.2
Zero-shot CoT
61.4
68.1
62.1
60.7
33.0
51.6
40.9
27.7
Automatic Prompt Engineer
63.4
36.2
Plan-and-Solvve
60.9
66.0
58.5
63.5
37.7
46.8
38.0
37.4
Generated Knowledge Prompting
63.2
67.5
59.9
06.8
47.5
47.3
41.6
55.3
83 Agent
60.8
66.6
59.6
62.0
19.2
62.9
60.0
11.8
MiniCPM-V2
2.8B
DMSD-CL
47.5
43.1
50.3
Croes: ToT GoT
46.4
37.9
48.5
57.9
42.1
41.6
95.4
59.2
43.3
42.9
95.5
Ours
72.5 (9.31)
74.5
65.9
80.8
72.0 (12.81)
73.9
66.8
78.1
So LLMs Commander
Zero-shot CoTT
71.4
62.8
70.5
59.4
61.1
53.9
66.1
Automatic Prompt Engineer
68.7
72.8
64.1
73.9
60.0
60.4
53.1
68.1
Plan-and-Solvy
66.3
70.2
62.5
71.8
53.7
56.5
49.6
58.5
Generated Knowledge Prompting Agent
69.5
73.6
66.3
74.2
65.
61.7
53.6
82.8
71.9
67.8
75.7
67.4
66.5
58.0
80.3
Gemini Pro
60013
DMSD-CL
48.2
57.9
Croes- ToT
67.5
66.6
66.0
69.0
59.5
59.3
60.6
58.3
GoT
70.5
71.8
75.4
66.2
66.4
66.0
70.8
62.6
Ours
73.8 (3.31)
76.2
70.1
78.3
71.9 (4.5f)
73.8
08.4
76.7
Zero-shot
78.5
71.3
7T.4
68.9
74.2
65.3
Automatic Prompt Engineer
72.9
78.9
70.3
75.6
67.1
73.8
Plan-aud-Solve
73.6
77.8
70.2
77.1
67.4
73.1
64.2
71.0
Generated Knowledge Prompting Agent
76.3
79.7
73.5
79.2
71.8
76.4
68.9
75.1
S0.2
74.6
81.5
73.2
77.8
70.3
GPT-4o
3001
DMSD-CL
91.2
Croes: ToT GoT
71.9
72.4
78.7
66.2
68.5
62.3
95.5
53.5
74.7
77.4
76.7
72.9
71.4
73.2
76.7
66.9
Ours
81.4 (3.51)
83.1
78.9
84.2
76.5 (3.31)
79.7
74.6
79.4
3.7
Subtask Invocation Frequency Analysis
Fig. shows the invocation frequency for each subtask agent. Context Modeling, Sentiment Analysis, and Image Summarization are triggered in all cases (2,409 times each), reflecting their necessity for every input. By contrast, Rhetorical Device Recognition, Facial Expression Recognition, and Scene Text.
11

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Table 3: Ablation study on the MMSD dataset.
MiniCPM-V2
DoepSeek-VE VL
GPT--0
Ablation Setting
Acc.
F1
Acc.
F1
Acc.
w/o Context Modeling
72.4
73.9
56.9
70.9
78.9
81.2
Sentiment Analysis
71.3
72.2
60.2
71.9
79.1
81.4
w/o Rhetorical Device Recognition
06.0
72.4
27.9
63.9
77.2
79.8
w/o Facial Expression Recognition
72.2
41.7
78.5
w/o Tmagg Summarisation
71.4
73.1
43.8
67.3
78.8
81.3
w/o Sceue Text Recognition
71.5
28.8
63.4
77.8
80.1
Full Model
72.5
74.5
61.1
69.4
81.4
83.1
Table 4: Ablation study on the MMSD 2.0 dataset.
Ablation Setting
MiniCPM-V2
DeepSeek-VV VL
GPT-4o
Asc.
F1
Acc.
F1
Acc.
Modeling
59.4
w/o Sentiment Analysis
71.2
71.6
59.0
42.9
74.3
77.4
Rhetorical Device Recognition
67.9
72.3
60.0
42.5
73.8
77.0
w/o Facial Expression Recognition
71.5
72.4
59.7
43.2
74.0
77.1
w/o Image Summerization
71.9
73.3
59.5
13.0
74.2
77.3
w/o Some Text Recognition
67.91
62.2
60.2
43.1
73.2
76.4
Full Model
72.0
73.9
60.5
46.7
76.5
79.7
Recognition are conditionally invoked, with call counts of 1,479, 1,285, and 935, respectively. The lower frequencies for the latter agents are due to the absence of rhetorical devices, facial expressions, or scene text in significant proportion of samples.
This distribution highlights two important findings. First, the frequency of subtask invocation is closely linked to the nature of the input data (i.e., whether textual or visual clues are present). Second, agents such as Facial Expression Recognition and Scene Text Recognition though not always activated capture critical cues for sarcasm detection when relevant. These results further support the rationality of designing dedicated subtasks and specialized agents, as each contributes unique information to the overall multi-agent sarcasm understanding framework.
3.8 Distrf--iitributtion Results
evaluate the robustness and generalization of Commander-GPT, we conducted out-of-distributiio sarcasm detection experiments on the SemEval 2018 Task 3 dataset. We selected two LLMs as the commander: the open-source MiniCPM-V2, and the SoTA model Claude-3.5 (serving
commander).
Fig. 2 reports the Fl and Acc. scores of all methods. On MiniCPM-V2, Gom achieves an Fl of 73.6% and an Acc. of 72.6%, outperforming all prompt-based and agent-based baselines. Specifically, com- pared to the strongest baseline (53 Agent, 69.5%, Acc. 67.5%), Commander-GPP brings improvements of 4.1% and 5.1%, respectively.
When using Claude 3.5 as the commander, Commander- still maintains the leading performance, achiev- ing 62.1% F1 and 51.7% Acc. This exceeds the best baseline (Zero-shot CoT, F1 59.8%, Acc. 46.8%) by 2.3% and 4.9%, respectively. It is worth noting that the performance boost remains consistent even when switching to more powerful closed-source model as the commander, demonstrating the flexibility and model-agnostic nature of our framework.
Overall, these results confirm that Commander-GPP can robustly generalize to new domains and leverage the advantages of different backbone LLMs. Whether with open-source or SOTA commercial models as the commander, our framework consistently delivers state-of-the-art results in sarcasm detection.
3.9
The Impact The Number of Subtasks
From the experimental results Fig. g can be seen that as the number of submodules increases, the model's F1 score on the MMSD and MMSD 2.0 datasets follows three-stage trend of "slow start, rapid rise, then gentle plateau with final slight boost." The first two basic modules Context Modeling and
12

---

Published in Transactions on Machine Learning Research (MM/ MYYY
2500
24019
2409
2000
1500
1479
1285
1000
935
500
0
Context Modeling
Sentiorst Anolysi sis Rbetorical ler Recognitlinn
Facial Espression Recogaltisn
Image Sammarication
Sceue Text Revugnition
Figure 4: Agent call froquency on the MMSD 2.0 dataset.
Fon-dbar Cuf
Plan
Sun
Sarvtber Call
Auaeerti Pronge
80
90
70-
70
00:3
F1(%)
60
15.4
59.8
12.7
ACC(%)
(0
$1.2
60-
90
40
60
32
50
20-
180
19
10
MisiCPM-Y-2
S.nnne3.5-Sonnct
Mi-VCPM--V-2
Claule-3.-Somme
Figure 5: The experimental results of MiniCPM-V-2 and Claude-3 on SemEval 2018 Task 3 were statistically evaluated.
Sentiment Analysis-brine only limited gains until the introduction of Image Summarization and Scene Text. Recognition triggers a substantial jump, with Scene Text Recognition especially filling the gap bet ween textual and visual information. Adding Rhetorical Device Recognition and Facial Expression Recognition thercafter still yiclds improvements, but with diminishing marginal returns. On MMSD 2.0, thanks to higher data quality, the modules cooperate more smoothly and deliver steadier performance gains. Therefore, under resource constraints, priority should be given to Image Summarization and Scene Text Recognition, followed by Rhetorical Device Recognition and Facial Expression Recognition, while cross-modal fusion strategies can be explored to further unlock the potential of later modules.
3.10 Routing Score Visualization
To further interpret the decision mechanism of Commander-GPPT, we visualize the routing scores assigned to each subtask agent for representative samples from the MMSD 2.0 dataset. Fig. 0 presents a heatmap where each column corresponds to an input sample and each row corresponds to one of the six subtask agents. The color intensity indicates the routing score (i.e., the normalized weight assigned by the router to each agent).
The heatmap reveals clear patterns in the agent assignment. For samples containing explicit rhetorical devicos, the router assigns high scores to the Rhetorical Device Recognition agent. Samples with prominent facial expressions or scene text result in higher weights for the Facial Expression Recognation and Scene Tert Recognation agents, respectively. In contrast, samples lacking such cues rely more on the Contert
13

---

Published in Transactions on Machine Learning Research MYYYY
81.5
76.5
81.0
91)%)
76.0
80.5
F1(%)
80.0
15.5
79.5
75.0
79.0
78.5
74.5
78.0
77.5
73,5
3
5
2
5
(MMSDD 2.0)
Fl score variation curves for the MMSD and MMSD 2.0 datasets with varying numbers of sub- modules, where the modules are added in the following order: Context Modcling, Sentiment Analysis, Image Summarization, Scene Text Recognition, Rhetorical Device Recognition, and Facial Expression Recognition.
Figure 6:
Modeling and Sentiment Analysis agents. This dynamic and adaptive score distribution demonstrates that the router effectively identifies and leverages the most relevant subtasks for each input, thereby improving interpretability and robustness.
Overall, the routing score visualization provides direct evidence that the proposed multi-agent archilecture selectively integrate different information sources based on the imput characteristics, supporting both the transparcncy and effectiveness of our approach.
1.0
Scene Text Recognition
0.8
Sestiment Analysis
Facial Expreasion Evaogaiiin
3.6
Image Summarization
0.4
Rhetorical Device Recognition
0.2
Content Modeling
241 481
721 961 1201 1441 1681 1921 2161 2401
Figure 7: Heatmap of subtask agent counts on the MMSD 2.0 dataset.
3.11 Error Analysis
In the first three examples, Commander-GPP demonstrates its strength in multimodal sarcasm detection by accurately identifying sarcastic or non-sarcastic content. The first example, "when my best friend tells me he got full marks in the exam," contains sarcasm, which GPT-4o and Claude-4 both fail to detect, labeling it as non-sarcastic. However, Commander-GPPT successfully recognizes the sarcastic intent by utilizing combination of subtasks, including Context Modeling, Sentiment Analysis, Facial Expression Recognition, Scene Text Recognition, and Image Summarization. This highlights Commander- GPT's ability to capture muanced contextual cues across both text and images, which are crucial for accurate sarcasm detection. Similarly, in the second example, "this is reality...lol,, GPT-4o and Claude-4 misclassify it as non-sarcastic, but Commander- GPT correctly identifies it as sarcastic, again by leveraging multimodal cues and the com- bination of its subtasks. These successes illustrate the advantage of breaking down the task into subtasks that handle specific aspocts of sarcasm, like emotional tone, context, and visual cues, enhancing the model's overall performance.
14

---

Published in Transactions on Machine Learning Research (MM/YYYY)
In the fourth example, "when you say I love you to person you hate," Commander-GGP misclassifies the sarcastic input non-sarcastic, while both GPT-4o and Claude-4 correctly classify it as sarcastic. This failure suggests a limitation in Commander-GPIT's model, especially in scenarios where certain subtasks might fail to capture the sarcasm cucs. For instance, despite the clear sarcastic nature of the statement, GPT might have missed critical contextual or emotional cues from the text or visual content. While the model effectively uses Context Modeling and Sentiment Analysis, it appears to misinterpret the sarcasm in this specifie scenario, likely due to a failure in one of the subtasks. This highlights the potential weakness of multimodal models when handling certain kinds of sarcasm that do not have clear visual or textual cues, emphasizing the need for further refinement of the task decomposition and routing strategy to better handle such cases.
reality!
T
when my best friend tells be got full marks in the exam
this reality ... Iol
lowkey miss this outfit and this night
when you say love you person you hate !...
True Label
Sarcastic
Sarcastic
Non-Sarcastic
Sarcastic
GPT-4o
Non-Sareastic
Non-Sarcastic
Sarcastic
Surcastic
Claude-4
Non-Sarcastle
X
Non-Sureastic
X
Sareastic
Sarcastic
Commander
Context Modeling
Context Modeling
Context Modeling
Context Modeling
GPT:
Sentiment Analysis
Sentiment Analysis
Sentiment Aualysis
Facial Expression
Sentiment Aualysis
Image Summarization
Scene Text Recognition
Facial Expression
Image Summarization
Scene Text Recognition
Image Summarization
Image Summarization
Figure 8: The figure shows four examples in a multimodal sarcasm detection task, along with the classification results of different models (GPTT-4o, Claude-4, Commander-GPP for these examples.
4 Conclusion
In this paper, we proposed GPT, a novel multi-agent routing framework for multimodal sar- casm detection. Our approach decomposes the complex sarcasm detection task into cognitively meaningful subtasks, dynamically routing each input to the most suitable specialist agent. The central commander inte- grates information from all subtask agents, enabling adaptive and fine-grained reasoning across both textual and visual modalities. Extensive experiments on multiple benchmarks demonstrate that Commander-GPP consistently achieves state-of-the--rt performance. Further analyses confirm the effectiveness, robustness, and strong generalization ability of our framework. We believe this work offers new insights into explainable and modular design for sarcasm and sentiment analysis.
Discussion on Hallucination. A potential risk when applying large language models to sarcasm detection is the occurrence of hallucinations, where the model invents context, emotions, or visual details that are not supported by the input. For example, we observed cases where commander LLM attributed sarcasm to an "angry facial expression" even though no face was present in the image, or generated an explanation that contradicted the literal text.
! further illustrates this issue. The input was an image of hamburgers and fries with the caption "feeding my abs nothing but the best quality beef." Humans can casily recognize the sarcastic intent here: the mention of "abs" contrasts ironically with the unhealthy fast food in the picture. However, the LLM produced hallucinations across multiple analysis steps:
Our own case study (Fig.
15

---

Published in Transactions on Machine Learning Research (M//YYY)
8
Contest Modaling The burgers and frie, match the caption literally, showigg
healthy proten.
You labeling whether spocific andiyii slep necessry decide
Sentiment The phrase quality regal idty detected.
Kheorrccal Device This simele hyperbole for advertsing-cot
feeding my nothing
quality beef
(magee Summnrizttion:
shows hamburgers and afgned
the praise quality
Final Judgment:AA
segged mon-sarcastic protise, nol armsom
Figure 9: Commander-GPT Hallucination Case
Context Modeling: it claimed the burgers and fries "match the caption literally," inventing advertising context about healthy protein that does not exist.
Sentiment Analysis: it highlighted "best quality" as a "positive, admiring tone," hallucinating a purely positive sentiment while ignoring the ironic undertone.
Rhetorical Device: it labeled the statement "simple hyperbole for advertising" rather than irony, a clear misinterpretation of the rhetorical device.
Image Summarization: it concluded that the picture was "aligned with the praise of beef quality," overlooking the tension between junk food and the notion of feeding one's abs.
As shown in the figure, these hallucinations accumulated in the final stage, leading the model to incorrectly judge the overall message as "non-sarcastic praise." This example demonstrates how subtle pragmatic cues can be lost when the model hallucinates supportive evidence, producing confident but misleading explana- tions. Such hallucinations are particularly problematie in sarcasm detection, the task already relies subtle pragmatic cucs and fino-grained multimodal alignment. While our modular design partially mit- igales this issue by constraining each agent to well-defined subtask, hallucinations can still emerge in the integration stage when the commander synthesizes final decisions. Future research could incorporate uncertainty estimation, factual consistency checks, or human-in-the-loop validation to reduce the impact of hallucinations in real-world deployment.
Limitations. Although Commander shows strong effectiveness in multimodal sarcasmn detection, sev- eral limitations remain. First, scalability poses challenge: incrcasing the number of agents or adopting larger commanders inevitably raises inference latency and memory consumption, which may hinder real- time or large-scale applications. While our analysis in $3.3 reports efficiency trade-offs, more systematic profiling is needed. Second, the framework relies on external specialist models (e.g., BLIP-2, OCR-2.0, RoBERTa, GPT-4o). This dependency makes stability and reproducibility sensitive to changes in upstream APTs or pretrained checkpoints. Finally, for real-world deployment, the current design is resource-intensive. Promising directions to make the framework lighter include agent pruning and parameter sharing across subtasks, compression and distillation of commanders, or integrating all subtasks into unified but modular multimodal backbone. We view these important next steps toward reducing resource consumption while maintaining interpretability.
Acknowledgements. This paper is supported by The National Social Science Fund of China (No. 25AYY001), Hong Kong RGC Theme-based Research Scheme 4015-400/22-N) and a grant under the scheme of Collaborative Research with World-leading Rescarch Groups The Hong Kong Polytechnic University (project no. G-SACF), Natural Science Foundation of Henan Province of China (242300421412), National Natural Science Foundation of China (No.62201572).
16

---

Published in Transactions on Machine Learning Research (MM/YYY)
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almoida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint MiXiv:2303.08774, 2023.
Sajal Aggarwal, Ananya Pandey, and Dinesh Kumar Vishwakarma. Modelling visual semantics via image captioning to extract enhanced multi-level cross-modal semantic incongruity representation with attention for multimodal sarcasm detection, arXiv preprint arXiv:2408.0255 2024.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in neural information processing systems, 35:23716 23736, 2022.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision- language model with versatile abilities. arXiv preprint arXiv:2308.12965 2023.
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pp. 17682 17690, 2024.
Yitao Cai, Huiyu Cai, and Xiaojun Wan. Multi-modal sarcasm detection in twitter with hierarchical fusion model. In Proceedings of the 57th annual meeting of the association for computational linguistics, pp. 2506 2515, 2019.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual- linguistic tasks. In Proceedings of the IEEE/CV /CF conference on computer vision and pattern recognition, 2024.
pp.
DeepSeck-AT, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyn Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Hnang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang. Xinnan Song. Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang. Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia Hc, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseck-rl: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL https: /mapsi///rxivv wttpp/wwwww...p//....////// 12948
17

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewci Fang, Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06395, 2024.
Mengzhao Jia, Can Xie, and Liqiang Jing. Debiasing multimodal sarcasm detection with contrastive learning. In Proccodings of the AAAI conference on artificial intelligence, volume 38, pp. 18354-18362, 2024.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199 -00-992-21111-11 2022.
Jumman Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning. pp. 22888-12900 PMLR, 2022.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lec. Visual instruction tuning. Advances in neural information processing systems, 11344822-49911 2023a.
Jiacheng Liu, Alisa Liu, Ximing Lu, Sean Welleck, Peter West, Roman Le Bras, Yejin Choi, and Hannaneh Hajishirzi. Generated knowledge prompting for commonsense reasoning, arXiv preprint Axiv:211000838) 2021.
Yaochen Liu, Yazhou Zhang, and Dawei Song. A quantum probability driven framework for joint multi-modal sarcasm, sentiment and emotion analysis. IEEE Transactions on Affective Computing, 15(1):326 341, 2023b.
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang. Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, al. Deepseek-vl: towards real-world vision-language understanding. arXiv preprint arXiv:2403.0557 2024.
Libo Qin, Shijue Huang, Qiguang Chen, Chenran Cai, Yudi Zhang, Bin Liang, Wanxiang Che, and Ruifeng Xu. Mmsd2. O: towards reliable multi-modal sarcasm detection system. arXiv preprint 0771: 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkinn Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8888 8763. PmLR, 2021.
Leonardo Ranaldi, Giulia Pucci, Federico Ranaldi, Elena Sofia Ruzzetti, and Fabio Massimo Zanzotto. A tree-of-thoughts to broaden multi-step reasoning across languages. In Findings of the Association for Computational Linguistics: NAACL 2024, pp. 1229-1241, 2024.
Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 14398 14409, 2024.
Binghao Tang, Boda Lin, Haolong Yan, and Si Li. Leveraging generative large language models with visual instruction and demonstration retrieval for multimodal sarcasm detection. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 1732 1742, 2024.
Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka- Lee, and Ee-Peng Lim. Plan- and-solve prompting: Improving zero-shot chain-of thhuuugt reasoning by large language models. arXiv preprint arXiv:2305.04099. 2023.
Peng Wang, Yongheng Zhang, Hao Fei, Qiguang Chen, Yukai Wang, Jiasheng Si, Wenpeng Lu, Min Li, and Libo Qin. S3 agent: Unlocking the power of vllm for zero-shot multi-modal sarcasm detection. ACM Transactions Multimedia Computing, Communications and Applications, 2024a.
18

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Tongguan Wang, Junkai Li, Guixin Su, Yongcheng Zhang, Dongyu Su, Yuxue Hu, and Ying Sha. Relmufn: Relational context learning and multiplex fusion network for multimodal sarcasm detection. arXiv preprint arXiv:2412.13008, 2024b.
Haoran Wei, Chenglong Liu, Jinyue Chen, Jia Wang, Lingyu Kong, Yanming Xu, Zheng Ge, Liang Zhao, Jianjian Sun, Yuang Peng, et al. General ocr theory: Towards ocr-2.0 via a unified end-to-end model. 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng. Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https://t//aatte org.oogg//aooo aat/a/b////000 09388
Qu Yang, Mang Ye, and Bo Du. Emollm: Multimodal emotional understanding meets large language models. arXiv preprint 2xXiv:2406116442 2024.
Ben Yao, Yazhou Zhang, Qiuchi Li, and Jing Qin. Is sarcasm detection step-by-step reasoning process in large language models? In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 25651-25659, 2025.
Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangeheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.0465 2024.
Tan Yue, Rui Mao, Heng Wang, Zonghai Hu, and Erik Cambria. Knowlenet: Knowledge fusion nelwork for multimodal sarcasm detection. Information Fusion, 100:101921, 2023.
Yazhou Zhang, Chunwang Zou, Qimeng Liu, Lu Rong, Ben Yao, Zheng Lian, Qiuchi Li, Peng Zhang, and Jing Qin. Are mims trapped in the visual room?, 2025. URL
23272.
Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.019910 2022.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Emhancing vision- language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023.
A Appendix
A.1
Related Work
A.1.1 Multimodal Large Language Models
Multimodal large language models (MLLMs) aim to unify language and vision understanding throngh joint pretraining and alignment. across modalities. Early frameworks such as CLIP (Radford al,, 2021) and BLIP (Li et al., 2022) align vision and text representations through contrastive learning or caption supervision viiion..
These models laid the groundwork for subsequent. generative MLLMs like Flamingo (Alayrac et. al., 2022). MiniGPT-4 (Zhu et al. 2023), and LLaVA (Liu et al. 2023a), which integrate vision encoders with pretrained LLMs via lightweight adapters projection lavers. These models support open-ended visual question answering and caption generation. Qwen-VL (Bai et al. 2023), InternVL (Chen et al., 2024), and Emu2 (Sun
at al. 2024) further explore multi-granular alignment, spatial grounding, and tool-augmented multimodal reasoning.
While MLLMs such as GPT-4o and Gemini have achieved strong performance in visual grounding and factual question answering, they consistently underperform on high-level pragmatic tasks like sarcasm and irony Yao
19

---

Published in Transactions on Machine Learning Research (MM/YYY)
et a 2025; Yang et al, 2024). Sarcastic expressions often rely rhetorical contradiction and contextual incongruity, which current models struggle to capture. Notably, Yao et al. (2025) proposed advanced prompting strategies, Le., graph-of-cue and bagging-of-cue to improve sarcasm detection, but found them less effective than standard input-output approach. This suggests that increasing prompt complexity alone cannot overcome the reasoning limitations monolithic models. contrast, our proposed Commander-GP1 framework explicitly decomposes sarcasm detection into specialized sub-tasks, each handled by expert agents under centralized coordination, enabling more effective and interpretable multimodal sarcasm understanding.
A.1.2 Multimodal Sarcasm Detection
Multimodal sarcasm detection is a complex task that requires the integration of linguistic, visual, and contextual information to resolve non-literal intent and pragmatic ambiguity. In recent years, researchers have explored variety of neural architectures and multimodal fusion strategies to address the inherent challenges of this task.
Recent work has shifted toward more advanced architectures that better capture the nuanced interplay between modalities. For example, Wang et al (2024a) proposed the $ Agent framework, which employs visual large language models and integrates multi-perspeetive analysis to enhance zero-shot multimodal sarcasm detection. Similarly, Tang et al. (2024) utilized generative MLLMs equipped with instruction templates and demonstration retrieval to improve the understanding of complex sarcastic cues. Aggarwal et al. (Aggarwal et al., 2024) designed a framework capable of processing multimodal input hiples-inchuding text, images, and descriptive captions highlighting the benefit of incorporating multiple contextual sources.
To further enhance semantic understanding, researchers have explored the integration of external knowledge bascs. KnowleNet (Yue et al.. 2023). for instance, leverages ConceptNet to inject prior knowledge and assesses cross-modal semantic similarity at both sample and word levels. Other recent models, such as RCLMuFN (Wang et al.. 2024b), introduce relational context learning and multi-path fusion networks to improve generalization and robustness in sarcasm detection.
Despite these advances, most existing methods either rely on monolithic model architectures or simple cross- modal fusion, which often fail to capture the composite and context-dependent nature of sarcasm. In this context, our proposed fommander-GGPT addresses these challenges by decomposing multinodal sarcasm detection into cognitively meaningful sub-tasks, each handled by specialized agent under centralized coor- dination. This design aims to improve the performance of multimodal sarcasm detection.
A.2 Algorithm
The algorithm is shown in Alg,
A.3 Prompt Construction
For instruction-tumed LLMs (e.g., GPT-4o, DeepSeek VL), agent routing is formulated as a natural language inference task. For each (x, Tk), construct the prompt:
20

---

Published in Transactions on Machine Learning Research (MM/YYYY)
Algorithm 1 Commander-GPII Modular Multimodal Sarcasm Understanding
Input: Multimodal imput (t,
Output: Sarcasm prediction
3: 1. Agent Routing:
4: Obtain agent activation vector (x))))...,TT using routing scorer.
Ag < {Ax (x)==
2. Subtask Execution:
for all activated agent A Ar in parallel do
8: Compute subtask output 2μ- = (x)x)
9:
end for
10: Collect all outputs z = [zk A E Ar]
11: 3. Result Integration and Final Decision:
12: if commander is encoder-based then
13: Fuse outputs: hused + concat(z)
14:
Compute hidden: Doontext +- [ERT((CLS; hiused SEP))
15:
Predict: softmax W Wouthe hcontext + bout)
16: else
17: Format prompt Pinput from zµ
18: Predict: CLLMM (Pinput)
19: end if
System: You are a military commander analyzing intelligence to deploy specialized units for sarcasm delection.
Input: Text: "(<t_i>}", Image: {<v_i>}
Task: Determine which of the following units should be deployod:
"ontext morreett_mdeeligg Analyze broader conversational context
"sentiment analysis": Extract emotional polarity
"rhetorical device": Identify irony, hyperbole, etc.
Dnccialexpresssoon" Analyze facial emotions in image
"imagc_summarizatioo Describe visual content
"scene text": Extract text from image
Output:
0/1, anciimment__aalliiii 0/1, ...}
A.4 Comparison of Commander Models
Fig. 10 (App. A4) presents the Fl scores of various commander models on the MMSD and MMSD 2.0 datasets. The results reveal significant performance differences among LLM backbones. BERT achieves the highest Fl scores (86.7 on MMSD, 85.8 on MMSD 2.0), primarily due to its fine-tuned nature the sarcasm detection task. In contrast, mainstream vision-language models, such as Yi-VL (59.1/55.3) and DeepSeek-VL (61.1/60.5), perform substantially worse, likely due to their weaker language modeling capability or suboptimal adaptation to sarcasm-rich domains. Larger multimodal models like Qwen- VL, MiniCPM-V2, Gemini Pro, and GPT-4o show a clear performance hicrarchy: GPT-4o delivers the strongest results among MLLMs (81.4/76.5), benefiting from both superior language modeling and robust multi-modal reasoning. Overall, these results highlight the importance of both model architecture and scale for effective
21

---

Published in Transactions on Machine Learning Research
80
70
DeepSeck-
Figure 10: Comparison of commander models on the MMSD and MMSD 2.0 datasct.
sarcasm detection, and retonstrate that Go can flexibly leverage wide range of L.MMs, with significant performance gain observed as model quality improves.
A.5 Prompt Design
We reformulate agent routing as a binary decision task: given the text, image, and candidate analysis step, the model outputs whether this step is necessary for sarcasm detection, in JSON format.
System: You are labeling whether a specific analysis step is necessary to decide sarcasm.
User:
TEXT: "[<t>}"
IMAGE: {<v>} (attached)
TASK: "(<T k>}" e.g., "facial expression", "scene text", sumaaasaummiiittinnn" "con- text pot__modeinng,, "sentiment_anaapp" "rhetorical_deee device"
Instruction: Reply in JSON ONLY: {"need"; "Yes" or "No"]
22


Reviews:
[{"review_id": "pJKTwGPZRr", "reviewer": "Reviewer_xkLK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes Commander-GPT, a modular multi-agent framework for multimodal sarcasm detection. Instead of relying on a single monolithic model, it decomposes the task into six cognitively grounded subtasks. A central “commander” dynamically routes each input to appropriate specialist agents, collects their outputs, and integrates them to make the final decision. The framework is evaluated on the MMSD and MMSD 2.0 benchmarks across various commander models (from fine-tuned BERT to large models like GPT-4o and Gemini Pro). Experiments show consistent improvements in F1 scores over several prompting-based baselines, with the largest gains observed for smaller and mid-sized models. \n\n**Strengths**\n\n- The paper identifies a well-motivated problem by focusing on the difficulty of multimodal sarcasm detection. The proposed approach shows that decomposing sarcasm understanding into six sub-modules is cognitively plausible and helps explainability, addressing the black box nature of existing LLM-based sarcasm detectors.\n- The proposed framework is quite flexible and model agnostic. Experiments show that the CommanderGPT architecture can orchestrate various LLM/MLLM backbones (from small to large), showing robustness and adaptability.\n- The authors evaluate across multiple commander sizes and provide ablations, subtask invocation frequency analysis, routing visualizations, and out-of-distribution tests, which strengthen the empirical claims.\n- Experiments indicate sizable F1 score improvements, especially in resource-constrained settings, showing that structured reasoning can compensate for smaller model capacity.\n\n**Weaknesses**\n\n- While modularity can bring performance gains, the paper does not report latency or computational overhead of routing and invoking multiple agents, which could be significant in real applications.\n- The routing classifier is trained on ~5,000 GPT-4o-labeled samples, but the quality and diversity of these annotations are not analyzed. If these labels are biased or limited, routing could overfit or fail in unseen scenarios.\n- Details about how the commander models are trained (especially the encoder-based version) are somewhat sparse.\n- While performance gains are reported, there is little qualitative or category-wise error analysis showing where Commander-GPT helps or still fails (eg., subtle sarcasm, implicit sentiment contradiction, etc.).", "claims_and_evidence_yn": "Yes", "claims_explanation": "The core performance claims of the paper are largely supported by clear and convincing evidence: Commander-GPT consistently outperforms prompting-based and monolithic baselines on MMSD and MMSD 2.0, with substantial gains across different commander backbones (from BERT to GPT-4o and Gemini Pro). Ablation studies support the contribution of each subtask module, and an additional evaluation on SemEval 2018 Task 3 provides evidence of generalization. However, some claims, particularly about efficiency, scalability, and the reliability of routing are less substantiated, as the paper does not report computational overhead, analyze the quality of the GPT-4o–distilled routing labels, or evaluate on noisy real-world data. Overall, the experimental evidence is convincing for accuracy improvements, but less so for the broader claims of practicality and robustness.", "audience_yn": "Yes", "audience_explanation": "The paper focuses on a problem setting of interest to the broader ML community with practical application scenarios, and the proposed approach would provide an interesting case study of a simplistic approach to leveraging multiple models and prompt structure for achieving success in an otherwise difficult task.", "requested_changes": "Following should be addressed by authors for an accept recommendation: \n\n1. Report inference time, number of model calls per sample, and GPU/CPU memory usage.\n\n2. Show examples where Commander-GPT succeeds while other models fail, and vice versa, to illustrate what the subtask decomposition captures.\n\n3. Describe the GPT-4o distillation process in more detail (prompt design, inter-annotator agreement if any, label quality checks).\n\n4. Clarify Commander-GPT model training details with chosen hyperparameters and training duration.\n\n5. Explain the limitations more explicitly by including discussion of scalability challenges, reliance on external models, and how the framework could be made lighter or faster for real-world deployment.", "broader_impact_concerns": "The work poses no major ethical risks, but since Commander-GPT is trained on social media datasets (e.g. MMSD, MMSD 2.0), it may inherit demographic or cultural biases and show uneven performance across groups. It also relies on large models like GPT-4o and Gemini Pro, which can propagate existing biases. A short Broader Impact Statement acknowledging these risks and outlining mitigation plans would be appropriate.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ipVB2WqBeL", "reviewer": "Reviewer_FhKc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces Commander-GPT, a modular multi-agent framework for multimodal sarcasm detection. It decomposes the sarcasm detection task into $6$ subtasks and the commander dynamically choose which of these subtasks should be carried out given the input. Empirical experiments on MMSD and MMSD 2.0 datasets are carried out to exhibit its improvement compared to the baselines.", "claims_and_evidence_yn": "No", "claims_explanation": "- Optimization clarity: Eq. (2) states joint minimization over $\\alpha,\\theta_C$​, but the method trains a router via Eq. (3)–(4) with GPT-4o-distilled labels plus threshold gating at inference. Clarify whether $\\alpha$ is jointly optimized with $\\theta_C$​ or learned in a staged manner only.\n- Agent dependence: Performance hinges on external specialists (BLIP-2, OCR-2.0, RoBERTa, etc.); robustness to swapping/weaker agents is not evaluated.\n- Overhead: Multi-agent orchestration likely increases latency/memory; no runtime/resource comparison vs. single strong LLM baselines (e.g., fine-tuned Llama-3-8B). (Table 2 includes GPT-4o/Gemini Pro but not a monolithic fine-tuned open LLM.)", "audience_yn": "Yes", "audience_explanation": "- This paper proposes a pipeline, Commander-GPT, that employ different large language models to conduct $6$ subtasks that help sarcasm detection.\n- The experiments demonstrate the efficacy of the proposed framework, and that each subtask contribute meaningful impact to the final detection.", "requested_changes": "- In equation (1), I suppose the right-hand side should be indexed up to $K$?\n- The curly brackets are missing in some expressions, e.g., second last line on page 3, second line on page 4, equation (1), etc. When using the curly brackets in tex, it should be companied by back slash. \n- Please explicitly describe the training pipeline (router via (3)–(4), commander via (5)–(7)), and state whether any joint or end-to-end optimization of $\\alpha$ and $\\theta_C$​ is performed. If not, rephrase Eq. (2) or present a bilevel/alternating optimization view.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wkWO1iiPuS", "reviewer": "Reviewer_sW8L", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces Commander-GPT, a modular multi-agent framework designed for multimodal sarcasm detection, inspired by military command theory. Instead of relying on a single large language model, the system decomposes sarcasm detection into six subtasks, context modeling, sentiment analysis, rhetorical device recognition, facial expression recognition, image summarization, and scene text recognition, each handled by specialized agents. A central commander model dynamically routes tasks to the most suitable agents, integrates their outputs, and makes the final sarcasm judgment. Experiments on MMSD and MMSD 2.0 benchmarks show that Commander-GPT outperforms state-of-the-art baselines by up to 11.7% in F1 score, demonstrating improved accuracy, robustness, and generalization.", "claims_and_evidence_yn": "Yes", "claims_explanation": "the claims are well supported by clear evidence. The paper justifies the need for subtask decomposition, details the Commander-GPT design, and evaluates it thoroughly on MMSD and MMSD 2.0 against strong baselines. Results consistently show significant F1 gains, and ablation plus out-of-distribution tests further confirm robustness. Together, these analyses convincingly support the authors’ claims.", "audience_yn": "Yes", "audience_explanation": "TMLR’s audience would be interested because sarcasm detection is a challenging and unsolved problem in NLP, especially in multimodal settings. The paper introduces a novel modular framework, Commander-GPT, which not only advances performance but also provides insights into explainable, task-decomposed reasoning with LLMs. These contributions are valuable to researchers in language understanding, multimodal learning, and sentiment analysis, making the findings relevant to the community.", "requested_changes": "1. I think the authors should also mention and discuss other multi-modality Sarcasm Detection approaches. From paper [1] Table 2 we could find more existing researches for this task which also delivered comparable performances. These work should be compared and discussed in this paper.\n\n[1] Jia, Mengzhao, Can Xie, and Liqiang Jing. \"Debiasing multimodal sarcasm detection with contrastive learning.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 38. No. 16. 2024.\n\n2. In Sec 3.3 and Sec 3.4, the authors are encouraged to add more discussion regarding why the proposed method could work better than the others towards the novel design. This discussion will help to enrich the insights of the experimental analysis section. For the current version it is only the description of the numbers without in-depth analysis.\n\n\n3. Lack of the qualitative results. The authors are encouraged to enrich the qualitative results together with failure cases for different approaches on different LLMs. The failure cases can help the authors to understand the limitation of the proposed approach well.\n\n4. Most of the baselines selected (i.e., Plan-and-Solve (Wang et al., 2023), Zero-shot CoT (Kojima et al., 2022), Generated Knowledge Prompting (Liu et al., 2021), Automatic Prompt Engineer (Zhou et al., 2022), and  S3Agent (Wang et al., 2024a)) are the works before 2024, which are not very recent. The authors are encouraged to make comparison with some recent techniques, e.g., [2] and [3].\n\n[2] Ranaldi, Leonardo, et al. \"A tree-of-thoughts to broaden multi-step reasoning across languages.\" Findings of the Association for Computational Linguistics: NAACL 2024. 2024.\n\n[3] Besta, Maciej, et al. \"Graph of thoughts: Solving elaborate problems with large language models.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 38. No. 16. 2024.\n\n5. I am wondering the hallucination of the LLMs on the sarcasm detection. Could the authors elaborate more on discussion regarding this direction?", "broader_impact_concerns": "A potential concern is that applying sarcasm detection in sensitive areas, such as social media monitoring or surveillance, could result in misclassifications that cause misunderstandings or lead to unfair treatment of individuals.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)