Title: FlexAC: Towards Flexible Control of Associative Reasoning in Multimodal Large Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: hallucination

creativity

multimodal large language model
Abstract: Multimodal large language models (MLLMs) face an inherent trade-off between faithfulness and creativity, as different tasks require varying degrees of associative reasoning. However, existing methods lack the flexibility to modulate this reasoning strength, limiting MLLMs' adaptability across factual and creative scenarios. To bridge this gap, we propose equipping MLLMs with mechanisms that enable flexible control over associative reasoning. We begin by investigating the internal mechanisms underlying associative behavior in MLLMs and  find that: (1) middle layers play a pivotal role in shaping model’s associative tendencies,  (2) modifying representations in these layers effectively regulates associative reasoning strength, and  (3) hallucinations can be exploited to derive steering vectors that guide this modulation. Building on these findings, we introduce Flexible Association Control (FlexAC), a lightweight and training-free framework for modulating associative behavior in MLLMs. FlexAC first induces hallucination-guided intermediate representations to encode associative directions. Then, it selects high-association instances to construct effective associative steering vectors, whose strengths are adaptively calibrated to balance creative guidance with output stability. Finally, recognizing the multi-dimensional nature of associative reasoning, FlexAC incorporates task-specific associative vectors derived from a forward pass on a few target-domain samples, enabling models to follow diverse associative directions and better adapt to creative tasks. Notably, our method achieves up to a 5.8× improvement in creativity on Creation-MMBench and a 29\% reduction in hallucination rate on CHAIR, surpassing existing baselines and demonstrating its effectiveness in enabling flexible control over associative reasoning in MLLMs. Our code is available at https://github.com/ylhz/FlexAC.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents FlexAC, a lightweight method for steering associative reasoning in multimodal LLMs by modifying middle-layer model parameters. Using parameter vectors extracted from hallucinated outputs, the approach adjusts the model’s balance between factual accuracy and creativity. Experiment results validate the steerability on hallucination and creativity.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The identification of middle layers as critical for hallucination control is novel and well-supported by empirical analysis. The proposed approach for steering between faithfulness and creativity is thoughtfully designed.\n\n2. The analysis of associative behavior is well-motivated and convincingly executed, especially the layer-wise representation study. These analyses provide solid empirical grounding for the design of the control mechanism.\n\n3. The introduction of the VDAT benchmark fills an evaluation gap for associative reasoning in MLLMs.\n\n\n\n**Weaknesses**\n\n1. **Data quality**: The construction of the steering vector depends on hallucinated input-response pairs, which are derived via blurred inputs and specific prompting. This may introduce distributional biases, e.g., capturing only specific hallucination types. Besides that, top-k selection can be sensitive to outlier data. Some robustness or sensitivity analysis of the vector construction would strengthen the approach.\n\n2. **Connection between hallucination vector and associative reasoning**: While the extracted steering vector clearly controls the factual grounding, it's less certain whether it directly reflects a model’s associative reasoning capability. The paper seems to assume that hallucinated features correspond to associative reasoning, but this isn't explicitly validated. For instance, in Figure 9 (b)–(c), FlexAC-P (presumably more powerful on associative reasoning) performs similarly or worse than FlexAC-C on reasoning-heavy tasks like MMMU and Math. Therefore, it seems to me this method is for hallucination control instead of associative reasoning control.\n\n3. **Model Architecture**: The empirical analysis on layer-wise impact on associative reasoning are demonstrated only on LLaVA model. It remains unclear whether the same behavior holds for other architectures, particularly mixture-of-experts (MoE) models.\n\nQUESTIONS:\n1. Since the non-middle layers show minimal change on hallucination, have the authors explored applying steering vectors to all layers as an ablation?\n\n2. Does FlexAC operate solely on the language-side layers, or is the vision encoder also considered? A brief discussion would clarify the scope of parameter mixing.\n\n3. Equations (7) and (9) suggest both unnormalized and normalized vector mixing. Given the presence of LayerNorm in the transformer, can the authors comment on whether there are redundant computations involved?\n\n4. While most prior efforts on steerable LLMs focus on trainable approaches, they are still relevant and should be discussed in the related work section. Including these works would help position FlexAC more clearly within the broader landscape of controllable and steerable LLM/MLLMs.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes FlexAC (Flexible Association Control), a training-free framework for modulating associative reasoning strength in multimodal large language models (MLLMs). The authors investigate how associative behavior emerges in MLLMs and find that middle layers are critical for shaping associative tendencies. Based on these findings, they develop a method that uses hallucination-guided steering vectors to control the trade-off between faithfulness and creativity. The approach is evaluated on hallucination benchmarks (CHAIR, POPE), creativity benchmarks (VDAT, Creation-MMBench), and general-purpose benchmarks (MME, MMMU, MMStar).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper provides an interesting unified view linking hallucination and creativity through associative reasoning mechanisms, which offers a fresh theoretical perspective on MLLM behavior.\n- The layer-wise analysis identifying middle layers as critical for associative behavior is well-conducted and provides valuable insights into MLLM internal mechanisms.\n- The proposed method requires no additional training, making it practically appealing and computationally efficient.\n\nWeaknesses:\n- The central premise that creativity and hallucination represent two ends of a single \"associative strength\" spectrum is a significant oversimplification.\n- The core technique of manipulating model behavior through activation steering is well-established. This work is essentially a direct application of existing representation steering concepts to MLLMs.\n- A significant portion of creativity evaluation depends on VDAT, a new benchmark that measures ability to generate nouns unrelated to an image. This is an extremely narrow and potentially misleading proxy for divergent thinking. A model could score highly on VDAT by simply ignoring the image, which contradicts the goal of image-grounded creativity.\n\nQUESTIONS:\n1. The steering vector is defined as the difference between hallucinated and grounded representations. Could this vector be capturing \"non-factualness\" or \"context-ignorance\" rather than genuine creativity? Have you explored alternative methods for deriving creativity-enhancing vectors using high-quality human-generated creative content instead of model-induced hallucinations?\n2. How does VDAT performance correlate with functional, grounded creative tasks? Is there evidence that pushing VDAT scores higher doesn't simply train the model to ignore visual input, undermining its utility as a multimodal model?\n3. The method requires empirical identification of optimal control layers for each model. How sensitive is performance to layer selection? For practitioners with new MLLMs, is there a principled way to identify these layers beyond exhaustive testing?\n4. How well does the method transfer across different types of creative tasks beyond the evaluated benchmarks? What evidence supports that the observed behavior represents genuine creativity rather than reduced factual grounding?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents FlexAC, a training-free framework for controlling associative reasoning in MLLMs. Through layer-wise analysis, the authors identify that middle layers are critical for encoding associative behavior. FlexAC constructs steering vectors from the difference between hallucinated and grounded representations in these layers, then applies them during inference to modulate the model's behavior between factuality and creativity. The paper also introduces VDAT, a new benchmark for measuring associative reasoning strength. Experiments on three MLLMs demonstrate that FlexAC can significantly reduce hallucination and enhance creativity depending on the steering direction, while maintaining general task performance.**Strengths:**\n\n1. **Well-executed empirical analysis**: The layer-wise localization experiments clearly demonstrate that middle layers control associative behavior, using both feature analysis and intervention studies.\n\n2. **Practical method**: Training-free, works on existing MLLMs, computationally efficient - just adds vector operations during inference.\n\n3. **Effective bidirectional control**: Successfully reduces hallucination and enhances creativity based on steering direction, addressing different application needs.\n\n4. **New benchmark (VDAT)**: Provides a simple, direct metric for measuring visual associative reasoning strength, filling a gap between hallucination metrics and subjective creativity evaluation.\n5. **Consistent results across architectures**: The method shows improvements across three different model families with varying architectures.\n\n**Weaknesses:**\n\n1. **Similar findings in prior work**: Although well-executed empirical analysis, the observation that middle layers control hallucination has been reported in previous studies[1].  This paper builds upon these existing findings by adding steering vector techniques.\n\n2. **Model-specific tuning**: Optimal layers vary dramatically (DeepSeek: 4-6, Qwen: 15-17), requiring manual search for each architecture. \n\n3. **Heavy reliance on task-specific non-associative features**: The method depends on non-associative features extracted from specific hallucination scenarios, but these features vary significantly across tasks, limiting generalization to different applications.\n\n4. **Limited evaluation on general benchmarks**: The authors only evaluate on 3 general benchmarks (MME, MMMU, MMStar), which is insufficient to fully assess the impact on model's general capabilities. \n\n[1]. Wang, Sudong, et al. \"Towards understanding how knowledge evolves in large vision-language models.\" *Proceedings of the Computer Vision and Pattern Recognition Conference*. 2025.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. **Well-executed empirical analysis**: The layer-wise localization experiments clearly demonstrate that middle layers control associative behavior, using both feature analysis and intervention studies.\n\n2. **Practical method**: Training-free, works on existing MLLMs, computationally efficient - just adds vector operations during inference.\n\n3. **Effective bidirectional control**: Successfully reduces hallucination and enhances creativity based on steering direction, addressing different application needs.\n\n4. **New benchmark (VDAT)**: Provides a simple, direct metric for measuring visual associative reasoning strength, filling a gap between hallucination metrics and subjective creativity evaluation.\n5. **Consistent results across architectures**: The method shows improvements across three different model families with varying architectures.\n\n**Weaknesses:**\n\n1. **Similar findings in prior work**: Although well-executed empirical analysis, the observation that middle layers control hallucination has been reported in previous studies[1].  This paper builds upon these existing findings by adding steering vector techniques.\n\n2. **Model-specific tuning**: Optimal layers vary dramatically (DeepSeek: 4-6, Qwen: 15-17), requiring manual search for each architecture. \n\n3. **Heavy reliance on task-specific non-associative features**: The method depends on non-associative features extracted from specific hallucination scenarios, but these features vary significantly across tasks, limiting generalization to different applications.\n\n4. **Limited evaluation on general benchmarks**: The authors only evaluate on 3 general benchmarks (MME, MMMU, MMStar), which is insufficient to fully assess the impact on model's general capabilities. \n\n[1] Towards understanding how knowledge evolves in large vision-language models.\n\nQUESTIONS:\n1. **Model generalization and comprehensive evaluation**: My biggest concern is the generalization ability of FlexAC. As mentioned in W3 and W4, could you provide evaluation results on more diverse benchmarks to better assess the impact on general capabilities? (You don't need to evaluate on all of these, but results from a representative subset would help demonstrate that the non-associative steering doesn't harm performance on diverse tasks requiring different types of reasoning and association.)\n   - General multimodal benchmarks: MMBench, SEED-Image, MMVet.\n   - Vision-centric benchmarks: CVBench, RealWorldQA.\n   - OCR and document understanding: TextVQA, ChartQA, DocVQA.\n2. **Architectural dependence**: The optimal layers vary dramatically across models. How would you determine optimal control layers for a new model architecture? Is manual search always required, or can this be automated?\n3. **Steering vector stability**: How stable are the derived steering vectors across different prompts and domains? Do they need to be recomputed frequently, or can a fixed set be reused effectively?\n4. **VDAT validity**: Can you provide more rigorous validation of VDAT beyond CLIP similarity? How does it correlate with human judgments of creativity quality? Automatic metrics may not fully capture quality differences in creative tasks.\n\nLIMITATIONS:\nThe authors mention that FlexAC requires white-box access to hidden states. This issue is related to whether the model is open-sourced or not.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes FlexAC, a lightweight and training-free method for flexibly modulating associative reasoning strength in multimodal large language models (MLLMs) to balance faithfulness and creativity. The key contributions include: (1) identifying the critical role of middle layers in associative reasoning; (2) proposing a hallucination-guided steering vector for control; and (3) introducing the VDAT benchmark to evaluate associative reasoning capabilities. Experiments demonstrate that FlexAC outperforms existing methods on multiple benchmarks. The paper is well-structured and the experimental design is sound, though some formulations could be refined for greater rigor.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Offers a fresh perspective by unifying hallucination and creativity as manifestations of associative reasoning and explores modulation via middle layers.\n2. The FlexAC method is simple yet effective, requiring no additional training, making it practical for deployment.\n3. Thorough evaluations across three dimensions: hallucination mitigation (CHAIR, POPE), creativity enhancement (VDAT, Creation-MMBench), and general capabilities (MME, MMMU).\n4. Ablation studies validate key design choices.\n5. The proposed VDAT benchmark addresses the lack of metrics for visual-driven divergent thinking.\n6. The method is compatible with diverse MLLMs, demonstrating strong generalizability.\n\nWeaknesses:\n1. While the paper identifies middle layers as pivotal for associative behavior, it does not explain why these layers exhibit this property .\n\nQUESTIONS:\nWhy was no comparison made with closed-source models like GPT-4V?\n\nLIMITATIONS:\nThe experiments primarily relied on the COCO dataset (object-centric static images) and did not validate performance in video temporal reasoning, abstract art comprehension, or cross-cultural scenarios. If the model fails in dynamic or multimodal associative tasks , would FlexAC's modulation mechanism remain effective?", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-06"}