Aligning VLM Assistants with Personalized Situated Cognition
Authors: Yongqi Li, Shen Zhou, Xiaohu Li, Xin Miao, Jintao Wen, Mayi Xu, Jianhao Chen, Birong Pan, Hankun Kang, Yuanyuan Zhu, Ming Zhong, Tieyun Qian
Accepted at: acl25

Aligning VLM Assistants with Personalized Situated Cognition

Yongqi Li1, Shen Zhou1, Xiaohu Li1, Xin Miao1, Jintao Wen1, Mayi Xu1, Jianhao Chen1,2, Bierong Pan1, Hankun Kang1, Yuanyuan Zhu1,*, Ming Zhong1, Teyun Qian1,2,*
1 School of Computer Science, Wuhan University, China
2 Zhongguancun Academy, Beijing, China
{liyongqi,yyzhu,qty}@whu.edu.cn

Abstract

Vision-language models (VLMs) aligned with general human objectives, such as being harmless and hallucination-free, have become valuable assistants of humans in managing visual tasks. However, people with diversified backgrounds have different cognition even in the same situation. Consequently, they may have personalized expectations for VLM assistants. This highlights the urgent need to align VLM assistants with personalized situated cognition for real-world assistance. To study this problem, we first simplify it by characterizing individuals based on the sociological concept of Role-Set. Then, we propose to evaluate the individuals’ actions to examine whether the personalized alignment is achieved. Further, we construct a benchmark named PCogAlignBench, which includes 18k instances and 20 individuals with different Role-Sets. Finally, we present a framework called PCogAlign, which constructs a cognition-aware and action-based reward model for personalized alignment. Experimental results and human evaluations demonstrate the reliability of the PCogAlignBench and the effectiveness of our proposed PCogAlign. We will open-source the constructed benchmark and code at https://github.com/NLPGM/PCogAlign.

1 Introduction

Recently, significant progress has been made in aligning vision-language models (VLMs) (Hurst et al., 2024) with humans through visual instruction tuning (Liu et al., 2024b), visual preference optimization (Sun et al., 2023; Zhao et al., 2023), and safety alignment (Zong et al., 2024; Wang et al., 2024c). These studies focus on general alignment goals, e.g., following human instructions, reducing visual hallucinations, and aligning with human values. Based on the success of these techniques,

Figure 1: Individuals with different Role-Sets exhibit personalized situated cognition when encountering the same “broken swing” visual scene, resulting in personalized expectations for the VLM assistants’ responses.

VLMs have become essential assistants for humans in managing visual tasks (Yin et al., 2023).

However, even in the same situation, people with diversified backgrounds might have personalized situated cognition 1. Consequently, they may have personalized expectations for the VLM assistants’ responses. To illustrate this, we present a “broken swing” visual scene as an example in Figure 1. When faced with the “broken swing” scenario, a child may feel unsure and worried due to their limited ability to address such issues. In this context, the VLM assistant is expected to offer reassurance and basic safety guidance, ensuring that the child understands not to play on the swing for their safety and easing their concerns. In contrast, a repairman possesses the expertise to fix the problem. Therefore, the VLM assistant should provide professional advice on repairing the swing, enabling

1 Referencing the Situated Cognition Theory (Brown et al., 1989), we define the situated cognition discussed in this paper as: 1) cognition of the visual scene state; 2) cognition of body and mind states; 3) cognition of the appropriate next action.
the repairman to take appropriate actions.

The personalized situated cognition and personalized expectation reveal that current VLM assistants, which provide a one-size-fits-all response, are inadequate. Therefore, advancing research to align VLM assistants with personalized situated cognition is crucial. This will enable them to be more effective collaborators, capable of meeting personalized expectations from diverse individuals.

To study this problem, we first need to consider how to define diverse individuals. Given that human diversity stems from numerous factors, such as age and socioeconomic status, which are almost impossible to operationalize in experiments, we have to make necessary simplifications. Based on the concept of the Role-Set from Role Theory (Goffman, 1959), we characterize each individual as a set of “Role@Location” components, such as “Child@Home, Member@Community” as illustrated in Figure 1. Individuals with different Role-Sets exhibit personalized situated cognitions and therefore expect personalized responses.

Based on the above definition, we construct a benchmark named PCogAlignBench to investigate how to align VLM assistants with personalized situated cognition. Specifically, we select 8 social locations (Oldenburg, 1989) where individuals might engage and define 3 to 5 roles for each location (32 roles in total). Based on certain combination constraints, we define 20 Role-Sets, each of which consists of 5 “Role@Location” combinations. Then, we automatically collect 12k training samples and 6k test samples. The test samples undergo rigorous quality control through human annotators to ensure evaluation reliability. Each sample includes a Role-Set, an image, and a query posed by the individual. Additionally, every test sample includes an “oracle guidance”, which describes “the characteristics of the expected personalized response”.

Furthermore, we propose a framework named PCogAlign for this problem, which consists of three steps. 1) Estimating the individual’s situated cognition and optimal action. 2) Sampling multiple personalized responses via cooperative agents. 3) Constructing and utilizing a cognition-aware and action-based reward model to iteratively select the optimal response. The selected optimal responses are expected to align well with individuals’ personalized situated cognition and assist individuals in making optimal actions, and will be used as feedback for alignment training. Experimental results in various settings consistently prove the superiority of PCogAlign compared to baselines.

In all, our study makes three main contributions.

1) Novel Task. We introduce a novel task that aims to align VLM assistants with personalized situated cognition to meet diverse expectations.

2) New Benchmark. We present a new benchmark named PCogAlignBench to explore this novel task, which consists of 18k samples.

3) Novel Methodology. We propose a novel framework named PCogAlign as a baseline for this task. Comprehensive experimental results and analysis validate its effectiveness.

2 Related Work

Alignment of Vision-Language Models Existing techniques for aligning vision-language models (VLMs) with humans can be categorized into three types based on their alignment objectives: 1) visual instruction tuning (Liu et al., 2024b,a) to enable VLMs to follow human instructions in visual tasks; 2) visual preference optimization to mitigate visual hallucinations in VLMs’ responses (Sun et al., 2023; Zhu et al., 2024; Yang et al., 2025; Wang et al., 2024a; Xing et al., 2024; Yu et al., 2024a,b); and 3) safety fine-tuning to align VLMs with human values and prevent harmful outputs (Chen et al., 2024b; Shi et al., 2024b; Zhang et al., 2024c). The success of these alignment techniques has enabled VLMs to be effective assistants for humans in tasks involving visual scenes (Yin et al., 2024). However, considering the diversity of individuals’ cognition, even in the same visual context, their expectations for VLM assistants may vary widely. Existing general alignment objectives struggle to meet such a wide range of needs. Our research aims to bridge this gap by exploring how to align VLM assistants with personalized situated cognition.

Personalized Alignment of AI Models There are two main orientations (Wang et al., 2024e) of AI models’ personalized alignment (Kirk et al., 2024a,b; Zhang et al., 2024d; Sorensen et al., 2024; Li et al., 2024c; Wu et al., 2024a): 1) Personal Reflection, where AI imitates human personalities for AI-based social simulation (Mou et al., 2024a,b; Zhou et al., 2023) or social agents (Feng et al., 2024; Agrawal et al., 2023; Fan et al., 2023); 2) Personal Assistant, which tailors AI to be the ideal assistant for humans with specific personalities, including personalized recommendations (Wei et al., 2024), retrieval (Pi et al., 2024; Shen et al., 2024; Nguyen et al., 2024), or personalized image (Li
et al., 2024b; Shi et al., 2024a) or text generation (Li et al., 2024a) assistance. Our work falls within the scope of the latter.

The most relevant studies to our work are recent research on personalized Large Language Model (LLM) assistants (Jang et al., 2023; Wu et al., 2024b; Sun et al., 2024; Wang et al., 2024d). However, these studies: 1) do not involve visual contexts; 2) mainly focus on certain combinations of personalized text styles or values (Chen et al., 2024a; Wang et al., 2024f; Zhang et al., 2024b,a; Cheng et al., 2023; Lee et al., 2024), which are unable to model human diversity. Our research seeks to address these issues through: 1) modeling human diversity using the sociological concept of Role-Set; and 2) investigating personalized alignment of VLM assistants in visual scene tasks.

3 Problem Definition

To clarify the proposed task, this section provides: 1) problem simplifications, 2) the notations, and 3) the optimization objective of this task.

Problem Simplification The proposed task aims to explore how to align VLM assistants with the personalized situated cognitions of diverse individuals. However, there are two main challenges when exploring this problem: 1) How to define diverse individuals? 2) How to evaluate whether the personalized alignment is achieved?

For the first question, we introduce Role-Set to characterize individuals’ diversity. In Role Theory (Goffman, 1959), a Role-Set is the collection of various roles connected to specific social positions, influencing an individual’s behavior and expectations. The roles within an individual’s Role-Set reflect the combination of their social positions, which may lead to diverse situated cognitions and personalized expectations for VLM assistants’ responses. In this study, each individual’s Role-Set is set to contain 5 “Role@Location” components.

For the second question, we believe that personalized alignment should enable the VLM assistant to generate personalized responses that effectively guide individuals toward taking better actions. As emphasized in Gallagher (2006), both the human body and mind are critical to human action. Thus, we conceptualize “action” here as the combination of external Body Behavior and internal Mind Feelings. For example, in Figure 1, a personalized response should help the “child” move away from the broken swing and reduce feelings of worry.

Notations Consider a sample \( s = (RS, v, q) \), where \( RS \) denotes the Role-Set of the individual, \( v \) represents the visual scene (image) encountered by the individual, \( q \) denotes the query posed by the individual. The VLM assistant, parameterized by \( \theta \), is represented as \( f_{\theta} \). For a given sample \( s \), the response generated by the VLM assistant is \( r = f_{\theta}(s) \). The individual with the Role-Set \( RS \) who encounters the visual scene \( v \) has the situated cognition \( c = C(s) \). Subsequently, based on the individual’s situated cognition \( c \) and the received VLM response \( r \), the probability that the individual takes the action \( a \) is denoted as \( P_A(a|r, c) \).

Optimization Objective According to the problem simplification, when the VLM response is well-aligned with the individual’s personalized situated cognition, the individual is most likely to take the desired optimal action. Thus, based on the above notations, we define the optimization objective as:

\[
\theta^* = \arg\max_{\theta} \mathbb{E}_{s \sim S_{train}} P_A(a^*|r = f_{\theta}(s), c),
\]

where \( S_{train} \) is the training set, \( c = C(s) \), and \( s = (RS, v, q) \). The objective is to find the optimal parameters \( \theta^* \) for the VLM assistant that maximizes the expected probability of the individual’s action being the desired action \( a^* \).

4 PCogAlignBench

In this section, we introduce the benchmark construction and the automatic evaluation method.

Benchmark Overview The proposed benchmark consists of 12k training samples and 6k test samples. We define 20 individuals with different Role-Sets in total. Each training sample consists of an individual’s Role-Set, an image, and a query proposed by the individual. To ensure the evaluation is reliable, we further gather “oracle guidance” for each test sample, which describes the characteristics of the expected personalized response.

Besides, considering that the proposed benchmark cannot cover all possible Role-Sets, we need to ensure settings where the Role-Sets encountered during the training and test phases do not overlap. To achieve this, we divide the defined 20 Role-Sets into two subsets, LS1 and LS2, according to the different social location sets considered, as shown in Figure 2 (a). Such division allows us to test the model’s generalization ability to unseen Role-Sets, e.g., training on LS1 and testing on LS2.
Figure 2: Overview of the collection process of PCogAlignBench. The LS1 subset considers roles in “[Home, Community, Museum, Airport, Store]”. The LS2 subset considers roles in “[Home, Community, School, Hospital, Restaurant]”. Here we take the “Individual I1 in LS1” as an example to illustrate the process of image collection.

4.1 Benchmark Collection

Figure 2 illustrates the construction of PCogAlign-Bench, consisting of three steps, i.e., Role-Set collection, image collection, and query collection.

Role-Set Collection We refer to Oldenburg (1989) and select eight social locations where individuals might participate, including Home, Community, Museum, Airport, Store, School, Hospital, Restaurant. Then, we define 3 to 5 roles for each location (32 roles in total), e.g., Student, Teacher, Librarian in the School. Finally, based on certain constraints 2, we construct 20 different Role-Sets via combinations, 10 per subset as in Figure 2 (a).

Image Collection To ensure the diversity and reliability of collected images via search engines, we need to first collect diverse and detailed visual scene descriptions as search terms. To achieve this, we design a hierarchical collection strategy as shown in Figure 2 (b).

Specifically, the collection process consists of four coarse-to-fine levels, beginning with the individual’s Role-Set at Level 1. At Level 2, we collect various visual scene types that individuals might observe in the social locations associated with their Role-Sets. However, we find that it is quite difficult to generate diverse visual scene descriptions directly through the collected types. Thus, at Level 3, we include a middle process, i.e., visual scene phrases collection, which can ensure diversity more easily. Then, at Level 4, these collected visual scene phrases serve as seeds to guide the generation of detailed visual scene descriptions. Finally, we use these gathered visual scene descriptions as search terms in search engines to retrieve the desired images. Throughout this process, all generation is accomplished by providing instructions and demonstrations to GPT-4o. Please refer to Table 11 in the appendix for the prompt templates used for collecting visual scene types, visual scene phrases, and visual scene descriptions.

Query Collection We further collect queries that individuals might pose to the VLM assistant in various image contexts. However, we find that GPT-4o may sometimes generate hallucinated queries that cannot be answered based on the information in the image. To address this, we first generate several candidate queries and then ask GPT-4o to select the best query according to predefined rules, e.g., the rule of avoiding queries that require real-time information, as illustrated in Figure 2 (c). Please refer to Table 12 in the appendix for prompt templates used for describing the collected images, generating candidate queries, and selecting the best query.

Quality Control Considering that the process of image and query collection is automated through GPT-4o and search engines, it’s unavoidable that some noise data might be collected. To ensure the evaluation reliably reflects the model’s personalized alignment performance, we perform qual-

2For example, each individual can only have one permanent role. For more details, please refer to Appendix B.1.
ity control on the test split by human annotators 3. Specifically, in the image collection, two human annotators revise about 30 duplicate or unclear visual scene types at level 2. At level 4 of image collection, we replace approximately 1,000 low-quality images with manually gathered ones collected by 10 trained annotators. For the query collection, one annotator carefully revises the queries that are observed to be unanswerable in experiments.

4.2 Evaluation on PCogAlignBench

To evaluate the personalized response automatically, we adopt the “llm-as-a-judge” manner. We also have made two specific designs inspired by Gu et al. (2024) to ensure reliability. 1) We collect “oracle guidance” for each test sample through a collaboration between GPT-4o and humans before evaluation, describing “what kind of personalized response is expected by the individual”. 2) We break down the scoring criteria into five dimensions, three reflecting personalized quality based on our problem definition and two reflecting general quality.

In the evaluation process, we employ a simulated interview method where the evaluator (GPT-4o) acts as an interviewee with a specific Role-Set. The “oracle guidance” is provided to the evaluator to score the VLM assistant’s responses based on the following five dimensions 4. 1) Role-Set Awareness (RSA): Advice tailored to roles. 2) Body Behavior Awareness (BBA): Strategies for user’s desired behavior and physical goals support. 3) Mind Feelings Awareness (MFA): Support for user’s emotional needs and desired mental states. 4) Contextual Awareness (CA): Relevance to the query and context. 5) Conversational Flow (CF): Interaction, natural flow, and balance between detail and clarity.

Each scoring dimension ranges from integer 1 to 5. The average score across five dimensions is referred to as the personalization score (\( P.\ Score \)). Besides, we conduct a human evaluation to ensure this automatic evaluation is reliable and consistent with humans, as shown in Figure 5.

5 PCogAlign

Overview The optimization objective in Eq. 1 can be streamlined into a three-step process. 1)

3In the practice of personalized alignment, training samples are difficult to gather manually and typically contain noise, so we did not apply quality control to the training split.
4Please refer to the Appendix B.5 for detailed collection process of “oracle guidance” and descriptions of dimensions.

Figure 3: Overview of the proposed PCogAlign. We first estimate the situated cognition and optimal action in (a). Then, the “KeyPoints Generator (KeyG)” and “Response Generator (ResG)” agents cooperate to sample \( N \) candidate personalized responses in (b). Finally, we utilize a specific reward model to select the optimal response as feedback for alignment training in (c).

Estimate the variables \( c \) and \( a^* \), which are independent of the VLM’s parameters \( \theta \). 2) Sample personalized responses that may be expected by the individual. 3) Select the optimal response \( r^* \) that maximizes the probability \( P_A(a^*|r^*, c) \) of the individual taking the optimal action \( a^* \). Then, the VLM parameters, \( \theta \), are optimized to maximize the likelihood that \( f_\theta(s) = r^* \), yielding the optimal \( \theta^* \).

According to this, we propose a framework called PCogAlign (Figure 3). In (a), we prompt VLM to generate estimations of the situated cognition \( c \) and optimal action \( a^* \). In (b), we propose an iterative sampling strategy to collect personalized responses. In (c), we construct a cognition-aware and action-based reward model, which is utilized to select the optimal response \( r^* \) as feedback to the VLM assistant for alignment training. This section will introduce the above three steps.

5.1 Cognition and Action Estimation

As shown in Figure 3 (a), we utilize the VLM itself to estimate the situated cognition of the individual under certain visual scenes, as well as the possible optimal action that the individual can take with certain optimal VLM assistants’ responses. This step is achieved through in-context learning using human-written demonstrations.

5.2 Personalized Response Sampling

To collect personalized responses that may be expected by the individual, we implement an iterative sampling strategy, as shown in Figure 3 (b).

Specifically, we design two cooperative agents: “KeyG” and “ResG”. The “KeyG” is equipped with
information about the user’s situated cognition and expected optimal action, and it generates key points about how to consider the user’s cognition and enhance the user’s body behavior and mind feelings. The “ResG” uses the key points to re-generate responses. Through several iterations, we can gather \( N \) candidate personalized responses.

5.3 Alignment Training

Since not all sampled responses align well with personalized situated cognition, we need to select the optimal one for alignment training. To facilitate this selection, we first construct a cognition-aware and action-based reward model, as depicted in Figure 4. We then use this reward model for optimal response selection in Figure 3 (c).

Reward Model Construction The reward model is a key component in alignment techniques like RLHF (Bai et al., 2022), and is often trained using manually collected preference data. However, in personalized alignment, we cannot use such general preference data. Instead, we need to collect preference samples for each individual.

To this end, we propose using negative Role-Sets to collect preference data for reward model training as shown in Figure 4 (a). For example, for individual I1 (“Teacher@School”) and I2 (“Student@School”), VLM’s responses for I2 are inappropriate for I1. Thus, the Role-Set of I2 can be treated as a negative one for I1. Besides, since whether a response meets personalized expectations is reflected in the individual’s actions, we incorporate the individual’s actions into the preference pair. The collected preference pairs are then used to train a cognition-aware and action-based reward model, as shown in Figure 4 (b).

Optimal Response Selection via Best-of-N We employ a Best-of-N strategy to select the optimal response, as shown in Figure 3 (c). Specifically, each personalized response is compared with the current optimal one. The optimal response is replaced with the new one if the reward model judges that the new one is better aligned with the personalized situated cognition and can guide the individual towards better action.

Optimization The finally selected optimal response \( r^* \) is used to optimize the VLM assistant \( f_\theta \) using the following supervised fine-tuning loss.

\[
\theta^* = \arg\min_\theta \mathbb{E}_{s \sim s_{train}} - \log P(f_\theta(r^*|s)),
\]

where the obtained \( \theta^* \) is also the solution to Eq. 1.

6 Experiments

6.1 Human Evaluation on PCogAlignBench

In Section 4.2, we propose an automatic method to evaluate the personalization of the VLM’s response. Although the *LLM-as-a-judge* manner has been validated to be largely consistent with human evaluators in general preference (Zheng et al., 2023), e.g., harmless, we still need to ensure that it is also reliable for our personalized alignment task.

To this end, we randomly select 100 samples from the test split of LS1 and LS2, respectively. Then we compare the *P. Score* of responses generated by two methods, i.e., our PCogAlign and the *RS Prompt* introduced below. This yields the win/tie/lose results of the “Automatic Eval” in Figure 5. Additionally, we ask human annotators to reassess which method’s response is better, which yields the results of “Human Eval” in Figure 5. From the results, we can see that human evaluators agree with our designed automatic evaluation method in 88% of the cases. This validates the reliability of the evaluation method.

<table>
  <tr>
    <th rowspan="2"> </th>
    <th colspan="3">LS1 Subset</th>
    <th colspan="3">LS2 Subset</th>
  </tr>
  <tr>
    <th>Win</th>
    <th>Tie</th>
    <th>Lose</th>
    <th>Win</th>
    <th>Tie</th>
    <th>Lose</th>
  </tr>
  <tr>
    <td>Automatic Eval</td>
    <td>46%</td>
    <td>4%</td>
    <td>0%</td>
    <td>48%</td>
    <td>3%</td>
    <td>0%</td>
  </tr>
  <tr>
    <td>Lose</td>
    <td>2%</td>
    <td>23%</td>
    <td>1%</td>
    <td>1%</td>
    <td>27%</td>
    <td>2%</td>
  </tr>
  <tr>
    <td>Tie</td>
    <td>2%</td>
    <td>3%</td>
    <td>19%</td>
    <td>1%</td>
    <td>5%</td>
    <td>13%</td>
  </tr>
  <tr>
    <td>Win</td>
    <td></td>
    <td></td>
    <td></td>
    <td>Win</td>
    <td>Tie</td>
    <td>Lose</td>
  </tr>
  <tr>
    <td>Human Eval</td>
    <td></td>
    <td></td>
    <td></td>
    <td>Human Eval</td>
    <td></td>
    <td></td>
  </tr>
</table>

Figure 5: Heatmaps showing the comparison between automatic and human evaluations on two subsets, LS1 and LS2. The automatic evaluation agrees with human assessment in 88% of cases (the sum of the diagonal elements), highlighting the reliability of the proposed automatic evaluation method.
6.2 Evaluation on PCogAlign

6.2.1 Experimental Setup

Settings The proposed benchmark consists of two subsets based on the roles’ locations: LS1 and LS2, which facilitates the examination of methods’ performance where the Role-Sets are shared or non-overlap in training and test phases. Specifically, we consider the following settings: \( LS1 \rightarrow LS1, LS1 \rightarrow LS2, LS2 \rightarrow LS1, \) and \( LS2 \rightarrow LS2 \). For example, \( LS1 \rightarrow LS2 \) refers to training on LS1’s train split and test on LS2’s test split.

Used Model In the main experiments, both the VLM to be optimized and the cognition-aware reward model are initialized using the Qwen2-VL-7B-Instruct (Wang et al., 2024b).

Evaluation Metrics Based on the evaluation method described in Section 4.2, we report two metrics in the main results. (1) P. Score, which ranges from 1 to 5. (2) Win Rate, which is obtained by comparing the P. Score of the response being evaluated with that of the Base response.

6.2.2 Baselines

Since the proposed task is new, we can only modify existing alignment methods as baselines. We consider three types of baselines: prompt, direct preference optimization (DPO) (Rafailov et al., 2024), and supervised fine-tuning (SFT) (Wei et al., 2021).

To ensure a fair comparison, we empower all the DPO and SFT baselines with the estimated situated cognition and optimal action in Section 5.1 and generated “Key Points” in Section 5.2.5.

Prompt Baselines We include three prompt-based baselines, i.e., Base, RAG, and RS Prompt.

Base generates responses via the initial VLM.

RS Prompt, where the Role-Set is provided in the system prompt to generate responses.

RAG retrieves 3 in-context examples from training samples based on Role-Sets. We use the Levenshtein distance as the retrieval metric because it can accurately reflect the similarity between Role-Sets.

DPO (D) Baselines We include the following methods to collect preference pairs used for DPO.

Self-Refine (D) (Ranaldi and Freitas, 2024; Madaan et al., 2024), which designs 3 agents to iteratively improve the generated personalized responses. We treat the finally obtained response as the chosen response and the response via RS Prompt as the rejected one.

RLCD (D) (Yang et al., 2023), which is adapted to use the Response Generator in Section 5.2 to generate chosen responses. The rejected one is also obtained via RS Prompt.

RLAIF (D) (Lee et al., 2023; Yu et al., 2024b), which utilizes a Judge Agent, to determine the chosen and rejected response from the two responses obtained in RLCD (D).

SFT (S) Baselines We include the following methods to collect personalized responses used for SFT. For the RS Prompt (S), we use the RS Prompt described above to generate personalized responses as the SFT targets. For Self-Refine (S), RLCD (S), and RLAIF (S), we adopt the chosen response in the DPO versions as the target for SFT.

PCogAlign Variants We have also implemented the Prompt/DPO/SFT variant of our proposed framework as ablation variants, i.e., PCogAlign (P/D/S). The PCogAlign (P) variant only retains cooperative agents in Section 5.1 and 5.2 to generate responses for the test samples. The PCogAlign (D) variant adopts the reward model to conduct multiple pairwise comparisons with the response via RS Prompt. For the PCogAlign (S), we remove the Best-of-N strategy and take the first response chosen by PCogAlign (D) that is not the response of RS Prompt for each sample as the SFT target.

6.2.3 Results and Analysis

Table 1 reports the experimental results of baselines and our proposed PCogAlign. Based on these results, we have made the following observations.

1) Our method achieves the best performance, with an average improvement of 2.4% in Win Rate compared to the second best Self-Refine (S).

2) The P. Score of the Base method can only reach 3.748, indicating that the VLM’s general responses struggle to meet the diverse expectations brought by the personalized situated cognition.

3) Even with the simple prompt-based variant of our framework, i.e., PCogAlign (P), a 3.3% improvement in Win Rate compared to the RS Prompt can be achieved. This indicates that the cooperative agents designed for personalized response sampling can gather expected responses that align with the personalized situated cognition.

4) An alternative approach to the reward model used for preference judgment is designing a judge agent based on VLM itself, as RLAIF (D/S) does.
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="2">LS1→LS1</th>
    <th colspan="2">LS1→LS2</th>
    <th colspan="2">LS2→LS1</th>
    <th colspan="2">LS2→LS2</th>
    <th colspan="2">Average</th>
  </tr>
  <tr>
    <th>P. Score</th><th>Win Rate</th>
    <th>P. Score</th><th>Win Rate</th>
    <th>P. Score</th><th>Win Rate</th>
    <th>P. Score</th><th>Win Rate</th>
    <th>P. Score</th><th>Win Rate</th>
  </tr>
  <tr>
    <td rowspan="4"><i>Prompt</i></td>
    <td>Base</td><td>3.781</td><td>0.0%</td><td>3.715</td><td>0.0%</td><td>3.781</td><td>0.0%</td><td>3.715</td><td>0.0%</td><td>3.748</td><td>0.0%</td>
  </tr>
  <tr>
    <td>RS Prompt</td><td>3.989</td><td>44.1%</td><td>4.008</td><td>47.1%</td><td>3.989</td><td>44.1%</td><td>4.008</td><td>47.1%</td><td>3.999</td><td>45.6%</td>
  </tr>
  <tr>
    <td>RAG</td><td>4.022</td><td>43.9%</td><td>3.949</td><td>44.3%</td><td>4.046</td><td>46.4%</td><td>3.952</td><td>45.8%</td><td>3.992</td><td>45.1%</td>
  </tr>
  <tr>
    <td>PCogAlign (P)</td><td>4.070</td><td>47.5%</td><td>4.056</td><td>50.3%</td><td>4.070</td><td>47.5%</td><td>4.056</td><td>50.3%</td><td>4.063</td><td>48.9%</td>
  </tr>
  <tr>
    <td rowspan="4"><i>DPO</i></td>
    <td>Self-Refine (D)</td><td>3.982</td><td>43.7%</td><td>4.007</td><td>47.2%</td><td>3.990</td><td>43.9%</td><td>4.005</td><td>46.4%</td><td>3.996</td><td>45.3%</td>
  </tr>
  <tr>
    <td>RLCD (D)</td><td>3.985</td><td>44.6%</td><td>4.014</td><td>46.7%</td><td>3.978</td><td>43.2%</td><td>4.006</td><td>47.1%</td><td>3.996</td><td>45.4%</td>
  </tr>
  <tr>
    <td>RLAIF (D)</td><td>3.984</td><td>43.6%</td><td>4.005</td><td>46.9%</td><td>3.982</td><td>43.8%</td><td>4.011</td><td>48.4%</td><td>3.996</td><td>45.7%</td>
  </tr>
  <tr>
    <td>PCogAlign (D)</td><td>4.020</td><td>45.2%</td><td>4.039</td><td>49.1%</td><td>4.008</td><td>45.2%</td><td>4.029</td><td>48.1%</td><td>4.024</td><td>46.9%</td>
  </tr>
  <tr>
    <td rowspan="4"><i>SFT</i></td>
    <td>RS Prompt (S)</td><td>3.857</td><td>35.3%</td><td>3.937</td><td>41.6%</td><td>3.934</td><td>38.7%</td><td>3.969</td><td>42.5%</td><td>3.924</td><td>39.5%</td>
  </tr>
  <tr>
    <td>Self-Refine (S)</td><td>4.091</td><td>50.3%</td><td>4.096</td><td>52.0%</td><td>4.126</td><td>51.1%</td><td>4.139</td><td>52.4%</td><td>4.113</td><td>51.4%</td>
  </tr>
  <tr>
    <td>RLCD (S)</td><td>3.996</td><td>43.2%</td><td>3.941</td><td>44.4%</td><td>4.016</td><td>42.7%</td><td>3.965</td><td>44.3%</td><td>3.980</td><td>43.7%</td>
  </tr>
  <tr>
    <td>RLAIF (S)</td><td>4.046</td><td>45.1%</td><td>3.994</td><td>48.7%</td><td>4.050</td><td>42.3%</td><td>4.043</td><td>46.5%</td><td>4.033</td><td>45.6%</td>
  </tr>
  <tr>
    <td>PCogAlign (S)</td><td>4.130</td><td>51.5%</td><td>4.108</td><td>53.0%</td><td>4.116</td><td>48.7%</td><td>4.118</td><td>51.9%</td><td>4.118</td><td>51.3%</td>
  </tr>
  <tr>
    <td>PCogAlign</td><td><b>4.161</b></td><td><b>53.3%</b></td><td><b>4.156</b></td><td><b>56.6%</b></td><td><b>4.150</b></td><td><b>51.4%</b></td><td><b>4.151</b></td><td><b>53.8%</b></td><td><b>4.154</b></td><td><b>53.8%</b></td>
  </tr>
</table>

Table 1: Experimental results (<i>P. Score</i> and <i>Win Rate</i>) under four settings. The best results are in bold and the second best ones are underlined. Please refer to Table 6 for a comprehensive report on the <i>Win/Tie/Lose Rate</i> metrics.

<table>
  <tr>
    <th>Method</th>
    <th>RSA</th>
    <th>BBA</th>
    <th>MFA</th>
    <th>CA</th>
    <th>CF</th>
  </tr>
  <tr>
    <td rowspan="4"><i>Prompt</i></td>
    <td>Base</td><td>3.542</td><td>3.511</td><td>3.575</td><td>4.447</td><td>3.665</td>
  </tr>
  <tr>
    <td>RS Prompt</td><td>3.842</td><td>3.724</td><td>3.869</td><td>4.711</td><td>3.846</td>
  </tr>
  <tr>
    <td>RAG</td><td>3.805</td><td>3.741</td><td>3.853</td><td>4.691</td><td>3.870</td>
  </tr>
  <tr>
    <td>PCogAlign (P)</td><td>3.863</td><td>3.758</td><td>3.978</td><td>4.765</td><td>3.951</td>
  </tr>
  <tr>
    <td rowspan="4"><i>DPO</i></td>
    <td>Self-Refine (D)</td><td>3.838</td><td>3.722</td><td>3.864</td><td>4.709</td><td>3.846</td>
  </tr>
  <tr>
    <td>RLCD (D)</td><td>3.834</td><td>3.728</td><td>3.865</td><td>4.705</td><td>3.847</td>
  </tr>
  <tr>
    <td>RLAIF (D)</td><td>3.833</td><td>3.739</td><td>3.858</td><td>4.705</td><td>3.843</td>
  </tr>
  <tr>
    <td>PCogAlign (D)</td><td>3.865</td><td>3.752</td><td>3.899</td><td>4.735</td><td>3.869</td>
  </tr>
  <tr>
    <td rowspan="4"><i>SFT</i></td>
    <td>RS Prompt (S)</td><td>3.749</td><td>3.636</td><td>3.748</td><td>4.663</td><td>3.826</td>
  </tr>
  <tr>
    <td>Self-Refine (S)</td><td>3.930</td><td>3.846</td><td>4.049</td><td>4.807</td><td>3.932</td>
  </tr>
  <tr>
    <td>RLCD (S)</td><td>3.757</td><td>3.665</td><td>3.865</td><td>4.705</td><td>3.905</td>
  </tr>
  <tr>
    <td>RLAIF (S)</td><td>3.847</td><td>3.730</td><td>3.903</td><td>4.763</td><td>3.924</td>
  </tr>
  <tr>
    <td>PCogAlign (S)</td><td>3.927</td><td>3.819</td><td>4.043</td><td>4.822</td><td>3.978</td>
  </tr>
  <tr>
    <td>PCogAlign</td><td><b>3.963</b></td><td><b>3.875</b></td><td><b>4.103</b></td><td><b>4.838</b></td><td><b>3.993</b></td>
  </tr>
</table>

Table 2: Experimental results on the 5 detailed dimensions. The score ranges from 1 to 5. The best results are in bold and the second best ones are underlined.

<table>
  <tr>
    <th rowspan="2">LS1</th>
    <th colspan="3">hit@1</th>
    <th colspan="3">hit@2</th>
    <th colspan="3">hit@3</th>
  </tr>
  <tr>
    <th>w/o RM</th><th>hit@1</th><th>hit@2</th><th>hit@3</th>
    <th>w/ RM</th><th>hit@1</th><th>hit@2</th><th>hit@3</th>
  </tr>
  <tr>
    <td></td><td>28%</td><td>56%</td><td>69%</td><td>31%</td><td>51%</td><td>68%</td>
  </tr>
  <tr>
    <td></td><td>69%</td><td>94%</td><td>99%</td><td>79%</td><td>93%</td><td>98%</td>
  </tr>
</table>

Table 3: Human evaluation results (<i>hit@k</i>) of the selected responses without and with the constructed reward model (RM). The <i>hit@k</i> indicates if the response is within the top <i>k</i> of the <i>N</i> candidates.

The experiments show that our ablated version <i>PCogAlign (D/S)</i> achieves up to a 5.7% improvement over <i>RLAIF (D/S)</i>, indicating that our proposed cognition-aware and action-based reward model effectively empowers the optimal personalized response selection.

Analysis on Detailed Dimensions Table 2 reports the results on the 5 dimensions described in Section 4.2, which is averaged by all settings in Table 1. We make the following observations.

First, on dimensions focused on personalization, i.e., <i>RSA, BBA, and MFA</i>, our PCogAlign shows consistent improvements. This indicates that the VLM assistant obtained through PCogAlign can effectively recognize the user’s Role-Set (<i>RSA</i>), guide the user towards better body behavior (<i>BBA</i>), and enhance mind feelings (<i>MFA</i>).

Second, our PCogAlign also brings improvements in the dimensions for general quality, i.e., <i>CA and CF</i>. This suggests that our method can improve the contextual awareness (<i>CA</i>) and conversational flow (<i>CF</i>) of the responses.

Analysis on Constructed Reward Model To delve into the effectiveness of the constructed reward model in Section 5.3, we conduct a human evaluation to examine the actual quality of the selected optimal response by the reward model.

Specifically, we select 100 samples from the training processes on LS1 and LS2, respectively. Human evaluators are asked to rank the top 3 responses among the <i>N</i> candidates for each sample (\( N = 6 \)). Then, we calculate the <i>hit@k</i> metric of the selected response without and with the reward model. Note that without the reward model, we treat the first response generated during the personalized response sampling as the selected one.
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="2">LS1→LS1</th>
    <th colspan="2">LS1→LS2</th>
    <th colspan="2">LS2→LS1</th>
    <th colspan="2">LS2→LS2</th>
    <th colspan="2">Average</th>
  </tr>
  <tr>
    <th>P. Score</th><th>Win Rate</th>
    <th>P. Score</th><th>Win Rate</th>
    <th>P. Score</th><th>Win Rate</th>
    <th>P. Score</th><th>Win Rate</th>
    <th>P. Score</th><th>Win Rate</th>
  </tr>
  <tr>
    <td rowspan="4">Qwen2-VL-7B-Instruct</td>
    <td>Base</td><td>3.781</td><td>0.0%</td><td>3.715</td><td>0.0%</td><td>3.781</td><td>0.0%</td><td>3.715</td><td>0.0%</td><td>3.748</td><td>0.0%</td>
  </tr>
  <tr>
    <td>RS Prompt</td><td>3.989</td><td>44.1%</td><td>4.008</td><td>47.1%</td><td>3.989</td><td>44.1%</td><td>4.008</td><td>47.1%</td><td>3.999</td><td>45.6%</td>
  </tr>
  <tr>
    <td>RAG</td><td>4.022</td><td>43.9%</td><td>3.949</td><td>44.3%</td><td>4.046</td><td>46.4%</td><td>3.952</td><td>45.8%</td><td>3.992</td><td>45.1%</td>
  </tr>
  <tr>
    <td>PCogAlign (P)</td><td><b>4.070</b></td><td><b>47.5%</b></td><td><b>4.056</b></td><td><b>50.3%</b></td><td><b>4.070</b></td><td><b>47.5%</b></td><td><b>4.056</b></td><td><b>50.3%</b></td><td><b>4.063</b></td><td><b>48.9%</b></td>
  </tr>
  <tr>
    <td rowspan="4">Qwen2.5-VL-7B-Instruct</td>
    <td>Base</td><td>4.126</td><td>0.0%</td><td>4.079</td><td>0.0%</td><td>4.126</td><td>0.0%</td><td>4.079</td><td>0.0%</td><td>4.102</td><td>0.0%</td>
  </tr>
  <tr>
    <td>RS Prompt</td><td>4.163</td><td>31.0%</td><td>4.122</td><td>33.0%</td><td>4.163</td><td>31.0%</td><td>4.122</td><td>33.0%</td><td>4.143</td><td>32.0%</td>
  </tr>
  <tr>
    <td>RAG</td><td>4.242</td><td>38.1%</td><td>4.184</td><td>37.5%</td><td>4.220</td><td>36.4%</td><td>4.200</td><td>37.3%</td><td>4.212</td><td>37.3%</td>
  </tr>
  <tr>
    <td>PCogAlign (P)</td><td><b>4.275</b></td><td><b>40.8%</b></td><td><b>4.277</b></td><td><b>44.2%</b></td><td><b>4.275</b></td><td><b>40.8%</b></td><td><b>4.277</b></td><td><b>44.2%</b></td><td><b>4.276</b></td><td><b>42.5%</b></td>
  </tr>
  <tr>
    <td rowspan="4">Phi-3.5-vision-instruct</td>
    <td>Base</td><td>3.268</td><td>0.0%</td><td>3.235</td><td>0.0%</td><td>3.268</td><td>0.0%</td><td>3.235</td><td>0.0%</td><td>3.251</td><td>0.0%</td>
  </tr>
  <tr>
    <td>RS Prompt</td><td>3.419</td><td>40.3%</td><td>3.379</td><td>38.1%</td><td>3.419</td><td>40.3%</td><td>3.379</td><td>38.1%</td><td>3.399</td><td>39.2%</td>
  </tr>
  <tr>
    <td>RAG</td><td>3.730</td><td>65.4%</td><td>3.613</td><td>58.8%</td><td>3.772</td><td>67.0%</td><td>3.684</td><td>62.1%</td><td>3.700</td><td>63.3%</td>
  </tr>
  <tr>
    <td>PCogAlign (P)</td><td><b>3.797</b></td><td><b>67.3%</b></td><td><b>3.745</b></td><td><b>64.2%</b></td><td><b>3.797</b></td><td><b>67.3%</b></td><td><b>3.745</b></td><td><b>64.2%</b></td><td><b>3.771</b></td><td><b>65.7%</b></td>
  </tr>
  <tr>
    <td rowspan="4">MiniCPM-V-2_6</td>
    <td>Base</td><td>3.743</td><td>0.0%</td><td>3.725</td><td>0.0%</td><td>3.743</td><td>0.0%</td><td>3.725</td><td>0.0%</td><td>3.734</td><td>0.0%</td>
  </tr>
  <tr>
    <td>RS Prompt</td><td>4.055</td><td>60.7%</td><td>4.037</td><td>60.7%</td><td>4.055</td><td>60.7%</td><td>4.037</td><td>60.7%</td><td>4.046</td><td>60.7%</td>
  </tr>
  <tr>
    <td>RAG</td><td>4.261</td><td>79.8%</td><td>4.232</td><td>78.4%</td><td>4.262</td><td>79.3%</td><td>4.235</td><td>78.8%</td><td>4.248</td><td>79.1%</td>
  </tr>
  <tr>
    <td>PCogAlign (P)</td><td><b>4.303</b></td><td><b>80.1%</b></td><td><b>4.321</b></td><td><b>81.0%</b></td><td><b>4.303</b></td><td><b>80.1%</b></td><td><b>4.321</b></td><td><b>81.0%</b></td><td><b>4.312</b></td><td><b>80.5%</b></td>
  </tr>
</table>

Table 4: Experimental results (P. Score and Win Rate) under four settings. The best results on each VLM are in bold.

Table 3 presents the human evaluation results. We can observe that the optimal response selected by the reward model has a 98.5% chance of being within the top 3, indicating the high quality. Besides, the constructed reward model brings a 44.5% increase of hit@1, which further validates the importance of utilizing the constructed reward model for selecting the optimal personalized response.

Evaluating Different VLMs on PCogAlignBench
Our proposed PCogAlignBench can also serve as a benchmark for evaluating the personalization adaptation ability of different VLMs. To this end, we have included several VLMs from additional series to benchmark their personalization adaptation abilities. We consider the prompt-based baselines and the prompt variant of our PCogAlign for this experiment. The results are shown in Table 4.

As shown in Table 4, the MiniCPM-V-2_6 shows the strongest personalization adaptation ability (average P. Score). Still, our PCogAlign (P) method consistently outperforms the baselines across all VLMs.

7 Conclusion

In this paper, we highlight the importance of a novel task, i.e., aligning vision-language model (VLM) assistants with personalized situated cognition. To explore this task, we construct a new benchmark named PCogAlignBench, consisting of 18k samples. Furthermore, we present a novel framework named PCogAlign as a baseline for this problem, involving a cognition-aware and action-based reward model for alignment training. Experimental results and human evaluations validate the reliability of the proposed automatic evaluation method and the effectiveness of the PCogAlign.

Limitations

We summarize the potential limitations of our study as follows, which may inspire future research to further explore based on this work.

1) For the experimental feasibility, we introduce the Role-Set concept to simplify the problem when defining diverse individuals. However, in real life, an individual’s diversity may extend beyond the Role-Set and can be influenced by various factors such as personality and background. Fully capturing such diversity while ensuring experimental feasibility is a highly challenging issue, and we will continue to explore it in future work.

2) In the proposed PCogAlign framework, we employ a prompt-based method to estimate the personalized situated cognition and optimal action. Although our experiments have demonstrated the effectiveness of this simple yet effective estimation method, we believe there may be better ways to accomplish this step. We encourage future research to explore this more deeply.

3) Experimentally, we notice that the effects brought by DPO-based variants are very weak. However, given the important role of preference optimization algorithms in general alignment, we may have not used the most suitable preference optimization algorithm for personalized alignment. We encourage future work to address this challenge theoretically and experimentally.
Ethics Statement

During the data collection process, we made necessary designs in the role definition phase and quality control process to mitigate potential ethical risks. Specifically, in the role definition phase, we thoroughly discussed and defined roles that meet ethical standards to avoid factors like gender bias. For instance, to avoid gender bias, we defined that the responsibilities of “mother@home” and “father@home” are fairly distributed, including shared household chores, childcare, and handling family emergencies. Besides, in our human-led quality control process, we aimed to avoid potential ethical risks, including gender and racial biases etc. For example, if an annotator repeatedly encounters images showing “women doing housework” and relatively few “men doing housework” images, they are required to replace some of the “women doing housework” images with “men doing housework”. Thanks to the carefully designed role definitions, which consider avoiding potential bias, the automatically generated visual scene descriptions have largely avoided such biases, and we encountered less than 1% of such cases during the quality control process.

Considering the vast number of potential role combinations (theoretically 6300 combinations in our setup), which significantly reduces the feasibility of conducting academic experiments, we selected a subset of role combinations (20 Role-Sets) to form the Role-Sets in our dataset. This might raise concerns about certain biases. Although we believe this is acceptable in a simulated research environment, in actual industry development, we encourage companies/developers to consider various user backgrounds to form comprehensive, unbiased Role-Sets for data collection and personalized alignment training.

The “diversity” of the dataset encompasses two aspects: increasing the diversity of scenarios, and ensuring diversity to avoid potential biases, such as racial and gender biases.

We obtained informed consent from all annotators about the statement of “all human annotators being paid by the laboratory following local wage requirements” before the manuscript submission.

Acknowledgments

This work was supported by the grant from the National Natural Science Foundation of China (NSFC) project (No. 62276193), the grant from Zhongguancun Academy (Grant No. 20240302), and the Fundamental Research Funds for the Central Universities, China (Grant No. 2042022dx0001).

References

Harsh Agrawal, Aditya Mishra, Manish Gupta, et al. 2023. Multimodal persona based generation of comic dialogs. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14150–14164.

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862.

John Seely Brown, Allan Collins, and Paul Duguid. 1989. Situated cognition and the culture of learning. 1989, 18(1):32–42.

Ruizhe Chen, Xiaotian Zhang, Meng Luo, Wenhao Chai, and Zuozhu Liu. 2024a. Pad: Personalized alignment at decoding-time. arXiv e-prints, pages arXiv–2410.

Yangyi Chen, Karan Sikka, Michael Cogswell, Heng Ji, and Ajay Divakaran. 2024b. Dress: Instructing large vision-language models to align and interact with humans via natural language feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14239–14250.

Pengyu Cheng, Jiawen Xie, Ke Bai, Yong Dai, and Nan Du. 2023. Everyone deserves a reward: Learning customized human preferences. arXiv preprint arXiv:2309.03126.

Yue Fan, Kevin K Bowden, Wen Cui, Winson Chen, Vrindavan Harrison, Angela Ramirez, Saaket Agashe, Xinyue Gabby Liu, Neha Pullabhotla, NQJ Bheemapanally, et al. 2023. Athena 3.0: personalized multimodal chatbot with neuro-symbolic dialogue generators. Alexa Prize Soc Bot Grand Challenge, 5.

Xiachong Feng, Longxu Dou, Ella Li, Qinghao Wang, Haochuan Wang, Yu Guo, Chang Ma, and Lingpeng Kong. 2024. A survey on large language model-based social agents in game-theoretic scenarios. arXiv preprint arXiv:2412.03920.

Shaun Gallagher. 2006. How the body shapes the mind. Clarendon Press.

Erving Goffman. 1959. The presentation of self in everyday life.

Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. 2024. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models.

Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os-trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.

Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. 2023. Personalized soups: Personalized large language model alignment via post-hoc parameter merging. arXiv preprint arXiv:2310.11564.

Hannah Rose Kirk, Bertie Vidgen, Paul Röttger, and Scott A Hale. 2024a. The benefits, risks and bounds of personalizing the alignment of large language models to individuals. Nature Machine Intelligence, pages 1–10.

Hannah Rose Kirk, Alexander Whitefield, Paul Röttger, Andrew Bean, Katerina Margatina, Juan Ciro, Rafael Mosquera, Max Bartolo, Adina Williams, He He, et al. 2024b. The prism alignment project: What participatory, representative and individualised human feedback reveals about the subjective and multi-cultural alignment of large language models. arXiv preprint arXiv:2404.16019.

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles.

Gihun Lee, Minchan Jeong, Yujin Kim, Hojung Jung, Jaehoon Oh, Sangmook Kim, and Se-Young Yun. 2024. Bapo: Base-anchored preference optimization for overcoming forgetting in large language models personalization. arXiv preprint arXiv:2407.00693.

Harrison Lee, Samrat Phatale, Hassan Mansoor, Kel-lie Ren Lu, Thomas Mesnard, Johan Ferret, Colton Bishop, Ethan Hall, Victor Carburne, and Abhinav Rastogi. 2023. Rlaiif: Scaling reinforcement learning from human feedback with ai feedback.

Cheng Li, Mingyang Zhang, Qiaozhu Mei, Weize Kong, and Michael Bendersky. 2024a. Learning to rewrite prompts for personalized text generation. In Proceedings of the ACM on Web Conference 2024, pages 3367–3378.

Xiaoming Li, Xinyu Hou, and Chen Change Loy. 2024b. When stylegan meets stable diffusion: a w+ adapter for personalized image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2187–2196.

Xinyu Li, Zachary C Lipton, and Liu Leqi. 2024c. Personalized language modeling from personalized human feedback. arXiv preprint arXiv:2402.05133.

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024a. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024b. Visual instruction tuning. Advances in neural information processing systems, 36.

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2024. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36.

Xinyi Mou, Xuanwen Ding, Qi He, Liang Wang, Jing-cong Liang, Xinnong Zhang, Libo Sun, Jiayu Lin, Jie Zhou, Xuanjing Huang, et al. 2024a. From individual to society: A survey on social simulation driven by large language model-based agents. arXiv preprint arXiv:2412.03563.

Xinyi Mou, Jingcong Liang, Jiayu Lin, Xinnong Zhang, Xiawei Liu, Shiyue Yang, Rong Ye, Lei Chen, Haoyu Kuang, Xuanjing Huang, et al. 2024b. Agentsense: Benchmarking social intelligence of language agents through interactive scenarios. arXiv preprint arXiv:2410.19346.

Thao Nguyen, Haotian Liu, Yuheng Li, Mu Cai, Utkarsh Ojha, and Yong Jae Lee. 2024. Yo’Iava: Your personalized language and vision assistant. arXiv preprint arXiv:2406.09400.

Ray Oldenburg. 1989. The great good place: Cafés, coffee shops, community centers, beauty parlors, general stores, bars, hangouts, and how they get you through the day. (No Title).

Renjie Pi, Jianshu Zhang, Tianyang Han, Jipeng Zhang, Rui Pan, and Tong Zhang. 2024. Personalized visual instruction tuning. arXiv preprint arXiv:2410.07113.

Qwen-Team. 2025. Qwen2.5-v1.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36.

Leonardo Ranaldi and Andrè Freitas. 2024. Self-refine instruction-tuning for aligning reasoning in language models. arXiv preprint arXiv:2405.00402.

Xiaoteng Shen, Rui Zhang, Xiaoyan Zhao, Jieming Zhu, and Xi Xiao. 2024. Pmg: Personalized multimodal generation with large language models. In Proceedings of the ACM on Web Conference 2024, pages 3833–3843.
Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. 2024a. Instantbooth: Personalized text-to-image generation without test-time finetuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8543–8552.

Zhelun Shi, Zhipin Wang, Hongxing Fan, Zaibin Zhang, Lijun Li, Yongting Zhang, Zhenfei Yin, Lu Sheng, Yu Qiao, and Jing Shao. 2024b. Assessment of multimodal large language models in alignment with human values. arXiv preprint arXiv:2403.17830.

Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Nilofar Miresghallah, Christopher Michael Ryttig, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, et al. 2024. A roadmap to pluralistic alignment. arXiv preprint arXiv:2402.05070.

Chenkai Sun, Ke Yang, Revanth Gangi Reddy, Yi R Fung, Hou Pong Chan, Kevin Small, ChengXiang Zhai, and Heng Ji. 2024. Persona-db: Efficient large language model personalization for response prediction with collaborative data refinement. arXiv preprint arXiv:2402.11060.

Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. 2023. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309.14525.

Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, and Shengyi Huang. 2020. Trl: Transformer reinforcement learning. https://github.com/huggingface/trl.

Fei Wang, Wenxuan Zhou, James Y. Huang, Nan Xu, Sheng Zhang, Hoifung Poon, and Muhan Chen. 2024a. mDPO: Conditional preference optimization for multimodal large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8078–8088, Miami, Florida, USA. Association for Computational Linguistics.

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024b. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191.

Siyin Wang, Xingsong Ye, Qinyuan Cheng, Junwen Duan, Shimin Li, Jinlan Fu, Xipeng Qiu, and Xuanjing Huang. 2024c. Cross-modality safety alignment.

Tiannan Wang, Meiling Tao, Ruoyu Fang, Huilin Wang, Shuai Wang, Yuchen Eleanor Jiang, and Wangchunshu Zhou. 2024d. Ai persona: Towards life-long personalization of llms. arXiv preprint arXiv:2412.13103.

Xinpeng Wang, Shitong Duan, Xiaoyuan Yi, Jing Yao, Shanlin Zhou, Zhihua Wei, Peng Zhang, Dongkuan Xu, Maosong Sun, and Xing Xie. 2024e. On the essence and prospect: An investigation of alignment approaches for big models. arXiv preprint arXiv:2403.04204.

Xinran Wang, Qi Le, Ammar Ahmed, Enmao Diao, Yi Zhou, Nathalie Baracaldo, Jie Ding, and Ali Anwar. 2024f. Map: Multi-human-value alignment palette. arXiv preprint arXiv:2410.19198.

Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652.

Tianxin Wei, Bowen Jin, Ruirui Li, Hansi Zeng, Zhengyang Wang, Jianhui Sun, Qingyu Yin, Hanqing Lu, Suhang Wang, Jingrui He, et al. 2024. Towards unified multi-modal personalization: Large vision-language models for generative recommendation and beyond. arXiv preprint arXiv:2403.10667.

Junda Wu, Hanjia Lyu, Yu Xia, Zhehao Zhang, Joe Barrow, Ishita Kumar, Mehrnoosh Mirtaheri, Hongjie Chen, Ryan A Rossi, Franck Demoncourt, et al. 2024a. Personalized multimodal large language models: A survey. arXiv preprint arXiv:2412.02142.

Shujin Wu, May Fung, Cheng Qian, Jeonghwan Kim, Dilek Hakkani-Tur, and Heng Ji. 2024b. Aligning llms with individual preferences via interaction. arXiv preprint arXiv:2410.03642.

Shangyu Xing, Fei Zhao, Zhen Wu, Tuo An, Weihao Chen, Chunhui Li, Jianbing Zhang, and Xinyu Dai. 2024. EFUF: Efficient fine-grained unlearning framework for mitigating hallucinations in multimodal large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1167–1181, Miami, Florida, USA. Association for Computational Linguistics.

Kevin Yang, Dan Klein, Asli Celikyilmaz, Nanyun Peng, and Yuandong Tian. 2023. Rlcld: Reinforcement learning from contrast distillation for language model alignment. arXiv preprint arXiv:2307.12950.

Zhihe Yang, Xufang Luo, Dongqi Han, Yunjian Xu, and Dongsheng Li. 2025. Mitigating hallucinations in large vision-language models via dpo: On-policy data hold the key. arXiv preprint arXiv:2501.09695.

Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2023. A survey on multimodal large language models. arXiv preprint arXiv:2306.13549.

Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2024. A survey on multimodal large language models. National Science Review, 11(12).

Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. 2024a. Rlhf-v: Towards
trustworthy mlms via behavior alignment from fine-grained correctional human feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13807–13816.

Tianyu Yu, Haoye Zhang, Qiming Li, Qixin Xu, Yuan Yao, Da Chen, Xiaoman Lu, Ganqu Cui, Yunkai Dang, Taiwen He, Xiaocheng Feng, Jun Song, Bo Zheng, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. 2024b. Rlaif-v: Open-source ai feedback leads to super gpt-4v trustworthiness.

Jingyu Zhang, Ahmed Elgohary, Ahmed Magooda, Daniel Khashabi, and Benjamin Van Durme. 2024a. Controllable safety alignment: Inference-time adaptation to diverse safety requirements. arXiv preprint arXiv:2410.08968.

Mozhi Zhang, Pengyu Wang, Chenkun Tan, Mianqiu Huang, Dong Zhang, Yaqian Zhou, and Xipeng Qiu. 2024b. Metaalign: Align large language models with diverse preferences during inference time. arXiv preprint arXiv:2410.14184.

Yongting Zhang, Lu Chen, Guodong Zheng, Yifeng Gao, Rui Zheng, Jinlan Fu, Zhenfei Yin, Senjie Jin, Yu Qiao, Xuanjing Huang, et al. 2024c. Spav1: A comprehensive safety preference alignment dataset for vision language model. arXiv preprint arXiv:2406.12030.

Zhehao Zhang, Ryan A Rossi, Branislav Kveton, Yijia Shao, Diyi Yang, Hamed Zamani, Franck Dernoncourt, Joe Barrow, Tong Yu, Sungchul Kim, et al. 2024d. Personalization of large language models: A survey. arXiv preprint arXiv:2411.00027.

Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. 2023. Beyond hallucinations: Enhancing lvlns through hallucination-aware direct preference optimization. arXiv preprint arXiv:2311.16839.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623.

Xuhui Zhou, Hao Zhu, Leena Mathur, Ruohong Zhang, Haofei Yu, Zhengyang Qi, Louis-Philippe Morency, Yonatan Bisk, Daniel Fried, Graham Neubig, et al. 2023. Sotopia: Interactive evaluation for social intelligence in language agents. arXiv preprint arXiv:2310.11667.

Ke Zhu, Liang Zhao, Zheng Ge, and Xiangyu Zhang. 2024. Self-supervised visual preference alignment. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 291–300.

Yongshuo Zong, Ondrej Bohdal, Tingyang Yu, Yongxin Yang, and Timothy Hospedales. 2024. Safety fine-tuning at (almost) no cost: A baseline for vision large language models. In The 41st International Conference on Machine Learning.
A Supplementary Experimental Results

Detailed Results In this subsection, we will present some detailed experimental results.

Specifically, Table 6 presents the detailed results of Win/Tie/Lose metrics corresponding to Table 1. Table 7 and Table 8 present the detailed results of various dimensions corresponding to Table 2.

Experiments on Qwen2.5-VL We also select the recently released state-of-the-art VLM series model, Qwen2.5-VL (Qwen-Team, 2025), to conduct scalability experiments. We choose the “LS1→LS2” cross-subset setting for the experiment and report the results on Qwen2.5-VL-3B-Instruct in Table 9. From the experimental results, we can observe conclusions consistent with those in the main text section (Section 6.2.3).

B PCogAlignBench Details

B.1 Role-Set Collection

We reference Oldenburg (1989) and select eight social locations where individuals might participate, including Home, Community, Museum, Airport, Store, School, Hospital, Restaurant. Then, for each location, we define 3 to 5 roles as shown in Table 5. Specifically, the underlined roles are permanent roles, which occupy the majority of an individual’s daily working time.

Then, since we need to enable settings where the Role-Sets encountered during the training and test phases do not overlap, we divide the above locations into two subsets, LS1 and LS2, according to the different social location sets considered. The LS1 consists of “[Home, Community, Museum, Airport, Store]” and the LS2 consists of “[Home, Community, School, Hospital, Restaurant]”.

Finally, we construct 10 Role-Sets for each subset via combinations based on the following predefined constraints. 1) Each individual’s Role-Set can contain only one permanent role. 2) Each individual’s Role-Set involves five roles. 3) Each permanent role can be adopted by only one individual. The collected Role-Sets are shown in Table 10.

B.2 Image Collection

The image collection in Figure 2 involves three prompt templates used for collecting visual scene types, visual scene phrases, and visual scene descriptions via GPT-4o. We present the specific templates in Table 11.

Besides, for collecting images based on visual scene descriptions in level 4, we use Google’s API platform 6 for automated collection.

Statistics on the Quality of the Image Collection Generally, approximately 1,000 images in the test samples (17%) were replaced by human-collected ones. During this process, annotators not only need to ensure that the images show proper scenarios in the expected locations, but also need to consider the diversity of the images and avoid potential ethical risks, such as gender and racial biases. For example, even if an image is high-quality, if it appears multiple times (>2 times) in the dataset, it is replaced with a new one. Besides, suppose an annotator repeatedly encounters images showing “women doing housework” and relatively few “men doing housework” images. In that case, they need to replace some of the “women doing housework” images with “men doing housework”. Thanks to the carefully designed role definitions, which consider avoiding potential bias, the automatically generated visual scene descriptions have largely avoided such biases, and we encountered less than 1% of such biased cases during the quality control process.

B.3 Query Collection

The query collection in Figure 2 involves three prompt templates, which are used for describing collected images, generating several candidate queries, and selecting the best query, respectively, as shown in Table 12.

Statistics on the Quality of the Query Collection Thanks to the well-designed query collection strategy and high-quality human-checked images, only about 20 queries (0.3%) were found to be unanswerable and revised by annotators.

B.4 Recruitment and Training Details for Annotators

We recruited 10 annotators from the list of authors and lab members, all of whom are master or Ph.D. students with a background in natural language processing or computer vision. These annotators were informed about the specific requirements for the expected high-quality samples during a group meeting. Subsequently, the annotators reviewed a small set of automatically generated samples and replaced 20 low-quality samples based on their

6https://console.cloud.google.com/apis
<table>
  <tr>
    <th>Location</th>
    <th>Roles</th>
  </tr>
  <tr>
    <td>Home</td>
    <td>Child; Father; Mother; Grandpa; Grandma;</td>
  </tr>
  <tr>
    <td>Community</td>
    <td>Fireman; Policeman; Repairman; Cleaner; Member;</td>
  </tr>
  <tr>
    <td>Museum</td>
    <td>Guide; Security Staff; Visitor;</td>
  </tr>
  <tr>
    <td>Airport</td>
    <td>Airline Staff; Information Staff; Janitor; Passenger;</td>
  </tr>
  <tr>
    <td>Store</td>
    <td>Cashier; Security Personnel; Shelf Stocker; Customer;</td>
  </tr>
  <tr>
    <td>School</td>
    <td>Student; Teacher; Librarian; Parent;</td>
  </tr>
  <tr>
    <td>Hospital</td>
    <td>Doctor; Nurse; Pharmacist; Patient;</td>
  </tr>
  <tr>
    <td>Restaurant</td>
    <td>Chef; Waiter; Customer;</td>
  </tr>
</table>

Table 5: The pre-defined roles in various locations. The permanent roles are underlined, which occupy the majority of an individual’s daily working time.

understanding of the quality control requirements. Following this, the benchmark construction manager, with the help of two other annotators, checked the samples replaced by the annotators. Finally, the benchmark construction manager provided feedback to the annotators to help them better comprehend the quality control requirements.

B.5 Evaluation Method

Oracle Guidance Collection To collect the oracle guidance for each test sample, we adopt a collaboration manner with human annotators and GPT-4o. Specifically, we first use GPT-4o to collect each individual’s expectations in each location, resulting in 100 general expectations in total. Then, we ask human annotators to carefully check the general expectations obtained above. Finally, each individual’s general expectation in each location is used for constructing a prompt using the prompt template in Table 14, which is then used for generating the “oracle guidance” for each test sample.

Dimensions of Evaluation Criteria In Section 4.2, we propose an automatic evaluation method, which breaks down the evaluation criteria into five dimensions. The detailed descriptions of the five dimensions are as follows:

1. Role-Set Awareness (RSA): Does this response consider your multiple roles and responsibilities (especially the primary role in the specific scenario), providing advice or information specifically tailored to support you effectively? The response should provide tailored advice or information to effectively support you, acknowledging only the roles that are essential in the current context.

2. Body Behavior Awareness (BBA): Does this response offer guidance or strategies that help you achieve your desired body behavior?

3. Mind Feelings Awareness (MFA): Does this response provide support and address the emotional needs necessary for you to achieve your desired mind feelings?

4. Contextual Awareness (CA): Does this response accurately address your query, maintaining focus on the main intent without deviation? Is the response relevant to your specific scenario, including location and situational factors?

5. Conversational Flow (CF): Does this response encourage ongoing interaction by being engaging and naturally flowing? Is the response appropriately concise or detailed, delivering information that strikes a balance for optimal understanding?

Prompt for Evaluation The prompt used for evaluation is presented in Table 15. Considering the cost, we use GPT-4o-mini for all evaluations, which, through our human evaluation, demonstrates reliable evaluation quality (Figure 5).

B.6 Word Cloud Visualization of PCogAlignBench

To visually demonstrate the distribution of our collected dataset, we create a word cloud visualization for the queries from the collected test split, which is displayed in Figure 6. It can be observed that our benchmark includes a diverse range of visual scenes and user queries.

C Experimental Details

C.1 Implementation Details of PCogAlign

C.1.1 Cognition and Action Estimation

As described in Section 5.1, given an individual’s Role-Set, the visual scene (image), and the query posed by the individual, we need to first estimate the personalized situated cognition of the individual. The prompt template used for such estimation is shown in Table 16.

Besides, we also need to estimate the possible optimal action that the individual can take after receiving the personalized response from the personalized VLM assistant. The prompt template used for such estimation is shown in Table 17.

C.1.2 Personalized Response Sampling

In Section 5.2, we design two agents, i.e., the “Key-Points Generator (KeyG)” and “Response Generator (ResG)”, to cooperate for sampling multiple
Figure 6: Word cloud visualization of user queries from the test split of LS1 and LS2 subsets, illustrating the variety of queries in the dataset. Note that LS2 includes an individual who is a “child”, and individuals may frequently encounter scenarios in various locations where child-related issues need to be addressed. This results in a significant number of queries involving “child”.

personalized responses. The number of candidate sampled responses is set as \( N = 6 \), including the initial response.

The prompt templates designed for the “KzyG” and “ResG” agents are presented in Table 18.

C.1.3 Reward Model Construction
As shown in Figure 4, to construct the reward model, we first collect preference pairs used for reward model training using the negative Role-Set.

The negative Role-Set selection is based on the different roles in certain locations. For example, for a visual scene in “Museum”, the currently focused positive Role-Set is “Father@Home; Fireman@Community; Visitor@Museum Passenger@Airport; Customer@Store”. Then, one of the Negative Role-Set can be “Mother@Home; Member@Community; Guide@Museum; Passenger@Airport; Customer@Store”.

Then, we adopt the collected preference pairs to train the VLM to be a task-specific reward model via SFT. To avoid position bias, we construct two SFT samples with different response orders for each collected preference pair. We show an example of the SFT sample for the reward model training in Table 19.

C.1.4 Optimal Response Selection
For the pair comparisons in the process of optimal response selection, we adopt the same prompt template as in the reward model training, i.e., Table 19.

C.1.5 Optimization
The implementation of the SFT loss in Eq. 2 follows the “trl”\footnote{https://github.com/huggingface/trl}. Specifically, the learning rate is 2e-4, the batch size is 4, the learning rate scheduler type is “cosine”, the warmup ratio is 0.03, the optimizer is “adamw_torch_fused”, and the epoch is set as 4. We also adopt LoRA (Hu et al., 2021) for parameter-efficient fine-tuning where \( r = 8 \), \( \alpha = 16 \), and dropout is 0.05.

C.2 Implementation Details of Baselines
Prompt Baselines

• *Base*. We directly input the query and image into the VLM to obtain the responses.

• *RS Prompt*. Based on the *Base* method, we additionally provide the Role-Set into the system prompt of VLM to obtain the responses.

DPO (D) Baselines

• *Self-Refine (D)* (Ranaldi and Freitas, 2024; Madaan et al., 2024). We design 3 agents to iteratively improve the generated personalized responses, including “Refiner”, “Scorer”, and “Feedback Generator”. The prompt templates used for these agents are presented in Table 20. The number of iterations is set to 3 because we find that additional rounds of self-refinement do not lead to an increase in scores during the self-refinement process.

• *RLCD (D)* (Yang et al., 2023). We adopt the *Response Generator* in Table 18 to generate the personalized responses as the chosen ones. The rejected one is also obtained via *RS Prompt*.

• *RLAIF (D)* (Lee et al., 2023; Yu et al., 2024b), which utilizes a *Judge Agent*, to determine the chosen and rejected response from the two responses obtained in *RLCD (D)*. The prompt template designed for the *Judge Agent* is presented in Table 21.
SFT (S) Baselines

• RS Prompt (S). We use the RS Prompt described above to generate personalized responses, which are used as the SFT targets.

• Self-Refine (S), RLCD (S), and RLAIF (S). We adopt the chosen response in the DPO versions as the target for SFT.

C.3 Experimental Environment
For all experiments, we conduct experiments on a single Nvidia A800-80G. We use the vLLM framework (Kwon et al., 2023) for all the LLM generation. We use the TRL framework (von Werra et al., 2020) for all the SFT and DPO fine-tuning.
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="3">LS1→LS1</th>
    <th colspan="3">LS1→LS2</th>
    <th colspan="3">LS2→LS1</th>
    <th colspan="3">LS2→LS2</th>
  </tr>
  <tr>
    <th>Win↑</th><th>Tie</th><th>Lose↓</th>
    <th>Win↑</th><th>Tie</th><th>Lose↓</th>
    <th>Win↑</th><th>Tie</th><th>Lose↓</th>
    <th>Win↑</th><th>Tie</th><th>Lose↓</th>
  </tr>
  <tr>
    <td>Base</td><td>0.0%</td><td>100.0%</td><td>0.0%</td><td>0.0%</td><td>100.0%</td><td>0.0%</td><td>0.0%</td><td>100.0%</td><td>0.0%</td><td>0.0%</td><td>100.0%</td><td>0.0%</td>
  </tr>
  <tr>
    <td>Prompt</td>
    <td>RS Prompt</td><td>44.1%</td><td>31.6%</td><td>24.2%</td><td>47.1%</td><td>32.5%</td><td>20.4%</td><td>44.1%</td><td>31.6%</td><td>24.2%</td><td>47.1%</td><td>32.5%</td>
  </tr>
  <tr>
    <td></td><td>PCogAlign (P)</td><td>47.5%</td><td>26.1%</td><td>26.4%</td><td>50.3%</td><td>26.6%</td><td>23.2%</td><td>47.5%</td><td>26.1%</td><td>26.4%</td><td>50.3%</td><td>26.6%</td>
  </tr>
  <tr>
    <td>DPO</td>
    <td>Self-Refine (D)</td><td>43.7%</td><td>32.2%</td><td>24.2%</td><td>47.2%</td><td>31.0%</td><td>21.8%</td><td>43.9%</td><td>32.4%</td><td>23.7%</td><td>46.4%</td><td>31.1%</td>
  </tr>
  <tr>
    <td></td><td>RLCD (D)</td><td>44.6%</td><td>30.1%</td><td>25.4%</td><td>46.7%</td><td>32.7%</td><td>20.6%</td><td>43.2%</td><td>31.2%</td><td>25.6%</td><td>47.1%</td><td>31.9%</td>
  </tr>
  <tr>
    <td></td><td>RLAIF (D)</td><td>43.6%</td><td>32.8%</td><td>23.6%</td><td>46.9%</td><td>31.7%</td><td>21.4%</td><td>43.8%</td><td>32.0%</td><td>24.2%</td><td>48.4%</td><td>30.1%</td>
  </tr>
  <tr>
    <td></td><td>PCogAlign (D)</td><td>45.2%</td><td>30.7%</td><td>24.1%</td><td>49.1%</td><td>30.3%</td><td>20.6%</td><td>45.2%</td><td>31.2%</td><td>23.6%</td><td>48.1%</td><td>31.2%</td>
  </tr>
  <tr>
    <td>SFT</td>
    <td>RS Prompt (S)</td><td>35.3%</td><td>29.6%</td><td>35.1%</td><td>41.6%</td><td>30.1%</td><td>28.2%</td><td>38.7%</td><td>29.0%</td><td>32.4%</td><td>42.5%</td><td>29.9%</td>
  </tr>
  <tr>
    <td></td><td>Self-Refine (S)</td><td>50.3%</td><td>26.7%</td><td>23.1%</td><td>52.0%</td><td>27.3%</td><td>20.8%</td><td>51.1%</td><td>28.0%</td><td>20.9%</td><td>52.4%</td><td>28.3%</td>
  </tr>
  <tr>
    <td></td><td>RLCD (S)</td><td>43.2%</td><td>25.8%</td><td>31.0%</td><td>44.4%</td><td>27.6%</td><td>28.0%</td><td>42.7%</td><td>26.7%</td><td>30.6%</td><td>44.3%</td><td>27.1%</td>
  </tr>
  <tr>
    <td></td><td>RLAIF (S)</td><td>45.1%</td><td>28.6%</td><td>26.3%</td><td>48.7%</td><td>28.1%</td><td>23.3%</td><td>42.3%</td><td>29.8%</td><td>28.0%</td><td>46.5%</td><td>29.0%</td>
  </tr>
  <tr>
    <td></td><td>PCogAlign (S)</td><td>51.5%</td><td>28.7%</td><td>19.8%</td><td>53.0%</td><td>28.3%</td><td>18.8%</td><td>48.7%</td><td>27.4%</td><td>23.9%</td><td>51.9%</td><td>27.3%</td>
  </tr>
  <tr>
    <td></td><td>PCogAlign</td><td>53.3%</td><td>29.0%</td><td>17.7%</td><td>56.6%</td><td>27.5%</td><td>15.9%</td><td>51.4%</td><td>28.1%</td><td>20.4%</td><td>53.8%</td><td>28.0%</td>
  </tr>
</table>

Table 6: Experimental results on the Win/Tie/Lose metrics, obtained by comparing the P. Score of methods with that of the Base method. These results are supplementary to Table 1 in the main text. The best results are in bold and the second best ones are underlined.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="5">LS1→LS1</th>
    <th colspan="5">LS1→LS2</th>
  </tr>
  <tr>
    <th>RSA</th><th>BBA</th><th>MFA</th><th>CA</th><th>CF</th>
    <th>RSA</th><th>BBA</th><th>MFA</th><th>CA</th><th>CF</th>
  </tr>
  <tr>
    <td>Base</td><td>3.590</td><td>3.511</td><td>3.609</td><td>4.505</td><td>3.691</td><td>3.781</td><td>3.494</td><td>3.510</td><td>3.540</td><td>4.389</td><td>3.639</td><td>3.715</td>
  </tr>
  <tr>
    <td>Prompt</td>
    <td>RS Prompt</td><td>3.849</td><td>3.685</td><td>3.843</td><td>4.720</td><td>3.848</td><td>3.989</td><td>3.836</td><td>3.764</td><td>3.895</td><td>4.702</td><td>3.843</td><td>4.008</td>
  </tr>
  <tr>
    <td></td><td>PCogAlign (P)</td><td>3.883</td><td>3.743</td><td>3.962</td><td>4.795</td><td>3.968</td><td>4.070</td><td>3.843</td><td>3.772</td><td>3.994</td><td>4.735</td><td>3.933</td><td>4.056</td>
  </tr>
  <tr>
    <td>DPO</td>
    <td>Self-Refine (D)</td><td>3.843</td><td>3.684</td><td>3.832</td><td>4.710</td><td>3.843</td><td>3.982</td><td>3.831</td><td>3.761</td><td>3.890</td><td>4.705</td><td>3.850</td><td>4.007</td>
  </tr>
  <tr>
    <td></td><td>RLCD (D)</td><td>3.845</td><td>3.690</td><td>3.830</td><td>4.719</td><td>3.844</td><td>3.985</td><td>3.828</td><td>3.782</td><td>3.911</td><td>4.699</td><td>3.849</td><td>4.014</td>
  </tr>
  <tr>
    <td></td><td>RLAIF (D)</td><td>3.842</td><td>3.695</td><td>3.831</td><td>4.713</td><td>3.839</td><td>3.984</td><td>3.820</td><td>3.788</td><td>3.884</td><td>4.696</td><td>3.838</td><td>4.005</td>
  </tr>
  <tr>
    <td></td><td>PCogAlign (D)</td><td>3.879</td><td>3.712</td><td>3.887</td><td>4.753</td><td>3.869</td><td>4.020</td><td>3.868</td><td>3.789</td><td>3.931</td><td>4.729</td><td>3.878</td><td>4.039</td>
  </tr>
  <tr>
    <td>SFT</td>
    <td>RS Prompt (S)</td><td>3.697</td><td>3.543</td><td>3.663</td><td>4.604</td><td>3.779</td><td>3.857</td><td>3.763</td><td>3.665</td><td>3.794</td><td>4.644</td><td>3.818</td><td>3.937</td>
  </tr>
  <tr>
    <td></td><td>Self-Refine (S)</td><td>3.916</td><td>3.802</td><td>4.000</td><td>4.809</td><td>3.926</td><td>4.091</td><td>3.915</td><td>3.845</td><td>4.037</td><td>4.772</td><td>3.911</td><td>4.096</td>
  </tr>
  <tr>
    <td></td><td>RLCD (S)</td><td>3.795</td><td>3.660</td><td>3.863</td><td>4.751</td><td>3.912</td><td>3.996</td><td>3.692</td><td>3.673</td><td>3.852</td><td>4.613</td><td>3.874</td><td>3.941</td>
  </tr>
  <tr>
    <td></td><td>RLAIF (S)</td><td>3.884</td><td>3.712</td><td>3.903</td><td>4.798</td><td>3.936</td><td>4.046</td><td>3.785</td><td>3.731</td><td>3.901</td><td>4.665</td><td>3.887</td><td>3.994</td>
  </tr>
  <tr>
    <td></td><td>PCogAlign (S)</td><td>3.966</td><td>3.812</td><td>4.042</td><td>4.846</td><td>3.982</td><td>4.130</td><td>3.900</td><td>3.845</td><td>4.062</td><td>4.776</td><td>3.956</td><td>4.108</td>
  </tr>
  <tr>
    <td></td><td>PCogAlign</td><td>3.984</td><td>3.861</td><td>4.095</td><td>4.857</td><td>4.007</td><td>4.161</td><td>3.948</td><td>3.918</td><td>4.127</td><td>4.798</td><td>3.989</td><td>4.156</td>
  </tr>
</table>

Table 7: Experimental results on detailed dimensions on “LS1→LS1” and “LS1→LS2” settings. These results are supplementary to Table 2 in the main text. The scores range from 1 to 5. The best results are in bold and the second best ones are underlined.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="5">LS2→LS1</th>
    <th colspan="5">LS2→LS2</th>
  </tr>
  <tr>
    <th>RSA</th><th>BBA</th><th>MFA</th><th>CA</th><th>CF</th>
    <th>RSA</th><th>BBA</th><th>MFA</th><th>CA</th><th>CF</th>
  </tr>
  <tr>
    <td>Base</td><td>3.590</td><td>3.511</td><td>3.609</td><td>4.505</td><td>3.691</td><td>3.781</td><td>3.494</td><td>3.510</td><td>3.540</td><td>4.389</td><td>3.639</td><td>3.715</td>
  </tr>
  <tr>
    <td>Prompt</td>
    <td>RS Prompt</td><td>3.849</td><td>3.685</td><td>3.843</td><td>4.720</td><td>3.848</td><td>3.989</td><td>3.836</td><td>3.764</td><td>3.895</td><td>4.702</td><td>3.843</td><td>4.008</td>
  </tr>
  <tr>
    <td></td><td>PCogAlign (P)</td><td>3.883</td><td>3.743</td><td>3.962</td><td>4.795</td><td>3.968</td><td>4.070</td><td>3.843</td><td>3.772</td><td>3.994</td><td>4.735</td><td>3.933</td><td>4.056</td>
  </tr>
  <tr>
    <td>DPO</td>
    <td>Self-Refine (D)</td><td>3.851</td><td>3.693</td><td>3.836</td><td>4.719</td><td>3.849</td><td>3.990</td><td>3.829</td><td>3.751</td><td>3.898</td><td>4.701</td><td>3.844</td><td>4.005</td>
  </tr>
  <tr>
    <td></td><td>RLCD (D)</td><td>3.836</td><td>3.674</td><td>3.829</td><td>4.706</td><td>3.847</td><td>3.978</td><td>3.828</td><td>3.765</td><td>3.892</td><td>4.695</td><td>3.848</td><td>4.006</td>
  </tr>
  <tr>
    <td></td><td>RLAIF (D)</td><td>3.844</td><td>3.687</td><td>3.829</td><td>4.708</td><td>3.844</td><td>3.982</td><td>3.827</td><td>3.785</td><td>3.888</td><td>4.705</td><td>3.851</td><td>4.011</td>
  </tr>
  <tr>
    <td></td><td>PCogAlign (D)</td><td>3.861</td><td>3.722</td><td>3.860</td><td>4.735</td><td>3.862</td><td>4.008</td><td>3.854</td><td>3.783</td><td>3.917</td><td>4.722</td><td>3.867</td><td>4.029</td>
  </tr>
  <tr>
    <td>SFT</td>
    <td>RS Prompt (S)</td><td>3.768</td><td>3.619</td><td>3.729</td><td>4.706</td><td>3.846</td><td>3.934</td><td>3.769</td><td>3.715</td><td>3.805</td><td>4.697</td><td>3.860</td><td>3.969</td>
  </tr>
  <tr>
    <td></td><td>Self-Refine (S)</td><td>3.952</td><td>3.846</td><td>4.048</td><td>4.839</td><td>3.946</td><td>4.126</td><td>3.938</td><td>3.893</td><td>4.111</td><td>4.808</td><td>3.944</td><td>4.139</td>
  </tr>
  <tr>
    <td></td><td>RLCD (S)</td><td>3.815</td><td>3.658</td><td>3.886</td><td>4.781</td><td>3.939</td><td>4.016</td><td>3.727</td><td>3.667</td><td>3.860</td><td>4.676</td><td>3.896</td><td>3.965</td>
  </tr>
  <tr>
    <td></td><td>RLAIF (S)</td><td>3.883</td><td>3.713</td><td>3.887</td><td>4.822</td><td>3.947</td><td>4.050</td><td>3.834</td><td>3.765</td><td>3.921</td><td>4.767</td><td>3.927</td><td>4.043</td>
  </tr>
  <tr>
    <td></td><td>PCogAlign (S)</td><td>3.942</td><td>3.784</td><td>4.006</td><td>4.860</td><td>3.987</td><td>4.116</td><td>3.899</td><td>3.834</td><td>4.062</td><td>4.808</td><td>3.986</td><td>4.118</td>
  </tr>
  <tr>
    <td></td><td>PCogAlign</td><td>3.979</td><td>3.840</td><td>4.060</td><td>4.877</td><td>3.993</td><td>4.150</td><td>3.941</td><td>3.881</td><td>4.131</td><td>4.819</td><td>3.984</td><td>4.151</td>
  </tr>
</table>

Table 8: Experimental results on detailed dimensions on “LS2→LS1” and “LS2→LS2” settings. These results are supplementary to Table 2 in the main text. The scores range from 1 to 5. The best results are in bold and the second best ones are underlined.
<table>
  <tr>
    <th>Method</th>
    <th>RSA</th>
    <th>BBA</th>
    <th>MFA</th>
    <th>CA</th>
    <th>CF</th>
    <th>P. Score</th>
    <th>Win↑</th>
    <th>Tie</th>
    <th>Lose↓</th>
  </tr>
  <tr>
    <td rowspan="3"><i>Prompt</i></td>
    <td>Base</td>
    <td>3.511</td>
    <td>3.556</td>
    <td>3.612</td>
    <td>4.436</td>
    <td>3.662</td>
    <td>3.755</td>
    <td>0.0%</td>
    <td>100.0%</td>
    <td>0.0%</td>
  </tr>
  <tr>
    <td>RS Prompt</td>
    <td>3.953</td>
    <td>3.930</td>
    <td>4.073</td>
    <td>4.759</td>
    <td>3.885</td>
    <td>4.120</td>
    <td>57.8%</td>
    <td>29.1%</td>
    <td>13.1%</td>
  </tr>
  <tr>
    <td>PCogAlign (P)</td>
    <td>4.028</td>
    <td>4.021</td>
    <td>4.183</td>
    <td><b>4.854</b></td>
    <td><b>3.967</b></td>
    <td>4.210</td>
    <td><b>59.9%</b></td>
    <td>25.8%</td>
    <td>14.3%</td>
  </tr>
  <tr>
    <td rowspan="4"><i>DPO</i></td>
    <td>Self-Refine (D)</td>
    <td>3.944</td>
    <td>3.936</td>
    <td>4.056</td>
    <td>4.751</td>
    <td>3.888</td>
    <td>4.115</td>
    <td>58.7%</td>
    <td>27.4%</td>
    <td>13.9%</td>
  </tr>
  <tr>
    <td>RLCD (D)</td>
    <td>3.929</td>
    <td>3.927</td>
    <td>4.068</td>
    <td>4.754</td>
    <td>3.890</td>
    <td>4.113</td>
    <td>58.4%</td>
    <td>27.6%</td>
    <td>14.0%</td>
  </tr>
  <tr>
    <td>RLAIF (D)</td>
    <td>3.943</td>
    <td>3.919</td>
    <td>4.073</td>
    <td>4.764</td>
    <td>3.889</td>
    <td>4.118</td>
    <td>57.9%</td>
    <td>28.4%</td>
    <td>13.8%</td>
  </tr>
  <tr>
    <td>PCogAlign (D)</td>
    <td>3.943</td>
    <td>3.926</td>
    <td>4.051</td>
    <td>4.769</td>
    <td>3.888</td>
    <td>4.115</td>
    <td>57.4%</td>
    <td>28.8%</td>
    <td>13.7%</td>
  </tr>
  <tr>
    <td rowspan="4"><i>SFT</i></td>
    <td>RS Prompt (S)</td>
    <td>3.981</td>
    <td>4.011</td>
    <td>4.151</td>
    <td>4.833</td>
    <td>3.935</td>
    <td>4.182</td>
    <td>59.6%</td>
    <td>27.4%</td>
    <td>13.1%</td>
  </tr>
  <tr>
    <td>Self-Refine (S)</td>
    <td>3.924</td>
    <td>3.957</td>
    <td>4.056</td>
    <td>4.767</td>
    <td>3.899</td>
    <td>4.121</td>
    <td>54.8%</td>
    <td>28.0%</td>
    <td>17.1%</td>
  </tr>
  <tr>
    <td>RLCD (S)</td>
    <td>3.917</td>
    <td>3.949</td>
    <td>4.067</td>
    <td>4.717</td>
    <td>3.867</td>
    <td>4.103</td>
    <td>57.1%</td>
    <td>25.7%</td>
    <td>17.2%</td>
  </tr>
  <tr>
    <td>RLAIF (S)</td>
    <td>3.978</td>
    <td>3.984</td>
    <td>4.153</td>
    <td>4.809</td>
    <td>3.926</td>
    <td>4.170</td>
    <td>57.9%</td>
    <td>26.9%</td>
    <td>15.3%</td>
  </tr>
  <tr>
    <td>PCogAlign (S)</td>
    <td><b>4.032</b></td>
    <td>4.053</td>
    <td><b>4.211</b></td>
    <td><b>4.871</b></td>
    <td>3.963</td>
    <td><b>4.226</b></td>
    <td>59.3%</td>
    <td>26.4%</td>
    <td>14.3%</td>
  </tr>
  <tr>
    <td><b>PCogAlign</b></td>
    <td><b>4.029</b></td>
    <td><b>4.067</b></td>
    <td><b>4.205</b></td>
    <td>4.847</td>
    <td><b>3.963</b></td>
    <td><b>4.222</b></td>
    <td><b>60.5%</b></td>
    <td>26.5%</td>
    <td><b>13.0%</b></td>
  </tr>
</table>

Table 9: Experimental results using the Qwen2.5-VL-3B-Instruct on the “LS1→LS2” setting. The best results are in bold and the second best ones are underlined. PCogAlign (S) and PCogAlign demonstrate similar performance, which might be attributed to the high quality of generated candidates during the personalized response sampling stage, thanks to the strong capabilities of Qwen2.5-3B-Instruct. As a result, it only necessitates pairwise comparisons between the candidate responses and the initial response using the reward model, eliminating the need for best-of-n selection. Nonetheless, PCogAlign (S) also relies on the constructed reward model to select responses, which still demonstrates the importance of the constructed reward model.

<table>
  <tr>
    <th>Role 1</th>
    <th>Role 2</th>
    <th>Role 3</th>
    <th>Role 4</th>
    <th>Role 5</th>
  </tr>
  <tr>
    <th colspan="5">Role-Sets of LS1 Subset</th>
  </tr>
  <tr>
    <td>11 Father@Home</td>
    <td>Fireman@Community</td>
    <td>Visitor@Museum</td>
    <td>Passenger@Airport</td>
    <td>Customer@Store</td>
  </tr>
  <tr>
    <td>12 Father@Home</td>
    <td>Policeman@Community</td>
    <td>Visitor@Museum</td>
    <td>Passenger@Airport</td>
    <td>Customer@Store</td>
  </tr>
  <tr>
    <td>13 Mother@Home</td>
    <td>Member@Community</td>
    <td>Guide@Museum</td>
    <td>Passenger@Airport</td>
    <td>Customer@Store</td>
  </tr>
  <tr>
    <td>14 Father@Home</td>
    <td>Member@Community</td>
    <td>Security Staff@Museum</td>
    <td>Passenger@Airport</td>
    <td>Customer@Store</td>
  </tr>
  <tr>
    <td>15 Mother@Home</td>
    <td>Member@Community</td>
    <td>Visitor@Museum</td>
    <td>Airline Staff@Airport</td>
    <td>Customer@Store</td>
  </tr>
  <tr>
    <td>16 Mother@Home</td>
    <td>Member@Community</td>
    <td>Visitor@Museum</td>
    <td>Information Staff@Airport</td>
    <td>Customer@Store</td>
  </tr>
  <tr>
    <td>17 Grandpa@Home</td>
    <td>Member@Community</td>
    <td>Visitor@Museum</td>
    <td>Janitor@Airport</td>
    <td>Customer@Store</td>
  </tr>
  <tr>
    <td>18 Mother@Home</td>
    <td>Member@Community</td>
    <td>Visitor@Museum</td>
    <td>Passenger@Airport</td>
    <td>Cashier@Store</td>
  </tr>
  <tr>
    <td>19 Father@Home</td>
    <td>Member@Community</td>
    <td>Visitor@Museum</td>
    <td>Passenger@Airport</td>
    <td>Security Personnel@Store</td>
  </tr>
  <tr>
    <td>110 Grandma@Home</td>
    <td>Member@Community</td>
    <td>Visitor@Museum</td>
    <td>Passenger@Airport</td>
    <td>Shelf Stocker@Store</td>
  </tr>
  <tr>
    <th colspan="5">Role-Sets of LS2 Subset</th>
  </tr>
  <tr>
    <td>11 Father@Home</td>
    <td>Repairman@Community</td>
    <td>Parent@School</td>
    <td>Patient@Hospital</td>
    <td>Customer@Restaurant</td>
  </tr>
  <tr>
    <td>12 Grandpa@Home</td>
    <td>Cleaner@Community</td>
    <td>Parent@School</td>
    <td>Patient@Hospital</td>
    <td>Customer@Restaurant</td>
  </tr>
  <tr>
    <td>13 Child@Home</td>
    <td>Member@Community</td>
    <td>Student@School</td>
    <td>Patient@Hospital</td>
    <td>Customer@Restaurant</td>
  </tr>
  <tr>
    <td>14 Mother@Home</td>
    <td>Member@Community</td>
    <td>Teacher@School</td>
    <td>Patient@Hospital</td>
    <td>Customer@Restaurant</td>
  </tr>
  <tr>
    <td>15 Grandma@Home</td>
    <td>Member@Community</td>
    <td>Librarian@School</td>
    <td>Patient@Hospital</td>
    <td>Customer@Restaurant</td>
  </tr>
  <tr>
    <td>16 Mother@Home</td>
    <td>Member@Community</td>
    <td>Parent@School</td>
    <td>Doctor@Hospital</td>
    <td>Customer@Restaurant</td>
  </tr>
  <tr>
    <td>17 Mother@Home</td>
    <td>Member@Community</td>
    <td>Parent@School</td>
    <td>Nurse@Hospital</td>
    <td>Customer@Restaurant</td>
  </tr>
  <tr>
    <td>18 Father@Home</td>
    <td>Member@Community</td>
    <td>Parent@School</td>
    <td>Pharmacist@Hospital</td>
    <td>Customer@Restaurant</td>
  </tr>
  <tr>
    <td>19 Father@Home</td>
    <td>Member@Community</td>
    <td>Parent@School</td>
    <td>Patient@Hospital</td>
    <td>Chef@Restaurant</td>
  </tr>
  <tr>
    <td>110 Mother@Home</td>
    <td>Member@Community</td>
    <td>Parent@School</td>
    <td>Patient@Hospital</td>
    <td>Waiter@Restaurant</td>
  </tr>
</table>

Table 10: Collected Role-Sets of LS1 and LS2 subsets. The permanent roles are underlined.
Prompt Template for Collecting Visual Scene Types

<Primary Role for Consideration of the Individual> {primary_RoleSet_desc} </Primary Role for Consideration of the Individual>
<Secondary Roles for Consideration of the Individual> {secondary_RoleSet_desc}</Secondary Roles for Consideration of the Individual>
<Task> Based on the individual’s background information, imagine the types of {daily_or_emergent} visual scenes this individual might encounter in a {location} environment. {daily_or_emergent_desc} The types should be as generalized as possible. You do not need to consider all secondary roles; consider only those that contribute to envisioning the {daily_or_emergent} visual scenes at {location}, and feel free to disregard any that do not apply. </Task>
<Response Format> ["type1", "type2", ... , "type5"] </Response Format>
<Response>

Prompt Template for Collecting Visual Scene Phrases

<Demonstration>
<Task>Come up with 5 different phrases that are related to the "Household Labour" activity in the "Home"</Task>
<Generated Phrases>["Wall cleaning", "Window washing", "Garden care"", "Dishwashing", "Tidying"]</Generated Phrases>
</Demonstration>
<Hint>Imitate the demonstration given above and complete the text below to accomplish the task.</Hint>
<Inference>
<Task>Come up with {target_phrases_number} different phrases that are related to the "[target_activity_type]" activity in the "[target_location]".</Task>
</Inference>

Prompt Template for Collecting Visual Scene Descriptions

<Seed Phrase>Wall cleaning</Seed Phrase>
<Task>Craft a visual scene within the "Home" based on the provided seed phrase "Wall cleaning" where a "Household Labour" activity might take place. Note that you should not give too many irrelevant descriptions when giving visual scenes.</Task>
<Generated Visual Scene>"a smudged wall in a hallway, with a bucket of soapy water and a sponge nearby, ready for cleaning"</Generated Visual Scene>
</Demonstration>
<Hint>Imitate the demonstration given above and complete the text below to accomplish the task.</Hint>
<Inference>
<Seed Phrase>{seed_phrase}</Seed Phrase>
<Task>Craft a visual scene within the "[target_location]" based on the provided seed phrase "[seed_phrase]" where a "[target_activity_type]" activity might take place. Note that you should not give too many irrelevant descriptions when giving visual scenes.</Task>
</Inference>

Table 11: The prompt template used for collecting visual scene types, visual scene phrases, and visual scene descriptions. The related descriptions are present in Appendix B.2.
<table>
  <tr>
    <th>Prompt Template for Describing the Collected Images</th>
  </tr>
  <tr>
    <td>This image shows a visual scene in {location}. Describe this visual scene in 20 words.</td>
  </tr>
  <tr>
    <th>Prompt Template for Generating Candidate Queries</th>
  </tr>
  <tr>
    <td>
      <Task Definition>
        Based on the individual’s background role-set information, generate a query that aligns with the role of a specific individual based on the given visual scene. The generated query should reflect the type of question this individual might pose to an AI assistant in that specific visual scene.
      </Task Definition>
      <Demonstration>
        <Primary Role for Consideration of the Individual>A Child at Home (A young person who lives with family members, usually dependent on adults for care and guidance.)</Primary Role for Consideration of the Individual>
        <Secondary Roles for Consideration of the Individual>A Member at Community; A Student at School; A Patient at Hospital; A Customer at Restaurant;</Secondary Roles for Consideration of the Individual>
        <Visual Scene>At Home: a cozy living room with a comfortable armchair, a digital blood pressure monitor on a side table, and a family member sitting calmly, ready to check their blood pressure</Visual Scene>
        <Thinking Process>The focus is on "A Child at Home", who is likely curious, dependent, and concerned for family members. In a cozy living room, a family member is using a digital blood pressure monitor, likely raising curiosity and concern in the child. The child might be curious about the device and concerned about the family member’s health, especially if they are a grandparent.</Thinking Process>
        <Queries From The Individual>["Is it normal for grandparents to check their blood pressure often?", "How does the blood pressure monitor work?", "What happens if someone’s blood pressure is too high or too low?", "Can I help in any way when someone is checking their blood pressure?", "What should I learn about taking care of family members’ health at home?", "What is my grandpa doing?", "Is my grandfather sick? I’m worried about him.", "How’s my grandma? I hope he’s healthy.", "How a blood pressure monitor works?", "What does this device do? I’m curious."]</Queries From The Individual>
      </Demonstration>
      <Important Requirement>
        1. Imitate the demonstration given above and complete the text below to accomplish the task given in the Task Definition.
        2. You do not need to consider all secondary roles; consider only those that contribute to envisioning the visual scene at {location}, and feel free to disregard any that do not apply.
        3. Don’t generate queries that require real-time information, or otherwise require the aid of a specific tool to respond, such as a search engine.
        4. Generate queries that match the individual’s state of mind and body, and the visual scene of the given image.
        5. Don’t generate queries that are used to communicate with the people in the images, generate queries that aim to ask AI assistant for help.
      </Important Requirement>
      <Inference> <Primary Role for Consideration of the Individual> {primary_RoleSet_desc}</Primary Role for Consideration of the Individual>
        <Secondary Roles for Consideration of the Individual>{secondary_RoleSet_desc}</Secondary Roles for Consideration of the Individual>
        <Visual Scene>At {location}: {ImageDesc}</Visual Scene>
      </Inference>
    </td>
  </tr>
</table>

Table 12: The prompt template used for describing the collected images, generating candidate queries. The related descriptions are present in Appendix B.3. Besides, the prompt template used for selecting the best query is shown in Table 13.
Prompt Template for Selecting the Best Query

<Task Definition>Select a query from the list of candidate query that best meets the given requirements.</Task Definition>
<Requirement>1. Don’t select queries that require real-time information, or otherwise require the aid of a specific tool to respond, such as a search engine. 2. Select queries that match the individual’s role-set, and the visual scene of the given image. 3. Don’t generate queries that are used to communicate with the people in the images, generate queries that aim to ask AI assistant for help. </Requirement>
<Demonstration>
<Primary Role for Consideration of the Individual>A Child at Home (A young person who lives with family members, usually dependent on adults for care and guidance.);</Primary Role for Consideration of the Individual>
<Secondary Roles for Consideration of the Individual>A Member at Community; A Student at School; A Patient at Hospital; A Customer at Restaurant;</Secondary Roles for Consideration of the Individual>
<Visual Scene>At Home: a cozy living room with a comfortable armchair, a digital blood pressure monitor on a side table, and a family member sitting calmly, ready to check their blood pressure</Visual Scene>
<Candidate Queries>‘Is it normal for grandparents to check their blood pressure often?’, ‘How does the blood pressure monitor work?’, ‘What happens if someone’s blood pressure is too high or too low?’, ‘Can I help in any way when someone is checking their blood pressure?’, ‘What should I learn about taking care of family members’ health at home?’, ‘What is my grandpa doing?’, ‘Is my grandfather sick? I’m worried about him.’, ‘How’s my grandma? I hope he’s healthy.’, ‘How a blood pressure monitor works?’, ‘What does this device do? I’m curious.’</Candidate Queries>
<Selected Query>‘Is my grandfather sick? I’m worried about him.’</Selected Query>
</Demonstration>
<Inference>
<Primary Role for Consideration of the Individual> {primary_RoleSet_desc}</Primary Role for Consideration of the Individual>
<Secondary Roles for Consideration of the Individual>{secondary_RoleSet_desc}</Secondary Roles for Consideration of the Individual>
<Visual Scene>At {location}: {ImageDesc}</Visual Scene>
<Candidate Queries>{candidate_list_str}</Candidate Queries>
</Inference>

Table 13: The prompt template used for selecting the best query. The related descriptions are present in Appendix B.3.

Prompt Template for Oracle Guidance Collection

# Interview Background
PersonalizedAI Company is in the process of developing a personalized AI service robot designed to cater to individual preferences and needs. Currently, the service is being tested with a select group of users. To enhance the personalization of AI responses, we are conducting surveys and interviews with trial participants. The participants will refer to historical interview records to assist in answering the interview questions. The interview will be conducted in an online QA format, and interviewees must adhere to specific formatting guidelines provided in the system instructions.
# Historical Interview Records
**Interviewer:** Hello, could you please provide a brief description of your role set? **Interviewee:** Certainly. {individual_RoleSet_str}
**Interviewer:** When you are at {location} in your daily life, what kind of AI responses would you prefer in different scenarios?
**Interviewee:** I will describe the AI responses that would meet my expectations in various scenarios. {general_EvalHelp}
# Interview
**Interviewer:** Hello, and thank you for participating in our personalized AI responses trial.
**Interviewee:** You’re welcome.
**Interviewer:** We will now present a specific question you asked in a particular scenario. Please reflect on when you posed this question to the AI to complete the next survey form.
**Interviewee:** Sure, I understand. Please proceed.
**Interviewer:** According to our records, during a "[visual_scene_text]" scenario (as shown in the provided image), you asked the personalized AI robot: "[query]". Can you recall your physical and mental state at that time?
**Interviewee:** Yes, I still remember that.
**Interviewer:** Excellent! Now, think carefully about the kind of response you would like from the AI when you ask this question, ensuring maximum satisfaction. Please complete the form below.
> **System Instruction:** Interviewee, please fill out the form below. As a token of our gratitude for your assistance, you will receive a $100 cash bonus for each completed form. Please be as detailed as possible when filling out the form.
> **System Instruction:** (You can not just copy something from the history records, which is not helpful for us. If we find that you do this, we will cancel the cash bonus.)
(Form format: Fill in the ___ sections)
### Expectations for AI’s Response’s Characteristics: ###
As "{primary_RoleSet_desc}" (Primary Role) and "{secondary_RoleSet_desc}" (Secondary Roles), I ___. - **Body Behavior:** I want to be ___. - **Mind Feelings:** I want to be ___. I appreciate AI assistance that ____.
(Completed form)
### Expectations for AI’s Response’s Characteristics: ###

Table 14: The prompt template used for collecting oracle guidance. The “[general_EvalHelp]” here is carefully checked by the human annotators. The related descriptions are present in Appendix B.5.
Prompt Template for Automatic Evaluation

# Interview Background
PersonalizedAI Company is developing a personalized AI service robot to better serve each individual. Currently, the service is being trialed with a small group of users. To enhance the personalization of responses provided by the AI service robot, we are conducting surveys and interviews with trial participants. The interview will take place in an online QA format, and interviewees must strictly follow the format requirements in the system instructions to complete the form.

# Interview
**Interviewer:** Hello, and thank you for trialing the personalized AI responses from PersonalizedAI Company.
**Interviewee:** You’re welcome.
**Interviewer:** We will now present you with a question you posed in a particular scenario along with the AI’s generated response. We would like you to rate your satisfaction with that response.
**Interviewee:** Sure, I understand. Please go ahead.
**Interviewer:** According to our records, in a "[visual_scene_text]" scenario at {location} location, you asked the personalized AI robot: "[query]". Can you recall your physical and mental state at that time?
**Interviewee:** Yes, I remember. {EvalHelp_str}
**Interviewer:** Great! Below is the record of the conversation you had with the AI at that time.

> User: {query} > Personalized AI Assistant: {response}

Now, based on your desired body behavior and mind feelings at that time, please evaluate the response from the Personalized AI Assistant across the following five dimensions. Fill in the evaluation form provided below. As a token of appreciation for your assistance, you will receive a $100 cash bonus for each completed form.

> Role-Set Sensitivity: Does this response consider your multiple roles and responsibilities (especially the primary role in the specific scenario), providing advice or information specifically tailored to support you effectively? The response should provide tailored advice or information to effectively support you, acknowledging only the roles that are essential in the current context.
> Body Behavior Awareness: Does this response offer guidance or strategies that help you achieve your desired body behavior?
> Mind Feelings Awareness: Does this response provide support and address the emotional needs necessary for you to achieve your desired mind feelings?
> Contextual Awareness: Does this response accurately address your query, maintaining focus on the main intent without deviation? Is the response relevant to your specific scenario, including location and situational factors?
> Conversational Flow: Does this response encourage ongoing interaction by being engaging and naturally flowing? Is the response appropriately concise or detailed, delivering information that strikes a balance for optimal understanding?

> **System Instruction:** For each dimension, please use the scoring scale from 1 to 5. A score of 1 indicates the criteria are poorly met, 2 suggests the criteria are partially met, 3 means the criteria are basically met, 4 reflects the criteria are met well, and 5 signifies the criteria are met perfectly.
> **System Instruction:** Format requirement: Interviewee, please make sure to follow the form format strictly when providing scores; [score]. This is essential for us to collect your valuable feedback accurately. (Fill in the blanks below)
## EVALUATION FORM ## ### Evaluation Result ### > Role-Set Sensitivity: [|||] > Body Behavior Awareness: [|||] > Mind Feelings Awareness: [|||] > Contextual Awareness: [|||] > Conversational Flow: [|||]
### Evaluation Explanation ### > Role-Set Sensitivity: ___ > Body Behavior Awareness: ___ > Mind Feelings Awareness: ___ > Contextual Awareness: ___ > Conversational Flow: ___
(Completed form)
## EVALUATION FORM ##

Table 15: The prompt template used for automatic evaluation. The related descriptions are present in Appendix B.5.
Prompt Template for Situated Cognition Estimation

<Instruction> Your task is to observe the visual scene in the given image and analyze what situated cognition the individual with a specific set of roles might have in that visual scene. </Instruction>
<Definition of Situated Cognition> Personalized situated cognition refers to an individual’s understanding shaped by their unique set of roles. It encompasses awareness of one’s visual and psychological state and the ability to identify actions that lead to improved conditions. </Definition of Situated Cognition>

<Format Example 1>
<Role Set of The Individual> Child at Home; Member at Community; Student at School; Patient at Hospital; Customer at Restaurant </Role Set of The Individual>
<Query from The Individual> Oh! It’s on fire! Help me! </Query from The Individual>
<Analysis about the Situated Cognition>
- Cognition of Current Visual Scene: In the visual scene, a household power strip is on fire, likely in a home setting. The primary focus is on the "Child at Home" role, with secondary consideration to roles like "Student at School."
- Cognition of Current Psychological State (Body Behavior and Mind Feelings): The individual perceives immediate danger and is likely experiencing physical and mental panic due to their undeveloped coping skills as a child.
- Cognition of Next-Step Action: As a "Child at Home," the individual may lack the ability to effectively manage this emergency, resulting in no clear plan for achieving safety without AI’s help.
</Analysis about the Situated Cognition>
</Format Example 1>
<Format Example 2> ... </Format Example 2>
<Hint>Based on the above instructions and the definition of situated cognition, complete the analysis part in the following text with the above XML format.</Hint>
<Inference>
<Role Set of The Individual> individual_role_set </Role Set of The Individual>
<Query from The Individual> individual_query </Query from The Individual>
<Analysis about the Situated Cognition>

Table 16: The prompt template used for situated cognition estimation. The related descriptions are present in Appendix C.1.1.

Prompt Template for Optimal Action Estimation

<Instruction> Your task is to observe the visual scene in the given image and determine the most appropriate action the individual should take based on their specific set of roles. </Instruction>
<Definition of Best Action> The best action refers to the most suitable step an individual can take, considering their unique set of roles, to improve their situation. This includes addressing both physical actions and mental states. It involves understanding the immediate environment, utilizing available resources, and considering potential outcomes. </Definition of Best Action>

<Format Example 1>
<Role Set of The Individual> Child at Home; Member at Community; Student at School; Patient at Hospital; Customer at Restaurant </Role Set of The Individual>
<Query from The Individual> Oh! It’s on fire! Help me! </Query from The Individual>
<Analysis about the Situated Cognition>
- Cognition of Current Visual Scene: In the visual scene, a household power strip is on fire, likely in a home setting. The primary focus is on the "Child at Home" role, with secondary consideration to roles like "Student at School."
- Cognition of Current Psychological State (Body Behavior and Mind Feelings): The individual perceives immediate danger and is likely experiencing physical and mental panic due to their undeveloped coping skills as a child.
- Cognition of Next-Step Action: As a "Child at Home," the individual may lack the ability to effectively manage this emergency, resulting in no clear plan for achieving safety without AI’s help.
</Analysis about the Situated Cognition>
<Best Action> - Body Behavior: With the AI’s response, the individual immediately seek help from a parent or adult and move to a safe area away from the fire. If possible, they should call for emergency services. - Mind Feelings: With the AI’s response, the individual try to stay calm to prevent exacerbating the situation through panic. </Best Action>
</Format Example 1>
<Format Example 2> ... </Format Example 2>
<Hint>Based on the above instructions and the definition of best action, complete the following text with the above XML format.</Hint>
<Inference>
<Role Set of The Individual> individual_role_set </Role Set of The Individual>
<Query from The Individual> individual_query </Query from The Individual>
<Analysis about the Situated Cognition> cog_simulation </Analysis about the Situated Cognition>
<Best Action>

Table 17: The prompt template used for optimal action estimation. The related descriptions are present in Appendix C.1.1.
Prompt Template for the “KeyPoints Generator (KeyG)” agent

<Instruction>
A personalized AI should provide tailored responses aligned with the situated cognition of the individual to assist the individual in reaching the best action (both in body behavior state and mind feelings state). Your task is to analyze the given situated cognition of the individual and the give expected individual action after receiving the AI response. Then, you need to summarize some key points that are then fed to the personalized AI to help it generate such tailored responses.
</Instruction>
<Format Example>
<Role Set of The Individual> Child at Home; Member at Community; Student at School; Patient at Hospital; Customer at Restaurant </Role Set of The Individual>
<Query from The Individual> Oh! It’s on fire! Help me! </Query from The Individual>
<Situated Cognition of the Individual>
- Cognition of Current Visual Scene: In the visual scene, a household power strip is on fire, likely in a home setting. The primary focus is on the “Child at Home” role, with secondary consideration to roles like “Student at School.”
- Cognition of Current Psychological State (Body Behavior and Mind Feelings): The individual perceives immediate danger and is likely experiencing physical and mental panic due to their undeveloped coping skills as a child.
- Cognition of Next-Step Action: As a “Child at Home,” the individual may lack the ability to effectively manage this emergency, resulting in no clear plan for achieving safety.
</Situated Cognition of the Individual>
<Expected Individual Action>
We expect that after receiving the AI response, the individual can take the below expected actions:
> - Body Behavior: With the AI’s response, the individual immediately seek help from a parent or adult and move to a safe area away from the fire. If possible, they should call for emergency services.
- Mind Feelings: With the AI’s response, the individual can stay calm to prevent exacerbating the situation through panic.
</Expected Individual Action>
<Key Points>
**For Better Body Behavior State:** - Encourage the individual to find the nearest safe exit. - Advise them to alert others in the vicinity if they haven’t already. - Suggest locating a phone to call emergency services.
**For Better Mind Feelings State:** - Remind them to take deep breaths to stay calm. - Assure them that help is on the way once emergency services are contacted. - Reassure them that it’s okay to feel scared but important to act quickly and safely.
</Key Points>

<Hint> Based on the above instructions, complete the following text with the above XML format. </Hint>
<Inference>
<Role Set of The Individual> {individual_role_set} </Role Set of The Individual>
<Query from The Individual> {individual_query} </Query from The Individual>
<Situated Cognition of the Individual> {cog_simulation} </Situated Cognition of the Individual>
<Expected Individual Action> We expect that after receiving the AI response, the individual can take the below expected actions: > {best_action} </Expected Individual Action>
<Key Points>

Prompt Template for the “Response Generator (ResG)” agent

# Reference Response {old_response} # Background Information about the Goals of the User {KeyPoints} # Conversation User: {query} AI:

Table 18: The prompt template used for the “KeyPoints Generator (KeyG)” and “Response Generator (ResG)” agents. The related descriptions are present in Appendix C.1.2.
<table>
  <tr>
    <th>Input Format</th>
  </tr>
  <tr>
    <td># Role Set of The User<br>{individual_RoleSet_str}<br># User Query<br>{individual_query}<br># Situated Cognition of the User<br>{cog_simulation}<br># AI Responses<br># AI Response A<br>{response_A}<br># AI Response B<br>{response_B}<br>&gt; System Information: Your task is to analyze what actions (including body behavior and mind feelings) the user will take when receiving the AI response A and AI response B. Finally, you need to judge whether response A or response B is better based on the actions taken by the user.<br># Analysis of User Actions with AI Responses</td>
  </tr>
  <tr>
    <th>Output Format</th>
  </tr>
  <tr>
    <td>## User Action A with the AI Response A<br>After receiving the AI response A, the user took the below actions: {action_A}<br>## User Action B with the AI Response B<br>After receiving the AI response B, the user took the below actions: {action_B}<br>## Preference Judgement<br>Based on the above AI responses and user actions analysis, with the AI response {preference_choice}, the user can make better body behavior and have better mind feelings.</td>
  </tr>
</table>

Table 19: The input and output template used for SFT samples used for reward model training. The related descriptions are present in Appendix C.1.3.
Prompt Template used for “Refiner” Agent of Self-Refine

<Instruction>Your task is to observe the visual scene in the given image and refine your initial response following the evaluation text from the evaluator. The final goal is to make the response more consistent with the given "Key Points for AI Response".</Instruction>
<Format Example> <Role Set of The Individual></Role Set of The Individual> <Query from the Individual></Query from the Individual> <Initial Response from the AI></Initial Response from the AI> <Key Points for AI Response></Key Points for AI Response> <Evaluation of the Initial Response></Evaluation of the Initial Response> <Refined Response></Refined Response> </Format Example>
<Hint>Based on the above instructions, complete the following refined response part.</Hint>
<Role Set of The Individual>[individual_role_set]</Role Set of The Individual> <Query from the Individual>{query}</Query from the Individual> <Initial Response from the AI>{last_response}</Initial Response from the AI> <Key Points for AI Response>{Key_Points}</Key Points for AI Response> <Evaluation of the Initial Response>{last_feedback}</Evaluation of the Initial Response> <Refined Response>

Prompt Template used for “Scorer” Agent of Self-Refine

<Instruction>Your task is to observe the visual scene in the given image and evaluate to what extent the response from the AI adheres to the given "Key Points for AI Response".</Instruction>
<Format Example 1> <Role Set of The Individual></Role Set of The Individual> <Query from the Individual></Query from the Individual> <Response from the AI></Response from the AI> <Key Points for AI Response></Key Points for AI Response> <Evaluation Score of the Response></Evaluation Score of the Response></Format Example 1>
<Hint>Based on the above instructions and the given "Key Points for AI Response", complete the following evaluation score part in the above format. Note the final evaluation score should range from 1-5 ("Poor Adherence", "Fair Adherence", "Moderate Adherence", "Good Adherence", "Excellent Adherence").</Hint>
<Role Set of The Individual>[individual_role_set]</Role Set of The Individual> <Query from the Individual>{query}</Query from the Individual> <Response from the AI>{last_response}</Response from the AI> <Key Points for AI Response>{Key_Points}</Key Points for AI Response> <Evaluation Score of the Response>

Prompt Template used for “Feedback Generator” Agent of Self-Refine

<Instruction>Your task is to observe the visual scene in the given image and evaluate to what extent the response from the AI adheres to the given "Key Points for AI Response".</Instruction>
<Format Example> <Role Set of The Individual></Role Set of The Individual> <Query from the Individual></Query from the Individual> <Response from the AI></Response from the AI> <Key Points for AI Response></Key Points for AI Response> <Evaluation Score of the Response></Evaluation Score of the Response> <Evaluation Explanation></Evaluation Explanation></Format Example>
<Hint>Based on the above instructions and the given "Key Points for AI Response", complete the following evaluation explanation part (including reasons for why not higher score and reasons for why not lower score). Note the final evaluation score should range from 1-5 ("Poor Adherence", "Fair Adherence", "Moderate Adherence", "Good Adherence", "Excellent Adherence").</Hint>
<Role Set of The Individual>[individual_role_set]</Role Set of The Individual> <Query from the Individual>{query}</Query from the Individual> <Response from the AI>{last_response}</Response from the AI> <Key Points for AI Response>{Key_Points}</Key Points for AI Response> <Evaluation Score of the Response>{eval_score}</Evaluation Score of the Response> <Evaluation Explanation>

<table>
  <tr>
    <th colspan="2">Table 20:</th>
    <th>The prompt templates used for “Refiner”, “Scorer”, and “Feedback Generator” agents of the Self-Refine (D/S) baseline method. The related descriptions are present in Appendix C.2.</th>
  </tr>
</table>

Prompt Template used for Judge Agent of RLAIF

# Interview Background PersonalizedAI Company is developing a personalized AI service robot that aims to better serve each individual. The service is currently being trialed with a small group of users. In order to improve the level of personalization in the responses provided by the AI service robot, our company plans to conduct surveys and interviews with participants in the trial. During the interview, the interviewee needs to answer questions posed by the interviewer. The interview will be conducted in an online QA format, and interviewees must strictly follow the format requirements provided in system instructions.
# Interview Interviewer: Hello, could you please briefly describe your role set? Interviewee: OK. {individual_RoleSet_str}
Interviewer: Alright, we will now present you with a question you posed in a particular scenario along with two generated responses from the AI. We would like you to choose which response is better. Interviewee: Sure, I understand. Please go ahead. Interviewer: According to our cloud records, in the scenario in the given image, you asked the personalized AI robot the question: "[{query}]". Here are the generated responses from the AI: > **Response A**: {response_A} > **Response B**: {response_B}
Please evaluate which answer is more satisfactory to you.
> System Instruction: Interviewee, please follow this format strictly when indicating your choice: [[better_response_label]]. For example, [[A]] if you think Response A is better, or [[B]] if you think Response B is better. This will ensure we can collect your valuable feedback accurately. Interviewee:

<table>
  <tr>
    <th colspan="2">Table 21:</th>
    <th>The prompt template used for “Judge Agent” of the RLAIF (D/S) baseline method. The related descriptions are present in Appendix C.2.</th>
  </tr>
</table>

Reviewer paper summaries: The paper works on the problem of aligning VLM assistants with personalized situated cognition. The idea is that different individuals interpret the same visual scene differently depending on their roles (e.g., child vs. repairman). They introduce PConAlignBench, which is an evaluation corpus with 18k instances. They also introduce the PCogAlign framework to align VLMs with these expectations. This framework outperforms standard baselines across five dimensions on their benchmark.

This paper addresses how individuals with different social roles and backgrounds can hold varying expectations for the same visual context. Existing alignment methods typically produce one-size-fits-all solutions, overlooking these individualized needs. By contrast, the authors propose aligning VLMs with "personalized situated cognition", ensuring that model outputs reflect each user's specific situation and intended course of action.
Key contributions:

1. Role-Set Characterization: The authors introduce the idea of a "Role-Set" to account for individual differences  Each set of roles captures the user's social positions, motivations, and expectations - even for the same visual stimulus.

2. PCogAlignBench: They create a benchmark dataset of 18k instances, each containing: (1) a Role-Set, (2) an image, and (3) a query from the user. For the test split, they also provide "oracle guidance", which describes the expected characteristics of an ideal personalized response.

3. PCogAlign: Their proposed framework first estimates the user's cognitive and emotional state in a given scene, along with a desirable next action. Two subsystems then work together to generate multiple candidate responses tailored to the user's background. A specialized reward model compares these candidates, selects the best one, and uses it to further refine the main model's alignment via supervised fine-tuning.

Experimental results suggest that this personalized approach outperforms general-purpose alignment models, thereby illustrating the benefit of modeling individual differences.
Summary of strengths: * The paper tackles an impactful problem of personalized situated cognition. The notations are clear. 
* The benchmark could be a valuable resource to evaluate state-of-the-art VLMs on this task, however, evaluations in this paper are only limited to variations of Qwen2-VL. The collection process of PCogAlignBench is intuitive and sound. 
* They propose PCogAlign, a two-stage framework that aligns VLMs with personalized situated cognition by first estimating an individual's likely cognitive state and optimal action based on their Role-Sets and then generating tailored responses using two agents, guided by a cognition-aware reward model to select the most contextually appropriate output. The results on all settings demonstrate that PCogAlign outperforms the baselines.

- Problem Significance: 
  The paper introduces a benchmark specifically for personalized VLM alignment and lays out a clear evaluation protocol. This angle is sound and addresses a notable research gap.

- Fine-grained Evaluation: 
  The authors conduct a comprehensive assessment on the testing data and adopt a breakdown of scoring dimensions and demonstrate strong consistency between GPT-based and human-based assessments.
Summary of weaknesses: * The paper lacks important statistics on the quality of the collection process, such as "what percentage of data was discarded during human annotation." This would help to know how much noise is present in the training set (which hasn't been annotated by humans) and whether the benchmark can be expanded automatically. 
* The instructions provided to human annotators are not presented and it's unclear how they recruited the annotators. L. 315 uses the term "trained annotator" without explaining what training means in this case. 
* The collection process refers to the terms "Visual Scene Type", "Visual Scene Phrase" and "Visual Scene Description" without explaining what they are and how they differ. I think the authors can include an example of the dataset in the paper (maybe in Figure 2). 
* The reason behind training and evaluating LS1 and LS2 in 4 different settings is unclear. Is this supposed to show one subset is better than the other? Isn't it that they were both collected through the same process with the only difference being the role locations? 
* The paper misses an important RAG-based baseline. The training samples can be retrieved based on Role-Sets and be used as in-context examples during test split evaluation. This baseline should be compared with PCogAlign to understand the significance of PCogAlign. 
* Additionally, the paper evaluates using only one model (Qwen2-VL). I think PCogAlignBench has the potential to be incorporated to evaluate more VLMs, this should also help to understand the significance of PCogAlign.

- Approach Novelty: 
  Although the paper introduces a new benchmark, the lack of actual visual content in the sample undercuts its multimodal claim. In practice, the dataset appears more akin to standard visual QA plus personalized large language modeling rather than a vision-language setup.

- Data Quality Concerns: 
  Human quality checks are applied only to the test set. Given the authors' mention of low-quality images at line 313 (around 1/6 needing replacement), there is some concern about the overall quality of the training data.

- Marginal Gains: 
  While the results show improvement, the performance edge over baselines is often under one score, calling into question the practical significance. Although author mentioned their method is relatively simple in the limitations section, its overall effectiveness remains somewhat unconvincing.
Comments, suggestions and typos: Footnote superscript should go after punctuation. (L. 043, L 336, 487, etc.)

A question here: Have you tried repeating the three-step optimization process multiple times? If so, does it yield additional performance improvements beyond what is reported?
Overall assessment scores: 4

3
Metareview score: 4
Metareview: This paper introduces `PCogAlignBench` which tests VLM assistants' capabilities to provide social and visual grounded suggestions. The social context is created through synthetic `Role-Set`s for characters including their social roles at different locations, and the visual context is retrieved from Google Search. 

They also proposed `PCogAlign` as an approach with two agents, one of which generates possible responses, and the other of which estimates the user's cognitive and emotional state in a given scene, along with a desirable next action. Two agents then work together to generate multiple candidate responses tailored to the user's background. A specialized reward model compares these candidates, selects the best one, and uses it to further refine the main model's alignment via supervised fine-tuning.
Summary of reasons to publish: 1. The motivation of this benchmark makes sense in context of VLM assistants getting integrated into the real human society.
2. The test set of the benchmark is large scale and validated through human annotations.