Towards a Formal Characterization of User Simulation Objectives in Conversational Information Access

Venue: ACM_org_SIGIR_ICTIR_2024_Conference
Published: 2024-04-26 11:59:32

Abstract: User simulation is a promising approach for automatically training and evaluating conversational information access agents, enabling the generation of synthetic dialogues and facilitating reproducible experiments at scale. However, the objectives of user simulation for the different uses remain loosely defined, hindering the development of effective simulators.
In this work, we formally characterize the distinct objectives for user simulators: training aims to maximize behavioral similarity to real users, while evaluation focuses on the accurate prediction of real-world conversational agent performance.
Through an empirical study, we demonstrate that optimizing for one objective does not necessarily lead to improved performance on the other. This finding underscores the need for tailored design considerations depending on the intended use of the simulator. 
By establishing clear objectives and proposing concrete measures to evaluate user simulators against those objectives, we pave the way for the development of simulators that are specifically tailored to their intended use, ultimately leading to more effective conversational agents.

Main paper: 
 user simulation for training: 
 The objective of training a CIA agent is to learn a dialogue policy that selects the best action to take given the current dialogue state [30] . Several approaches have been proposed to tackle this problem including deep learning using human-human [36] , synthetic, or hybrid corpora [13] as well as reinforcement learning (RL) [17] . User simulators can be used to generate a synthetic corpus for offline training, or mimic a human a dialogue participant in a reinforcement learning setting. In the first case, user simulation is a fast and cheap way to generate large amounts of data and to help overcome the problem of data sparsity. Indeed, the collection of dialogues involving humans is cumbersome and expensive. The Simulated Agenda Dataset [23] is an example of a synthetic corpus built with the help of user simulators. While the creation of synthetic corpora is an interesting question, it is not the focus of this work. In the case of RL, the decision-making process of selecting the next action is commonly modeled as a Markov Decision Process (MDP) [17] . In this context, user simulation allows for dynamic interactions with the conversational agent. Furthermore, it allows for an exhaustive exploration of the state-action space, which is otherwise limited when using a fixed corpus [30] . Schatzmann et al. [29] train a policy using a user simulator that does not require training data. Conversely, Lin et al. [20] perform a comparison of dialogue policies trained with the PPO algorithm [31] using different user simulators. The approach of training a user simulator first and then using it in a RL setting is often used in the field [17] .The evaluation of CIA agents is an open challenge and is thus an active area of research [3, 38] . None of the existing evaluation methodologies (offline, online, and user studies) enables the comparison of multiple CIA systems using reproducible experiments. Offline test collections are static in nature and fail to capture the interactive nature of conversations, while there is an inherent lack of reproducibility, coupled with challenges of performing evaluation at scale, when real users are involved. User simulation is presented as a promising solution to complete existing evaluation methodologies [1] . Indeed, with a user simulator it is possible to quickly and inexpensively perform evaluation based on a large number of dialogues. Moreover, simulated users can be controlled, thereby enabling reproducible experiments. In the literature, user simulation has been used to evaluate CIA agents at different granularity levels. For example, Zhang and Balog [39] performed an evaluation at the conversation level, while others [24, 28, 32] evaluated the mixed-initiative abilities of CIA agents at the turn level.Evaluation metrics may be divided into two categories: singleturn and multi-turn metrics (also referred as end-to-end metrics) [26] . One major limitation of single-turn metrics is that they do not consider the dialogue structure, which is an important aspect of CIA and can impact the overall user experience [38] . However, they are useful when a specific feature of a CIA agent is in focus. For example, Sekulić et al. [32] used BLEU [25] and ROUGE [18] to automatically assess the quality of their simulator's generated utterance; these particular metrics have limitations when applied to dialogues, yet they are widely used due to the lack of better alternatives [22] . In the case of conversational search, previous work [24, 28] have used information retrieval metrics such as normalized discounted cumulative gain and mean reciprocal rank to evaluate the relevance of retrieved items. Conversely, multi-turn metrics take into consideration multiple dialogue turns or entire dialogues. These metrics commonly evaluate the conversational agent's ability to achieve a given goal. Examples include the broadly used success rate and the expected conversation satisfaction metrics [21] . While they are insightful, they do not consider the quality of the dialogue in terms of structure, naturalness, and coherence. Out of these aspects, naturalness has attracted the most attention and metrics such as D-BLEU [14] and SUPER [27] were proposed to assess it.Even though there is a large body of work leveraging user simulation for training and evaluation of CIA agents, the work by Zhang and Balog [39] is the only one, to the best of our knowledge, that mathematically formalizes the requirements for a user simulator for the purpose of system evaluation. We believe that such formalization is important to identify the requirements and desiderata for user simulation in CIA and provide a solution to verify if they are met. Balog and Zhai [3] also argue that interpretability of user simulators is an important requirement for evaluation as it allows for a comprehensive understanding of the results. On the other hand, interpretability is not critical for training. Additional work have discussed requirements and desiderata for user simulation [1, 5, 24] , however, these are articulated in qualitative terms, lacking quantifiable measures for assessing their fulfillment. 
problem statement: 
 We formally define the two main uses of user simulation in conversational information access: training and evaluation. This formalization aims to make the objectives for each use of user simulation explicit, thereby providing a framework that (i) enables a comparative analysis of different uses of simulation with regards to objectives and (ii) guides the design of appropriate measures for evaluating simulators. We emphasize the importance of studying user simulation not in isolation, but in relation to how it interacts with the conversational agent within the specific use context. In this section, we start in Section 3.1 by defining fundamental concepts. Then, we propose a formalization for training and evaluation in Sections 3.2 and 3.3, respectively.We define key concepts used in the remainder of this work. For ease of reference, their notation and description are included in Table 1 .We consider a task-based dialogue setting, where a user 𝑢 interacts with a conversational agent 𝐶𝐴 for the purpose of completing Measure of the completion of goal 𝐺 in dialogue 𝑑 𝑀 (𝐶𝐴, 𝑈 ) Measure of the performance of a conversational agent 𝐶𝐴 when used by a set of users 𝑈 sim Similarity function some goal 𝐺. For example, the goal may be to learn about astrophysics or to find a new laptop under $700. We assume that the completion of the goal may be measured by some metric 𝑚 to assess the performance of 𝐶𝐴. In the course of the dialogue, participants take turns 𝑡 in issuing utterances𝑥: 𝑑 = [𝑥 𝐶𝐴 0 , 𝑥 𝑢 0 , 𝑥 𝐶𝐴 1 , 𝑥 𝑢 1 , . . . , 𝑥 𝐶𝐴 𝑛 , 𝑥 𝑢 𝑛 ].With user simulation, the aim is to create a simulated user 𝑢 * that can substitute the real user 𝑢 in the dialogue with 𝐶𝐴. In practice, we are typically interested in a set of users 𝑈 that are representative of the expected user base of the system, rather than relying on a single individual. Thus, we let 𝑈 * denote a user simulator that is meant to mimic the behavior of the user population 𝑈 . We assume that each user is characterized by a set of properties, such as personality traits and preferences. (Some properties might be defined on the population level, while others are specific to each individual.) The user simulator 𝑈 * then serves as model predicting the actions of a particular user 𝑢 * with specific characteristics in a given context.Training a conversational agent consists in learning a dialogue policy that dictates how it should respond in a given dialogue situation. Without loss of generality, we can define the policy as a mapping from a set of possible states S to a set of possible actions A, 𝜋 𝐶𝐴 : S → A. The state 𝑠 ∈ S may take different forms, such as an explicit knowledge structure, where the user goal is represented as a set of slots and values (as in frame-based dialogue systems [15] ), or as a latent representation in an embedding space (see, e.g., [11, 19] ). Actions 𝑎 ∈ A represent the intent behind the response, which is ultimately given as a natural language utterance 𝑥. In traditional state-based dialogue architectures, there is a separate natural language generation component that creates the utterance from the action [11, 15, 39] . More end-to-end simulator architectures might generate utterances jointly with the underlying actions [19] , or generate only the natural language utterance [8, 16] . In the latter case, we assume that the action can be inferred from the utterance. The mapping defined by the policy may be deterministic, corresponding to a rule-based system, or stochastic and adaptive, which is typical for probabilistic policies that may be defined based on empirical observations or be automatically learned. We focus on this latter category of learned policies, which allows for more flexible and context-sensitive responses [6] . Specifically, reinforcement learning is often employed as a learning paradigm, where the conversational agent continuously refines its policy through trial-anderror based on feedback from user interactions; see Fig. 1 . The task is expressed as an optimization problem, based on the notion of reward. The reward function 𝑅 assigns a numerical value to transitions from one state to another due to an action: 𝑅 : S × A ×S → R. The goal then is to find an optimal policy 𝜋 * 𝐶𝐴 that maximizes the expected cumulative reward:EQUATIONwhere 𝛾 is a discount factor (0 ≤ 𝛾 < 1) that determines the present value of future rewards. Notice that the formulation in Eq. ( 1) does not explicitly consider the user simulator as part of the learning process. Since the conversational agent and the user interact to achieve a shared goal through a series of utterances, the actions taken by the user play a pivotal role in determining the subsequent state of the conversational agent (𝑠 𝑡 +1 ). Consequently, the reward function may be redefined such that it considers the action space of the conversational agent (A 𝐶𝐴 ) as well as the action space of the user (A 𝑈 ), when making state transitions 𝑅 : S × A 𝐶𝐴 × A 𝑈 × S → R. The objective for the optimal policy may then be written as:EQUATION)With the role of the user policy 𝜋 𝑈 made explicit in Eq. ( 2), we now turn to the examination of objectives for effectively simulating the policy of real users.In order to facilitate the learning of the dialogue policy of a conversational agent, simulated users should act the same way as real users would act in a given dialogue situation. Formally, the similarity of the simulated user policy 𝜋 𝑈 * and the real user policy 𝜋 𝑈 need to be maximized with regards to some similarity function:EQUATIONGiven the above, a relative ranking of user simulators for the training objective may be established based on their similarity to the real user policy:sim 𝜋 (𝜋 𝑈 * 1 , 𝜋 𝑈 ) > sim 𝜋 (𝜋 𝑈 * 2 , 𝜋 𝑈 ) =⇒ 𝑈 * 1 ≻ 𝑈 * 2 .A significant challenge with this formulation lies in the fact that 𝜋 𝑈 is not directly observable. This is because we lack direct insight into the thoughts of real users during their interactions with the conversational agent. Hence, a proxy is needed for estimating the similarity between 𝜋 𝑈 and 𝜋 𝑈 * .As 𝜋 defines the conversational strategy, it can be indirectly observed through the generated dialogues. We thus consider historical dialogues with real users (𝐷 𝑈 ) and with simulated users (𝐷 𝑈 * ) as proxies of the respective policies:EQUATIONwhere sim 𝐷 is a similarity function for comparing dialogues. Representing dialogues as a sequence of actions(𝑑 = [𝑎 𝐶𝐴 0 , 𝑎 𝑢 0 , 𝑎 𝐶𝐴 1 , 𝑎 𝑢 1 , . . . , 𝑎 𝐶𝐴 𝑛 , 𝑎 𝑢 𝑛 ]), similarity may be defined on the level of entire conversations or on the level of individual turns.Specifically, conversation-level similarity measures could adapt methods from the fields of automatic summarization and machine translation, such as ROUGE-L and METEOR. The idea is that a dialogue may be seen as a sequence of words, where the words are actions. Then, the aforementioned measures can assess the extent with which simulated users exhibit the same behavioral patterns as reals user by comparing the subsequences of actions. Note that the notion of ordering is important and should be taken into consideration by the chosen similarity measure. Commonly, the measures from automatic summarization and translation compare the generated text against a set of reference translations/summaries. However, in our case, we want to compare two sets of dialogues (𝐷 𝑈 and 𝐷 𝑈 * ) which do not have pairwise correspondence. Thus, a possible adaptation is to compute the measure for all possible pairs of dialogues and then aggregate the results:sim 𝐷 (𝐷 𝑈 , 𝐷 𝑈 * ) = 𝑎𝑔𝑔𝑟 (𝑠𝑖𝑚 𝑑 (𝑑, 𝑑 * ) | 𝑑 ∈ 𝐷 𝑈 , 𝑑 * ∈ 𝐷 𝑈 * ) , (5)where 𝑠𝑖𝑚 𝑑 is the similarity measure between two dialogues, e.g., ROUGE or METEOR, and 𝑎𝑔𝑔𝑟 is an aggregation function, e.g., average, median, or maximum. For turn-level similarity measures, we can characterize each system action in terms of the distribution of user actions that follow that action, i.e., 𝑃 𝜋 (𝑎 𝑢 |𝑎 𝐶𝐴 ) where 𝑎 𝑢 ∈ A 𝑈 and 𝑎 𝐶𝐴 ∈ A 𝐶𝐴 . This distribution could correspond to the empirical distribution from dialogues or be estimated based on heuristics and intuition. The distributions 𝑃 𝜋 𝑈 and 𝑃 𝜋 𝑈 * may be compared using similarity measures of probability distributions, such as Jensen-Shannon divergence. Unlike conversation-level measures, the notion of ordering is lost here. Indeed, the distribution of user actions is independent of the conversational context except for the current system action. Note that one can consider using both conversationand turn-level metrics to get a more comprehensive assessment of the similarity between the user policies. Indeed, as they take into account either the global or local context of a dialogue, they capture different aspects of user behavior, such as overall strategy versus response to a specific system action.User simulation allows for a time-and cost-efficient evaluation of conversational agents by reducing the reliance on real users, thereby enabling automatic evaluation at scale [3] . However, the validity of the user simulator is critical to ensure the trustworthiness and reliability of the findings. The objective of evaluation using user simulation has previously been defined by Zhang and Balog [39] as: "an automatic assessment of the agent such that it is predictive of its performance with real users. " In order to verify this, they propose to compare the performance of two conversational agents 𝐶𝐴 1 and 𝐶𝐴 2 when used with 𝑈 and 𝑈 * , and formulate the following condition to be satisfied:𝑀 (𝐶𝐴 1 , 𝑈 ) < 𝑀 (𝐶𝐴 2 , 𝑈 ) =⇒ 𝑀 (𝐶𝐴 1 , 𝑈 * ) < 𝑀 (𝐶𝐴 2 , 𝑈 * ), where 𝑀 (𝐶𝐴, 𝑈 ) represents some measure of performance when 𝐶𝐴 is used by a set of users 𝑈 .A main issue with the above pairwise comparison that it does not take the absolute performance differences into account. As such, it cannot be used to establish a relative ranking of user simulators. For example, let us assume that we obtain the performance measurements for two conversational agents with real users 𝑀 (𝐶𝐴 1 , 𝑈 ) = 0.1 and 𝑀 (𝐶𝐴 2 , 𝑈 ) = 0.2. Let one user simulator 𝑈 * 1 estimate the performances to be 𝑀 (𝐶𝐴 2 is a better and more useful simulator as it produces estimates that are closer to the real performance measures in absolute terms. Based on this observation, we propose an adaption of the requirements from [39] -one that facilitates the comparison between user simulators with regards to the system evaluation objective.In order to automatically evaluate conversational agents, the performance measures obtained from simulated users should closely approximate those obtained from real users, that is:EQUATIONThis notion of near equivalence, however, is challenging to operationalize directly; therefore, we introduce a threshold parameter 𝜀 to define acceptable levels of approximation with regards to a given evaluation metric:EQUATIONDepending on the use case and aim for evaluation, 𝜀 can be set to different values, hence, quantifying the notion of a "good enough" user simulator [2] . The relative ranking of user simulators for the evaluation objective can be established by comparing their performances to that of real users:|𝑀 (𝐶𝐴, 𝑈 )-𝑀 (𝐶𝐴, 𝑈 * 1 )| < |𝑀 (𝐶𝐴, 𝑈 )-𝑀 (𝐶𝐴, 𝑈 * 2 )| =⇒ 𝑈 * 1 ≻ 𝑈 * 2 .Simulation. Now that we have defined the objectives, we focus on how to assess the performance of the conversational agent for a given set of users. Here, we focus on measuring performance with regards to the agent's ability to aid users in completing some goal 𝐺, based on a set of dialogues 𝐷 𝑈 between theAssess dialogue policies similarity:Assess agent's performances: , Compare and rank user simulators agent and users:EQUATIONwhere 𝑚(𝑑, 𝐺) is a metric that assesses if the goal 𝐺 was completed or not. This assessment can be binary, e.g., completed or not, or graded, corresponding to the degree of completion. While this formulation implies that 𝑚 is a conversation-level metric, one can choose to use a turn-level metric 𝑚(𝑡, 𝐺), such as recall in conversational search [32] , if it is better suited for their use case. In that case, Eq. ( 8) should be adapted to iterate over the turns 𝑡 of all the dialogues in 𝐷 𝑈 . Additionally, it is possible to use statistical significance testing to evaluate the objectives with a certain confidence level. Let 𝑃 𝑀 (𝐶𝐴,𝑈 ) and 𝑃 𝑀 (𝐶𝐴,𝑈 * ) be the distribution of performances for real and simulated users, 𝑈 and 𝑈 * , respectively (based on the dialogue-level performances 𝑚(𝑑, 𝐺) that are averaged in Eq. ( 8)). Taking the formulation in Eq. ( 6), we can extract the null hypothesis 𝐻 0 = 𝑃 𝑀 (𝐶𝐴,𝑈 ) ≁ 𝑃 𝑀 (𝐶𝐴,𝑈 * ) and the alternative hypothesis 𝐻 1 = 𝑃 𝑀 (𝐶𝐴,𝑈 ) ≃ 𝑃 𝑀 (𝐶𝐴,𝑈 * ) . Then, a statistical test can be employed to determine whether the requirement is satisfied by either rejecting or not rejecting the null hypothesis 𝐻 0 , with a specified level of confidence. 
analysis: 
 In this section, we study the relationships between the objectives for training and evaluation. More specifically, we are interested in the potential implications between these objectives. Consequently, we seek to answer the following research question: If user simulator A outperforms user simulator B on the training objective, does this imply that A will also outperform B on the evaluation objective, and vice versa? We start in Section 4.1 by describing the methodology applied. Then, we introduce the user simulator used in our experiments (Section 4.2), followed by our experimental setup (Section 4.3). Finally, we present the results and discuss the findings with regards to our research question (Section 4.4).The methodology considers 𝑁 conversational agents and corresponding user populations. For each conversational agent 𝐶𝐴, we perform the following steps (Fig. 2 ):(1) Create various user simulators 𝑈 * 𝑖 to be compared with regards to the training and evaluation objectives.(2) Generate synthetic dialogues (𝐷 𝑈 * 𝑖 ) between the reference conversational agent 𝐶𝐴 and each user simulator 𝑈 * 𝑖 . The dialogues are generated by sampling the actions of each participant according to their dialogue policy. Moreover, they have an associated (binary) success outcome: successful or not.(3) Compute the different metrics associated with the training and evaluation objectives. For training, we consider two metrics to measure the similarity between the reference and simulated user populations at the turn-and conversationlevels. The turn-level similarity is computed with the Jensen-Shannon divergence (JSD) between the policies of the user simulator and the reference user population. The conversationlevel similarity is an aggregation of the ROUGE-L scores between the synthetic dialogues 𝐷 𝑈 * 𝑖 and the reference dialogues 𝐷 𝑈 . For evaluation, we use the success rate to assess the performance of the conversational agent. (4) Establish a relative ordering of user simulators 𝑈 * 𝑖 based on the computed metrics.For our experiments, we employ a user simulator that is characterized by three elements: (1) user model, (2) interaction model, and (3) outcome prediction.User model. The user model defines the characteristics of simulated users with respect to patience and inclination. Patience is associated with the time a user is willing to be engaged to complete the task. Inclination refers to the overall attitude of the user towards goal completion; higher inclination increases the likelihood of a successful dialogue outcome. Interaction model. The interaction model is based on the QRFA (query, request, feedback, and answer) model [35] . The model defines two user actions, i.e., query and feedback, and agent actions, i.e., request and answer; see Fig. 3 . It has the advantage of being simple and generalizable to a wide range of applications. Using this model, we define transition probabilities between the agent and user actions, that is, each agent action conditions the next user action following a probability distribution.Outcome prediction. The outcome is predicted by a logistic regression model 𝑓 that assigns a binary outcome to a dialogue, i.e., successful or not. 1 Our model uses five input features: dialogue length 𝑙, last action from the user 𝑎 𝑡 𝑢 and from the conversational agent 𝑎 𝑡 𝐶𝐴 , patience 𝑝 and inclination 𝑖 of the user. We chose this particular combination of features as they capture the notion of effectiveness of the conversational agent (with 𝑙, 𝑎 𝑡 𝑢 , and 𝑎 𝑡 𝐶𝐴 ), while also considering the user's perspective (with 𝑝 and 𝑖). Formally, we define 𝑓 based on the sigmoid function:EQUATIONwhere ℎ computes a score based on input features. It can be a simple linear model or a more complex model, e.g., a neural network. For this analysis, we define ℎ as a linear model:EQUATIONwhere 𝑤 1 , 𝑤 2 , and 𝑤 3 are feature weights, and 1 is the indicator function. Patience and dialogue length go hand in hand as they are related to the time a user is engaged in the dialogue. Therefore, we combine them in the first term of the model to account for this correlation. Within the QRFA interaction model, the feedback action Impatient and critical -0.9 -0.9 𝑈 * 2 Impatient and positive -0.9 0.9 𝑈 * 3 Patient and critical 0.9 -0.9 𝑈 * 4 Patient and positive 0.9 0.9𝑈 * 5 Neutral 1e-5 1e-5does not distinguish between positive and negative feedback, hence, we assume that 𝑖 can be used to polarize the feedback action. In this analysis, we use a hyperbolic tangent function for the polarization, noting that other functions (e.g., sigmoid) could also be used.For the analysis, we set 𝑤 1 and 𝑤 2 to 1, and 𝑤 3 to 0.5 inspired by the empirical study by Siro et al. [33] . However, we acknowledge that the choice of the weights is somewhat arbitrary and may not be optimal with regards to real-word scenarios.For the experiments, we consider five conversational agents {𝐶𝐴 1 , 𝐶𝐴 2 , . . . , 𝐶𝐴 5 } with corresponding user populations, and five user simulators {𝑈 * 1 , 𝑈 * 2 , . . . , 𝑈 * 5 } parametrized to simulate these user populations. These are associated with five datasets that have each been annotated according to the QRFA interaction model; see Table 2 . These datasets allow us to ground our analysis in real data, hence, integrating the notion of realistic user behavior. Furthermore, we can assume that each dataset reflects different user behavior regarding the completion of an information-seeking task. Indeed, the datasets are from different domains, e.g., restaurant and opendomain search, and have different dialogue lengths indicating that users employ different strategies to complete their tasks.Using the QRFA annotations, we can extract the transition probabilities between the actions for both the user simulator and conversational agent from each dataset. The transition probabilities are used as a proxy for the dialogue policy. Note that some dialogues in the datasets are not annotated with a success label that is required to assess the success rate, hence, the evaluation objective. Then, we characterize each user simulator with regards to patience and inclination (Table 3 ). For the sake of simplicity, we consider that each user within the simulated user population is associated with the same personality. 2 The values for 𝑝 and 𝑖 are bounded in the range [-1, 1] and chosen to reflect extreme cases (e.g., very impatient or very positive), in addition to a neutral case (i.e., 𝑈 5 ). Based on these values and the dialogues, we enrich the datasets with success labels. The labels are assigned with the function 𝑓 defined in Section 4.2.Our experiments are inspired by the leave-one-out cross validation approach: we consider one conversational agent and its corresponding user population as reference, and compare the four user simulators against this reference. (For example, considering 𝐶𝐴 1 as the reference, the user simulators compared are 𝑈 * 2 , 𝑈 * 3 , 𝑈 * 4 , and 𝑈 * 5 .) The number of synthetic dialogues generated for each simulated user population is set to 500 (i.e., second step of the methodology). We repeat this process for all possible reference conversational agents. The results are reported in Table 4 . 3 Our interest lies in the comparison of the user simulators with respect to the training and evaluation objectives in order to determine if optimising for one objective would also lead to improvements on the other objective. From the results presented in Table 4 , we make two main observations. First, focusing on the training objective alone, we see that the turn-and conversation-level evaluation measures (JSD and aggregated ROUGE-L) do not agree on which simulator is ranked first. To quantify the agreement in the rankings, 2 This choice was made in the interest of simplicity. A straightforward extension to simulating variations in personalities within a user population would be to characterize the personality traits, i.e., patience and inclination, as distributions. 3 The code and data to run these experiments are made available at: https://anonymous. 4open.science/r/usersim-objectives-D177/ we compute the Kendall's 𝜏 correlation between JSD and ROUGE-L, see Table 5 , and find that it varies between -0.333 and 0.667, indicating that the metrics are not strongly correlated. This supports the idea that the level of context considered by the metrics provides different insights into the behavior of the simulated user populations. It also highlights that the choice of the similarity metric is an important design decision and also suggests that further research is needed on measures that can integrate local and global behavior.Second, comparing simulators on the training vs. evaluation objectives, we observe several cases where there is a disagreement in the relative ordering for a pair of user simulators. For example, for the reference conversational agent 𝐶𝐴 1 , we find 𝑈 * 2 ≻ 𝑈 * 4 for the training objective (according to both JSD and aggregated ROUGE-L), while for the evaluation object (success rate) it is 𝑈 * 4 ≻ 𝑈 * 2 . In Table 5 , we report the pairwise Kendall's 𝜏 correlation between the metrics for the training and evaluation objectives. Interestingly, we see that Jensen-Shannon divergence and the absolute difference of success rate tend to disagree on the rank of a user simulator (𝜏 is negative in 80% of the cases studied). In case of the ROUGE-L measure, we do not observe a clear trend for 𝜏, suggesting that additional experiments are needed to understand its correlation with the evaluation metric. Overall, our results provide evidence that training and evaluation objectives are not always aligned, hence, designing/selecting a user simulator should consider its use, i.e., training vs. evaluation.We acknowledge that the high-level requirements/desiderata of interpretability and realism have been identified in the literature and are considered important for a widespread adoption of user simulation [2, 3] . Below, we discuss how our proposed objectives related to these. Interpretability is an objective that is observable but is very challenging to mathematically formalize or quantify. Therefore, we argue that it should be considered during the design of the user simulator before assessing the proposed objectives, and especially the evaluation objective. On the other hand, realism is a complex objective that may be decomposed into several lower-level objectives. In this work, we propose the formalization of one of these. Indeed, a user simulator sharing a highly similar behavior with real users is one indicator of realism. However, other objectives should be formalized in future work to ensure a comprehensive assessment, e.g., consistency of actions taken given a context [5] and ability to learn and forget information [1] . This also includes the development of associated methodologies and metrics to evaluate the fulfillment of these objectives. It is possible that some of these lower-level objectives may not be critical for training and/or evaluation, but still improve the overall realism of the simulation.The focus of the current work has been on the objectives of user simulation in the context of conversational information access. However, the general concepts defined here could be adapted to the broader context of interactive information access. Indeed, a session can be represented by replacing the sequence of utterances by a sequence of interactions between an information access system and a user (such as querying, clicking, and liking/bookmarking documents), suggesting a potential generalization to interactive information systems. 
conclusion: 
 In this work, we have presented a formal characterization of the distinct objectives for user simulation for training and evaluation purposes in the context of conversational information access. We have demonstrated empirically that optimization on the training objective-maximizing behavioral similarity to real users-does not necessarily imply improvement on the evaluation objectiveaccurately predicting the performance of the conversational agent in helping users accomplish some task. This highlights the need for distinct design considerations when developing user simulators for training vs. evaluating systems. Whether optimizing for one objective inherently benefits the other remains an open question, motivating further research.A key contribution of this work is the establishment of a formal framework for quantifying user simulator performance on these objectives. This framework enables direct comparison between different simulators and provides a foundation for guiding the development of simulators tailored to specific purposes.We identify two main directions for future work. First, we plan to conduct a more comprehensive empirical study where we evaluate the performance of multiple conversational agents that are trained using different user simulators. Second, we aim to refine the evaluation metrics to provide a more nuanced and comprehensive assessment of user simulator effectiveness. 
: 
 dialogue corpora involving real users is time-consuming and expensive, and the evaluation of interactive systems presents similar challenges due to the necessity of human participation for accurate assessment. User simulation has been proposed as a potential solution to automate the training and evaluation of CIA agents [3, 12] , offering the possibility to generate synthetic dialogues and conduct reproducible experiments at scale, which is notoriously difficult, if not impossible, with real users.User simulation has been informally defined as the process of mimicking the behavior of real users when interacting with a system in order to accomplish some task [3] . This description is applicable to both training and evaluating CIA agents. However, we argue that each of these uses, i.e., training and evaluation, has distinct objectives. Surprisingly, despite its importance and potential impact, there is limited work on the formal characterization of these objectives in the context of conversational information access. A notable exception is the work by Zhang and Balog [39] , who proposed a high-level formalization of requirements for evaluation, based on pairwise comparison of system performance when used with a simulator. However, this approach has limitations, as it does not account for absolute differences in evaluation scores, potentially masking substantial variations in system performance. For training, we are not aware of any work on formalizing objectives from a user simulation perspective, despite the widespread use of simulators.The current paper aims to fill this gap by formally characterizing the distinct objectives for different uses of user simulation in conversational information access, and by proposing corresponding metrics for measuring the extent to which these objectives are met.For training, we posit that the objective is to closely mirror the behavior of real users. This can be expressed in terms of the similarity of dialogue policies, which determine the next user action given the current context. In contrast, for evaluation, the objective shifts to accurately estimating the performance of real users when interacting with a CIA agent in order to accomplish some task. Given the apparent alignment of these objectives, it is natural to ask the question whether optimizing for one objective would inherently lead to improvements on the other objective. Our initial empirical study reveals that this is not necessarily the case. We find several instances where simulator A outperforms simulator B on the training objective, but it is the other way around for the evaluation objective. This suggest the need for distinct design considerations when developing user simulators for training versus evaluating systems.In summary, the main contributions of this work are twofold (1) a formalization of the problems related to the two uses of simulation in CIA, training and evaluation, identifying their distinct objectives and measures for evaluating simulators against those objectives, and (2) an empirical study showing that the two objectives are not necessarily aligned. This work thus lays the foundation for the development of more effective, purpose-built user simulators, ultimately leading to more user-centric conversational agents.Introduced in the fields of information retrieval in the 1970s [7] and dialogue systems a few decades later [10] , user simulation approaches have recently gained renewed attention due to their potential for training and evaluating conversational information access (CIA) agents [1, 2] . 
 
 Appendix: We note that other models, such as support vector machine or Bayesian network, could also be used instead. 


Decision: All reviewers agree that this paper offers a significant contribution to the analysis of user simulators in information accessing. The paper is well written and easy to follow, and the presentation of systematic results that are well-supported by the empirical experiments.

Though some reviewers have pointed out that the paper could be improved if more details are provided on models and applications, the overall merits of this paper clearly overwhelm its weakness. We recommend an accept and suggest the authors to revise the paper based on the review comments before submitting the camera-ready version.

Reviews: review: main_review: main_review: The paper offers a straightforward reading experience, presenting complex concepts in a clear and concise manner. The chosen chapter layout enhances readability, while the formal notations in use are adequately explained, making it easy for readers to follow the arguments of this study. The problem addressed is relevant, addressing an important subject for the growing trend of user simulation. Moreover, while some components of the elaborated methods draw from established methodologies or metrics, these are properly integrated in the current method. The results presented are interesting, shedding light on the complexities of user simulation objectives and their implications for system development. The observation that optimization for one objective does not necessarily lead to improvement on the other objective provides valuable insights for researchers and practitioners alike, prompting further exploration. As such, it is highly likely that others in the ICTIR community will find this work compelling. Only minor issues were identified. Firstly, the 2.1 subsection could be rewritten in a more concise way (e.g. The section discussing the "Simulated Agenda Dataset" lacks integration and coherence within the broader context. Improved organization and a clearer articulation of ideas through paragraph structuring would enhance readability and narrative cohesion. Refinement is necessary to ensure a seamless flow of concepts and topics.). Besides that, it would also be useful to provide more specific details in Chapter 4, regarding the interaction QRFA model in use, to facilitate the understanding of the reader.
main_review: main_review: Unfortunately, the article is far removed from my research interests. This certainly supports my scepticism about the real usefulness of such approaches. Nevertheless, I do not feel able to provide a critical review and will support what more experienced reviewers than me will provide as suggestions.
main_review: main_review: This paper provides nice formalizations for its methods. With respect to the empirical results, some of the metrics are grounded to specific characteristics (e.g., patience, inclination), but the paper provides a good starting point into further studies for discrepancies between user simulations objectives at train vs test time.

A few questions/comments:
- The number of dialogues used for each of the five user simulators vary drastically. Was the entire dataset used for each simulator?