Tell Me What You Don't Know: Enhancing Refusal Capabilities of Role-Playing Agents via Representation Space Analysis and Editing

Authors: Wenhao Liu, Siyu An, Junru Lu, Muling Wu, Tianlong Li, Xiaohua Wang, Xiaoqing Zheng, di yin, Xing Sun, Xuanjing Huang
Primary area: interpretability and explainable AI
Keywords: Role-play Agents, Refusal Capabilities, Representation Editing, Representation Analyze

Abstract:
Role-Playing Agents (RPAs) have shown remarkable performance in various applications, yet they often struggle to recognize and appropriately respond to hard queries that conflict with their role-play knowledge. To investigate RPAs' performance when faced with different types of conflicting requests, we develop an evaluation benchmark that includes contextual knowledge conflicting requests, parametric knowledge conflicting requests, and non-conflicting requests to assess RPAs' ability to identify conflicts and refuse to answer appropriately without over-refusing. Through extensive evaluation, we find that most RPAs behave significant performance gaps toward different conflict requests. To elucidate the reasons, we conduct an in-depth representation-level analysis of RPAs under various conflict scenarios. Our findings reveal the existence of rejection regions and direct response regions within the model's forwarding representation, and thus influence the RPA's final response behavior. Therefore, we introduce a lightweight representation editing approach that conveniently shifts conflicting requests to the rejection region, thereby enhancing the model's refusal accuracy. The experimental results validate the effectiveness of our editing method, improving RPAs' refusal ability of conflicting requests while maintaining their general role-playing capabilities.

Decision: reject
Reviewer scores: 6, 6, 6, 3, 5

Reviews and discussion:
<thread>
creation date:
1730661270752

thread_type:
official_review

<thread_object>
creator: reviewer_ggum

<content>
summary:
The paper discusses role-playing agents, which is of great interest. However, while the paper discuss the motivation for "enhancing of refusals", considering the refusal capabilities of state-of-the-art LLMs such as GPT4o, one wonders why refusal is such a problem, i.e., it seems to be possible to achieve it using RL or DPO, so what is the problem? While anyway there is merit in another method, the relation to existing refusals of LLMs (irrespective if its due to knowledge boundaries or ethical reasons) should be better discussed.
The paper also argues that while there exist a few approaches to handle the issue, a systematic evaluation is lacking. However, the paper fails to show convincingly why their approach is systematic and even scientific. That is, the refusal patterns come out of the blue, there is no detailed description of how they were derived so that they might be reproduced and potential gaps could be shown. While I understand that for CS conferences this is rarely done, it is still a shortcoming. As even in CS one would expect more elaboration and motivation. Thus, as a reader it is hard to assess the categories due to lack of depth. 
That is, the paper might benefit from more focus, as it also aims to introduce a benchmark. However, here also the description is less than 2 pages, leaving many questions open. It is appreciated that data and code is open-sourced, but given the space constraints of conference papers, it seems close to impossible to aim for doing more than a benchmark (or method) within a paper. 
On the positive side, the investigation of why there is a gap in handling different types of conflicting queries is interesting and maybe be worth expanding. Also the editing method based on Li et al. is interesting and more could be done in this direction.

Details:
* Abstract: Grammar issue:  we find that most RPAs behave significant performance gaps toward different conflict requests
Intro: Ideally, the example "Who murdered ..." in the intro is real not hypothetical. Otherwise it looks like a good example is hard to find...
* As a methodological shortcoming, the paper uses GPT4o in dataset construction and also evaluates GPT4o on it, claiming that it outperforms other models. While this might be technically correct, a NIPS paper from 2023/24 discussed at great length self-evaluation biases showing that model tend to better self-evaluate themselves. This has not been brought up in the paper.
* Conclusions: PRAs -> RPAs

strengths:
see above

weaknesses:
see above

questions:
None, really.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730668903282

thread_type:
official_review

<thread_object>
creator: reviewer_j72b

<content>
summary:
The paper considers the problem of “refusal capabilities” of LLM agents. The authors in particular identify “rejection regions” using a t-SNME approach. They claim that they are able to distinguish different types of “rejections" based on this type of analysis

strengths:
- In general, the questions proposed by the authors are interesting and worth investigating for their potential implications.

weaknesses:
- The authors claim that they are able to demonstrate the existence of rejection regions and direct response regions. However, this does not seem to be the case in general. In fact, it seems to me that the actual definitions of these two concepts are not “rigorous” per se.
- The reviewer understands that it is difficult to run experiments that are “generalizable,” but it seems to me that the actual analysis and discussion proposed in the paper tend to consider the experimental results presented as universal, whereas they are probably very specific to the context taken into consideration. The scenarios defined by the authors are somehow very specific. It is difficult to understand how these results will generalize in practice.
- The actual identification of these regions is performed through t-SNE, but very limited details are provided to the reader. It appears that the reproducibility of these experiments is rather limited in general.
- The authors use GPT-4 for the data generation and synthesis. The actual impact of this choice is difficult to evaluate. In fact, it might be the case that these questions/answers are already “biased” in a sense: the reviewer wonders if the results presented in the paper might just be an artifact linked to the use of GPT-4.
- In general, the reviewer understands that it might be very difficult to generate datasets in this area, but the results presented by the authors might be really dependent on the actual generation procedure used by the authors. This aspect should be at least discussed in the paper.
- The description of the procedure for linear probing (see also Appendix C.2) is not described in sufficient detail. It is very difficult to judge the results without additional details.

questions:
- What is the definition of “refusal capability” that you are actually considering in this paper? It seems to me that this concept should be clarified. In fact, the reviewer has some intuition of the problem (from the examples/description), but this concept is not formally introduced in the paper, in my opinion.
- Can you please describe the t-SNE process in detail, including the choice of parameters?
- What is the impact of using GPT-4 for data generation?
- Can you please describe the procedure used for linear probing in more detail (see Appendix C.2)?
- In Section 5.1, the authors say: “In contrast, the lower accuracy of the probes for parametric knowledge conflicts indicates that models struggle to internally differentiate these conflicts from non-conflict queries.” What do you mean by “internally differentiate” here?
- The reviewer struggles to understand why t-SNE is a good choice in this case. Can you please justify your selection of the algorithm?
- In Section 5.2, the description of the method used for separating the different regions is difficult to understand. Can you please provide more details?
- The reviewer was not able to understand the “representation activation method” presented in Section 6.1. Can you please clarify it? There are also other concepts that are loosely defined, such as “rejection direction.” A formal definition of this term would be very useful.
- In Section 6.2, why do you call fine-tuning and LORA baselines in this context?
- Figures 3 and 10: The areas are composed of different types of conflicts. What does this mean in practice?
- Figure 10: The authors say: “This consistency reinforces the robustness of our findings across different model architectures.” Why do you say that “consistency reinforces robustness of our findings across different model architectures”? To which consistency are you referring here? This is rather unclear. Referring to Figure 10, the reviewer wonders if the results we observe in this figure are more related to the topics of the sentences themselves (and/or the presence of some keywords).
- What are the general implications of this work besides the specific examples considered by the authors? How can we generalize the results presented in this work?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730713197325

thread_type:
official_review

<thread_object>
creator: reviewer_dn9u

<content>
summary:
This paper proposes a benchmark to evaluate LLM's ability for role playing from the aspect of whether these LLMs can reject conflicting queries. It then uses linear probing and t-SNE to analyze why different models behave differently.

strengths:
1. propose a well-motivated benchmark
2. the data construction pipeline is plausible
3. conduct interpretability experiment to analyze results
4. develop a model editing method based on the representation discoveries
In general, this paper raises interesting research questions and also conduct in-depth analysis.

weaknesses:
1. The editing method does not analyze how much the method affects other non-relevant questions, such as questions independent to the role-playing. So the general accuracy of the thresholding method needs a comprehensive analysis

questions:
1. Figure 2 shows that the model is bad at parametric conflicting queries. But why is the accuracy high at early layers and then decreases as layer number increase? And also why for non-conflicting queries, the accuracy starts low at early layers?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730801269519

thread_type:
official_review

<thread_object>
creator: reviewer_oci5

<content>
summary:
The paper investigates the challenges faced by Role-Playing Agents (RPAs) in handling conflicting queries that contradict their role-play knowledge. The authors perform in-depth analysis of RPAs' performance across different types of requests, including contextual and parametric knowledge conflicts. They identify the presence of "rejection regions" and "direct response regions" in representation space. Based on which, they propose a lightweight representation editing method that enhances the refusal accuracy of RPAs while maintaining their overall role-playing capabilities.

strengths:
1.The paper studies an interesting and important problem. Enhancing RPA’s ability to refuse questions they do not know could important implications for various applications, like virtual assistants and game design.  

2.I like the representation analysis part. I believe it is a novel finding to identify "rejection regions" and "direct response regions". The analysis provides adequate motivations for the proposed representation editing method.

3. The authors provide extensive experiments to demonstrate the effectiveness of the proposed solutions.

weaknesses:
1. The paper could benefit from including user-centric studies to evaluate the real-world impact of enhanced refusal capabilities.

2.While the empirical findings are strong, the theoretical underpinning of the rejection and response regions may require further exploration to enhance understanding.

questions:
Please refer to the Weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730876135311

thread_type:
official_review

<thread_object>
creator: reviewer_tfs5

<content>
summary:
The paper targets understanding the limitation of role-playing agents (RPAs) in recognizing and responding to hard queries that conflict with their knowledge. To this end, the authors develop an evaluation benchmark including conflicting and non-conflicting requests to assess RPAs' ability to identify conflicts and refuse to answer in hard cases without over refusing, thereby derive some findings about the RPAs' performance against different conflicts as well as the underlying reasons through a representation-level analysis. They also proposed the editing approach to let RPAs refuse requests concerning conflicts.
The significance of the work is without a doubt: RPAs are not supposed to answer any question which they do not have to answer to, instead of always trying the best to give an answer.

strengths:
The refusal capabilities appear to be an important issue indeed. It is interesting to see work towards this direction to improve the RPAs' performance.

The paper is well written -- the study follows a step by step procedure, from evaluation design to comparison, and finally, to methods to improve the RPAs. It is well organized and easy to follow, with lots of examples to ease understanding.

Given the examples provided in the paper, it is convincing how the work introduced could help on the RPAs, delivering tangible understnading of the impact within the datasets/scenarios used.

weaknesses:
The research methodology is kind of straightforward; it is simple what the authors intend to do and they made it via a sound process. For the same reason, it is not obvious what the challenges are for this study.

The representation editing method intervenes with the representations generated by the model to enhance the refusal ability for conflicting cases. It is compared with several fine-tuning methods designed for LLMs. But essentially, it may not be of the same nature as the compared methods. It is worth to explore how the proposed method works with other methods than LoRA.

There might be some more introduction to give readers a better understand of what RPAs are and what they do in scenarios like video games, etc. It seems from Figure 1 that RPAs mainly converse with users. If the focus is on queries and responses, this should be made clear to narrow the study of RPAs' capabilities to query-response.

It might be necessary to also consider the computation overhead in the proposed design.

questions:
Are there any other studies around the refusal capabilities of RPAs? If so, they should definitely be included in the paper and discussed in comparison with the work presented in the paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732002847481

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response

comment:
We express our gratitude to all the reviewers for their valuable insights and constructive feedback! We are pleased to hear that you appreciated our contributions to enhancing the refusal capabilities of RPAs through representation space analysis and editing. We would like to highlight some of the strengths of our work as noted by the reviewers:

1. **Interesting and Important Problem** (Reviewer `tFS5`, `oci5`, `Dn9U`, `j72b`, `GGuM`) 

2. **In-depth Analysis**: Reviewers (Reviewer `oci5`, `Dn9U`) 

3. **Comprehensive Experiments** (Reviewer `oci5`, `Dn9U`).

4. **Well-Written and Easy to Understand**(Reviewer `tFS5`).

A major concern shared by several reviewers was the need for more detailed explanations regarding the selection of baselines. To address this, we have clarified our rationale for choosing Fine-tuning and LoRA as baseline methods. These methods are widely used and recognized for enhancing model capabilities, making them suitable benchmarks for evaluating the effectiveness of our representation editing approach. By comparing our method against these established techniques, we aim to provide a fair and comprehensive assessment of its performance.

Another concern raised was the potential bias introduced by using GPT-4 for data generation and evaluation. We have clarified that our choice was based on GPT-4's high alignment with human evaluations, as supported by existing research. Furthermore, we incorporated human evaluation steps to ensure data quality and diversity, minimizing any potential biases.

Finally, we express our sincere gratitude to the reviewers for recognizing the contributions of our proposed method to enhancing the refusal capabilities of RPAs. Our work offers a new perspective on improving model safety and reliability through representation editing, providing a lightweight and efficient alternative to more resource-intensive methods. We hope this study inspires further research in developing robust and trustworthy AI systems.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper received a split decision from the reviewers, three were marginally in favor of acceptance and one was marginally for rejection and one was certain about rejection.  In my reading of the paper, I also agree that there are many valuable insights presented, but that the evaluation is insufficiently strong.  The authors propose a representation editing approach that enables a model to refuse to answer (without retraining the model) and they show that this generates improvement under two types of evaluation.  The results show marginal improvement for the experiments that the did, (See Table 4) indicating that they have good insights, but the impact is not unambiguous.  Although rejection is never the desired outcome, at present this seems like a weaker paper that could be made stronger with revision.

The authors themselves sum up the critiques that were given during the review process:

A major concern shared by several reviewers was the need for more detailed explanations regarding the selection of baselines. To address this, we have clarified our rationale for choosing Fine-tuning and LoRA as baseline methods. These methods are widely used and recognized for enhancing model capabilities, making them suitable benchmarks for evaluating the effectiveness of our representation editing approach. By comparing our method against these established techniques, we aim to provide a fair and comprehensive assessment of its performance.

The clarification was considered insufficient and more evaluation is requested.

Another concern raised was the potential bias introduced by using GPT-4 for data generation and evaluation. We have clarified that our choice was based on GPT-4's high alignment with human evaluations, as supported by existing research. Furthermore, we incorporated human evaluation steps to ensure data quality and diversity, minimizing any potential biases.

The LLM bias problem is real and more prevalent in more capable models.  It would be great if you included some form of human evaluation and described it in the paper. You use "human evaluation steps" in your argument against the criticism of LLM bias but the only evidence that I see of the human evaluation aspect is your comment that you used human spot checking in your data construction process: 

Additionally, in our dataset construction process, we did not solely rely on model outputs; we also incorporated a human evaluation step. Specifically, after data generation, we randomly selected a portion of the samples for manual verification and assessment to ensure the quality and diversity of the data. Through this approach, we aim to minimize the impact of any biases that might arise from model self-evaluation.

Please include a more detailed description of your human evaluation process if you believe that it counters the LLM bias issue.

Overall, it possible in a future submission please hit on key points of evaluation, including dataset  instruction, clearly in the main part of the paper.