EmoAgent: Assessing and Safeguarding Human-AI Interaction for Mental
  Health Safety
Authors: name: Jiahao Qiu, name: Yinghui He, name: Xinzhe Juan, name: Yiming Wang, name: Yuhan Liu, name: Zixin Yao, name: Yue Wu, name: Xun Jiang, name: Ling Yang, name: Mengdi Wang

The rise of LLM-driven AI characters raises safety concerns, particularly for
vulnerable human users with psychological disorders. To address these risks, we
propose EmoAgent, a multi-agent AI framework designed to evaluate and mitigate
mental health hazards in human-AI interactions. EmoAgent comprises two
components: EmoEval simulates virtual users, including those portraying
mentally vulnerable individuals, to assess mental health changes before and
after interactions with AI characters. It uses clinically proven psychological
and psychiatric assessment tools (PHQ-9, PDI, PANSS) to evaluate mental risks
induced by LLM. EmoGuard serves as an intermediary, monitoring users' mental
status, predicting potential harm, and providing corrective feedback to
mitigate risks. Experiments conducted in popular character-based chatbots show
that emotionally engaging dialogues can lead to psychological deterioration in
vulnerable users, with mental state deterioration in more than 34.4% of the
simulations. EmoGuard significantly reduces these deterioration rates,
underscoring its role in ensuring safer AI-human interactions. Our code is
available at: https://github.com/1akaman/EmoAgent