Bot or Human? Detecting ChatGPT Imposters with A Single Question
Authors: name: Hong Wang; name: Xuan Luo; name: Weizhi Wang; name: Xifeng Yan
Published: 2023-05-10 19:09:24

Abstract: Large language models like ChatGPT have recently demonstrated impressive
capabilities in natural language understanding and generation, enabling various
applications including translation, essay writing, and chit-chatting. However,
there is a concern that they can be misused for malicious purposes, such as
fraud or denial-of-service attacks. Therefore, it is crucial to develop methods
for detecting whether the party involved in a conversation is a bot or a human.
In this paper, we propose a framework named FLAIR, Finding Large language model
Authenticity via a single Inquiry and Response, to detect conversational bots
in an online manner. Specifically, we target a single question scenario that
can effectively differentiate human users from bots. The questions are divided
into two categories: those that are easy for humans but difficult for bots
(e.g., counting, substitution, positioning, noise filtering, and ASCII art),
and those that are easy for bots but difficult for humans (e.g., memorization
and computation). Our approach shows different strengths of these questions in
their effectiveness, providing a new way for online service providers to
protect themselves against nefarious activities and ensure that they are
serving real users. We open-sourced our dataset on
https://github.com/hongwang600/FLAIR and welcome contributions from the
community to enrich such detection datasets.