Papers
arxiv:2311.06772

ChatAnything: Facetime Chat with LLM-Enhanced Personas

Published on Nov 12, 2023
· Submitted by akhaliq on Nov 14, 2023
#2 Paper of the day
Authors:
,
,
,

Abstract

In this technical report, we target generating anthropomorphized personas for LLM-based characters in an online manner, including visual appearance, personality and tones, with only text descriptions. To achieve this, we first leverage the in-context learning capability of LLMs for personality generation by carefully designing a set of system prompts. We then propose two novel concepts: the mixture of voices (MoV) and the mixture of diffusers (MoD) for diverse voice and appearance generation. For MoV, we utilize the text-to-speech (TTS) algorithms with a variety of pre-defined tones and select the most matching one based on the user-provided text description automatically. For MoD, we combine the recent popular text-to-image generation techniques and talking head algorithms to streamline the process of generating talking objects. We termed the whole framework as ChatAnything. With it, users could be able to animate anything with any personas that are anthropomorphic using just a few text inputs. However, we have observed that the anthropomorphic objects produced by current generative models are often undetectable by pre-trained face landmark detectors, leading to failure of the face motion generation, even if these faces possess human-like appearances because those images are nearly seen during the training (e.g., OOD samples). To address this issue, we incorporate pixel-level guidance to infuse human face landmarks during the image generation phase. To benchmark these metrics, we have built an evaluation dataset. Based on it, we verify that the detection rate of the face landmark is significantly increased from 57.0% to 92.5% thus allowing automatic face animation based on generated speech content. The code and more results can be found at https://chatanything.github.io/.

Community

家人们,谁懂啊

今天遇到个Down head man

![canvas-oil-Wolfgang-Amadeus-Mozart-Barbara-Krafft-1819.jpg]({"error":"Only GIF, JPEG, JPG, MOV, MP3, MP4, MPGA, PNG, QT, WAV, WEBM, WEBP files are supported"})

image (2).png

Sign up or log in to comment

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2311.06772 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2311.06772 in a dataset README.md to link it from this page.

Spaces citing this paper 2

Collections including this paper 10