Package: aiavatar
AIAvatarKit🥰 Building AI-based conversational avatars lightning fast ⚡️💬✨ FeaturesLive anywhere: VRChat, cluster and any other metaverse platforms, and even devices in real world.Extensible: Unlimited capabilities that depends on you.Easy to start: Ready to start conversation right out of the box.🍩 RequirementsVOICEVOX API in your computer or network reachable machine (Text-to-Speech)API key for Speech Services of Google or Azure (Speech-to-Text)API key for OpenAI API (ChatGPT)Python 3.10 (Runtime)🚀 Quick startInstall AIAvatarKit.$pipinstallaiavatarMake the script asrun.py.importloggingfromaiavatarimportAIAvatar,WakewordListenerGOOGLE_API_KEY="YOUR_API_KEY"OPENAI_API_KEY="YOUR_API_KEY"VV_URL="http://127.0.0.1:50021"VV_SPEAKER=46# Configure root loggerlogger=logging.getLogger()logger.setLevel(logging.INFO)log_format=logging.Formatter("[%(levelname)s]%(asctime)s:%(message)s")streamHandler=logging.StreamHandler()streamHandler.setFormatter(log_format)logger.addHandler(streamHandler)# Promptsystem_message_content="""あなたは「joy」「angry」「sorrow」「fun」の4つの表情を持っています。特に表情を表現したい場合は、文章の先頭に[face:joy]のように挿入してください。例[face:joy]ねえ、海が見えるよ！[face:fun]早く泳ごうよ。"""# Create AIAvatarapp=AIAvatar(google_api_key=GOOGLE_API_KEY,openai_api_key=OPENAI_API_KEY,voicevox_url=VV_URL,voicevox_speaker_id=VV_SPEAKER,# volume_threshold=2000,    # <- Set to adjust microphone sensitivitymodel="gpt-3.5-turbo",system_message_content=system_message_content,)# Create WakewordListenerwakewords=["こんにちは"]asyncdefon_wakeword(text):logger.info(f"Wakeword:{text}")awaitapp.start_chat()wakeword_listener=WakewordListener(api_key=GOOGLE_API_KEY,volume_threshold=app.volume_threshold,wakewords=wakewords,on_wakeword=on_wakeword,device_index=app.input_device)# Start listeningww_thread=wakeword_listener.start()ww_thread.join()# Tips: To terminate with Ctrl+C on Windows, use `while` below instead of `ww_thread.join()`# while True:#     time.sleep(1)Start AIAvatar.$pythonrun.pyWhen you say the wake word "こんにちは" the AIAvatar will respond with "どうしたの？".
Feel free to enjoy the conversation afterwards!If you want to set face expression on some action, configure as follows:# Add face expresionsapp.avatar_controller.face_controller.faces["on_wake"]=10app.avatar_controller.face_controller.faces["on_listening"]=11app.avatar_controller.face_controller.faces["on_thinking"]=12# Set face when the character is listening the users voiceasyncdefset_listening_face():awaitapp.avatar_controller.face_controller.set_face("on_listening",3.0)app.request_listener.on_start_listening=set_listening_face# Set face when the character is processing the requestasyncdefset_thinking_face():awaitapp.avatar_controller.face_controller.set_face("on_thinking",3.0)app.chat_processor.on_start_processing=set_thinking_faceasyncdefon_wakeword(text):logger.info(f"Wakeword:{text}")# Set face when wakeword detectedawaitapp.avatar_controller.face_controller.set_face("on_wake",2.0)awaitapp.start_chat(request_on_start=text,skip_start_voice=True)🐈 Use in VRChat2 Virtual audio devices (e.g. VB-CABLE) are required.Multiple VRChat accounts are required to chat with your AIAvatar.First, run the commands below in python interpreter to check the audio devices.$%python

>>>fromaiavatarimportAudioDevice
>>>AudioDevice.list_audio_devices()Availableaudiodevices:0:HeadsetMicrophone(OculusVirt:6:CABLE-BOutput(VB-AudioCable7:Microsoftサウンドマッパー-Output8:SONYTV(NVIDIAHighDefinition:13:CABLE-AInput(VB-AudioCableA:In this example,To useVB-Cable-Afor microphone for VRChat, index foroutput_deviceis13(CABLE-A Input).To useVB-Cable-Bfor speaker for VRChat, index forinput_deviceis6(CABLE-B Output). Don't forget to setVB-Cable-B Inputas the default output device of Windows OS.Then editrun.pylike below.# Create AIAvatarapp=AIAvatar(GOOGLE_API_KEY,OPENAI_API_KEY,VV_URL,VV_SPEAKER,model="gpt-3.5-turbo",system_message_content=system_message_content,input_device=6# Listen sound from VRChatoutput_device=13,# Speak to VRChat microphone)You can also set the name of audio devices instead of index (partial match, ignore case).input_device="CABLE-B Out"# Listen sound from VRChatoutput_device="cable-a input",# Speak to VRChat microphoneRun it.$run.pyLaunch VRChat as desktop mode on the machine that runsrun.pyand log in with the account for AIAvatar. Then setVB-Cable-Ato microphone in VRChat setting window.That's all! Let's chat with the AIAvatar. Log in to VRChat on another machine (or Quest) and go to the world the AIAvatar is in.🟦 Use Azure ListenersWe strongly recommend using AzureWakewordListener and AzureRequestListner that are more stable than the default listners. Checkexamples/run_azure.pythat works out-of-the-box.Install Azure SpeechSDK.$pipinstallazure-cognitiveservices-speechChange script to use AzureRequestListener and AzureWakewordListener.YOUR_SUBSCRIPTION_KEY="YOUR_SUBSCRIPTION_KEY"YOUR_REGION_NAME="japanwest"# Create AzureRequestListenerfromaiavatar.listeners.azurevoicerequestimportAzureVoiceRequestListenerrequest_listener=AzureVoiceRequestListener(YOUR_SUBSCRIPTION_KEY,YOUR_REGION_NAME,)# Create AIAVater with AzureRequestListenerapp=AIAvatar(openai_api_key=OPENAI_API_KEY,system_message_content=system_message_content,request_listener=request_listener,voicevox_url=VV_URL,voicevox_speaker_id=VV_SPEAKER,)# Create AzureWakewordListnerasyncdefon_wakeword(text):logger.info(f"Wakeword:{text}")awaitapp.start_chat()fromaiavatar.listeners.azurewakewordimportAzureWakewordListenerwakeword_listener=AzureWakewordListener(YOUR_SUBSCRIPTION_KEY,YOUR_REGION_NAME,on_wakeword=on_wakeword,wakewords=["こんにちは"])To specify the microphone device by settingdevice_nameargument.
See Microsoft Learn to know how to check the device UID on each platform.https://learn.microsoft.com/en-us/azure/ai-services/speech-service/how-to-select-audio-input-devicesWe providea script for MacOS. Just run it on Xcode.Device UID: BuiltInMicrophoneDevice, Name: MacBook Proのマイク
Device UID: com.vbaudio.vbcableA:XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX, Name: VB-Cable A
Device UID: com.vbaudio.vbcableB:XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX, Name: VB-Cable BFor example, the UID for the built-in microphone on MacOS isBuiltInMicrophoneDevice.Then, set it as the value ofdevice_name.request_listener=AzureVoiceRequestListener(YOUR_SUBSCRIPTION_KEY,YOUR_REGION_NAME,device_name="BuiltInMicrophoneDevice")wakeword_listener=AzureWakewordListener(YOUR_SUBSCRIPTION_KEY,YOUR_REGION_NAME,on_wakeword=on_wakeword,wakewords=["こんにちは"],device_name="BuiltInMicrophoneDevice")⚡️ Function CallingUsechat_processor.add_functionto use ChatGPT function calling. In this example,get_weatherwill be called autonomously.# Add functionasyncdefget_weather(location:str):awaitasyncio.sleep(1.0)return{"weather":"sunny partly cloudy","temperature":23.4}app.chat_processor.add_function(name="get_weather",description="Get the current weather in a given location",parameters={"type":"object","properties":{"location":{"type":"string"}}},func=get_weather)And, afterget_weathercalled, message to get voice response will be sent to ChatGPT internally.{"role":"function","content":"{\"weather\": \"sunny partly cloudy\", \"temperature\": 23.4}","name":"get_weather"}🎤 Testing audio I/OUsing the script below to test the audio I/O before configuring AIAvatar.Step-by-Step audio device configuration.Speak immediately after start if the output device is correctly configured.All recognized text will be shown in console if the input device is correctly configured.Just echo on wakeword recognized.importasyncioimportloggingfromaiavatarimport(AudioDevice,VoicevoxSpeechController,WakewordListener)GOOGLE_API_KEY="YOUR_API_KEY"VV_URL="http://127.0.0.1:50021"VV_SPEAKER=46VOLUME_THRESHOLD=3000INPUT_DEVICE=-1OUTPUT_DEVICE=-1# Configure root loggerlogger=logging.getLogger()logger.setLevel(logging.INFO)log_format=logging.Formatter("[%(levelname)s]%(asctime)s:%(message)s")streamHandler=logging.StreamHandler()streamHandler.setFormatter(log_format)logger.addHandler(streamHandler)# Select input deviceifINPUT_DEVICE<0:input_device_info=AudioDevice.get_input_device_with_prompt()else:input_device_info=AudioDevice.get_device_info(INPUT_DEVICE)input_device=input_device_info["index"]# Select output deviceifOUTPUT_DEVICE<0:output_device_info=AudioDevice.get_output_device_with_prompt()else:output_device_info=AudioDevice.get_device_info(OUTPUT_DEVICE)output_device=output_device_info["index"]logger.info(f"Input device: [{input_device}]{input_device_info['name']}")logger.info(f"Output device: [{output_device}]{output_device_info['name']}")# Create speakerspeaker=VoicevoxSpeechController(VV_URL,VV_SPEAKER,device_index=output_device)asyncio.run(speaker.speak("オーディオデバイスのテスターを起動しました。私の声が聞こえていますか？"))# Create WakewordListenerwakewords=["こんにちは"]asyncdefon_wakeword(text):logger.info(f"Wakeword:{text}")awaitspeaker.speak(f"{text}")wakeword_listener=WakewordListener(api_key=GOOGLE_API_KEY,volume_threshold=VOLUME_THRESHOLD,wakewords=["こんにちは"],on_wakeword=on_wakeword,verbose=True,device_index=input_device)# Start listeningww_thread=wakeword_listener.start()ww_thread.join()⚡️ Use custom listenerIt's very easy to add your original listeners. Just make it run on other thread and invokeapp.start_chat()when the listener handles the event.Here the example ofFileSystemListenerthat invokes chat whentest.txtis found on the file system.importasyncioimportosfromthreadingimportThreadfromtimeimportsleepclassFileSystemListener:def__init__(self,on_file_found):self.on_file_found=on_file_founddefstart_listening(self):whileTrue:# Check file every 3 secondsifos.path.isfile("test.txt"):asyncio.run(self.on_file_found())sleep(3)defstart(self):th=Thread(target=self.start_listening,daemon=True)th.start()returnthUse this listener inrun.pylike below.# Event handlerdefon_file_found():asyncio.run(app.chat())# Instantiatefs_listener=FileSystemListener(on_file_found)fs_thread=fs_listener.start():# Wait for finishfs_thread.join()