IEEE MetaCom 2023

Avatar Fusion Karaoke:
Research and development on
multi-user music play VR
experience in the metaverse

June 26-28

Alexandre BERTHAULTREALITY Metaverse Laboratory
REALITY, Inc.

1

(IEEE MetaCom 2023)
2023/6/27 14:40 ～ 14:55

Avatar Fusion Karaoke:
Research and developmenton multi-user music play VR
experience in the metaverse
❏❏❏

❏

❏

Introduction and backgroundOur Focus
Pre production research and tools❏ Performance benchmark❏ Reflection on LookDev and character
design❏ QueTra
AI assisted music play
❏ AI Fusion principle❏ Challenges❏ Proof of conceptsConclusion / Future Evaluations

Alexandre BERTHAULT / Takuma Kato / Akihiko SHIRAIREALITY Metaverse Laboratory
REALITY, Inc.

2

Members of the Laboratory
Mr.Nakano
Fourth year student in the Department of Advanced Media Science at Meiji University's School ofComprehensive Mathematics and Science.
Joined the lab in December 2020, and has worked as an assistant director throughout the UXDev
series, taking care of a wide range of tasks from storyline design to Unity engineering. Belongs toan a cappella circle. Loves movies.

Mr.Yahagi
Second year student in the Department of Digital Content at the College of Digital Communication
at Digital Hollywood University, he is in charge of Unity engineering and various scripting for theUxDev series.
A genuine resident of the Metaverse who has been in and out of the lab since high school, Plays alot of Valorant

Mr. Horibe
First-year master's student in the Department of Advanced Media Science, Graduate School of
Advanced Mathematical Sciences, Meiji University.
Started working at GREE VR Studio Laboratory in October 2019 on voice analysis of users of theexperimental voice changer service "Tensei Koe Uranai" and iOS implementation and accelerationof the voice changer. In the UXDev team, he is mainly responsible for voice and facial expressions.
Level 70+ at Pikmin Bloom3

Members of the Laboratory
Mr. Alexandre Berthault
Graduated from ISART Digital, a French video game school, and became a research student at
Tokyo University of Technology. Started working at GREE VR Studio Laboratory in June 2022, heis mainly responsible for UX prototyping and effect design in the UXDev series. Loves ramen.
Doesn’t like Kanjis.

Mr.Yamaoka
Fourth-year student in the Department of Advanced Media Science, School of Integrated
Mathematical Sciences at Meiji University, he joined the GREE VR Studio Laboratory in October
2022. He is researching technology that allows avatars to blend in with the real world without
discomfort.
He is in charge of R&D related to avatars, making use of his production experience with Meta
Quest, Hololens2, Web AR, etc. Recently, he has been working on HMD devices such as Quest towork on avatar benchmarking tools for Quest and other HMD devices.

Mr.Akihiko Shirai (Director)
Director of GREE VR Studio Laboratory. He has over 25 years of experience in R&D andeducation, visual arts, VR entertainment, and scientific communication. He brings diversity to theGREE Group's development culture through his diverse talents.
Outside of work, he writes for the GREE Technical Book Department and the book "Collaborate
with AI to Become a Divine Artist".

4

「REALITY」User Scale
Over 10 million downloads worldwide by the end of 2022
Continued growth expected in the futureOver

100 000 000
Users worldwide
World N°1
smartphone metaverse
application

Over 10 Millions
downloads
worldwide!

End of theyear 2022

Plans for202X© REALITY, Inc.

55

References: VibeShare and Directional Haptics
1.

Yuichiro Kume. 1998. Foot interface: fantastic phantom slipper. In ACM SIGGRAPH 98 Conference abstracts andapplications (SIGGRAPH '98). Association for Computing Machinery, New York, NY, USA, 114.
DOI:https://doi.org/10.1145/280953.284801

2.

Yusuke Yamazaki, Shoichi Hasegawa, Hironori Mitake, and Akihiko Shirai. 2019. NeckStrap Haptics: An Algorithm
for Non-Visible VR Information Using Haptic Perception on the Neck. InACM SIGGRAPH 2019 Posters(Los Angeles,California)(SIGGRAPH ’19). Association for Computing Machinery, New York, NY, USA, Article 60, 2 pages.
https://doi.org/10.1145/3306214.333

3.

Yusuke Yamazaki and Akihiko Shirai. 2021. Pseudo Real-Time Live Event: Virtualization for Nonverbal LiveEntertainment and Sharing. In2021: Laval Virtual VRIC, ConVRgence Proceedings 2021(Laval, France)(VRIC ’21).
https://doi.org/10.20870/IJVR.2021.1.1.479

6

References: Map and Walking interactions
1.

Makiko Suzuki Harada, Hidenori Watanave and Shuuichi Endou, : "Tuvalu Visualization Project - Net Art on Digital
Globe: Telling the Realities of Remote Places", page 559- 572, 2011.

2.

Akihiko Shirai, Kiichi Kobayashi, Masahiro Kawakita, Shoichi Hasegawa, Masayuki Nakajima, and Makoto Sato.
2004. Entertainment applications of human-scale virtual reality systems. In Proceedings of the 5th Pacific Rim
conference on Advances in Multimedia Information Processing - Volume Part III (PCM'04). Springer-Verlag, Berlin,Heidelberg, 31–38. DOI:https://doi.org/10.1007/978-3-540-30543-9_5

3.

Akihiko Shirai, Yuki Kose, Kumiko Minobe, and Tomoyuki Kimura. 2015. Gamification and construction of virtual fieldmuseum by using augmented reality game "Ingress". In Proceedings of the 2015 Virtual Reality International
Conference (VRIC '15). Association for Computing Machinery, New York, NY, USA, Article 4, 1–4.
DOI:https://doi.org/10.1145/2806173.2806182

7

References: Avatar and metaverse in Education1.

Rex Hsieh, Akihiko Shirai, and Hisashi Sato. 2019. Evaluation of Avatar and Voice Transform in Programming
E-Learning Lectures. In Proceedings of the 19th ACM International Conference on Intelligent Virtual Agents (IVA '19).
Association for Computing Machinery, New York, NY, USA, 197–199. DOI:https://doi.org/10.1145/3308532.3329430

2.

Rex Hsieh, Akihiko Shirai, and Hisashi Sato. 2019. Effectiveness of facial animated avatar and voice transformer inelearning programming course. In ACM SIGGRAPH 2019 Posters (SIGGRAPH '19). Association for Computing
Machinery, New York, NY, USA, Article 82, 1–2. DOI:https://doi.org/10.1145/3306214.3338540

3.

Liudmila Bredikhina, Toya Sakaguchi, and Akihiko Shirai. 2020. Web3D Distance LiveWorkshop for Children in Mozilla
Hubs. In The 25th International Conference on3D Web Technology(Virtual Event, Republic of Korea)(Web3D ’20).
Association for Computing Machinery, New York, NY, USA, Article 27, 2 pages.
https://doi.org/10.1145/3424616.342472

4.

Stewart Culin. 1920. THE JAPANESE GAME OF SUGOROKU.The Brooklyn Museum Quarterly 7, 4 (1920), 213–233.
http://www.jstor.org/stable/26458

References: XR Live Entertainment
1.

Akihiko Shirai. 2019. REALITY: broadcast your virtual beings from everywhere. In ACM SIGGRAPH 2019 Appy Hour
(SIGGRAPH '19). Association for Computing Machinery, New York, NY, USA, Article 5, 1–2.
DOI:https://doi.org/10.1145/3305365.3329727

2.

Bredikhina, Liudmila et al. “Avatar Driven VR Society Trends in Japan.” 2020 IEEE Conference on Virtual Reality and 3D
User Interfaces Abstracts and Workshops (VRW) (2020): 497-503.

3.

Akihiko Shirai, et.al. 2019. Global Bidirectional Remote Haptic Live Entertainment by Virtual Beings. ACM SIGGRAPH
ASIA 2019 Real-Time Live!

SIGGRAPH
ASIA 20189Latest work : Generative AI in the Metaverse●Metaverse Mode Maker●AITuber #MetaChatNews

Words to Textile using Stable Diffusion + Motion UGCMulti-Agents chat play with voice and motion expressions

https://youtu.be/Lrd5i8EZxTI

https://bit.ly/MCNKWS2310

Introduction and backgroundhttps://www.matthewball.vc/all/forwardtothemetaverseprimer●No definite definition of the “Metaverse”●Can be seen as an iteration of our internetsystem

Seven core enablers of the metaverse by Matthew
Ball

11

OurOurfocus
focus●Create Research and development protocol for new metaverse user experiences and use it toanswer our hypothesis●Conclude on the hypothesis that it should be possible to design the avatars with only thenecessary functions and rendering costs.

●

Objective : create a school life karaoke experience

R&D procedure. This paper focuses on completing UX R&D for the HMD
experience12Pre production research and tools
Performance benchmark●VR Benchmarking is critical !

●Avatar average polygon count in REALITY : 30 000, Materials : 6-14

Benchmark application

Result graph13

Preandtools
Preproduction
productionresearchresearchandtools
Reﬂection on LookDev and character design●

Limitate change in visuals●

Optimization and compatibility●

Coherence between appearance and use

Example of different character models used in our application. The same character can wear a vastvariety of clothes making the compatibility and look dev that more challenging.

14

Pre
Preproduction
productionresearchresearchandandtools
tools
QueTra●Motion capture system using Meta Quests●Creates animations compatible with Unity●

Tested in a children’s workshop

Kids workshop
https://youtu.be/bXxnUdpaX8M

QueTra application

15

AI
AIassisted
assistedmusic
musicplay
play
AI Fusion principle●Uses Standard MIDI files●Algorithms transforms midi files into usable animations●Layers allow for a mix of multiple animations
Example of Midi generated music play
https://youtu.be/LrB5aNikLvA

16

AI
AIassisted
assistedmusic
musicplay
play
Challenges●

Face tracking impossible with VR headsets●

Research on RandomForest●Research on Audio2Face

The process to generate exaggerated facial expressions with BlendShape + LipSync from real-time
voice and depth-camera input.

17

AIAIassisted
assistedmusic
musicplayplay
Proof of concepts●

MetaDreamers objectives, limitations and future focus●

GV Band, a VR music play experience

Meta Dreamers short film

GV Band VR application

18

Conclusion / Future Evaluations●Results : two proofs of concepts which answer the initial hypothesis●

Created an atavistic experience which checks three type of validations :

1.

2.

3.

Quality ofcommunication basedon hardware limitations
Identification ofchallenges in the user
collaboration experience
Verify user motion
representation andcomplementarytechnologies

We started with the objective of an avatar-driven music play. Through various R&D, we createdan APK application (GVBand) and a short film (MetaDreamers), which can then be evaluated.

19

Thank you for your attention
Website : https://reality.inc/lab Twitter : @VRStudioLab
Alexandre BERTHAULT
apro.berthault@gmail.com

Takuma KATO
takuma.katou42@gmail.com

Akihiko SHIRAIshirai@mail.com

https://www.youtube.com/watch?v=mMrNWEe0IOs
20