AI-powered Immersive
Virtual Reality (VR)
Yasunobu Uchiyama (Rikkyo University & Galaxies, inc.)

ISCO2023, OIST in Okinawa, Japan, 27th Feb. — 3rd MarchIntroducing myself
2003:
Received Ph D. from Univ. of Tokyoin X-ray Astronomy(Supervisor: Prof. Tad Takahashi)

2018—present:
Founder of Galaxies, inc.
in AI & VR

2008—2013:2016—present:
Panofsky Fellow at SLAC (Stanford) Professor at Rikkyo U.
in Gamma-ray Astronomy
High-Energy Astrophysics

2020—present:
Founder & Chairof Rikkyo Graduate
School of AI and Science

Interdisciplinarity
First President of OIST:Jonathan Dorfan
2Interdisciplinary and Transdisciplinary
According to “The Application of Transdisciplinary Theory and Practice to STEM Education”(S. M. Back, H. Greenhalgh-Spencer, K M. Frias):

Interdisciplinarity
A joint e ort to solve a problem or develop a product in which experts fromdi erent disciplines exchange theories and methods in a process of sharing
their areas of expertise.

Physics

egndaohhct
ex /meaedifo

MedicalScience
Mathematics

Transdisciplinarity
A fusing of theories, methods and expertise across disciplinary boundaries in which
each discipline merges with the others in the formation of a whole that is greaterthan the sum of its parts. New disciplines may emerge as in the case of the “NewBiology” which is the result of collaborative e orts on behalf of traditional biology,
physical and chemical sciences, computational science, mathematics, andengineering. Transdisciplinarity may also include perspectives and methods fromsuch disciplines as social science, economics, and public administration as well asfrom civil society and a wide range of stakeholders.

New discipline

Civil Societyff

ffff

3

1. What AI can do today4Some examples

Nichol et al. (2021)

ff

Stable Di usion (2022)

5

AI Technologies in World Cup
42 cameras for Qatar World Cup (2022)

To track movement and posture of each player, 3D
positions of 29 joints are estimated (every ~0.1 sec) bymachine learning from video (Hawk-Eye Innovations)credit: FIFA
Semi-automatic o side judgement(credit: Hawk-Eye Innovations)ff6

2. Rikkyo-AI Projects

7

Applied AI in Professional Baseball (Preview)

X

ViTPose: Simple Vision Transformer Baselines for
Human Pose Estimation (Xu et al. 2022 October)

Bubble sizerepresents model size

(Not Samsung Lions…)

Precision

Our choice Since 2022 Oct

Our choice in 2021 22-Pose estimate AI: estimate the positions of 17 joints(keypoints) such as wrists, elbows, and shoulders from asingle camera taking pitcher’s motion.

Throughput (fps)8Applied AI in Nursing Care
Smile Project: using AI/VR/robots, we try to increase the smiles of elderly
people living in special nursing homes.
This project is a joint industry-academia project among NTT East Japan, Social Welfare
Corporation: Frontier, and Rikkyo University Graduate School of Arti cial Intelligence Science.

fi9

Applied AI in Nursing Care
Smile Project: using AI/VR/robots, we try to increase the smiles of elderly
people living in special nursing homes.
This project is a joint industry-academia project among NTT East Japan, Social Welfare
Corporation: Frontier, and Rikkyo University Graduate School of Arti cial Intelligence Science.

fi

10

3. Immersive Virtual Reality

11Virtual Reality (VR)(stereo) Head-mounted Display (HMD)
3D CG (Computer Graphics)(credit: Meta Platforms)

Key elements of(immersive) VR

・3D spatiality
・Real-time interaction (able to act freely in the created environment)
・Self-projectiveness (immersion in the created environment)
12

VR Device and Software
Key elements of(immersive) VR

・3D spatiality
・Real-time interaction (able to act freely in the created environment)
・Self-projectiveness (immersion in the created environment)
Simulator(VR PC)

VR system w/ HMD
user

sensory generation(vision)pseudo sense

displayuserʼs real-time

actioninformation
tracking of userʼs motion
recognition of userʼs voice/face・Operation of 3DCG objects
・Physics simulations
・Rendering

13

Avatar and Virtual EmbodimentAvatar(stereo) Head-mounted Display (HMD)(credit: Meta Platforms)

Virtual Embodiment・Body Ownership
・Agency

Powered by AI
15

Metaverse“Embodied Internet”“The ‘metaverse’ is a set of virtual spaces
where you can create
and explore with other people who aren’t inthe same physical space as you.”Meta Platforms (formally Facebook)

Facebook Japan

Taken from Wikipedia
16

4. AI x VR
Recent AI revolution leading to VR revolution

17

Real Avatar in Real Environment
With the advent of high-performance device (HMD) and revolutionary powerful AI technologies(deep learning), it becomes possible for VR to be more and more “real”.
photoreal avatar of myself

a photoreal desk of our classroom(reconstructed from its pictures)a photoreal room
(historical room in UK)

18

Avatars of Hatsuda-san & Nagataki-san
Photogrammetry:
Combining images from ~50 cameras surrounding aperson, 3D photorealistic model can be generated.
→ “classical” computer vision technique
AI (Deep Learning) canimprove the 3D model:
・image inpainting・mesh optimization
・etc.
In the near future,
deep learning may take overfrom current photogrammetry
algorithm.
20

Transformation of Virtual Body and Objects
Bayesian Coherent Point Drift

A. Hayashi (OIST, grad student) is working on thisin my company (Galaxies, inc.)21

Facial Expression Synchronization

avatar22

Posture Synchronization

23

SPH Simulation for VR
Smoothed Particle Hydrodynamics (SPH) Simulation
Deep Learning may realize much faster, real-time, simulation
SPHere of Galaxies

SPH (Smoothed Particle Hydrodynamics)
on GPUAI x VR (overview)
Key elements of(immersive) VR

・3D spatiality
・Real-time interaction (able to act freely in the created environment)
・Self-projectiveness (immersion in the created environment)
Simulator(VR PC)VR system w/ HMD
user

sensory generation(vision)pseudo sense

displayuserʼs real-time

actioninformation
tracking of userʼs motion

Powered by AI recognition of userʼs voice/face・Operation of 3DCG objects
・Physics simulations
・Renderingerew
PoybdIA25