ISSN 2186-7437

NII Shonan Meeting Report

No. 2018-8

Augmented Reality in
Human-Computer Interaction
Yuta Itoh
Kai Kunze
Alexander Plopski
Christian Sandor
June 4–7, 2018

National Institute of Informatics
2-1-2 Hitotsubashi, Chiyoda-Ku, Tokyo, Japan

Augmented Reality in
Human-Computer Interaction
Organizers:
Yuta Itoh, Tokyo Institute of Technology, Japan
Kai Kunze, Keio Univerisity, Japan
Alexander Plopski, Nara Institute of Science and Technology, Japan
Christian Sandor, Nara Institute of Science and Technology, Japan
June 4–7, 2018
The first Shonan Meeting on Perceptual Issues in Augmented Reality (AR)was successfully organized by Christian Sandor, Dieter Schmalstieg, and Edward
J. Swan II in 2016. Building on the success of the first meeting, we propose to
conduct a Shonan Meeting on Augmented Reality in Human-Computer
Interaction (HCI) to investigate yet another key issue that remains in the
AR community.
AR expands the users world by presenting information, entertainment, orinteraction surfaces as virtual content in the environment. Research into AR
began more than 50 years ago with the vision of the Ultimate Display by Sutherland (1968). Over the years, research on head-worn AR has been complemented
by work on new platforms such as handheld AR (Dey et al., 2012) and projectorcamera systems (Bandyopadhyay et al., 2001). With the rapid advent of applications on cell phones, AR has become almost mainstream.
One major goal of AR is to expand the users perception of the surroundings
by visualizing invisible data, like connections between different objects and thecorresponding information (Sandor et al., 2005; Heun et al., 2013). By nature,
AR is a technology for creating intuitive human-computer interfaces, using
computers, sensors, and mobile devices. AR and HCI communities often focus
on similar questions, e.g., eye-gaze tracking (Sewell and Komogortsev, 2010),
haptic feedback (Bidmon et al., 2007), sensor tracking (Wagner et al., 2008) andperception of virtual content by the users (Moser et al., 2015).
Many premier conferences on HCI, such as ACM Conference on Human
Factors in Computing Systems (CHI), ACM Symposium on User Interface Software and Technology (UIST), and ACM Special Interest Group on Computer
Graphics and Interactive Techniques (SIGGRAPH), recognized the impact ofAR, with many of their best papers focusing on AR. However, these papers commonly do not develop novel technologies that can enable AR, but apply alreadywell-known algorithms and commercial devices to explore the human factor side of AR. On the contrary, the IEEE International Symposium on Mixed
and Augmented Reality (ISMAR) community focuses on the technologicalaspects of AR. The ISMAR community has been pushing the boundaries ofwhat’s possible with papers including PTAM (2007) and KinectFusion (2011).

1

In that sense, both communities complement each other.
The goal of this meeting is to build a bridge between leading researchers fromthe HCI and ISMAR communities. Exchanging ideas in an open meeting
will promote the dissemination of results and technologies between the communities. This will improve the integration of AR technology for HCI, lead to thedevelopment of novel interaction techniques, and provide new research questionsof interest for both communities.

References
Deepak Bandyopadhyay, Ramesh Raskar, and Henry Fuchs. Dynamic Shader Lamps: Painting on Movable Objects. In Proceedings of the IEEE/ACM International Symposium on
Augmented Reality, pages 207–216, Washington, DC, USA, October 2001.
Katrin Bidmon, Guido Reina, Fabian Bös, Jürgen Pleiss, and Thomas Ertl. Time-Based
Haptic Analysis of Protein Dynamics. In Proceedings of the Joint EuroHaptics Conference
and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems,
pages 537–542, Tsukuba, Japan, March 2007.
Arindam Dey, Graeme Jarvis, Christian Sandor, and Gerhard Reitmayr. Tablet versus Phone:
Depth Perception in Handheld Augmented Reality. In Proceedings of the IEEE International Symposium on Mixed and Augmented Reality, pages 187–196, Atlanta, GA, USA,
November 2012.
Valentin Heun, James Hobin, and Pattie Maes. Reality Editor: Programming Smarter
Objects. In Proceedings of the UbiComp (Adjunct Publication), pages 307–310, Zurich,
Switzerland, September 2013.
Georg Klein and David Murray. Parallel Tracking and Mapping for Small AR Workspaces. In
Proceedings of the IEEE/ACM International Symposium on Mixed and Augmented Reality,
pages 225–234, Nara, Japan, November 2007.
Kenneth Moser, Yuta Itoh, Kohei Oshima, Edward Swan, Gudrun Klinker, and Christian
Sandor. Subjective Evaluation of a Semi-Automatic Optical See-Through Head-Mounted
Display Calibration Technique. IEEE Transactions on Visualization and Computer Graphics, 21(4):491–500, March 2015.
Richard A Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J Davison, Pushmeet Kohi, Jamie Shotton, Steve Hodges, and Andrew Fitzgibbon.
KinectFusion: Real-Time Dense Surface Mapping and Tracking. In Proceedings of theIEEE International Symposium on Mixed and Augmented Reality, pages 127–136, Basel,
Switzerland, October 2011.
Christian Sandor, Alex Olwal, Blaine Bell, and Steven Feiner. Immersive Mixed-Reality Configuration of Hybrid User Interfaces. In Proceedings of the IEEE/ACM International Symposium on Mixed and Augmented Reality, pages 110–113, Washington, DC, USA, October
2005.
Weston Sewell and Oleg Komogortsev. Real-time Eye Gaze Tracking with an Unmodified
Commodity Webcam Employing a Neural Network. In Proceedings of the ACM Conference on Human Factors in Computing Systems: Extended Abstracts on Human Factors in
Computing Systems, pages 3739–3744, Atlanta, GA, USA, April 2010.
Ivan E. Sutherland. A Head-Mounted Three Dimensional Display. In Proceedings of the Fall
Joint Computer Conference, Part I, pages 757–764, New York, NY, USA, December 1968.
Daniel Wagner, Gerhard Reitmayr, Alessandro Mulloni, Tom Drummond, and Dieter Schmalstieg. Pose Tracking from Natural Features on Mobile Phones. In Proceedings of theIEEE/ACM International Symposium on Mixed and Augmented Reality, pages 125–134,
Cambridge, UK, September 2008.

2

Meeting Schedule• Sunday: Check In• Monday: PechaKucha-Style Self-Introductions• Tuesday–Thursday: Free Discussion• Thursday: Check Out

Group Drawing

3

Participants• Gerd Bruder (University of Central Florida)• Nicko Reginio Caluya (Nara Institute of Science and Technology)• Cedric Carêmel (Keio University)• Alvaro Cassinelli (Tokyo University)• Stephen Ellis (Mississippi State University)• Morten Fjeld (Chalmers University of Technology)• Jens Grubert (Coburg University of Applied Science)• Takumi Hamasaki (Keio University)• Keita Higuchi (The University of Tokyo)• Teresa Hirzle (Ulm University)• Takeo Igarashi (The University of Tokyo)• Yuta Itoh (Tokyo Institute of Technology, Japan)• Yoshiki Kudo (Tohoku University)• Kai Kunze (Keio University)• Tobias Langlotz (University of Otago)• Katsutoshi Masai (Keio University)• Keigo Matsumoto (The University of Tokyo)• Nisho Matsushita (The University of Tokyo)• Alex Olwal (Google Inc.)• Alexander Plopski (Nara Institute of Science and Technology)• Damien Constantine Rompapas (Nara Institute of Science and Technology)• Enrico Rukzio (Ulm University)• Daniel Saakes (KAIST)• Daisuke Sakamoto (Hokkaido University)• Christian Sandor (Nara Institute of Science and Technology)• Maki Sugimoto (Keio University)• Yasufumi Tanaka (The University of Tokyo)• Yuji Uema (The University of Tokyo)• Gregory Welch (University of Central Florida)• Shigeo Yoshida (The University of Tokyo)• Pai Yun Suen (Keio University)• Yuchong Zhang (Chalmers University of Technology)

4

PechaKucha Talks
Participants’ slides for their PechaKucha talks are appended on the remaining
pages of this report. Order is in alphabetic order of last names.

Mr. Carêmel’s slides were excluded due to confidentiality.

5

"

File

Size

Word list

Gap size

#

$

Shape

Mask %

Theme

Colors−

Font

Wizard4

If you like WordClouds.com (seems like you do!), you might also want to check our free online Poster Maker (https://www.postermaker.com). Design and create yourown posters, flyers, memes and typography design in your browser, for fun, to share, to print.

& 78 words not drawn

Tag CloudPerceptually Inspired User Interfaces
in Virtual and Augmented Reality
Gerd BRUDER
Research Assistant Professor
The University of Central Florida
Institute for Simulation & Training
Orlando, Florida
3 June 2018(based on 110 peer-reviewed paper titles)© 2018 Gerd BRUDER • UCF

2
1024×768© Zygomatic (http://zygomatic.nl/) · Home (/) · Disclaimer (/disclaimer/)

Redirected Walking• Detection thresholds• Redirection algorithms• Multimodal interfaces• Locomotion planning• Path prediction• Redirected driving• Cognitive load estimation• Simulator sickness reduction• Perceptual models• Walkthrough applications•…

'

2D-3D Touch User Interfaces

physical pathvirtualpath• Stereoscopic 3D touching [CHI’11]• Touching floating objects [JVRC’10]• To touch or not to touch [IN’13]

m

45

22m
radius45m© 2018 Gerd BRUDER • UCF

1.25

0.75

Real Human

Virtual Human

time

0.5

ce

tan

dis

0.5

Speed

Time
Distance

Speed = Distance / Time ?

0.25

4

Perception of/with AR Agentsactualestimated1© 2018 Gerd BRUDER • UCF

3

Perceptual Differences in VR/AR• 3D touch performance [3DUI’13]• Applications•…

“Ghost-like”
visual conflicts(occlusion, etc.)

0.25
0
0

0.25

spe
ed

0.5
0.75

0.75
1

1

1.25

1.25

Mediated Physicality• Manipulate zeitgebers• Change optic flow
• Warp field of view
• Artificial eye separation• Calibration methods• Perceptual models• Applications•…
© 2018 Gerd BRUDER • UCF• Create illusion of physicality• Touch w/ intermediate obj.
• Avoid visual distortions• Increase social presence• Change motion perception• Increase cognitive load
5

AR “Physicality” Conflicts• Affect decision making• Leverage perceptual priming• Apply social facilitation
theory• Applications•…
© 2018 Gerd BRUDER • UCF

6

Applications in AR Training
Giver

Beyond Augmented Reality

Receiver
Instructor
Patient

Pararescue Jumper Training

AR Medical Training (Handoffs,
Combat Casualty Care)“Active Shooter” Police Training

AR Stressful/Realistic Environments• Portable ad hoc in situ AR training approaches• Improve training fidelity and efficiency• Develop handoff protocols, training
environments• Methods to increase realism, stress, mental load
• Develop instructor interfaces (real time,after action review)•…
© 2018 Gerd BRUDER • UCF

MR “Blend Over” and“Afterglow” Effects

AR “Super Powers” — new social norms© 2018 Gerd BRUDER • UCF

7

8“Alexa” Demo
We are hiring!
PhD students & PostDocs(starting fall/end of year)
Send applications to…
welch@ucf.edu
bruder@ucf.edu© 2018 Gerd BRUDER • UCF

9© 2018 Gerd BRUDER • UCF

10

770385

www.alvarocassinelli.com

ALVAROCASSINELLI

Augmented Perception

Human-centered AR?
From the inside••

Augmented perception / expression
Augmented Self

From the outside••

Augmented information / aesthetic
AR or SAR?

Augmented Expression & Body Schema

Minimal / Supernormal stimuli

Augmentd Space (surfaces & volumes)

Laser-based Spatial Augmented Reality...special effects
in the Real World?

Towards a “function projector”?
1. Is AR best developed with democratization in mind, or
as a high-end product eventually declined for themasses? [think graphic cards]
2. And who should do the bulk of the research on this new
media? public or private sector? [think cinema analogy]

www.alvarocassinelli.com“ Augmented Reality as the graphic front-end of Ubiquity.
And Ubiquity as the killer-app of Sustainability.”, B. Sterling (Wired Blog) Dynamic Duo: Multi-Devices Int.

Morten Fjeld
Professor of HCI
t2i Lab, Interaction Design Division
CSE, Chalmers, Sweden
t2i Lab @ www.t2i.se!
Tommaso Piazza, Morten Fjeld, Gonzalo Ramos, Asim Evren Yantac, and Shengdong Zhao. 2013. Holy smartphones and tablets, Batman!: mobile interaction's
dynamic duo. In Proceedings of the 11th Asia Pacific Conference on Computer Human Interaction (APCHI '13). ACM, New York, NY, USA, 63-72.
DOI: https://doi.org/10.1145/2525194.2525205.

Connecting distributed teams; telepresenceDistributed physical tasks; physical presence=> Computer vision=> Augmentation techniques

Khanh-Duy Le, Morten Fjeld, Ali Alavi, and Andreas Kunz. 2017. Immersive environment for distributed creative collaboration. In Proceedings ofthe 23rd ACM Symposium on Virtual Reality Software and Technology (VRST '17). ACM, New York, NY, USA, Article 16, 4 pages.

Khanh-Duy Le, Kening Zhu, and Morten Fjeld. 2017. Mirrortablet: exploring a low-cost mobile system for capturing unmediated hand gestures in remotecollaboration. In Proceedings of the 16th International Conference on Mobile and Ubiquitous Multimedia (MUM '17). ACM, New York, NY, USA, 79-89.

DOI: https://doi.org/10.1145/3139131.3139163.

DOI: https://doi.org/10.1145/3152832.3152838.

3

Using Crowdsourcing for Scientific Analysis
of Industrial Tomographic Images

4

Smart and actuated work environments=> Computer vision=> Actuated screens

IPSJ Interaction (2017, URL: http://www.interaction-ipsj.org/proceedings/2017/data/pdf/17INT001.pdf.
IPSJ Best Paper Award 2017.

Chen Chen, Paweł W. Woźniak, Andrzej
Romanowski,
MohammadObaid,Tomasz Jaworski, Jacek Kucharski,
Krzysztof Grudzień, Shengdong Zhao,
and Morten Fjeld. 2016. Using
Crowdsourcing for Scientific Analysis ofIndustrial Tomographic Images. ACM
Trans. Intell. Syst. Technol. 7, 4, Article
52 (July2016), 25 pages.
DOI: https://doi.org/10.1145/2897370.

56

IPSJ Interaction 2018, URL: http://www.interaction-ipsj.org/proceedings/2018/data/pdf/INT18015.pdf.

Wearable devices & smart environments

Wearable devices & smart environments

Rocsole Inc. Flow Watch

Kening Zhu, Morten Fjeld, and Ayça Ünlüer. 2018. WristOrigami: Exploring Foldable Design for Multi-Display Smartwatch. In Proceedings ofthe 2018 Designing Interactive Systems Conference (DIS '18). ACM, New York, NY, USA, 1207-1218.
77
DOI: https://doi.org/10.1145/3196709.3196713

Kening Zhu, Morten Fjeld, and Ayça Ünlüer. 2018. WristOrigami: Exploring Foldable Design for Multi-Display Smartwatch.
In Proceedings of the 2018 Designing Interactive Systems Conference(DIS '18). ACM, New York, NY, USA, 1207-1218.
8/12
DOI: https://doi.org/10.1145/3196709.3196713.

Professorship of Human-Computer Interaction in the Internet of Things
Prof. Dr. Jens Grubert et al.

Professorship of Human-Computer Interaction in the Internet of Things
Prof. Dr. Jens Grubert et al.

Jens Grubert

Coburg

10 years of AR experience
2 PhD students
2 children
Since 2016:
Associate Professor for Human-Computer Interaction in the Internet of Things
at Coburg University of Applied Sciences
Previous Positions:
University of Passau (PostDoc at Matthias Kranz‘s group), Graz University ofTechnology (PhD at Dieter Schmalstieg‘s group), Fraunhofer Society, University of
Magdeburg.

Professorship of Human-Computer Interaction in the Internet of Things
Prof. Dr. Jens Grubert et al.

2Professorship of Human-Computer Interaction in the Internet of Things

Prof. Dr. Jens Grubert et al.
Stationary Displays

Motivation

Need for efficient interaction methods for AR/VR
On-Body Displays

Handheld Displays

Pimp existing commodity
devices to utilize them for
AR/VR interaction
04.06.2018

3

Professorship of Human-Computer Interaction in the Internet of Things
SpatialInputusing Smartphones
Prof. Dr. Jens Grubert et al.

Professorship of Human-Computer Interaction in the Internet of Things
Prof. Dr. Jens Grubert et al.

Spatial Input using Smartphones
Idea:
use eye reflections to enlarge sensing area of front camera[Grubert, J., Ofek, E., Pahud, M., Kranz, M., & Schmalstieg, D.. GlassHands: Interaction Around Unmodified Mobile Devices Using Sunglasses. In Proc. ACM ISS ‘16, 215-224][Grubert, J., Ofek, E., Pahud, M., Kranz, M., & Schmalstieg, D. (2016). Towards Interaction Around Unmodified Camera-equipped Mobile Devices. In CS. W. at ACM ISS ‘16]
[Schneider, D., & Grubert, J. (2017). Feasibility of Corneal Imaging for Handheld Augmented Reality. In Adj. Proc. IEEE ISMAR ‘17, 44-45][Schneider, D., & Grubert, J. (2017). Towards Around-Device Interaction using Corneal Imaging. In Proc. ACM ISS ‘17, 287-293]

Professorship of Human-Computer Interaction in the Internet of Things
Fishtank
VR using
Smartphones
Prof. Commodity
Dr. Jens Grubert et al.

Professorship of Human-Computer Interaction in the Internet of Things
Combining
HMDs
Smartphones
Prof. Dr. Jenswith
Grubert et al.

Body-Proximate Display Ecologies[Grubert, J. & Kranz, M. Headphones: Ad hoc mobile multi-display environments through head tracking. In Proc. ACM CHI ’17, 3966-3971][Grubert, J. & Kranz, M.. mpCubee: Towards a mobile perspective cubic display using mobile phones. In Proc. IEEE VR ‘17 IEEE, 459-460]

04.06.2018

8[Grubert, J., Heinisch M., Quigley, A. & Schmalstieg, D.. MultiFi: Multi Fidelity Interaction with Display on and Around the Body. In Proc. ACM CHI ’15, 3933-3942]

Professorship of Human-Computer Interaction in the Internet of Things
Prof. Dr. Jens Grubert et al.

Professorship of Human-Computer Interaction in the Internet of Things
Prof. Dr. Jens Grubert et al.

Text Entry in VR

Text Entry in VR?
[Grubert, J., Witzani, L. Ofek, E., Pahud, M., Kranz, M., & Kristensson, P.. GlassHands: Effects of Hand Representations for Typing in Virtual Reality. Proc. IEEE VR 2018][Grubert, J., Witzani, L. Ofek, E., Pahud, M., Kranz, M., & Kristensson, P.. Text Entry in Immersive Head-Mounted Display-based Virtual Reality using
Standard Keyboards. Proc. IEEE VR 2018][Grubert, J., Witzani, L. Ofek, E., Pahud, M., Kranz, M., & Kristensson, P.. GlassHands: Effects of Hand Representations for Typing in Virtual Reality. Proc. IEEE VR 2018][Grubert, J., Witzani, L. Ofek, E., Pahud, M., Kranz, M., & Kristensson, P.. Text Entry in Immersive Head-Mounted Display-based Virtual Reality using
Standard Keyboards. Proc. IEEE VR 2018]

Researches
Occlusion Leak Compensation

Takumi Hamasaki

HySAR

Keio University, Japan

2nd-year graduate student
AR / MR / Optical see-through HMD

Itoh, Yuta, Takumi Hamasaki, and Maki Sugimoto. "Occlusion Leak
Compensation for Optical See-Through Displays using a Single-layer
Transmissive Spatial Light Modulator." IEEE transactions on visualizationand computer graphics 23.11 (2017): 2463-2473.

Hamasaki, Takumi, et al. "HySAR: Hybrid Material Rendering by an Optical
See-Through Head-Mounted Display with Spatial Augmented Reality
Projection." IEEE transactions on visualization and computer graphics 24.4
2(2018): 1457-1466.

Occlusion Leak Compensation

Problem of the Simple LCD Approach

improves the realism of See-Through AR

Accommodation mismatch makes occlusion blurredNo
occlusion
No occlusion

Naiveocclusionmask

Expanded
occlusionmask

Insufficient occlusion

Leaked occlusion

Virtual object
3

4

MethodHySARSingle LCD + Image compensation Æ Sharp occlusion

reproduces the both components over multiple viewers

Viewpoint-independent (VI)

Viewpoint-dependent (VD)Real

OST-HMD

Result5

6

Problem of ordinary SAR

Ideal SAR

Projected images are viewpoint independent

Images must be tuned to the position of each viewer

7

8Results

MethodProjector (diffuse only)+
Projector (SAR)

OST-HMDHySAR

Phong shader+

Blinn-Phong shader

Ward shader

VI component

VD component

VI + VD component
9

Kajiya-Kay shader

10Recent Studies• AR for Remote Collaboration• Can Eye Help You?: Gaze Visualization [CHI2016]• Computer Vision for Video Browsing and Annotation• Ego Scanning: Adaptive Video Fast-forwarding with
Automatic Analysis and User Input [CHI2017]

Keita Higuchi• Visualizing Gaze Direction for Video-based Evaluation on
Assessment for Children with Autism Spectrum Disorder[IUI2018]

1. The University of Tokyo
2. CMU Robotics Institute

Background: Asymmetric Remote Collaboration

Gaze visualization for Remote Collaborationc

Support a local worker by remote collaborator’s instructions
How to transmit instructions?
Workspace

Workspace view

Instructions
Local Worker

Wearable System of Gaze Visualization

Remote Collaborator

How to browse long and redundant first-person videos

EgoScanning Interface

Similar Gaze Behavior Search by Registration of Sampes

Gaze Visualization for Video Coding of Social Attention

Teresa Hirzle
Ulm University

Human-Computer Interaction Group

Samsung Gear VR + Pi Cam

eye Tracking
Mobile Eye

Raspberry Pi (35 fps, 400x300)

2D
Samsung Galaxy S6 Edge+

3D

Monocular Eye Tracking

Eye-based Interaction in AR
F. Geiselhart, M. Rietzler, and E. Rukzio, „EyeVR: low-cost VR eye-based interaction,“Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct. ACM, 2016.

VRSpinning

Eye-based Interaction in AR

VRSpinning: Exploring the Design
Space of a 1D Rotation Platform
to Increase the Perception of SelfMotion in VR
M. Rietzler, T. Hirzle, J. Gugenheimer, J. Frommel, T. Dreja und E. Rukzio, „VRSpinning: Exploring the Design Space of a 1D
Rotation Platform to Increase the Perception of Self-Motion in VR,“ Designing Interactive Systems, June 2018.

WatchVR: Exploring the Usage ofa Smartwatch for Interaction in
Mobile Virtual Reality

WatchVR
T. Hirzle, J. Rixen, J. Gugenheimer und E. Rukzio, „WatchVR: Exploring the Usage of a Smartwatch for Interaction in Mobile
Virtual Reality,“ Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, p. LBW634, 2018.

Sharing Environments

Takeo Igarashi

Spatial Interaction
For Situated Design(The University of Tokyo)
User Interface / HCI
Making computers easier to use.
Computer Graphics
Supporting creativity.

Use AR/VR for design and creation.

CHI 2011

Actuated Puppet Input Device

Wataru Yoshizaki, Yuta Sugiura, Albert C. Chiou, Sunao Hashimoto,Masahiko Inami,
Takeo Igarashi, Yoshiaki Akazawa, Katsuaki Kawauchi,Satoshi Kagami, Masaaki Mochimaru• Puppet input device with actuators• Supported by human body constraints

VRST 2013

Facetons: Face Primitives with Adaptive Bounds for
Building 3D Architectural Models in Virtual Environment
Naoki Sasaki, Hsiang-Ting Chen, Daisuke Sakamoto, Takeo Igarashi• 3D modeling in VR.
• Place face primitives in empty space
puppetfacetonTEI 2012(e-tech 2015)

Dollhouse VR: An Asymmetric Collaborative
Environment for Architecture-scale Space Design
Hikaru Ibayashi, Yuta Sugiura, Daisuke Sakamoto, Natsuki Miyata, Mitsunori Tada
Takashi Okuma, Takeshi Kurata, Takeshi Shimmura, Masaaki Mochimaru, Takeo Igarashi

Occupant

Situated Modeling: A Shape-Stamping Interface
with Tangible Primitives

Manfred Lau, Masaki Hirose, Akira Ohgawara, Jun Mitani, Takeo Igarashi

Designer• Collaborative design of a living space• See-through ceiling / Giant’s finger• Use physical “proxies” for AR modeling.
dollhousevr

video

TEI 2012

DressUp: A 3D Interface for Clothing Design with a
Physical Mannequin
Amy Wibowo, Daisuke Sakamoto, Jun Mitani, and Takeo Igarashi

Visual Computer

Lighty: An Actuated Room Lighting
System with a Painting Interface
Seung-tak Noh, Sunao Hashimoto, Daiki Yamanaka,
Masahiko Inami, Takeo Igarashi,• The user “draws” surface around physical manikin.

The user control room lighting with painting
lightyvideo

Graffiti Fur

UIST 2014

SIGGRAPH 2015

Building made of Million Chopsticks(Collaboration with architecture department)

Yuta Sugiura, Takeo Igarashi, Masahiko Inami

Making an ordinary carpet into a display

Workers drop chopstick guided by computers.
furYuta Itoh

Tokyo Institute of Technology

Yuta Itoh“Overwrite the world in a smart way”

“Overwrite the world in a smart way”

See-Through HMD
AR/VR, Augmented Human

Optical See-Through HMDs

Tokyo Institute of Technology

See-Through HMD
AR/VR, Augmented Human

Calibration of OST-HMDs→Virtual images into the real view

Itoh, Y., & Klinker, G.
(2013, IEEE 3DUI)

Calibration of OST-HMDs

Vision Augmentation via OST-HMD

Itoh et al. Augmented Human 2016 & SIGGRPAH 2016 E-tech

An issue (the nature) of OST-HMDs

Occlusion-Capable OST-HMDs

Additive（＋）image construction

Transmissive LCD-> shut backgroundscene lightIEEE TVCG (presented at ISMAR 2017) “Occlusion Leak Compensation for Optical See-Through Displays
using a Single-layer Transmissive Spatial Light Modulator,” Yuta Itoh, Takumi Hamasaki, Maki Sugimoto.

Occlusion-Capable OST-HMDs

Occlusion-Capable OST-HMDs

Transmissive LCD-> shut backgroundscene lightIEEE TVCG (presented at ISMAR 2017) “Occlusion Leak Compensation for Optical See-Through Displays
using a Single-layer Transmissive Spatial Light Modulator,” Yuta Itoh, Takumi Hamasaki, Maki Sugimoto.

PechaKucha-style self introduction

Interactive Content Design Lab

Yoshiki Kudo
2nd year graduate student
Spatial design/
spatial user interface/entertainment content

http://www.icd.riec.tohoku.ac.jp/

PechaKucha-style self introduction

1

Various Workspace design and approach2

PechaKucha-style self introduction

Dynamics in common working style

Focused, private, quiet
workspace



Large screen forpresenter, circulararrangement etc.

Face-to-face formation,
large space for inviting
more people etc.

Different workspaces and display configurations desired for efficient work
PechaKucha-style self introduction

3

Traditional Solution

4

PechaKucha-style self introduction

Dynamic and Adaptive Workspace
More Active, Enjoyable,
Efficient, Comfortable, …

Manual furniture layout rearrangement→Need additional effort and time

Content,
Displays, Furniture,
Walls, …..

Verbal and/or
Nonverbal behaviors

ActiveSpace
Change rooms→Only if there is enough office space

Sensing

DesignInterior, Interaction
Information, …..

PechaKucha-style self introduction

5

Awareness

Motion,Bio-information,
Brain waves, …

Intention, Situation,
Atmosphere, ...

PechaKucha-style self introduction

6MovemenTable

Reconfigurable Large Interactive Tabletops

MovemenTable

PechaKucha-style self introduction

7

Shapeshifting wall display

PechaKucha-style self introduction

PechaKucha-style self introduction

8

PechaKucha-style self introduction

10

Living wall9

Kai’s Background
Associate Professor
Research Assistant Professor

My Research Group (and Friends)

Graduate School of Media Design, Keio University
Osaka Prefecture University

Visiting Researcher at MIT Media Lab
phD in Ubiquitous/Wearable Computing (2011)

MIT, Cambridge, USA
University Passau, Germany

Collaborations with PARC, SUN, Deutsche Börse …

Palo Alto, Grenoble, Frankfurt,Innsbruck

Smart Eyewear - Kai Kunze

My Workplaces

Smart Eyewear - Kai Kunze

1

2

J!NS Office
Research and Development for J!NS

Shibuya Lab, Keio Media Design
Workspace, Workshops with
Companies etc.

Hiyoshi Campus, Keio University
Teaching, Mentoring etc.

Smart Eyewear - Kai Kunze

http://eyewear.pro

Living Lab, Miraikan, Japanese Science Museum
Public Outreach, Workshops, Talks, Prototyping

3

!4

5

!6

How to edit our body schema?

Unconstrained Neck: Omnidirectional Observationfrom an Extra Robotic Neck, Lichao Shen, Yamen
Saraji, Kai Kunze, Kouta Minamizawa (Keio University)
Augmented Human 2018

Smart Eyewear - Kai Kunze

7

Smart Eyewear - Kai Kunze… now to sth. completely different

Under Review: How to edit our body schema?
Collaboration with Inami Sensei, Minamizawa Sensei
Yamen Saraiji and Tomoya Sasaki

Superhuman Design Challenge

Interest in Artificial Arm Proprioception

July 2-5, 2018 in Delft, Netherlands
Co-Organizers:
Stephan Lukosch (TU Delft)Kai
http://superhuman-sports.org/delft/

Smart Eyewear - Kai Kunze

8

9

Smart Eyewear - Kai Kunze

Smart Eyewear Workshop at
UbiComp (8th October)
Co-Organizers:
Christian Holz (Microsoft), Paul
Lukowicz (DFKI), Yuji Umea (J!NS)
Olivier Augerau (Osaka Prefecture
University), Benjamin Tag (KMD)
Keynotes: Thad Starner (Georgia
Tech), Ozan Camazikc (Google X)

10•2001-2007:
Bauhaus University Weimar•2008-2013:
Graz University ofTechnology

TOBIAS•2014-present:
University of Otago•Internships:•Osaka University•HITLabNZ•Lancaster University•Tokyo Institute ofTechnology

Tobias Langlotz
Human-Computer Interaction Laboratory (www.hci.otago.ac.nz)
University of Otago, New Zealand

Spatial Augmented Reality

Video-See Through Augmented Reality

Optical-See Through Augmented Reality

Oliver Bimber, Franz Corian, Alexander
Kleppe, Erich Bruns, Stefanie Zollmann, andTobias Langlotz, Superimposing Pictorial
Artwork with Projected Imagery
IEEE MultiMedia, Volume 12, Number 1, Jan.March 2005, 2005, 16-26.

Tobias Langlotz, Daniel Wagner, Allesandro Mulloni, andDieter Schmalstieg, Online Creation of Panoramic
Augmented Reality Annotation on Mobile Phones, In IEEE
Pervasive Computing 2012

Tobias Langlotz, Jonathan Sutton, Stefanie Zollmann,
Yuta Itoh, and Holger Regenbrecht (2018)
ChromaGlasses: Computational Glasses for Compensating
Colour Blindness
ACM CHI'18 Conference on Human Factors in
Computing Systems (ACM CHI 2018), Montreal, 2018,

University of Otago, New Zealand

8
01
I2 PH
H
A
C R
M G
AC SIGof M
ts C
Be at A

Computational Glasses Vision Augmentation Beyond Traditional Glasses

ChromaGlasses Computational Glasses for Compensating Colour Blindness

tobias langlotz, jonathan sutton, yuta itoh

tobias langlotz, jonathan sutton, stefanie zollmann, yuta itoh, holger regenbrecht

Immersive Telepresence and Remote Collaboration using
Mobile and Wearable Devices
tobias langlotz, jacob young, jörg müller, matthew cook, holger regenbrecht

Immersive Telepresence and Remote Collaboration using
Mobile and Wearable Devices
tobias langlotz, jacob young, jörg müller, matthew cook, holger regenbrecht

Katsutoshi Masai
Ph.D student at Keio University- other belongings: JSPS DC2 Fellow / JST
CREST- interest: HCI, wearable computing,human behavior sensing

Facial Gestureand Smart
EyewearEye Gesturesand Facial
Expresion

Posed
Expression

Accuracy
Dynamic time warping+ nearest neigbhor

９２.９% Average perindividual training

Measuring
Reading
Activity
Example of applying
ICA to the data while
the user reading an
English joke.

Accuracy ofdetecting the
rubbing to 10
areas

Input byrubbing Face
[Augmented Human 2018
(Short Paper)]]

AH2017.mp4

K E I G O M AT S U M O T O
Ph.D student at the University of Tokyo
JPSP DC1 Fellow

REDIRECTED WALKING

Nescher, Thomas, Ying-Yin Huang, and Andreas Kunz. "Planning redirection techniques for optimal free walking
experience using model predictive control." 3D User Interfaces (3DUI), 2014 IEEE Symposium on. IEEE, 2014.

VISUO-HAPTIC INTERACTION

Yuki Ban, Takuji Narumi, Tomohiro Tanikawa, Michitaka Hirose : “Magic Pot: Interactive Metamorphosis of
the Perceived Shape”, SIGGRAPH ’12: ACM SIGGRAPH 2012 Emerging Technologies, Aug. 2012

UNLIMITED CORRIDORI N F I N I T E S TA I R SSIMPLE DEMO

https://news.ameba.jp/entry/20180427-542

—-Y.Sato Laboratoryl”

Future Localization in First-Person Videos (CVPR2018)�
Key observations

u
g
n

f

ll
ロCan extract

h
t

゜m m
゜
e
f
゜e
c
゜

口First-Person v,s,on ,n ass,st,ve technology
ロ

t
s
a
s

Motivation

.

•

.. ..
.
..
項震 . . . .
- .. - ,''"�
.. . .
.���、 ,�..、- - 笠�長

Problem Setting

、

►

、

.

·•

.

"•

·1:;·, .. 1,
t
a
c

。cn

Input: location history of the target

I

..

―
―

― �� 咤'",, ...,, r >1 l--•,
.

-

Coo,
Decoo,＊ヽ-. -. -、 ＊Output sequence

Output: future location of successive frames, ri:, , ,,Futureしocaじzation in First-Person Videos (CVPR2018',,, ,,, , ,,'' . .. , ,.,ぐ，、）守..... ..

...

1 second prediction

,..

Input

�
.，，し、、'"" ·'.. �!'
r:--:..::.

ー· XーベXGT••• NNeighbor• •• Social LSTM••• Proposed冒〗；靡〗I;誓悶応信�t 〗t�i い�I喜l〗�

byしc�rnlllg

崎

Key idea
The transition of human attention 1s deeply related to the undergoingtask and can be explrnted to improve gaze prediction
MethodMerge
andInput video and Optical Flow

extracted from

二三：◄

•

•

Alex Olwal, Ph.D.
olwal.com

Technologies > Interaction• Augmented Reality• Dynamic Physical Form
• Electronic Textile & Skin
• Medical User Interfaces

Sr Research Scientist, Google
Lecturer, Stanford
Docent, KTH
Previously:• MIT Media Lab• Columbia University• KTH - Royal Institute of Technology• Microsoft Research• University of California• Rhode Island School of Design

MATTER

PHOTONS

Spatial optical seethrough display

Handheld display

Head-worndisplay

Retinal display

Body-worn projector

Body

3D window foraugmented reality

Interactive pixels
in mid-air

Mobile displays as
magic lenses

Dynamic physicalinterfaces

Head-worn
projector

MatterHandheld projector

REALISM
Spatially aligned
projector

Dynamic Physical Affordances

ButtonSlider (1D)Touch Surface (2D)HandleDynamic Physical Constraints

Well

Slot

Ramp

Surface1D shape display

Robotic, electronic façade signage + crowd-sourced weather sayings
Albin Karlsson and Alex Olwal

31μ4 μm mTouch Electrode

Shaper

Autocorrecting smart tools

LiftwareSelf-stabilizing utensils for tremor

Motoric

Addition / Subtraction
MatterBody

VisualEL Display

46
μm

•
•
•

Creating Large Scale, High-Fidelity
Collaborative AR Experiences

What you may know me for…
2014&15

ROMPAPAS DAMIEN CONSTANTINE
D3 STUDENT – 1661032
25 YEARS OLDADELAIDE, SOUTH AUSTRALIA, AUSTRALIA2016

INTERACTIVE MEDIA DESIGN LABORATORY, JAPANRESEARCH THEMES -> AUGMENTED REALITY IN ENTERTAINMENT &EDUCATION
Email: hyperlethalvector92@gmail.com

Fidelity

2017

Scale

[Registration]✔ Large Scale ✖ Fidelity

Vs

Contribution

Lessons learned
designing:

Scale

My work-Thomas, B., Close, B., Donoghue, J., Squires, J., De Bondi,
P., Morris, M. and Piekarski, W., ISWC 2000

✖ Large Scale ✔ Fidelity

Fidelity
Microsoft, “Roboraid”

SystemInteraction

G

M1 P1

P2
M2

Improved Precision
MountedCameraHoloLens Out of Box

My System

Future Work => Design
Encourage betweenplayer Interactions?

~20m

HoloLens
Center of ~40m
Estimated
Pose(Virtual
Crosshair)
~60m

~80m

Come join the fun during the demos!

Encourage largescale gameplay?
Prevent users fromentering unplayable
areas?

1 of 10

2 of 10

Enrico Rukzio (University of Ulm, Germany)

Enrico Rukzio (University of Ulm, Germany)

Ulm
Head of HCI group (15 PhD students)

15 years in Human-Computer-Interaction
5 years in Augmented and Virtual Reality
Best paper awards at CHI, EuroVR, ISWC,
ITS, MobileHCI and MUM
Many industry collaborations, e.g. with Airbus,
BMW, Bosch, Daimler, e.Solutions, MAN, Zeiss
Enrico Rukzio
Ulm University
Germany

Ulm Minster: world's
highest church steeple

3 of 10

Enrico Rukzio (University of Ulm, Germany)

4 of 10Birthplace ofAlbert Einstein

Enrico Rukzio (University of Ulm, Germany)

Ulm University

Research Methodology- Located at Donkey Hill in Ulm– Approach and scientific contributions•- Part of Ulm Science City (inspired by Tsukuba Science City)

Focused problem or opportunity in interaction, service,
application, or hardware- Highest-located University in Germany and European Union (?)•-

Development of corresponding concepts:
theoretical* contributions•

Iterative development of prototypes (incl. hardware):artefact* contributions•

Evaluation via user studies (field or laboratory):
empirical* research contributions– “People who are really serious about software shouldmake their own hardware.” (Alan Kay, 1982)* Jacob O. Wobbrock and Julie A. Kientz. Research contributions in human-computer interaction. ACM interactions. 2016.

5 of 10

Enrico Rukzio (University of Ulm, Germany)

VR: Need for co-located collaboration support

6 of 10

Enrico Rukzio (University of Ulm, Germany)

VR: Need for co-located collaboration support

Oasis. 2017 TV Series. Amazon Video. https://www.amazon.de/Oasis/dp/B06WPB55YF

7 of 109 of 10

Enrico Rukzio (University of Ulm, Germany)

8 of 10

Enrico Rukzio (University of Ulm, Germany)

Input Capabilities: FaceTouch

Need for co-located collaboration support: FaceDisplay

Jan Gugenheimer, David Dobbelstein, Christian Winkler, Gabriel Haas, Enrico Rukzio. FaceTouch:
Enabling Touch Interaction In Display Fixed UIs for Mobile Virtual Reality. UIST 2016.

Jan Gugenheimer, Evgeny Stemasov, Harpreet Sareen, Enrico Rukzio. FaceDisplay: Towards AsymmetricMulti-User Interaction for Nomadic Virtual Reality. CHI 2018.

Enrico Rukzio (University of Ulm, Germany)

FaceDisplay: collaboration vs. discomfort / dominance

Jan Gugenheimer, Evgeny Stemasov, Harpreet Sareen, Enrico Rukzio. FaceDisplay: Towards AsymmetricMulti-User Interaction for Nomadic Virtual Reality. CHI 2018.

10 of 10

Enrico Rukzio (University of Ulm, Germany)when design gets personal

http://www.boredpanda.com/this-is-why-you-should-always-checkproduct-dimensions-before-ordering-something-online/

Daniel Saakes saakes@kaist.ac.kr
department of Industrial Design. KAIST

from designing for people…

to designing for a person

a

c

a
b

“up to here”

simulatingabc

ruler gestures use the body to Measureed“from here
a to here”bc

acting gestures
use the body to Simulate
Embodied Thinking Gestures: Use Body to Think

9)VNBO$PNQVUFS*OUFSBDUJPO-BC%BJTVLF4BLBNPUP

1SPG5FUTVP0OPBOE1SPG%BJTVLF4BLBNPUP

3FTFBSDIUPQJDT˖ 6TFS*OUFSGBDFBOE*OUFSBDUJPO%FTJHO˖ )VNBO3PCPU*OUFSBDUJPO%JWJTJPOPG$PNQVUFS4DJFODFBOE
*OGPSNBUJPO5FDIOPMPHZ)$*MBC )PLLBJEP6OJW4FBSDI

%BJTVLF4BLBNPUP

4FBSDI0S

EBJTVLFTBLBNPUP!%BJTVLF4BLBNPUPIUUQTEBJTVLFTBLBNPUPKQ

IUUQIDJMBCKQ

'JFMEUSJBMBUBUSBJOTUBUJPOJO

<)3* *+43>-BSHFTDBMFFMETUVEZBOEBOBMZTJT(FNJOPJE)*<4BLBNPUP )3*ˏ>$PPLJOHXJUI3PCPU<$)*>)PNFSPCPUT

1IZCPUT<%*4>5PPMLJUGPSSPCPUJDUIJOHT

6*SFTFBSDIPO)PNFSPCPUT)VNBO3PCPU*OUFSBDUJPO

7PJDF"VHNFOUFE.BOJQVMBUJPO<.PCJMF)$*>.PCJMF)$*4FM1I<$)*>(SBQIJDBM6TFS*OUFSGBDF

6TFSJOUFSGBDFJOHFOFSBM

$3*45"-<4FJGSJFE 4*((3"1)&UFDI><4FJGSJFE *54ˏ><4FJGSJFE -BWBM7JSUVBM (SBOE1SJYEV+VSZ ><4V[VLJ .PCJMF)$*><:PTIJEB *&&&$(">%FTJHOJOHVTFSJOUFSBDUJPO<.ZSFTFBSDIJOUFSFTUT>

PODPNQVUJOHEFWJDFT.PCJMF XFBSBCMF UBCMFUPQ SPCPUT IVNBOPJE FUD

"373DPVMECFPOFPQUJPOGPSNF CVUOPUBGSJFOETPGBS

<*CBZBTIJ 4*((3"1)"TJB>1975-2005: TU MUNICH
2004: COLUMBIA UNIVERSITY (7 MONTHS)
2005-2007: CANON JAPAN
2008: TU GRAZ (2 MONTHS)
2008-2014: UNIVERSITY OF SOUTH AUSTRALIASINCE 2014: NARA INSTITUTE OF SCIENCE AND TECHNOLOGY

THE REAL VOYAGE OF DISCOVERYCONSISTS NOT IN SEEKING NEW LANDSCAPES,
BUT IN HAVING NEW EYES.
MARCEL PROUSTEDGE-BASED X-RAY

BENJAMIN AVERY, CHRISTIAN SANDOR, BRUCE H. THOMAS. IMPROVING SPATIALPERCEPTION FOR AUGMENTED REALITY X-RAY VISION. IN PROCEEDINGS OF THE IEEE VIRTUALREALITY CONFERENCE, PAGES 79–82. IEEE, MARCH 2009. LAFAYETTE, LOUISIANA, USA.

BURNAR: FEEL THE HEAT

MATT SWOBODA, THANH NGUYEN, ULRICH ECK, GERHARD REITMAYR, STEFAN HAUSWIESNER,
RENE RANFTL, AND CHRISTIAN SANDOR. DEMO AT IEEE INTERNATIONAL SYMPOSIUM ON MIXEDAND AUGMENTED REALITY, BASEL, SWITZERLAND, OCTOBER 2011. BEST DEMO AWARD

TRANSMEDIA CINEMATOGRAPHYHTTPS://WWW.YOUTUBE.COM/WATCH?V=9JPWITVR0GA

DJ CRAZE: NEW SLAVES ROUTINE. 2014

DJ CRAZE: NEW SLAVES ROUTINE. 2014

UNPUBLISHED PROTOTYPE. 12 MAY 2018

MY DAUGHTER (AGE: 2 MONTHS). 21 MARCH 2018

Interactive Media Lab

Augmented Reality in Human-Computer Interaction

Self Introduction
 Maki Sugimoto, Ph.D.
 Associate Professor, Department of Information and Computer
Science, Keio University
 Research Topics
 Interactive Systems

Towards Augmented Real-world Sensing Technologies for AR/VR

Maki Sugimoto
Interactive Media Lab, Keio University

Citation: http://www.nasa.gov/

 Contact:
mailto: sugimoto@ics.keio.ac.jp

 Hobbies CG/Web Design/Programming
 Driving Photograph

Department of Information and Computer Science

Interactive Media Lab

FuwaFuwa: Detecting Shape Deformation on Soft Object
Using Directional Photoreflectivity Measurement

Embedded Optical Sensors

Huge Information

LimitedInformation

Wide Target

Limited Target

General Purpose

Specialized
PurposeInteractive Media Lab

Facial expression mapping

Interactive Media Lab

Interactive Media Lab

Interactive Media Lab

Interactive Media Lab

嶋崎嵐，杉浦裕太，伊藤勇太，三上弾，木村聡貴，杉本麻樹，
反射型光センサとばねを用いた身体膨張の計測，第21回日本バーチャルリアリティ学会大会論文集，2016

Interactive Media Lab

BYU-BYU-View: a wind communication interface

Erika Sawada, Shinya Ida, Maki Sugimoto, Masahiko Inami, et al.

Interactive Media Lab

BYU-BYU-View: a wind communication interface

Erika Sawada, Shinya Ida, Maki Sugimoto, Masahiko Inami, et al.

Kendo Agent System ~2014 (Doctor Student)

Yasufumi Tanaka (University of Tokyo)● 2010: Tohoku University¾ Assembly Task Partner Robot (Master)¾ Attack Motion Prediction on Kendo (Doctor)

Computer 2
Virtual space

Computer 1
Real space(Offline)

Measuring human motion
using Motion Capture System● 2015: ZMP Inc.
¾ Autonomous Car, SLAM
● 2016: Sony Corp.
¾ Reinforcement Learning● 2017: Freelance Robot Creator

赤色Player’s
avatar● 2018: Project Researcher at Inami Lab.

青色kendo Agent
Player (Human)
Measurement data

Kendo Agent System ~2014 (Doctor Student)
Computer 2
Virtual space

3D-Printed Robot -2017- (Freelance)

Computer 1
Real space(Offline)

Measuring human motion
using Motion Capture System

赤色Player’s
avatar青色kendo Agent
Player (Human)
Measurement data

3D-Printed Robot -2017- (Freelance)

Yasufumi Tanaka (University of Tokyo)● 2010: Tohoku University¾ Assembly Task Partner Robot (Master)¾ Attack Motion Prediction on Kendo (Doctor)● 2015: ZMP Inc.
¾ Autonomous Car, SLAM
● 2016: Sony Corp.
¾ Reinforcement Learning● 2017: Freelance Robot Creator● 2018: Project Researcher at Inami Lab.

A
A:

@

:

D

@

.

: E

/1

I:E
EE

H

H :
?

?

>D

A

:J: E

0

0

A -

AD

0

10

,



-

3-point electrooculography

?

?
6

•

&?

 

When people concentrate?

4

4

1, 2

374

374

11-Sep-17

Monday and Friday results
Lowest performance

3

3

Midnight and early morning results
higher performance

11-Sep-17

3

8,4

7 ,4

,

,,0

Work/ove

rlap wit

Medical Manikins

h Prof. G

Physical Shape and Virtual Physiology

erd Bruder
Human Surrogates:
Real,Virtual, and Augmented
Shape, Appearance, and Intelligence

SimBaby™

Laerdal
SimMom™

CAE Healthcare BabySim
PediaSim

METImanGreg WELCH
Professor and Florida Hospital Endowed Chair in Healthcare Simulation

FidelisThe University of Central Florida
College Nursing
Computer Science, CECS
Institute for Simulation & Training
2 June 2018

SimMan® 3G© 2018 Greg WELCH • UCF CON/CS/IST

Physical-Virtual Patient (Adult)

2

Physical-Virtual Patients (2012)

Welch, Daher, Hochreiter, Gonzalez, Cendan, Anderson, Diaz, et al.

Human head shell

Upper/Lower Interchangeable Human-Shaped Rear-Projection “Shells”

IRcamera 3IRcamera 1
IR light
IR light

IRcamera 2IRcamera 4

IEEE Virtual Reality 2015

Projector

MirrorMirror (folded optics)

Projectors

IR Cameras & Lights

Rear-projection
imagery ofpatient

Touch Sensing

HVAC

TECHTILE toolkits

Body
Pulse, heart, etc.
Temperature© 2018 Greg WELCH • UCF CON/CS/IST

Projectors

Audio

AR Training for Combat Casualty Care

Prof. Gerd
BruderTemperature
Control
Pulse
Actuator

4

Jason
Salam
Hochreiter Daher

Physical-Virtual Child
Patient
Shell© 2018 Greg WELCH • UCF CON/CS/IST

3

Active Moulage

UCF Nursing: Laura Gonzalez, PhD, ARNP; Mindi Anderson, PhD, ARNP and
Desiree Diaz, PhD, RN-BC. UCF Medicine: Juan Cendan, MD.
US National Science Foundation 1564065© 2018 Greg WELCH • UCF CON/CS/IST

5© 2018 Greg WELCH • UCF CON/CS/IST

6

Augmented Reality Agentswith Pervasive Awareness, Appearance, and Abilities

Augmented Reality Agentswith Pervasive Awareness, Appearance, and Abilities

AR Alexa: A visual AR
agent integrated with
Amazon Echo

US National Science Foundation© 2018 Greg WELCH • UCF CON/CS/IST© 2018 Greg WELCH • UCF CON/CS/IST

7

8The Future: Allocentric Augmented Reality

Beyond Reality

Light

Virtual & Augmented Reality ➟ Pseudo Reality

Touch

Sound

Motion

Fluid
In

VR/AR + Awareness and Influence ➟ “Real” Reality

Out

Smell
In

Out

Distributed Sensing and Control ➟ Super Reality

Greg, you look very
handsome today!

Your shoe isuntied!

You left your bedroom
window open!

She is aware of me, myhouse, my schedule, myfriends, …

Your mother-in-law just arrived.
She is usually cold so I haveraised the temperature.

In

In
In

She can affect myhome, my appliances,
my car…

She can influence me…!
© 2017 Greg WELCH • UCF CON/CS/IST

Out

9

Out

Out
In

Out

Transform AR from a solely egocentric paradigm to a hybrid ego/allocentric paradigm.
Distributed network of AR Input/Output (ARIO) units carry out multi-modal signal
detection and estimation that enables virtual entities to be aware of real events, objects, andpeople throughout the environment; and multi-modal signal synthesis and display that enables
virtual entities to affect real events, objects, and people throughout the environment.

© 2018 Greg WELCH • UCF CON/CS/IST

10

Nii Shonan Meeting No.135
Augmented Reality in Human-Computer Interaction

Designing
Emotional & Perceptional
Experiences
Shigeo Yoshida
Assistant Professor
The University of Tokyo

Incendiary reﬂection"Pseudo-smile" enhances positive feelingPresented at Augmented Human (2013)
Exhibited at SIGGRAPH Emerging Technologies (2013)
and Ars Electronica (2015)

partner sees:
pseudo-smile

Teardrop glasses

Smart Face / FaceShare"Pseudo-tears" enhances sadness feeling

Exhibited at Ars Electronica (2015)

Artiﬁcial Resemblance Generator
Which is my "actual" face?

partner sees:
pseudo-smile

Being exhibited at ICC (2017-2019)"Pseudo-smile" augments communication

Presented at CSCW (2014) and CHI (2017)

Artiﬁcial Resemblance Generator
Which is my "actual" face?

Being exhibited at ICC (2017-2019)

AR marker
transformed faceinsert a smart phone

AR Board Game

Baketage (バケタージュ) Museum

Perceived shapeoriginaltransformed face
Purchase this board game on the link below:
https://gift10industry.myshopify.com/products/product-4

Actual shape

Actual shape

Perceived shapeComputational Design of Haptic
Equivalent VR Controller•

Transcalibur
Presented at VRST (2017)

Degree in CAD/M
Engineering atUniversity of
Malaya, Malaysia(2009 – 2012)•

Masters inEngineering
Sciences (2013 –
2015)

•

PhD in Keio Media
Design (2015 –2018…hopefully)

Weight moving VR controller for 2D haptic shape rendering

Will be presented at
SIGGRAPH Emerging
Technologies (2018)

Manufacturing

AR Manufacturing

Pai Yun Suen,
Keio University Graduate School of Media Design

Current Interests:
HCI, VR/AR/MR, physiological sensing
VR interaction

Physical Space + Digital Space
Convex Interactions:Physiological Signal-Driven
Virtual Reality in Social Spaces?

Most efficientMore efficient

Motions and gestures that utilize our proxemic andperipersonal space sense to shortcut interactions, both
spatially and through exploration of bodily mapping, cancreate a more efficient and intuitively superior form ofinteraction than what we are used to

Is there a possibility of the next step of interaction to exist that ismore efficient than what we are already used to by ourselves?
Does a shortcut exist, and in what way?

Approach

Convexed Locomotion: Arm-Based Gestures

Arm Swing

Emphasizing secondary
motion over primary motion
while keeping motion within
intimate proxemics- Energy- SpaceYS Pai, et al., ArmSwing: Using Arm Swings for Accessible and Immersive Navigation in AR/VR Spaces, published at MUM 2017

Approach

Convexed Locomotion: Head-Based Gestures

Approach

Convexed Locomotion: Arm-Based Gestures

PinchMove
Remapping primary motion
into secondary limbs while
keeping motion within
intimate proxemics- Time- Space+ Accuracy
YS Pai, et al., PinchMove: Improved Accuracy of User Mobility for Near-Field Navigation in Virtual Environments, accepted at MobileHCI 2018

Approach

Convexed Selection and Activation: Eye Gaze and Muscle

AnyOrbit

EMG + Eye Gaze

Remapping motion to
unrelated limbs andchanging expected output
directly while keeping
motion within peripersonal
space

Testing other forms ofinteraction by reducing
number of motions, while
keeping motion within
peripersonal space- space- motion sickness(2nd author) YS Pai, et al., AnyOrbit: Orbital Navigation in Virtual Environments with Eye Tracking, accepted at ETRA 2018
(2nd author) YS Pai, et al., AnyOrbit: Fluid 6DOF Navigation of Virtual Environments using Orbital Motion, published at SUI 2016 Posters
YS Pai, et al., GazeSphere: Navigating 360-Degree-Video Environment Using Head Rotation and Eye Gaze, published at Siggraph 2017 Posters- Space- TimeYS Pai, et al., Assessing Hands-Free Interaction for VR using Eye Gaze and Electromyography, submitted to Journal of Virtual Reality, Springer

Approach

Convexed Activation: Eye Gaze

Transparent Reality
Shortcutting the order offunction by combining thefirst input with the final
output while keeping motion
within peripersonal space- Space- TimeYS Pai, et al., Transparent Reality: Using Eye Gaze Focus Depth as Interaction Modality, published at UIST 2017 Posters

I do AR stuff too

Mr. Yuchong ZHANG

l Education:•

M.Sc., Research Associate

From Aug. 18

Ph.D. Candidate

Chalmers University of Technology, SE

l Current and Proposed Interests:• Data Science
Predictive Analysis for Medical Data

• July, 16 to Aug. 17 M.Sc.

Machine Learning and Data Visualisation

Nanyang Technological University, SGCurrently: CIL Lab, School of CSE,

Data Mining for Social Medias• Sep. 12 to Jun. 16 B.Eng.
Hangzhou Dianzi University, CN

Nanyang Technological University, Singaporel Work Experience• Apr. 18 to Present Research Associate

Starting Aug. 2018: Marie Curie ITN ESR (PhD-student)

Natural Language Processing• Human-Computer Interaction
AR/VR in Microwave Drying

Nanyang Technological University, SG

Visualisation in Industrial Tomography• Sep. 17 to Apr. 18 Research AssistantUser Interface Design

Singapore Management University, SG

Department of CSE, Chalmers, Sweden

Machine/Deep Learning with HCI• Oct. 15 to Apr. 16 Lab Intern
CNRS/Université Paris Descartes, FR

Date of Birth: 30 Oct. 1994, Nationality: Chinese

2

Predictive Analysis of ICU Data

20.05.18

Visualisation and Analysis• Predict ICU in-hospital Mortality
Modelling and training
Neural Networks• Propensity Scores to analyse
average treatment effect offactors in ICU
Tuning (hyper)parameters
Optimisation
20.05.18

3

20.05.18

4

20.05.18

Human-Computer Interaction

Data Mining and NLP for Social Medias

lTOMOCON: Smart Tomographic Sensors for Advanced
Industrial Process Control
lX-ray tomography: “Seeing” production flow

• Grab data of famous
accounts in Facebook,
Twitter• NLP in posted contents:
sentiment, word2vec......
• Based on current data,
predict upcoming posting
contents• Transfer learning: algorithm
implemented in Twitterfrom learned in Facebook
5

20.05.18

6

20.05.18

Exploring alternative interactions and visualizations• User Interface (UI) alternatives:

Tablet, Handheld AR

Augmented Reality (AR)Virtual Reality (VR)

Tablet, Handheld AR

Augmented Reality (AR)Virtual Reality (VR)7

20.05.18

l Demo in Lappeenranta University of Technology (Finland):
Batch Crystallization=> All three alternatives & 2D visualisation

Gas holdup by EIT *Heat Mapl Demo in Karlsruhe Institute of Technology (Germany):
Microwave Dying of Porous Materials=> Tablet (Handheld AR)

820.05.18

l Batch Crystallization=> All three alternatives & 3D visualisation

VR Heat Map Analytics
https://vrscout.com/news/vr-heat-map-analytics-designers-retailers/

9

20.05.18

10

20.05.18