Visuomotor Influence of Attached Robotic Neck Augmentation
Lichao ShenMHD Yamen Saraiji

Kai Kunze

Keio University
Graudate School of Media Design
shenlc@kmd.keio.ac.jp

Keio University
Graudate School of Media Design
yamen@kmd.keio.ac.jp

Keio University
Graudate School of Media Design
kai@kmd.keio.ac.jp

Kouta Minamizawa

Roshan L Peiris

Keio University
Graudate School of Media Design
kouta@kmd.keio.ac.jp

Rochester Institute of Technology
School of Informationroshan.peiris@rit.edu

ABSTRACTThe combination of eye and head movements plays a major part inour visual process. The neck provides mobility for the head motion
and also limits the range of visual motion in space. In this paper,a robotic neck augmentation system is designed to surmount thephysical limitations of the neck. It applies in essential a visuomotor
modification to the vision-neck relationship. We conducted two
experiments to measure and verify the performance of the neckalternation. The multiple test results indicate the system has apositive effect to augment the motions of vision. Specifically, therobotic neck can enlarge the range of neck motion to 200%, andinfluence the response motion, by overall 22% less in time and 28%
faster in speed.

CCS CONCEPTS• Human-centered computing → Displays and imagers.

KEYWORDS
visuomotor coordination, visual expansion, spatial perception, robotic
neck, human augmentation, wearable
ACM Reference Format:
Lichao Shen, MHD Yamen Saraiji, Kai Kunze, Kouta Minamizawa, and Roshan
L Peiris. 2020. Visuomotor Influence of Attached Robotic Neck Augmentation. In Symposium on Spatial User Interaction (SUI ’20), October 31-November
1, 2020, Virtual Event, Canada. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/3385959.3418460

1

INTRODUCTION

Our visual system plays an important role in perceiving our surroundings, not only imagery information but also spatial information, and the type of reactions we perform are based on the complex
sensory information we perceive. Our eyes capture a limited stereo
visual field of view (FoV) of approximately 120° horizontally and120° vertically. This field can be significantly expanded to cover 360°
Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than theauthor(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specific permission
and/or a fee. Request permissions from permissions@acm.org.
SUI ’20, October 31-November 1, 2020, Virtual Event, Canada© 2020 Copyright held by the owner/author(s). Publication rights licensed to ACM.
ACM ISBN 978-1-4503-7943-4/20/10. . . $15.00
https://doi.org/10.1145/3385959.3418460

Figure 1: Unconstrained Neck System Overview:
the artificial neck system substitutes human neck motion
limits, resulting a wider range of motion.

by combining our visuomotor system that consists of the eye-gaze,neck, torso, and overall body posture movements. The speed toobserve a specific object in our surroundings depends on a variety of factors including the number of the involved visuomotor
elements listed prior. If an object locates within the visual field, a
simple eye movement is sufficient to see the object, however, if it
locates behind us, then a full-body posture change is required tobe performed requiring more time to look at it. Many scenarios
demand rapid response and wide vision, thus achieving a wider andfaster visuomotor system can be beneficial for various application
scenarios, such as surveillance, sports, and entertainment.
To augment the spatial range of visual system, either 1) the fieldof view of the eyes can be expanded using optical methods (suchas wide FoV lenses), or 2) the limited motion range of the bodyand the neck can be altered to accommodate a new visual range.
In this paper, we developed a visuomotor alteration system named
Unconstrained Neck [27], using an artificial neck (Figure 1) that
attempts to expand the motor range as in the second approach.
Unconstrained Neck surpass the mechanical limits of the humanneck and expands the spatial range of the visual system by remapping the original neck motion and by mechatronic modification
to visuomotor coordination. In our user study, we demonstratethat our prototype speeds up locating objects surrounding the user
(covering 360°) without the need to move the entire body. We also

SUI ’20, October 31-November 1, 2020, Virtual Event, Canada

report the efficiency of using our hands to point at the surrounding
objects while using the system. Thus, our main contributions are:
• We present the design and implementation of a neck augmentation system that allows us to redefine the mapping of neckmotion covering 360°, while maintaining visual consistency
for the eyes.
• We report a user study evaluating the effectiveness of the
proposed system in terms of the usability as well as the time
to locate targets in space around the user.
• A second user study is presented that explores the augmentation of the movement of the hands with the operationalparameters of the system.

2

RELATED WORKIn an attempt to expand the spatial range of visual perception, themost direct approach is to extend the original visual field. The visual
field is the observable extent at any given moment. This visual field
can be expanded either by getting the image from a camera or alens with a wider field of view (FOV), or by stacking the images
from multiple camera array to construct wider FoV [21, 22, 24].
Such approaches of direct alteration of the visual system typicallyrequires training and adaptation by the user, which is a widespread
phenomenon in the visual system. Adaptation occurs on multiple
time-scales, ranging from seconds to hours, according to the typeof the change involved and its duration [14, 17].
A conventional solution is to use wide-angle camera, for example a 360-degree panoramic camera. In the project FlyVIZ, J.
Ardouin et al. uses a panoramic image acquisition system to obtainomni-directional vision and then squeezes the whole image into
the visual field of head mounted displays (HMD) [1]. The systemachieves a visual field 360° horizontal and 80° vertical. However,using such approaches requires the user to perform long periods oftraining to adapt to using it. In addition, it can typically eliminate
the stereoscopic vision and thus depth perception when using it.
Works such as [3, 13, 23] proposed the use of wide lenses cameras with video see-through HMD to expand the perceived visual
field. Similarly, wide angle lenses introduces large visual distortion
which as a result reduces depth perception and skewed direction
estimation compared with binocular field [2, 15].
E. Schoop et al. proposed a environment awareness system [26],which utilized deep neural network to locate and recognize objects
around the user through a head-worn panoramic camera. Yet, it
converted the visual information to semantic audio content, andhence the original format was not preserved. Gustafson et al. [10]and Gruenefeld et al. [9] developed methods to view off-screen
objects: using a wedge, and a HMD version of it respectively. In
addition, Kuhl et. al. and Li et. al. explored the effects of minification
and artificially reducing the field of view on the perception of
distance judgements [16, 19, 20].
K. Fan et al. developed the SpiderVision, a handset extends human visual field to mainly augment user’s awareness in the back [7].
The system enables user to focus on front view by the video-seethrough HMD, but it blends the back-view video in only when thesystem detects dynamic visual change. It keeps monitoring the surrounding environment and sets an intellectual trigger to activate
the augmented content. This system requires part of the visual field

L. Shen et al.

to display the back vision, which inevitably reduces the quality ofnormal vision: the image become blurred or narrowed. Its visualaugmentation based on peripheral monitoring and the trigger forthe visual blend is passive for the user so that the reliability of
automatic detection limits the whole system performance.
To address the previous limitations, we propose to use an alternative neck representation, Unconstrained Neck, which enhances
the range of motion our biological neck can achieve. This system
maintains the visual properties of our eyes (FoV and binocular vision) in order to reduce the adaptation time, and to sustain depth
perception when using this system.

3

DESIGN CONSIDERATIONS: VISION-NECK
RELATIONSHIPAs seen from the related work, an established body of work exist
for visual system augmentation using optical solutions. Given theadvantages and drawbacks of such methods, we adopted a different
approach by addressing the visuomotor system limitations. In thissection, we describe the design considerations and implementationof the proposed system.

Extension
53.9°~77.9°

Flexion~53.9°

Right Lateral Flexion
43.7° ~ 46.4°

Left Lateral Flexion
41.5° ~ 43.5°

Left Rotation
67.3° ~ 69.5°

Right Rotation
67.3° ~ 71.7°

Figure 2: Estimates of Normal Cervical Active Range of Motion (ROM), at the age of 30 years.
The visual sense contributes to spatial perception, and thus
involve visuomotor coordination (i.e., human vision and motion
works together). Human eyes have relatively narrow field of viewcompared with animals such as bird or rabbit. The monocular (oneeyed) visual field of a healthy human measures approximately 100°
temporally, 60° nasally, 60° superiorly and 75° inferiorly of eacheye [25, 29], and the cyclopean vision (total) is approximately 200°
wide and 135° tall, with a region of binocular (two-eyed) overlap of
approximately 120° wide [4, 5], which is crucial for stereo visionand depth perception. Hence, our eyes need assistance from therest of body, to expand the spatial range of eyesight, due to thenarrowness of visual field.
The neck influences the visual sense though it is not a visualorgan. It plays an important role in scanning our surroundings.
Scanning is defined as the action shifting the visual focus in a larger
scale, and with the essential purpose to expand the visual field. The
motion of one or some body parts including eyeball, neck, and torso,
are likely to involve in scanning the environment. When scanning,
the eye gaze can be considered as the easiest to manipulate, but theocularmotor range (OMR) of the eye gaze is restricted compared tothe torso and the neck, typically not exceeding 40° to 45° [8]. The
torso (trunk) is able to deform (twist and bend), and thus have larger
ROM. But the torso bears the weight of the whole upper body, andits motions involve several groups of muscles, so the torso is mostVisuomotor Influence of Attached Robotic Neck Augmentation

SUI ’20, October 31-November 1, 2020, Virtual Event, Canadadifficult to action. The neck is at a balanced point: it only bears anddrives the head; its ROM is relatively wide. The neck can perform
six types of movements, and the cervical active range of motion(ROM) varies with the age [30] and the approximate value is asshown in Figure 2. The head motion is directly driven by the neck,and therefore is limited by the range of the neck movements.
Existing research has shown that the behavior of the neck isdependent on the target location when scanning [18]. The eye-only
range (EOR), where no head motion tends to happen, is approx.
46.7°(±23°) [28]. The customary ocular motor range (COMR), where
the eye movements tend to happen with a probability of 90%, isapprox. 57.4°(±29°) [18]. As shown in Table 1, the type of motions
which is very likely to happen, depends on the different final target
position relative to the neutral position. The eye movements only
have a small probability to occur if the intended final position is
outside the COMR, that is, the human is likely to use only neckmovements when the intend rotation is relatively large.
Table 1: Different Motion Types Depend on Target Position
Target Position (T )T < 23°23° < T < 29°29° < T

Motion TypeEyes Only

Eye and Head

Head Only

Thus, to reduce the chain of visuomotor motion, we are focusing
on augmenting the neck with our system. The proposed artificial
neck would alter the ROM of human neck by remapping each axis
to a new motion range as shown in Figure 6. Here our goal is toenhance the visual field by reducing the effort and time required toreach targets beyond our typical visual field.

4 UNCONSTRAINED NECK SYSTEM
4.1 Implementation

In this paper we present our prototype Unconstrained Neck (Figure 3). The prototype contains a 3-DoF (degree of freedom) robotic
neck holding a stereo camera. The robotic neck consists of 3 Dongbu
Herkulex DRS-0201 servo motors, 3D-printed plastic supports / connectors, and other necessary mechanical components. Each motor
drives the rotation about one axis (i.e., yaw, pitch, and roll). A SparkFun Pro Micro-controller is embedded to control 3 servos by wire.
This external robotic neck consists of 3 rotational kinematic pairs;
thus, it has 3 degrees of freedom. An Ovrvision Pro camera set capturing stereo videos by dual lenses and sensors, streams the videos
to an Oculus DK2 / Rift HMD (head-mounted display) worn by theuser, as depicted in Figure 1. The distance between the two lateralplaced lens is 65 mm, close to the human interpupillary distance [6],which mean value is approximately 63 mm. The camera FoV resembles to normal human vision by adjusting the magnification so thatit can create a see-through immersive experience. A laptop running
Unity 3D software maintains wired connections to the camera, themicro controller and the HMD. The spatial position of the camera
mounted at the endpoint of the robotic neck is controlled by a customized developed Unity 3D software and is able to operate about
the 3-axis rotations: yaw, pitch, and roll. The resulting motion ofrotation is thus a combination of the movements of human neckand robotic neck. The robotic neck is attached to helmet to allow
the user to wear the prototype with ease. Due to the height of therobotic neck, the location of the camera set has a vertical offset of
approx. 450 mm from the user’s eye when wearing this prototype.
The overall system stands at approx. 400 mm when worn (from the
top of the head to the top of the camera) and weighs approx. 500 g
with the helmet, and 300 g without the helmet.
The Unconstrained Neck system can be described as a substituteneck system, that is, it controls the orientation of vision 1 with analterable and programmable mechanism. The input of the systemis the original vision with its inherent spatial information, and theoutput is the modified vision with new spatial information (Figure 4(left)). The input vision is typically sourced from human eyes on
the head, so its orientation is equivalent to the head orientation.
The head rotation is driven by the human neck, and thus the input
vision and the human neck can be studied by tracking the head. The
modified vision output is sourced from the camera, so its orientation
is equivalent to the camera orientation. The camera rotation isdriven by the Unconstrained Neck system (the human neck plus therobotic neck), and thus the output vision and the whole system canbe studied by tracking the camera motion. In brief, head orientation
as input and (camera) vision orientation as output.

4.2

Figure 3: Unconstrained Neck system consists of 3 servo motors that control the yaw, pitch, and roll angles, and a stereo
camera mounted atop a helmet

Mapping

The internal mechanism are defined in the controlling software (Figure 4 (right)), mainly focusing on the orientations of vision, while
the imagery content of vision remains unmodified. The spatial orientation of a rigid body can be parametrized by three independent
coordinates, and MEMS gyroscopes usually provide three values ofyaw, pitch and roll. All the orientation and components are represented with respect to the body frame. Thus orientation mappings
1 In this paper, the orientation of vision, eyesight, or visual orientation refers to theorientation of the outward normal vector of human visual field, i.e., the gaze direction
when looking straight forward.

SUI ’20, October 31-November 1, 2020, Virtual Event, Canada

L. Shen et al.

StereoCameraRaw

HMD

Orientation

Yaw
Pitch
Roll

(left)(right)NaturalOrientation

Euler Angle
Conversion

Environment

Eye

Euler Angle
Conversion

Nervous
SystemHead

VisionConfiguration

Human

Yaw
Pitch
Roll

Computer
Robotic
Neck

HMD
SensorMapping

Unconstrained Neck

Figure 4: (left) System I/O and (right) Software Mechanism
about 3 axes can be built respectively, between the orientation of
output vision (derived from camera position, and driven by the necksubstitution system) and the corresponding orientation of input
vision (derived from head position, and driven by the human neck).
A rotation is the change of orientation, so an angular displacement
can be defined as the difference in orientation, or angular position,from the origin to the current position. If the origin of the reference
orientation was set as the neutral position when facing straightforward, the general mappings can be denoted by Function Group 1:

roll:




pitch: yaw:
ϕ O = f 1 (ϕ I )θ O = f 2 (θ I ) (1)ψ O = f 3 (ψ I )Figure 5: (left) Parameter and axis definitions; (right) thesystem in use with augmentation of Linear ×2 mapping([Camera Rotation] = 2 × [Head Rotation])
Corresponding Vision Orientation(Degrees)

180°

90°


roll:




pitch: yaw:
ϕ O = 2ϕ I
θ O = 2θ I

(2)

-90°
90°

ψ O = 2ψ I

where the subscript of I denotes the input and O denotes theoutput. The ϕ I , θ I ,ψ I are the input orientations of vision, identical tothe motion displacements of human head in value, which the humanneck acts; the ϕ O , θ O ,ψ O are the output orientations, identical to theresulting motion displacements of camera vision in value, which the
Unconstrained Neck system actuates. By ϕ O = ϕ I , θ O = θ I ,ψ O = ψ I ,it defines the innate original mapping (i.e. no augmentation).
The prime feature of Unconstrained Neck is the controllable andprogrammable vision-neck relationship. One possible beneficial
application is visual expansion. When the system is worn by a user,
the maximum range of motion of the output vision is −180° to 180°
about each axis (limited by the servo motors and the mechanicalstructure), which is almost twice the range of human neck motion.
Therefore, it can be utilized to amplify or enlarge the neck’s rangeof motion by twice or more, and the human eyesight can hence
achieve omni-directional observation or simply 360° vision (i.e., ascanning range of 360° tall and 360° wide). For the scope of this paper,
we will concentrate on this application scenario. One of the mosttypical and basic augmented mapping can be Linear ×2 (Function
Group 2, Mapping Graph Figure 6): ϕ O = 2ϕ I , θ O = 2θ I ,ψ O = 2ψ I .
For example, as shown in Figure 5 (a) and (b), when the user’s head
(input) is at a yaw angle of Ψ, the resulting camera vision (output)
is at a yaw angle equivalent to 2Ψ due to the combined movementof the human neck and robotic neck. In this case, the eyesight canfully cover a task space of S 2 , with only neck movements.
In this paper, the neck mapping refers to mapping of the wholesystem: head rotation as input, system (camera) vision as output;
the hand mapping was defined similarly: physical hand position as
input, virtual hand position as output. All the following experiments
were conducted with the augmented mapping of Linear ×2 applied,that is, in plain languages, the rotational angles the camera (output)
travelled is always twice (×2) as large as the rotational angles thehead (input) travelled, in both Study 1 and Study 2.

Head Orientation (Degrees)

-90°

-180°

Before Augmentation (Neck)
After Augmentation(using Unconstrained Neck)

Figure 6: Graph of Original/Augmented Mapping (Linear ×2)
for each axis, (x: Input, Orientation of Human Head,y: Output, Corresponding Orientation of (Camera) Vision)

5

STUDY 1: USER EVALUATION OF
UNCONSTRAINED NECK SYSTEM

The main goal of Study 1 was to evaluate whether the users can scan
their surrounding space more efficiently using the Unconstrained
Neck, and to observe its influences on the users. The hypothesis
was that the system could reduce the response time and boost the
response speed for scanning motion with neck involved.

5.1

MethodWe took the scanning as a representation of visuomotor practice,and then measure the perform of the response reaction to evaluate
the effectiveness of the system. We presented several virtual targets
around the subject’s spatial range as shown in Figure 7. As themain task, the subject was required to locate the presented target
as quickly as possible. The targets were generated and presented
in augmented virtual environment through the study software referenced from an external global point in space, to measure thehead rotations and present the target location. Virtual target was
designed to be distinct and distinguishable from the surrounding
environment using color of yellow (Figure 8). As depicted in Figure 7, we presented a total of 20 targets distributed evenly aroundthe origin of the subject’s head. The spatial location of the target
was denoted in the form of (latitude, longitude): 0 latitude is thetransverse plane (horizontal), and 0 longitude is the sagittal plane(vertical); the original point of the coordinate is (0,0) at the rightVisuomotor Influence of Attached Robotic Neck Augmentation

SUI ’20, October 31-November 1, 2020, Virtual Event, Canada

Table 2: Condition Specification: Setup and Mapping / Motion († robotic neck deactivated; ‡ robotic neck activated.)
Abbr.

Setup

Mapping / Resulting Motion

System Drive(Camera) Vision

Virtual Hand

Study 1C1C2Human Neck†
Human Neck + Robotic Neck‡

Original
Augmented (×2)

n/a
n/a

Study 2C3C4C5Human Neck†
Human Neck + Robotic Neck‡
Human Neck + Robotic Neck‡

Original
Augmented (×2)
Augmented (×2)

Original
Original
Augmented (×2)(model: Oculus Touch) were introduced to track the hand locations
and optimize the experimental interface. The experimental setup
was as shown in the Figure 7(top).

Figure 7: Experiment Setup: (top) the subject sat in the experimental field, and (bottom) the distribution of targets along
three different latitudes / tilts: −30°, 0°, +30°)
front of human head which is also the intersecting line of transverse
plane and sagittal plane;
In Study 1, we presented 20 virtual targets in total, of which thelocations are (0, 30), (0, 60), (0, 90), (0, 120), (0, 150), (0, 180), (±30, 0),(±30, 30), (±30, 60), (±30, 90), (±30, 120), (±30, 150) and (±30, 180),
as per Figure 7, where lat. +30° refers to the upper plane, and lat.
−30° refers to the lower plane. The target was a virtual sphere witha radius of 50 mm, and located at a distance of 500 mm from theorigin subject’s head.
5.1.1 Study Design. The study was a within-subject design andconsisted of 2 main conditions (independent variables of the study),
as shown in Table 2. The condition when the subject wore the system but with robotic neck deactivated is Condition 1 (denoted as C1,
with original neck mapping). Under C1, the scanning motion of camera was only as a result of the subjects’ human neck rotations; The
condition when the subject wore the system with robotic activatedis Condition 2 (denoted as C2, with augmented neck mapping). In
C2, the scanning motion of camera was as a result of the rotation of
the subject’s human neck rotations plus the robotic neck rotations
as described in the previous section, and hence the augmentation
effect was on (enhanced by two times, Linear × 2). As for the measurements (dependent variables), the response time to locate andselect the target was recorded. Lastly, the participant answered thesimulator sickness questionnaire [12] after completing each condition. Each target was repeated 4 times; thus, each subject faced atotal of 160 task trails (20 tarдets × 4 repetitions × 2 conditions).
5.1.2 Subjects. 14 participants volunteered to participate in study
1 (9 females and 5 males, age range: 22 to 35 years, mean: 25.6, SD:3.59). The subjects were recruited from the university where theexperiment took place, and consented for the study. All subjects
had prior experience with using HMD and virtual reality.
5.1.3 Apparatus. This study utilized the Unconstrained Neck system as the main apparatus of the study. Plus, two remote controllers

5.1.4 Procedure. After collecting the informed consent and thebiographical data, each subject was briefed on the study. In addition, for safety reasons, the subjects were all queried through aquestionnaire to confirm that their physical condition fulfilled all
required criteria. The criteria were as follows: (1) no current neckpain, (2) no history of any neck medication, (3) no current eyesightillness, (4) no severe virtual reality related sickness, (5) unambiguous communication with the experimenter. Information of virtual
reality and motion sickness was introduced and explained for the
subjects prior to conducting the study.
During the study, the subject sat on a rigid chair, wearing therobotic neck and HMD; the subject held two remote controllers
with action keys. The subject could only move and twist his/her
upper body, neck or torso to scan. At the beginning of task, thesubject placed his/her neck and torso at the neutral position, andkept aiming at the origin point. Before the recording started, thesubject had time to practice and understand the task. Once thetask began, a ‘Find’ command was displayed on the HMD screen
instructing the subject to initiate localizing the target and finish
aiming. After each condition, a break of 5 minutes was provided forcompensation. Specific procedures of the conditions are as follows.
The experiment took approximately 45 minutes per participant.
5.1.5 Task Procedure. As shown in Figure 8, the sight (aimingmark) consisted of a cross and a circle in the center of visual field,
and the virtual target was a yellow 3D sphere anchored in external
coordinate. During the experiment, the participant was required
to move the field of view until the virtual target entered the circle
mark. A text indicator showed the progress of the experiment andthe status of current target. ‘Find’ indicated the presence of a newtarget, while in the meantime ‘< < <’ or ‘> > >’ below indicated thatthe target located in the left or right hemisphere. ‘Back’ meant thetarget had been destroyed, and the subject should resume his/her
position to the original position.
The subject pressed any action key to start the procedure. Oncethe action key was pressed, a target appeared and the text displayed‘Find’. Then the subject should move the field of view, to search forthe target and aim at it. To aim was to overlap the central circle withthe target. Once the aiming was finished, the subject pressed theaction key, and instantly the existing target would disappear andthe text displayed ‘Back’. Then the subject could resume his/her
position of head and torso to the origin, and should aim at the

SUI ’20, October 31-November 1, 2020, Virtual Event, Canada(a) Start

L. Shen et al.

for C2 is 171.99s (SD: 40.762s). It indicates the participants were able
to aim at the virtual targets significantly faster in C2 (augmented
neck mapping) than in C1 (original neck mapping). Similarly, theCondition Speed for C1 is 36.732°/s and for C2 is 46.344°/s. Results
are further analyzed using a paired t-test with α = 0.05. The t-test
results of condition time, t(14) = 3.51, p = .00384, and condition
speed, t(14) = −4.27, p < .001, show statistically significant differVisuomotor Influence of Attached Robotic Neck Augmentation
ences between two conditions, this is, time decreased and speed
increased.

(b) AimCondition, Latitude(C1, -30)(C2, -30)(C1, 0)(C2, 0)(C1, 30)
4(C2, 30)

Target Speed

2

2

20

1

(d) Resume & End

0

30

60
90
120
Target Longitude

150

1

180

(a) y: 150Target
Time(s),300
100
200
250
Time (s), User
x: Target Condition
Longitude(°)(a) Condition Time
8

Condition, Latitude(C1, -30)(C2, -30)

0

30

60
90
120
Target Longitude

150

180

(b)
y: 30Target
Speed(°/s),
20
40
50
60
70
Condition Speed (degree/s), User
x: Target
Longitude (°)350(b) Condition Speed

Condition, Latitude(C1, 0)(C2, 0)Condition, Latitude(C1, 30)(C2, 30)

7

Figure 9: (C1,C2) Box Plot: Mean Condition Time / Speedfor 14 Subjects. Mean of 4 Repetitions. (blue cross: scatteredpoint of each sample, red diamond: outlier, orange solid line:
median, green dash line: mean)
5
4Target Time

Target Time

6

Target Time

Figure 8: Procedure of Aim at the Virtual Targets: the subject(a) start at the initial point, was asked to find the target, (b)searched for the target and aimed, (c) shoot by pressing abutton, then was asked to return to the origin, and (d) ended
one target, then prepared for a next target

2

40
30

1

(c) Shoot70 Condition, Latitude(C1, -30)(C2, -30)(C1, 0)60(C2, 0)(C1, 30)(C2, 30)
50

Condition

3

Condition

Target Time

5

3
2
1

0

5.2

r =1 n=1

150

180

0

30

60
90
120
Target Longitude

150

180

0

30

60
90
120
Target Longitude

150

180

(c) y: Target Time (s), x: Target Longitude (°);

90 Condition, Latitude(C1, -30)(C2, -30)
80

Condition, Latitude(C1, 0)(C2, 0)Condition, Latitude(C1, 30)(C2, 30)

Target Speed

60
50
40

Target Speed

70

30

0

Study 1 consisted of 20 virtual targets that were distributed in thespace around the user, at varying angular distances from the initialstarting location. Therefore, to analyse the results, we initiallyobserved the results of the conditions (C1 and C2) as a whole using
the following definitions. Every participant completed 4 repetitions(R) under each condition (C), and 20 different targets (θ ) under each
repetition (R). “Target Time” (tθ ) is the time costed to complete aim
at one specific target from the initial location, that is, the interval
between the moment one target appears and the moment the next
target appears. “Condition time” (tC ) is defined as the total timecosted to complete all 4 repetitions for all 20 targets and is definedby the Equation 3. In addition, “Condition Speed” is defined as
the average head rotational speed of the whole condition, and issimilarly calculated by using the Equation 4 (∆θ is the angulardisplacement for each target from the initial location).

4 Õ
20
Õ

60
90
120
Target Longitude

20

ResultstC =30

from leftthe
to right:
lat -30,
lat. values
0, lat. +30
Figure 10 indicatestime andspeed
for each target.
Here, the “Target Time” is tθ as defined previously. The “Target
Speed” was defined as ω θ = ∆θt θ . For clarity, the Figure 10 results
are further presented separately by 3 latitude values. The results
are further analyzed using a two-way repeated measure ANOVA. A
significant main effect is found across condition (C1,C2) and acrosstarget, from both Target Time, pcondition < .001, ptarget < .001,
and Target Speed, pcondition < .001, ptarget < .001. Post-hoc pair(d) y: Target Speed (°/s), x: Target Longitude (°);
wise comparison
with Bonferroni and FDR (False Discovery Rate)from left to right: lat. -30, lat. 0, lat. +30
correction between each pairs of the targets (C1,C2) reveal theunevenness of the effect as shown in Table 3. A overall efficacy
Figure 10: (C1,C2) Mean Target Time / Speed for 20 Targets.
ratio index (R) is calculated to quantify the degree of effectiveMean of 4 Repetitions × 14 Subjects. (color area: the errorness, R t ime,C2/C1 = tC2 /tC1 = 0.782 = 78% and Rspeed,C2/C1 =bar with
standard deviation)
ωC2 /ωC1 = tC1 /tC2 = 1.28 = 128%.
Target Speed

original point. When ready, the subject could press the action key
again, and instantly a new target would appear. Then the subject
continued to next target. For this part, a subject was presented
with 4 repetitions of the 20 targets, at a distance of 500 mm. This
study presented conditions of C1 and C2 in a randomized order.
The subject answered the simulator sickness questionnaire (SSQ)
at the end of each condition.

t Θn,r

(3)

ωC =

Í4

r =1

150

180

0

30

60
90
120
Target Longitude

150

180

0

30

60
90
120
Target Longitude

150

180

(b) Target Speed(a) Target Time
Lon.

Latitude

Lon.

-30

0

30

Average

0

0.199

n/a

4.07 E −2

6.48 E −2

30
60

0.351
0.447

0.597
0.721

0.623
9.62 E −2

0.334
0.864

90
120

0.506
0.537

0.500
7.75 E −2

0.154
0.124

150

8.06 E −3

9.23 E −3

180

1.16 E −2

Avg.

8.81 E −3

n=1 ∆Θn,r(4)

The overall results of C1 and C2 from Study 1 are depicted in Figure 9. Here, the Condition Time for C1 is 219.72s (SD: 57.997s) and

60
90
120
Target Longitude

Table
Table 3:3: (C1,C2)(C1,C2) P-values
P-values ofof Posthoc
Posthoc Pairwise
Pairwise T-test
T-test forfor 2020
Targets
Targets//33Latitudes
Latitudes//77Longitudes
Longitudes(*(*naïve
naïvesignificance,
significance,****
FDR
FDRsignificance,
significance,******Bonferroni
Bonferronisignificance)
significance)

Í20

tC

30

∗
∗

∗∗∗

Latitude
-30

0

30

Average07.61 E −2n/a3.80 E −2

4.36 E −230

4.47 E −2

0.518

0.835

0.196

0.213
0.119

60
90
120

0.931
0.369
0.162

0.708
0.336
3.81 E −2

0.112
0.293
6.07 E −2

0.441
0.228
5.60 E −2

3.10 E −3

2.20 E −3150

1.13 E −2

1.15 E −3

1.81 E −3

7.63 E −4

9.51 E −3

6.99 E −4

4.35 E −4180

6.30 E −3

3.98 E −3

2.95 E −4

3.26 E −4

1.16 E −2

4.13 E −3

3.84 E −3

Avg.

1.59 E −2

3.18 E −3

3.99 E −4

1.18 E −3

∗
∗

∗∗∗

∗

∗∗

∗∗∗

∗∗∗

∗∗∗
∗∗∗

∗∗∗

∗

∗∗

∗∗

∗∗∗

∗

∗∗∗
∗∗

∗∗∗

∗

∗∗∗
∗∗∗
∗∗∗

∗

∗∗∗
∗∗∗
∗∗∗

Table 4: (C1,C2) Sickness Questionnaire for 10 Subjects
SSQ Score
C1
C2

SSQ T-Test

Neck Score

Neck T-Test

Mean

SD

Stats.

P-value

MeanSD

Stats.

P-value

430
518

338
332

-1.08

0.309

3.50
1.70

3.06
2.54

2.86

0.0187*t-test sho
existed beload exis
tended toever, as fo
so the au
despite o

5.3

D

The hypoin C2 (ausystem pr
get acqui
the condi
showed tdifference
target spe
target tim
vation fro
to 180° ar
those targ
look at thto C1 whin C2 thehelped fr
achieved
In sumof positiv
augmenta
ingly boo
relief thetation eff
benefit m

6

STU
INV

The maineye coord
strained Ntion wouof tasks i

6.1

M

We condu
Similar t
around t
the maintionally t
two virtu
sphere w
it was po
remote coat a close
the targe

Visuomotor Influence of Attached Robotic Neck Augmentation

Condition, Latitude(C1, -30)(C2, -30)(C1, 0)(C2, 0)(C1, 30)
4(C2, 30)

Target Speed

Target Time

5

3
2

SUI ’20, October 31-November 1, 2020, Virtual Event, Canada

70 Condition, Latitude(C1, -30)(C2, -30)(C1, 0)60(C2, 0)(C1, 30)(C2, 30)
50
40
30

1

20
0

30

60
90
120
Target Longitude

150

180

0

(a) y: Target Time (s),x: Target Longitude (°)
8

30

60
90
120
Target Longitude

150

180

(b) y: Target Speed (°/s),
x: Target Longitude (°)

Condition, Latitude(C1, -30)(C2, -30)

Condition, Latitude(C1, 0)(C2, 0)Condition, Latitude(C1, 30)(C2, 30)754Target Time

Target Time

Target Time

6

3
2
1
0

30

60
90
120
Target Longitude

150

180

0

30

60
90
120
Target Longitude

150

180

0

30

60
90
120
Target Longitude

150

180

150

180

(c) y: Target Time (s), x: Target Longitude (°);
from left to right: lat -30, lat. 0, lat. +30
90 Condition, Latitude(C1, -30)(C2, -30)
80

Condition, Latitude(C1, 0)(C2, 0)Condition, Latitude(C1, 30)(C2, 30)5040Target Speed

60

Target Speed

Target Speed

70

6

30
20
0

30

60
90
120
Target Longitude

150

180

0

30

60
90
120
Target Longitude

150

180

0

30

60
90
120
Target Longitude(d) y: Target Speed (°/s), x: Target Longitude (°);
from left to right: lat. -30, lat. 0, lat. +30

Figure 10: (C1,C2) Mean Target Time / Speed for 20 Targets.
Mean of 4 Repetitions × 14 Subjects. (color area: the errorbar with standard deviation)
Table 4: (C1,C2) Sickness Questionnaire for 10 Subjects
SSQ Score
C1
C2

SSQ T-Test

Neck Score

Neck T-Test

Mean SD Stats. P-value MeanSD

Stats. P-value

430
518

3.06
2.54

2.86

338
332

-1.08

0.309

3.50
1.70

0.0187*

We conducted surveys after C1 and C2 using the simulator sickness questionnaire (SSQ). The data summary is shown in Table 4,
and the higher a score was, the severer a sickness was. The result oft-test shows no statistically significant differences of sickness index
existed between two conditions, but significant difference of neckload existed. Thus, the modification of vision-neck relationship,tended to cause no significant VR / simulator sickness extra. However, as for the influence on neck, the score declined significantly,
so the augmentation was likely to reduce and relief the neck load,
despite of the extra weight and height brought by the system.

5.3

system provided enhanced neck movements resulted in faster target acquisition times (tθ ) than in C1 (original neck mapping), fromthe condition perspective. Further individual target-wise analysis
showed those target locations which contributed to the significant
differences, concentrated around longitude 150° and 180°. As for thetarget speed, its result showed more significant locations than thetarget time. This notion can be considered as an interesting observation from Study 1. We identify that targets around longitude 150°
to 180° are those that are typically beyond our visual field. Hence,
those targets would typically require neck and torso movements tolook at them, and in C2 were aimed significantly faster comparedto C1 where there was no enhancement from the system. However,in C2 the rest targets which were not in this range although were
helped from the system did not yield significant effects and wereachieved with similar effort and performance to C1.
In summary, in Study 1, all the results implied the existenceof positive influence was of high probability. This system and its
augmentation, generally reduced the response time and correspondingly boosted the speed of vision shifting, while in the meantime
relief the neck motion and expand the visual field. The augmentation effect was uneven across targets, and it applied significant
benefit mostly for the motion in rear hemisphere.

Discussion

The hypothesis was proved. These results clearly indicated thatin C2 (augmented neck mapping) where the Unconstrained Neck

STUDY 2: USER EVALUATION OF HAND
INVOLVEMENT

The main goal of this study was to observe the influences of handeye coordination in the surrounding space when using the Unconstrained Neck system. The hypothesis was that the neck augmentation would similarly have a positive influence on the performanceof tasks involved with the hand-eye coordination.

6.1

MethodWe conducted this study similar to Study 1 with a few adjustments.
Similar to previous study, we presented several virtual targets
around the subject’s spatial range as presented in Figure 7. As
the main task, the subject was required to locate and then additionally touch the presented target as quickly as possible using thetwo virtual hands. The virtual hand was represented by a virtual
sphere with a radius of 50 mm, same size as the virtual target, andit was positioned by the spatial position data collecting from theremote controller. We only presented targets at the 0° latitude, andat a closer distance of 300 mm so that the participant could reachthe targets with minimal effort.
In Study 2, we presented 6 virtual targets in total, and the locations are (0, 30), (0, 60), (0, 90), (0, 120), (0, 150), (0, 180); Thetarget was a virtual sphere with a radius of 50 mm, and located at a
distance of 300 mm from the origin subject’s head.
6.1.1 Study Design. The study was a within-subject design andconsisted of 3 main conditions (independent variables). In the system, the resulting neck motion outputs two levels, original neckmapping and augmented neck mapping, and the resulting virtual
hand motion outputs two levels, original hand mapping and augmented hand mapping, so the combination conditions are defined as
C3 (original neck mapping, original hand mapping) C4 (augmented
neck mapping, original hand mapping) and C5 (augmented neckmapping, augmented hand mapping), as shown in Table 2. Similar

SUI ’20, October 31-November 1, 2020, Virtual Event, Canada

Results are further analyzed using a Repeated Measures ANOVA reveals significant effects between the conditions, F (2, 15) = 7.89, p =.00454. Post-hoc pairwise tests reveals significant effects between
comparison groups of (C3,C5), t(6) = 6.88, p = .00801, and (C4,C5),t(6) = 4.26, p < .001, except (C3,C4), t(6) = −1.02, p = .354. Similarly, Condition Speed for C3 is 52.395°/s, for C4 is 50.485°/s and,for C5 is 63.418°/s.

6.1.2 Subjects. 6 participants volunteered to participate in study 2(6 males, age range: 24 to 36 years, mean: 29.8, SD: 4.96).
Condition

6.1.3 Apparatus. The experiment utilized the same Unconstrained
Neck system as the Study 1. Thus, the experimental setup was thesame as indicated in Figure 7(top).

5

5Condition

to the previous study, under C3, the robotic neck was deactivated,
and under C4 and C5, the robotic neck was activated and hence
the vision was enhanced. In C3, C4 and C5, hands were presented
in the visual system as described in Section 6.1. As for the measurements (dependent variables), the response time to locate andselect the target was recorded. Lastly, the participant answered
the NASA Task Load Index questionnaire [11]. Each target was
repeated 4 times; thus, each subject faced a total of 72 task trails(6 tarдets × 4 repetitions × 3 conditions).

L. Shen et al.

4

3

6.1.4 Procedure. The subject selection process was similar to the
Study 1 and the initial procedure (pre-study questionnaires, etc) is
the same as described in Study 1 in Section 5.1.4.

4

3
35

40

45
50
3-4-5 Condition Time (s), User(a) Condition Time

55

60

45

50
55
60
65
3-4-5 Condition Speed (degree/s), User

70(b) Condition Speed

Figure 12: (C3,C4,C5) Box Plot: Average Condition Time /Speed for 6 Subjects. Mean of 4 Repetitions. (blue cross: scattered point of each sample, red diamond: outlier, orange
solid line: median, green dash line: mean)(a) Hand Representations(b) Effective Reach

Figure 11: (a) The locations of virtual hands in AR were represented by two white spheres. (b) The subject was required
to overlap the spheres so to “reach” the target, then the hand
sphere would turn green
6.1.5 Task Procedure. The task followed the same procedure asshown in Figure 8 and described in Section 5.1.5. Besides, twovirtual hands (white spheres) represented the hand location. The
specific locating mechanism could be set to normal or augmented,
i.e., the movements of virtual hands (white spheres), relative tothe body, were twice the movements of real hands following theaugmentation principle of the neck. In addition, after aiming at thetarget, the subject was asked to control either hand to reach thetarget. Once the target was reached, the white sphere turned green,
to confirm an effective target acquisition, as shown in Figure 11.
Then the subject could press a key to destroy the target. The restof the procedures was similar to Study 1. In this part, a subject
performed 4 repetitions of the 6 targets in the horizontal plane.
This study presented the C3, C4, and C5 in a randomized order. At
the end of each condition, the participant answered the NASA Task
Load Index (TLX) questionnaire.

6.2

Results

The overall results of the C3, C4, and C5 from Study 2 are depicted in
Figure 12. Here, the Condition Time for C3 is 48.623s (SD: 5.8320s),
for C4 is 50.459s (SD: 5.6717s) and for C5 is 40.210s (SD: 4.8023s).

Figure 13 indicates the time and speed values for each target.
Here, the “Target Time” and “Target Speed” are defined similar to
Study 1, when the participant acquires the target. For clarity, theFigure 13 results are further presented separately in three comparison groups. The results are further analyzed using a series of
two-way repeated measure ANOVA tests. A significant main effect
was found across conditions and across targets, from both Target
Time and Target Speed. Post-hoc pairwise comparison with Bonferroni and FDR correction between each pairs of the targets revealed
the uneven effects as shown in Table 5. The result indicates thatin the comparison group of (C3,C4), every location did not havesignificant differences, but in the group of (C3,C5) and (C4,C5), significant differences occurred in the latitude range from 120 to 180.
This is similar to the observation of (C1,C2) in Study 1, where therear hemisphere yielded significant effects. As for the efficacy index,
R t ime,C4/C3 = 1.038, R t ime,C5/C3 = 0.827, R t ime,C5/C4 = 0.797.
We conducted surveys using the NASA TLX questionnaire after
each condition. The result in Table 6 showed that the task load of
C3 and C4 seems not to be statically different, and of C5 were likely
to have the smallest load. One-way ANOVA of 3 conditions andindividual pairwise t-test showed similar results.

6.3

Discussion

Hypothesis was partly proved when synchronizing the hand augmentation with the vision augmentation. Observing the result in
Figures 12 and 13, in C5 (augmented neck mapping, augmented
hand mapping) the participants responded best when the virtual
hands with augmentation followed the same neck augmentation
mapping (Linear × 2 in this scenario). However, when comparing
C3 (original neck mapping, original hand mapping) with C4 (augmented neck mapping, original hand mapping), the differences

Visuomotor Influence of Attached Robotic Neck Augmentation
Visuomotor Influence of Attached Robotic Neck Augmentation

SUI ’20, October 31-November 1, 2020, Virtual Event, Canada
SUI ’20, October 31-November 1, 2020, Virtual Event, Canada

90

6.3

Condition
C3
C4
90 80
ConditionC5
C3
80 70
C4
C5
70 60

Target Time
Target Time

Target Speed
Target Speed

Condition
C3
C4
3.0 ConditionC5
C3
C4
3.0
C5
2.5
2.5
2.0

60 50
50 40

2.0
1.5

40 30

1.5

0

30

0

30

60
90
120
150
(C3-C4-C5) Target Longitude

60
90
120
150
(C3-C4-C5) Target Longitude

30

180
180

(a) y: Target Time (°/s),(a) y: Longitude
Target Time
x: Target(°) (°/s),
x: Target Longitude (°)
4.0

3.0
2.5
2.0
1.5

60
90
120
150
(C3-C4-C5) Target Longitude

60
90
120
150
(C3-C4-C5) Target Longitude(b) y: Target Speed (°/s),(b) y: Target
Speed
x: Target
Longitude(°) (°/s),
x: Target Longitude (°)

Condition

Condition

C3
C4

Condition

C3
C5

C4
C5

Condition

Condition

Condition

C3
C4

C3
C5

C4
C5

3.0
2.5
2.0
1.5

180
180

Target Time
Target Time

Target Time
Target Time

3.5

3.5

30
30

Target Time
Target Time

4.0

0
0

1.0
0

30

1.0
0

30

60
90
120
Target Longitude
60
90
120
Target Longitude

150
150

180

0

180

30

0

30

60
90
120
Target Longitude
60
90
120
Target Longitude

150
150

180

0

180

0

30
30

60
90
120
Target Longitude
60
90
120
Target Longitude(c) y: Target Time (s), x: Target Longitude (°);
y: Target
Time(s), x:(C4,C5)
Target Longitude (°);
from left to (c)right:(C3,C4),(C3,C5),from left to right: (C3,C4), (C3,C5), (C4,C5)
100

80
70
60
50
40
30

Condition

C3
C4

C4
C5

Condition

Condition

C3
C5

C4
C5

C3
C4

70

Condition

C3
C5

80 Condition

60
50
40
30

180
180

Target Speed
Target Speed

Target Speed
Target Speed

90

Condition

90

Target Speed
Target Speed

100

150
150

20
0

30

20
0

30

60
90
120
Target Longitude
60
90
120
Target Longitude

150
150

180

0

180

0

30
30

60
90
120
Target Longitude
60
90
120
Target Longitude

150
150

180

0

180

0

30
30

60
90
120
Target Longitude
60
90
120
Target Longitude(d) y: Target Speed (°/s), x: Target Longitude (°);
y: Target
Speed(°/s), x:
Target Longitude (°);
from left to(d)right:(C3,C4),(C3,C5),(C4,C5)from left to right: (C3,C4), (C3,C5), (C4,C5)

150
150

180
180

Figure 13: (C3,C4,C5) Mean Target Speed / Time for 6 Targets.
FigureMeanTarget Speed/ TimeTargets.
Mean 13:of 4(C3,C4,C5)
Repetitions
× 6 Subjects.
(colorarea:forthe6error-bar
Mean
of
4
Repetitions
×6
Subjects.
(colorarea:
theerror-barwith standard deviation)with standard deviation)
Table 5: (C3,C4,C5) P-Values of Posthoc Pairwise T-test for 6
Table 5: (C3,C4,C5) P-Values of Posthoc Pairwise T-test for 6
Targets × 3 Comparison Groups (* naïve significance, ** FDR
Targets × 3 Comparison Groups (* naïve significance, ** FDRsignificance, *** Bonferroni significance)
significance, *** Bonferroni significance)(a) Target SpeedLon.

(b) Target TimeComparison Group
C3,C4

C3,C5

C4,C5

30
60
90
120

0.415
0.452
0.799
0.720

0.568
0.858
0.609
2.16 E −2

0.305
0.390
0.591
6.20 E −3

150

0.633

1.90 E −3

2.90 E −3

180

0.258

7.75 E −3

Avg.

0.374

1.37 E −2

∗∗

∗∗∗
∗∗∗
∗∗∗

Lon.

Comparison Group
C3,C4

C3,C5

C4,C5

30
60
90
120

0.482
0.470
0.657
0.546

0.948
0.943
0.786
1.80 E −2

0.533
0.473
0.551
1.61 E −2

150

0.699

2.02 E −3

4.29 E −3

4.26 E −4

180

0.389

2.14 E −2

3.55 E −3

1.23 E −3

Avg.

0.370

6.82 E −3

7.98 E −4

∗∗∗
∗∗∗
∗∗∗
∗∗∗

∗

∗∗∗
∗

∗∗∗

∗∗

∗∗∗

∗∗∗
∗∗∗

Table 6: (C3,C4,C5) Task Load Index for 6 Subjects
between 2 conditions were not statistically significant at any levels, whichTask
mean
theIndex
involvement of normalhands without augLoad
TLX T-Test
mentation nullified the influence brought by the augmented neck.
Mean
SD
𝐻0
T-Stats. P-value
The reason may be the proprioception conflict between neck and26.0 C36.42
𝜇𝐶4with 0.250
0.812
hand.C3Comparing
with C5,𝜇𝐶3
and=C4C5, the interventionof
C4 25.2
10.4
𝜇𝐶4performance,= 𝜇𝐶5
3.42and then
0.0188*augmented
hand improvedthereached a
C5 12.8
6.71
𝜇𝐶3 = 𝜇𝐶53.320.0209*

Table 6: (C3,C4,C5) Task Load Index for 6 Subjects

Discussion

HypothesisTask
was Load
partlyIndex
proved when synchronizing
TLX T-Test the hand augmentation with the vision augmentation. Observing the result in
Mean
SD
H0
T-Stats. P-value
Figures 12 and 13, in C5 (augmented neck mapping, augmented
C3 26.0 the participants
6.42µC3responded= µC4 best
0.250when 0.812
hand mapping)the virtual
25.2
10.4 followedµC4 =theµC5same3.42
0.0188*handsC4
with augmentationneck augmentation
C5 (𝐿𝑖𝑛𝑒𝑎𝑟
12.8 × 2 6.71
µC3 = µC5However,3.32 when
0.0209*mappingin this scenario).
comparing C3 (original neck mapping, original hand mapping) with C4(augmented neck mapping, original hand mapping), the differences
significant
promotion.
When
hand andsignificant
neck was augmented
between
2 conditions
werenotthestatisticallyat any levsynchronous,
thetheoveralleffect seems
to return.
Yet, without
we are unable
els,
which mean
involvementof normal
hands
augto decide exactly
part, neckor hand,playeda prior roleinmentation
nullifiedwhich
the influence
brought
by theaugmented
neck.
this
compoundedphenomenon
and
mixed
situation.
The reason may be the proprioception conflict between neck andhand. Comparing C3 with C5, and C4 with C5, the intervention of
7 DISCUSSION
AND
augmented
hand improved
theCONCLUSION
performance, and then reached a
In conclusion,
the Unconstrained
Neck andwasneck
provenenlarge thesignificant
promotion.
When the handwastoaugmented
spatial rangetheof overall
vision, effectand augmented
the response
motion.
Tosynchronous,
seems to return.
Yet, we areunablebespecific,
theenlarged
range
was
200%
as
wide
as
before(100%
to decide exactly which part, neck or hand, played a prior role inwider);
the speedphenomenonof augmentedand
response
was 128% as fast
thiscompounded
mixed motion
situation.
as before (28% faster); the time of augmented response motion was
78% as long as before (22% less). One possible factor that inhibited
7the DISCUSSIONCONCLUSION
response time fromAND
decreasing
greater to an ideal efficacy ratio
Inindexconclusion,theless)
Unconstrainedwasprovento enlarge
theof 0.5 (50%
may be the Neckhumanvisualsystem,which itself
spatialrange limitof vision,and augmented
response
motion.
To
had a upperof toleranceof visionthe
shifting.
Shifting
too fastbespecific,
rangecapability
was 200%toasreceive
wide asbefore (100%mayexceedthe
theenlarged
visual system
informationandwider);
speed of The
augmented
response
128%andas fastlead tothe
discomfort.
effect was
uneven:motion
target was
locationuser
asinfluenced
before (28%the timeof augmented
response
motionwasof
thefaster);efficiencyof theaugmentation.
The sideeffect78%
asloadlongandas before(22% less).
One possible
factorinhibitedneck
VR sickness,
resultingin fatigue
and thatdizziness,maythetimeforfrom
greateroftonegative
an ideal influence.
efficacy ratio
be response
responsible
the decreasingsporadic outlier
But
index
of 0.5 (50%less) maydid
be thevisualwhichitselfon average,
VR sickness
not humanincrease,and system,neck loaddecreased.
hadFora upper
limit of toleranceof vision
shifting.
fast
the hand-involved
tasks,
however,with Shiftinga normaltoo
virtualmay
exceedthe visualsystemof
capability
to receive information
and
hand,
the overall
influence
the Unconstrained
Neck was weaklead
discomfort.
The effectwastarget location
and userenedtoto
an insignificant
level;
byuneven:
an augmented
virtual hand,theinfluenced
the efficiency
therecalled.
augmentation.
side effect areofoverall positive
influenceofwas
PossibleTheconstructions
neck
load
and VRfirst,
sickness,resultinginhad
fatigueand dizziness,may
put asfollows:
the handalwaysa higherpriority totheberesponsible
sporadicoutlierprocessof negative
influence.
head/ neck infor
thethespatialperceptionof body;
second,But
theon
average,VR sickness
did notincrease,neckreference
load decreased.
disordercausedby a conflict
betweentwoandspatialobjects,
For the hand-involved
diminishedthe influence. tasks, however, with a normal virtual
hand,
the overallof theUnconstrained
Neck wasweakFeedback
frominfluenceparticipants
spotlighted
the generalexperience
ened
to
an
insignificant
level;
byanaugmented
virtualhand,
thewas smooth and immersive, and none felt severe sickness. Usually
overallpositive
influence wasrecalled.
Possibleconstructionsareusers could
accommodate
to thisnew visualaugmentation
experiput
aswithinfollows:
first, time.
the handhadof
a higherpriority
to theence
a short
The always
anisotropy
experience
existed:theheadin the spatial
perception
body;
second,theusers/ neckcommented
that extension
andprocess
flexionofwere
likelyto cause
disordercaused bya conflict
between
twoWe
spatialreference
objects,
more dizzinessthan
any othermotion.
suggested
it wascataldiminishedinfluence.
ysed by thethe
complete
flip of sky-ground reference (i.e., everything
Feedbackfrom participants
spotlighted
experiencewasupside down)
which totallyconflicted thewithgeneral
the common
sense.
wassmoothand immersive,
andwerenone
felt severe
sickness.
Usually
Different
patterns
of mapping
tried:
linear andnonlinear.
The
users
to andthis benewvisualexperiuserscould
were accommodate
able to recognize
aware
of augmentation
the difference between
ence
a short
Thefelt
anisotropyof experience
existed:thetwo withinmappings,
but time.
usually
more uncomfortable
about
the nonusers
thatmayextension
were
likely tocause
linearcommented
mapping. Thatbecauseandtheflexion
nonlinear
mapping
producedmoredizzinessthan any othermotion.
was catalrelatively
unpredictablebehaviors,andWethesuggested
nonlinearitvisuomotor
ysed
by the completeof sky-ground reference (i.e., everything
integration
was alienflip
to human.
wasTheupsidedown) which
totally
conflictedwith thevisioncommonsense.
Unconstrained
Neckefficiently
contrives
augmentaDifferent
patterns of the
mapping
were tried:
linear andand
nonlinear.
tion by modifying
vision-neck
relationshipalteringThe
the
users
were able
to recognize
be awarethe difference
betweenvisuomotor
coordination
asand
a bridge.
Theofvision-neck,vision-necktwo
mappings,
butrelationship
usually felt are
moreuncomfortable
the nonhand,
hand-eyetypicalexamplesaboutof visuomotor
linear mapping. That may because the nonlinear mapping produced

SUI ’20, October 31-November 1, 2020, Virtual Event, Canada

coordination or integration, and furthermore belong to sensorimotor coordination, the ability to operate different parts of the somatic
body together compatibly and effectively, both input and output.
The coordination is flexible but fragile, and could be interrupted by
the alternation from other human sense, function, or organ. Moreover, the body schema is inferred to possess the potential capacityof being modified. The human body, essentially the brain is adaptable enough to the alteration and modification. Rebuilding mappingof body schema could create a profound impact on human augmentation. Human behavior, mechanism and structure are complicated,
we need to take extra care of the overall body coordination when
applying alternation to a part. In some degree, coordination was
more essential than augmentation.

7.1

Future Potential Applications

As future applications, we are considering a similar technique to be
used to explore more novel vision-motor relationship, for instance,
concept of transplanting eyes to anywhere. Besides, the techniqueof alternating sensorimotor coordination and its further implementation could have beneficial application in research, entertainment,
sport and other scenarios. For example, researchers can make useof this programmable system in medical, bioengineering research,or assistance for rehabilitation therapy. When driving, riding andoutdoors exploring, the system can be worn to gain wider field ofview and quicker response action, for security and performance. As
for the entertainment, the technique can bring unusual and unreal
experience of vision-head relationship that is suitable for gameworld of exceeding the human sensation.

ACKNOWLEDGMENTS
Many thanks to all the volunteers participating in our user test,
for their neck rotations of more than 6000 times. This research is
supported by Embodied Media Project of Keio University.

REFERENCES[1] Jérôme Ardouin, Anatole Lécuyer, Maud Marchal, Clément Riant, and Eric Marchand. 2012. FlyVIZ: a novel display device to provide humans with 360 vision bycoupling catadioptric camera with hmd. In Proceedings of the 18th ACM symposium on Virtual reality software and technology. ACM, 41–44.
[2] Th Brandt, Jo Dichgans, and E Koenig. 1973. Differential effects of central versus
peripheral vision on egocentric and exocentric motion perception. Experimental
brain research 16, 5 (1973), 476–491.
[3] C Bill Chen. 2002. Wide field of view, wide spectral band off-axis helmet-mounted
display optical design. In International Optical Design Conference 2002, Vol. 4832.
International Society for Optics and Photonics, 61–67.
[4] Robert Cubbidge. 2005. Visual fields. Elsevier Health Sciences. 2 pages.
[5] Gislin Dagnelie. 2011. Visual prosthetics: physiology, bioengineering, rehabilitation.
Springer Science & Business Media.
[6] Neil A Dodgson et al. 2004. Variation and extrema of human interpupillary
distance. In Proceedings of SPIE, Vol. 5291. 36–46.
[7] Kevin Fan, Jochen Huber, Suranga Nanayakkara, and Masahiko Inami. 2014.
SpiderVision: extending the human field of view for augmented awareness. In
Proceedings of the 5th Augmented Human International Conference. ACM, 49.
[8] Edward G Freedman. 2008. Coordination of the eyes and head during visualorienting. Experimental brain research 190, 4 (2008), 369.
[9] Uwe Gruenefeld, Tim Claudius Stratmann, Lars Prädel, and Wilko Heuten. 2018.
MonoculAR: A Radial Light Display to Point Towards Out-of-view Objects on
Augmented Reality Devices. In Proceedings of the 20th International Conference on

L. Shen et al.

Human-Computer Interaction with Mobile Devices and Services Adjunct (Barcelona,
Spain) (MobileHCI ’18). ACM, New York, NY, USA, 16–22. https://doi.org/10.
1145/3236112.3236115[10] Sean Gustafson, Patrick Baudisch, Carl Gutwin, and Pourang Irani. 2008. Wedge:
Clutter-free Visualization of Off-screen Locations. In Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems (Florence, Italy) (CHI ’08).
ACM, New York, NY, USA, 787–796. https://doi.org/10.1145/1357054.1357179[11] Sandra G. Hart and Lowell E. Staveland. 1988. Development of NASA-TLX (Task
Load Index): Results of Empirical and Theoretical Research. 52 (1988), 139 – 183.
https://doi.org/10.1016/S0166-4115(08)62386-9[12] Robert S Kennedy, Norman E Lane, Kevin S Berbaum, and M G Lilienthal. 1993.
Simulator Sickness Questionnaire: An Enhanced Method for Quantifying Simulator Sickness. The International Journal of Aviation Psychology 3, 3 (1993),
203–220.
[13] Kiyoshi Kiyokawa. 2007. A wide field-of-view head mounted projective display
using hyperbolic half-silvered mirrors. In Mixed and Augmented Reality, 2007.
ISMAR 2007. 6th IEEE and ACM International Symposium on. IEEE, 207–210.
[14] Adam Kohn. 2007. Visual adaptation: physiology, mechanisms, and functional
benefits. Journal of neurophysiology 97, 5 (2007), 3155–3164.
[15] Ernst Kruijff, J Edward Swan, and Steven Feiner. 2010. Perceptual issues inaugmented reality revisited. In Mixed and Augmented Reality (ISMAR), 2010 9th
IEEE International Symposium on. IEEE, 3–12.
[16] Scott A. Kuhl, William B. Thompson, and Sarah H. Creem-Regehr. 2009. HMD
Calibration and Its Effects on Distance Judgments. ACM Trans. Appl. Percept. 6, 3,
Article 19 (Sept. 2009), 20 pages. https://doi.org/10.1145/1577755.1577762[17] MiYoung Kwon, Gordon E Legge, Fang Fang, Allen MY Cheong, and Sheng He.
2009. Adaptive changes in visual cortex following prolonged contrast reduction.
Journal of vision 9, 2 (2009), 20–20.
[18] R John Leigh and David S Zee. 2015. The neurology of eye movements. Vol. 90.
Oxford University Press, USA.
[19] Bochao Li, Anthony Nordman, James Walker, and Scott A. Kuhl. 2016. The
Effects of Artificially Reduced Field of View and Peripheral Frame Stimulation on
Distance Judgments in HMDs. In Proceedings of the ACM Symposium on Applied
Perception (Anaheim, California) (SAP ’16). Association for Computing Machinery,
New York, NY, USA, 53–56. https://doi.org/10.1145/2931002.2931013[20] Bochao Li, Ruimin Zhang, Anthony Nordman, and Scott A. Kuhl. 2015. The
Effects of Minification and Display Field of View on Distance Judgments in Realand HMD-Based Environments. In Proceedings of the ACM SIGGRAPH Symposium
on Applied Perception (Tübingen, Germany) (SAP ’15). Association for Computing
Machinery, New York, NY, USA, 55–58. https://doi.org/10.1145/2804408.2804427[21] Feng Liang, Stevanus Kevin, Holger Baldauf, Kai Kunze, and Yun Suen Pai. 2020.
OmniView: An Exploratory Study of 360 Degree Vision using Dynamic Distortion based on Direction of Interest. In Proceedings of the Augmented Humans
International Conference. 1–10.
[22] Feng Liang, Stevanus Kevin, Kai Kunze, and Yun Suen Pai. 2019. PanoFlex:
Adaptive panoramic vision to accommodate 360° Field-of-view for humans. In
25th ACM Symposium on Virtual Reality Software and Technology. 1–2.
[23] Hajime Nagahara, Yasushi Yagi, and Masahiko Yachida. 2006. A wide-field-ofview catadioptrical head-mounted display. Electronics and Communications inJapan (Part II: Electronics) 89, 9 (2006), 33–43.
[24] Yun Suen Pai, Benjamin I Outram, Benjamin Tag, Megumi Isogai, Daisuke Ochi,
and Kai Kunze. 2017. GazeSphere: navigating 360-degree-video environments in
VR using head rotation and eye gaze. In ACM SIGGRAPH 2017 Posters. 1–2.
[25] Peter J Savino and Helen V Danesh-Meyer. 2012. Color Atlas and Synopsis of
Clinical Ophthalmology–Wills Eye Institute–Neuro-Ophthalmology. Lippincott
Williams & Wilkins.
[26] Eldon Schoop, James Smith, and Björn Hartmann. 2018. HindSight: Enhancing
Spatial Awareness by Sonifying Detected Objects in Real-Time 360-Degree Video.
(2018).
[27] Lichao Shen, Mhd Yamen Saraji, Kai Kunze, and Kouta Minamizawa. 2018. Unconstrained Neck: Omnidirectional Observation from an Extra Robotic Neck.
In Proceedings of the 9th Augmented Human International Conference (Seoul, Republic of Korea) (AH ’18). Association for Computing Machinery, New York, NY,
USA, Article 38, 2 pages. https://doi.org/10.1145/3174910.3174955[28] John S Stahl. 2001. Without Title. Experimental brain research 136, 2 (2001),
200–210.
[29] William B Trattler, Peter K Kaiser, and Neil J Friedman. 2012. Review of Ophthalmology E-Book: Expert Consult-Online and Print. Elsevier Health Sciences.
[30] James W Youdas, Tom R Garrett, Vera J Suman, Connie L Bogard, Horace O
Hallman, and James R Carey. 1992. Normal range of motion of the cervical spine:
an initial goniometric study. Physical therapy 72, 11 (1992), 770–780.