Computational Design of Hand-Held VR Controllers Using
Haptic Shape Illusion
Eisuke Fujinawa

Shigeo Yoshida

Yuki Koyama

The University of Tokyo, Bunkyo-ku,
Tokyo, Japannawafuji@cyber.t.u-tokyo.ac.jp

The University of Tokyo, Bunkyo-ku,
Tokyo, Japanshigeodayo@cyber.t.u-tokyo.ac.jp

National Institute of Advanced
Industrial Science and Technology(AIST), Tsukuba, Ibaraki, Japan
koyama.y@aist.go.jp

Takuji Narumi

Tomohiro Tanikawa

Michitaka Hirose

The University of Tokyo, Bunkyo-ku,
Tokyo, Japan
narumi@cyber.t.u-tokyo.ac.jp

The University of Tokyo, Bunkyo-ku,
Tokyo, Japan
tani@cyber.t.u-tokyo.ac.jp

The University of Tokyo, Bunkyo-ku,
Tokyo, Japanhirose@cyber.t.u-tokyo.ac.jp

ABSTRACT
Humans are capable of haptically perceiving the shape of an object
by simply wielding it, even without seeing it. On the other hand,
typical hand-held controllers for virtual reality (VR) applications are
pre-designed for general applications, and thus not capable of providing appropriate haptic shape perception when wielding specificvirtual objects. Contradiction between haptic and visual shape perception causes a lack of immersion and leads to inappropriate object
handling in VR. To solve this problem, we propose a novel methodfor designing hand-held VR controllers which illusorily represent
haptic equivalent of visual shape in VR. In ecological psychology,
it has been suggested that the perceived shape can be modeledusing the limited mass properties of wielded objects. Based on thissuggestion, we built a shape perception model using a data-driven
approach; we aggregated data of perceived shapes against various
hand-held VR controllers with different mass properties, and derived the model using regression techniques. We implemented adesign system which enables automatic design of hand-held VR
controllers whose actual shapes are smaller than target shapes
while maintaining their haptic shape perception. We verified that
controllers designed with our system can present aimed shapeperception irrespective of their actual shapes.

Figure 1: The actual appearance (right) and perceived shape(left) of a hand-held VR controller designed with the proposed system. Though the controller’s actual appearance differs from the object’s appearance in VR, the user perceives
handling the object in VR.
users handle virtual objects through a hand-held VR controller, thequality of this controller is important for the user.
The appearance of VR controllers is application-dependent, andgives users the impression of controlling different objects. However, humans can estimate the shape of an object with haptic cues,
especially when wielding it. Since actual controller shapes are predefined, they cannot provide users with appropriate haptic shapeperception specific to wielded objects in VR. Moreover, a contradiction between haptic and visual shape perception can cause a lackof immersion and leads to inappropriate object handling in VR.
One direct approach to reproducing appropriate haptic shapeperception is to use an object identical to the virtual object in its
shape and the mass properties as a hand-held VR controller. A
controller with the identical properties should reasonably lead tothe perception that the real and virtual object are the same. However,
it is impractical to produce a controller with the same properties
as a large virtual object, owing to spatial restrictions due to thelimited room space and obstacles (e.g., wall, ceiling, shelf, lamp, etc)
in physical environments.
On the other hand, previous studies have investigated a shapeillusion in which humans perceive a shape that differs from an

KEYWORDS
Perception; computational design; data-driven; virtual reality

1

INTRODUCTION

As satisfying virtual reality (VR) hardware has become commercially available, VR has become increasingly popular. VR is expected
to be applied to a wide range of fields including medicine, education, advertising, and entertainment. Because the visual quality of
VR is now high, developers have shifted their focus to hand-held
VR controllers (e.g., Oculus Touch and Playstation Move). Since
VRST ’17, November 8–10, 2017, Gothenburg, Sweden© 2017 Copyright held by the owner/author(s). Publication rights licensed to Association for Computing Machinery.
This is the author’s version of the work. It is posted here for your personal use. Not
for redistribution. The definitive Version of Record was published in Proceedings of
VRST ’17, November 8–10, 2017, https://doi.org/10.1145/3139131.3139160.
1

VRST ’17, November 8–10, 2017, Gothenburg, Sweden

Fujinawa et al.
Sample controllers

object’s actual shape [Omosako et al. 2012]. Several studies in ecological psychology have suggested that the main factor in haptic
shape perception constitutes the limited mass properties of wieldedobjects (e.g., the moment of inertia and the static moment [Burton
et al. 1990; Kingma et al. 2004; Pagano et al. 1993; Turvey et al. 1998]).
This means there is a mapping that relates the specific mass properties to the perceived shape. By taking advantage of this mapping,
we can design a hand-held controller with haptic shape illusion,
with which a user haptically perceives an appropriate shape for an
object in VR that is, in reality, smaller and/or lighter.
In this paper, we propose a novel method for designing handheld VR controllers that appropriately represent visual shape inVR (Figure 1) using a haptic shape illusion. As the first step, we
specifically focused on controllers for symmetric planar objects up
to 1 m length or width, and implemented a proof-of-concept systemin this scope. Our primal target users are VR developers, including
indies as well as enterprises. When they demonstrate their content,
they can design the dedicated controller which makes their demo
or attraction more appealing to the audience. Individual players
can also use our system to design an additional dedicated controller
for their fun.
Figure 2 shows an overview of our approach. Based on the virtual
target shape and physical spatial requirements, a controller design is
automatically generated so that its haptic shape perception matches
the target shape. To do so, we first build a shape perception modelthat relates mass properties to haptically perceived shape using adata-driven approach. We aggregate perceived shape data against
hand-held VR controllers with different mass properties throughperceptual experiments and derive a mapping using regression
techniques. Using this mapping, we implement a design systemthat optimizes the perceived shape of the hand-held controller to be
the target shape. The designed model is easily fabricated through alaser cutter or a 3D printer and can be used as a VR controller. We
performed a user study using several VR controllers designed byour system to validate that these controllers reproduce the desired
shape perception with reduced sizes.
To summarize, our technical contributions include (1) a novel
design concept that uses a haptic shape illusion, (2) a data-driven
representation of a perceived shape based on the mass properties ofa wielded object, and (3) an interactive optimization-guided handheld VR controller design tool.

2?

Shape Perception Model

Target virtual
objectDesigned
controller

Figure 2: Our framework consists of a precomputation stage
and online design sessions. In the precomputation stage,a shape perception function that maps mass properties ofcontrollers to their perceived shapes is constructed. This isachieved by utilizing actual data from a perceptual experiment. This function is used in our design system to design
a custom hand-held VR controller for a specific target applications.

do not always corresponds to the perception of an object’s shape as
human perception is often nonlinear to stimuli. Some researchers
have leveraged this nonlinearity to easily present haptic perception
easily [Amemiya et al. 2008; Rekimoto 2013]. However, nonlinear
relationships in perception are complicated and unclear. We employ a data-driven approach to model the nonlinear relationshipbetween stimulation and perception.
Instead of rendering force, it is also possible to present local
shape outline around a user’s hand to present the perception oftouching objects. Such haptic displays are called shape displays.
For example, Benko et al. [Benko et al. 2016] developed two typesof mobile devices to present a local shape to a user’s finger, oneof which represent the rough shapes by tilting the surface while
the other renders the detailed textures by moving a set of tiny rods.
These types of shape displays still have problems with a limited
spatial resolution and noticeable latency. While these devices try topresent local shape perception when touching a surface, most partsof the mobile controller are not to be touched since users control itwith its handle. Therefore, we focused on the overall perception of
the whole shape when wielded with its handle while leaving thehandle shape alone as it is.
In contrast to the typical haptic displays using dynamic actuators,some methods uses passive haptic properties of a physical object.
Azmandian et al. [Azmandian et al. 2016] proposed a method toretargeting passive haptic properties of a single object for handling
multiple virtual objects. Their method uses a physical object withidentical size with rendered object to present appropriate haptic

RELATED WORK

Our method is based on established works from various researcharea, including haptic displays in VR, shape perception in ecologicalpsychology, and computational designs in computer graphics.

2.1

Perceived shapes

Haptic Displays

Various haptic displays, such as force displays and shape displays,
have been developed in order to present rich haptic perceptions tousers. Force displays aim to feedback force to the user’s hand as it
moves using actuators and other mechanisms [Massie and Salisbury
1994; Minamizawa et al. 2007; Nakagawara et al. 2005]. There arevarieties in the mechanism sizes that have a trade-off between its
structural complexity and the variety of the presented force. Although force displays try to precisely reproduce counter-force, they2Computational Design of Hand-Held VR Controllers Using Haptic Shape Illusion VRST ’17, November 8–10, 2017, Gothenburg, Swedencue. On the other hand, Zenner et al. [Zenner and Krüger 2017]
proposed Shifty, a hand-held device that dynamically changes thepassive mass properties by shifting weight along with its renderedshape in VR. Their approach can give feeling that the object’s
length/thickness is changing. However, it does not present appropriate passive properties corresponding to the rendered shape since
the mapping from the mass properties to perceived shapes is still
unclear. Similar to these approaches, our approach is fully passive; we do not use any actuators to render haptic feedback. Our
approach can present various shapes without employing a complicated actuators or mechanisms by switching a controller depending
on applications. Although this may sound inconvenient comparedto the multi-use controller, preparing the dedicated controllers arevery easy since the designed model is easily fabricated through alaser cutter or a 3D printer.

2.2

graphics have presented various computational design tools considering functionality of objects. For example, researchers havefocused on structural strength [Lu et al. 2014; Stava et al. 2012],
stability [Bächer et al. 2014; Prévost et al. 2013], or elasticity [Schumacher et al. 2015]. Many of previous studies leverage physicalsimulation to analyze the functionality of objects and to generatefunctional designs.
However, it is sometimes difficult to simulate functionality withfully analytic approaches; for this situation, data-driven approaches
have been taken to capture complex physical phenomena and material parameters. For example, Umetani et al. [Umetani et al. 2014]
proposed a data-driven aerodynamic simulation of hand-launchedglider airplanes, and then utilized it in an airplane design tool. They
gathered a sample set of glider flight trajectories and learned therelationship between the forces on the wings and wing shapes.
Data-driven approaches are also used to predict human’s perception. Lau et al. [Lau et al. 2016] presented a perceptual model
for predicting, given 3D shapes, which areas are more likely to be
touched by human. They gathered a large-scale training data using
crowdsourcing and utilized a deep learning technique to learn sucha perceptual model. Piovarci et al. [Piovarči et al. 2016] proposed aperceptual model for the compliance of nonlinearly elastic objects.
To construct the model, they conducted a psychophysical experiment in which a participant compared the compliance of various
3D-printed cubes, and then they evaluated various computationalmodels that predict perceived compliance.
In this work, we construct a mapping from the mass properties ofa hand-held object to its perceived shape to design a VR controller.
For this, we leverage a data-driven approach in which we collected
empirical data of perceived shapes for various hand-held objects
and learn a suitable mathematical representation for the mapping.
To our knowledge, this is the first attempt to model human’s haptic
shape perception and use it as a computational design criterion.

Haptic Shape Perception

Humans can perceive several spatial and other properties of objects,
without seeing them, simply by wielding them. Gibson called thisability as “Dynamic Touch” [Gibson 1966] and many researchers inecological psychology have worked to clarify its mechanism. Manyexperiments have been conducted to understand how humans perceive the length of an object. Studies have supported the idea thatperceived length and width of the object is mainly related to themoment of inertia (object’s tendency to resist angular acceleration)
about the position of the hand [Turvey 1996]. Burton et al. [Burton et al. 1990] suggested that humans can roughly estimate theshape of an object with a handle by blindly wielding it. Pagano et al.
[Pagano et al. 1993] showed that the perceived length is in proportion to the maximum principal moment of inertia and is in inverse
proportion to the minimum principal moment of inertia. Turvey etal. [Turvey et al. 1998] hypothesized that perceived width is also
related to the principal moment of inertia and conducted perceptual
experiments using solid rectangular parallelepipeds. Contrary toperceived length, the perceived width was found to be proportional
to the minimum principal moment of inertia and is inversely proportional to the maximum principal moment of inertia. Kingma etal. [Kingma et al. 2004] showed that the static moment (equivalent
to torque) of the object also contributes to length perception, andthat its weight is not related.
Based on these studies, we make the following assumptions
about haptic shape perception: (1) the perceived shape of a wieldedobject is represented by its length and width and (2) there exists a
mapping between the perceived shapes and specific mass properties,i.e., the principal moment of inertia and the static moment about thehandle position. Using this mapping, we propose a novel methodfor designing a VR controller with a desired shape perception. As
far as we know, there have been no previous attempt to explicitly
utilize this haptic shape perception model for engineering purposes,i.e., to design hand-held VR controllers.

2.3

3

FUNDAMENTALS AND OVERVIEWThe VR controllers used in this study are dedicated hand-held objects that functions as a tool or a hand that a user operates in VR
spaces. Based on previous studies in ecological psychology [Burton
et al. 1990; Kingma et al. 2004; Pagano et al. 1993; Turvey et al.
1998], we construct a shape perception model that can be utilized
to computationally design an appropriate VR controller that reproduces a desired haptic shape perception. With this approach, we
can make a controller with proper haptic shape perception that issmaller and/or lighter than a naïvely designed controller, i.e., theone shaped exactly like the target virtual object. This is beneficial,for example, when the physical space is limited compared withthe virtual environment but users want to handle a relatively large
virtual object with suitable haptic perception; this is a common
situation in indoor VR systems.

3.1

Computational Design and Data-Driven
Methods

Assumptions on Shape Perception Model

Based on the previous studies concerning the relationship between
the mass properties and the perceived shape [Burton et al. 1990;
Kingma et al. 2004; Pagano et al. 1993; Turvey et al. 1998], we assume
that there is a mapping from the mass properties of a wielded object
to its shape perception.

Digital fabrication devices such as 3D printers or laser cutters havebecome widely available, increasing the importance of accessible
design tools for functional objects. Recent studies in computer
3

VRST ’17, November 8–10, 2017, Gothenburg, Sweden

Fujinawa et al.

For a hand-held controller C, let the second moment (i.e., themoment of inertia) about the handle position be I ∈ R3×3 , the first
moment (i.e., the static moment [Kingma et al. 2004]) about thehandle position be M ∈ R3 , and the shape that a user perceives
when grasping it be S. Based on the previous studies [Burton et al.
1990; Kingma et al. 2004; Pagano et al. 1993; Turvey et al. 1998], we
assume that the perceived shape S is determined by the moment ofinertia I and the static moment M about the handle position, Thatis, we assume the existence of a shape perception function:f : (I, M) 7→ S.

?

(1)Mass Properties

In this study, we also put the following assumption about thecontroller C and its perceived shape S to make the shape perception
model tractable.

Perceived ShapeFigure 3: Illustration of the shape perception function thatmaps the mass properties to the perceived shape. Here, themass properties are described by three parameters: I x , Iy (themoment of inertia), and My (the static moment). The perceived shape is parametrized using the perceived length L
and the perceived width W .

(1) A controller C and its perceived shape S are both planar witha constant, sufficiently small thickness.
(2) A controller C and its perceived shape S are both reflective
symmetric with respect to the y-z plane.
(3) A perceived shape S can be represented as a rectangular thinplate made of a homogeneous material.

3.2

Under these assumptions, a perceived shape S is parametrized byits perceived length L and a perceived width W . Although these assumptions may sound strong, it is sufficient to verify our approach,since many hand-held tools can be seen as 2-dimensional planar
symmetric shapes.
We define the handle coordinate system using the handle position and the handle axes, where
the center of the handle is the origin, the handle axis y is parallelto the handle, and the handle axis
z is perpendicular to the planar
surface of the controller (see theinset figure). In this work, we assume that the length of the handle is approximately 10 cm.
Under these assumptions, a controller C need not be rectangular,
but its moment of inertia I always aligns with the axes (i.e., I =diag(I x , Iy , Iz )) and its static moment has only one non-zero entry(i.e., M = (0, My , 0)). In addition, as the controller is thin, we have
Iz = I x + Iy . As a result, the shape perception function can be
re-written asf : (I x , Iy , My ) 7→ (L,W ).

Mapped to

Designing Controller using the Shape
Perception Model

Using the shape perception function f , we can predict a perceived
shape (L,W ) = f (I x , Iy , My ) for an arbitrary controller C. To design
a VR controller, we solve an inverse problem for this prediction;
given a target shape perception S target = (Ltarget ,W target ), where
Ltarget is a target perceived length and W target is a target perceived
width, we want to find an optimal controller design C ∗ whose
perceived shape f (I x∗ , Iy∗ , My∗ ) is as similar to (Ltarget ,W target ) as
possible.
Our design workflow is as follows. First, a user inputs the target
virtual model and specifies its handle, from which our design systemdefines the target shape perception S target = (Ltarget ,W target ). Next,
the user can specify the maximum and minimum size constraints of
the resulting controller. Then, our system automatically solves anoptimization problem in which, intuitively, the distance between thetarget shape S target and the predicted perceived shape is minimized
while satisfying the constraints. The system then outputs a CAD
data for the optimal controller design C ∗ , which is easy to fabricate
using a laser cutter or a 3D printer. Details of these procedures are
explained later.

4

SHAPE PERCEPTION MODEL

We collected data on the perceived length and width (L, W ) ofvarious physical controllers with a certain mass properties (I x , Iy ,My ). These data are later used to regress the shape perception
model.

(2)

Figure 3 illustrates the shape perception model. To construct thefunction f , we took a data-driven approach; we collected a set ofpaired data between a controller with a mass properties (I x , Iy , My )and its perceived shapes (L,W ). For data collection, we conducted
an experiment in which participants were asked to wield a set ofsamplesample controllers {Ci} in a VR environment and to determine
their perceived lengths and widths {(Li ,Wi )} without seeing theiractual appearances. By leveraging function regression techniqueson these data, we constructed the function f . Details of the datacollection and the function construction are described later.

4.1

Collecting Perceived Shape Data
sample4.1.1 Sample Controllers. Sixteen sample controllers {Ci}(i = 1, . . . , 16) with various mass properties {(I xi , Iyi , Myi )} wereprepared, as shown in Figure 4. The sample controllers were madeof a 5.5-mm thick medium-density fibreboard (MDF), and multiple
9-mm diameter lead weights are attached on them in various ways.
The handle for all controllers is a 100 mm × 30 mm rectangle
4Computational Design of Hand-Held VR Controllers Using Haptic Shape Illusion VRST ’17, November 8–10, 2017, Gothenburg, Sweden

290 mm
360 mmFigure 5: Evaluation model in VR (left) that the users seeon the position of the sample controller. Participants adjust
the length Lvar and the width Wvar of the model. The modelis initialized with either the minimum size (center) or the
maximum size (right)

Figure 4: Sample controllers used in the data aggregation experiment.

and grab the handle of the selected controller with their dominantsamplehand. We selected one of the sample controllers, Ci, and displayed R var in the virtual environment. Evaluation model R var has
two initial sizes. One of them is the minimum size (L var , Wvar ) = (30mm, 30 mm) and the other is the maximum size (L var ,Wvar ) = (1500mm, 1500 mm). Participants began their evaluation with one of
the two. Figure 6 shows a participant and their view in the virtualenvironment. We chose magenta for the color of the evaluation
model R var because an expansive or contractive color would affectparticipant’s size perception.
sampleParticipants estimate the size of Ci
while wielding it in various directions, adjusting the L var and Wvar to the size that they
perceive using the game controller. In the experiment instructions,
we requested the participants to wield a sample controller in various direction (away from their body), to touch only the handle, andto assume the same material for all the controllers. When the evaluation of the participants (L var , Wvar ) was confirmed, we recorded
them as (Li ,Wi ), and retrieve the controller from the participants.
Evaluations were performed twice for each pair of a sample
controller and initial size (in total, 16 × 2 × 2 = 64 times) in the
random order. Each evaluation was limited to 30 seconds, and a
2-minute break was inserted in every 16 evaluations. When starting
and resuming the evaluation, participants wielded the reference
controller which has the same actual shape as the VR appearance
to control the material assumptions.
For this experiment, 10 participants were gathered, including 2
women and 2 left-handed people and ranged in age from their 20s
to 40s. There were 64 evaluations for each person, so we obtained
64 × 10 = 640 evaluation data pairs {(I xi , Iyi , Myi ), (Li ,Wi )}.

wrapped in curing tape to avoid giving any tactile clue about thecontroller material of the controllers to participants. The maximum
size of the controllers was limited to 360 mm in length and 290 mm
in width, and the maximum mass was set to 124 g so as to avoid
danger when swinging them in the room. The deflection whenswinging these sample controllers was negligibly small. For thedetails of the mass properties of the sample controllers, see thesupplemental material.
4.1.2 VR System. In the previous studies on evaluation of perceived shapes [Burton et al. 1990; Kingma et al. 2004; Pagano et al.
1993; Turvey et al. 1998], participants were blindfolded during wielding objects. Since we want to model the perceived shape whichis later used for VR applications, we aggregated data of perceived
shapes in a virtual environment. We developed a VR system forthis purpose in which participants handle a physical controller andresize its virtual shape so that the haptically perceived shape is as
close to the virtual appearance as possible. The system consistsof an head-mounted display (HMD, the Oculus Rift DK 2), motion
capture cameras (Optitrack Flex 13), a PC, a game controller (Xboxsample360 controller), and sample controllers {Ci}. Tracking markers
were attached to the HMD and the sample controllers, and 5 motion
capture cameras (Optitrack Flex 13) were placed around the 2 m ×
2 m working space to obtain the position and the posture of theparticipant’s HMD and the sample controllers. Through the HMD,
participants can see a virtual hand and a virtual handle of the same
size and position as the actual handle position, which they canmove naturally. To evaluate the perceived shape when wielded, avariable evaluation model R var was placed and displayed on the tipof the virtual handle. This model has a variable length L var (30 mm≤ L var ≤ 1500 mm) and a variable width Wvar (30 mm ≤ Wvar ≤1500 mm) with constant thickness 5.5 mm (Figure 5). Participants
can change the length L var and width Wvar of the evaluation model
R var using the game controller.

4.2

Regression Models and Results

Using the 640 data pairs, we performed regression analysis to construct a mapping from the mass properties (I x , Iy , My ) to the perceived length and width (L, W ). For regression model selection, we
tested linear regression (LR), quadratic regression (QR), and Gaussian process regression [Rasmussen and Williams 2006] (GPR), and4.1.3 Procedure. Participants were asked to sit on a chair, wear
an HMD, hold the game controller with their non-dominant hand,
5

VRST ’17, November 8–10, 2017, Gothenburg, Sweden

Fujinawa et al.
Table 1: Prediction errors with different regression models:linear (LR), quadratic (QR), and Gaussian process (GPR) regressions.

Error in L [mm]Error in W [mm]

LR

QR

GPR

117.5
99.2

116
98.5

115.9
97.9

computation, the system outputs a CAD data for a VR controllerthat can be fabricated by the user. In the following subsections, wedefine this input more specifically, and then describe the details of
the automatically generated design.

Figure 6: A participant wielding a sample controller (left)
and their view in the virtual environment (right).

5.1

User Input

A design session begins with a user’s specification of the 3D shapemodel of the target hand-held virtual object. The user must also
specify a handle part for grasping. From this information, the systemdefines the target shape S target ; the target shape is defined with the
maximum length Ltarget along the handle axis y from the handle
position, and the maximum width W target along the handle axis x.
Next, the user can specify a size constraint, which is explained in
the next subsection. Finally, the user must also specify the densityof the base material ρ for fabrication and the density ρ ′ (> ρ) ofadditional weights, which are used in the later optimization step.

5.2

Figure 7: Visualization of the perceptual shape function f :(I x , Iy , My ) 7→ (L,W ) constructed by Gaussian process regression (GPR).
compared their prediction performances. Figure 7 visualizes theregression results. Note that these results are not contradictory
to the report by Turvey [Turvey 1996], which suggests that L is
proportional to I x , Iy−1 , and My , and W is proportional to I x−1 and Iy .
Table 1 lists the prediction errors of these three regression models.
We calculated the prediction errors with the leave-one-out crossvalidation strategy, where each sample controller is left out once.
Although GPR has the smallest prediction errors, there is little
difference among these three models; thus we decided to use thefunction constructed with linear regression (LR) in this study forsimplicity.

5

Size Constraint

The user can specify a size constraint; by controlling a rectangle
in the interface, the user can specify a rectangular design regionΩ = {(x, y) | x min ≤ x ≤ x max , ymin ≤ y ≤ ymax }. This will
be used in the later computation so that the resulting controller
fits within this region. In addition, we also set a constraint thatthe mesh vertices of the handle part will not be moved in the latercomputation. As the handle is the only part of the controller that theuser touches, they notice large contradictions between the actualand virtual handle shapes; thus, we preserve its shape. Note that thecurvature of the other parts’ outline also follows the original shapewhile transfomed in case a user unexpectedly touch parts otherthan the handle. Optionally, instead of specifying size requirements,
the user can directly specify an arbitrary shape model for the output
model.

5.3

Shape Perception Cost

We require the perceived shape of the controller to be close to target
shape S target ; that is, the perceived length L and width W should
be close to the target length Ltarget and width W target , respectively.
We define a shape perception cost E S as the squared distance of theperceived scale and the target scales. Since the perceived length
and width are computed through the shape perception function,
this shape perception cost is also a function of the mass properties:

DESIGN SYSTEM

In this section, we describe our design system for designing custom
VR controllers, in which we utilize the shape perception model
built in the previous section. Figure 8 shows an overview of thesystem workflow. The input for our system consists of the target
virtual object and the target size constraint. After some automatic

E S = E L + EW ,6(3)

Computational Design of Hand-Held VR Controllers Using Haptic Shape Illusion VRST ’17, November 8–10, 2017, Gothenburg, Sweden
Target virtual model

CAD data of designed
VR controller

WeightDeformation

Size
constraint
HandleInitialization(cage deformation)

Input

Optimization(weight placement)

Optimization(inner carving)

OutputFigure 8: System workflow. The user input consists of the 3D shape model of the target hand-held virtual object, the handle part,and the target size constraint. The output is a CAD data of the designed VR controller for fabrication. The system computes
the following 3 steps: The first step is cage deformation, which generates an initial solution that satisfies the size constraints,
the second step is weight placement, in which the design is optimized using additional weights, and the third step is innercarving, in which the design is further refined by placing holes.
where
E L = f L (I x , Iy , My ) − Ltarget 2

EW = fW (I x , Iy , My ) − W

5.4,

target 2

the handle part are not updated. This provides a deformed modelthat can be used as the starting point of the later minimization
steps. Note that, when the user specifies the output controller shape
directly, this step is simply skipped.

(4)
.

(5)

5.5.2 Optimization by Weight Placement. To achieve the desired mass properties, we use a heavier material (e.g., metal) for
additional weight. Although it is possible to fabricate arbitrary
shapes with metal and combine them [Bächer et al. 2014], this is
too difficult for most end users. Therefore, we considered attaching
pre-manufactured metal balls on the base controller.
The system first discretizes the whole shape with regular 2dimensional grids (see Figure 9). The size of a grid cell is 10 mm ×
10 mm. With these grid cells, the system later computes the weightplacement and the inner carving. Since these grid cells are used forinner carving as well, we set a certain margin (∼ 5 mm) from theoutline to the grid cells so that the mesh model can be fabricatedsafely even when the cells are carved. Before the computationof weight placement and inner carving, since we want to reduce
unnecessary mass as much as possible, the grid cells are uniformly
carved with 9 mm × 9 mm rectangular holes. This is the maximum
carving size for each grid cell to maintain the rigidity.
Although this weight placement is a discrete optimization problem, we approximate it as a continuous optimization problem.
To each grid cell, we set weight density ratios α k ∈ [0, 1] (k =1, . . . , #cells). Suppose we replace the cells with the weight material
with density ratio α k ; then, the mass of the whole controller s ′ canbe represented with the mass of the carved mesh s, the weight density ratios α k , the ratio of weight material to base material ρw , andP
the mass of each cell sk : s ′ = s + k α k ρw sk . Since the derivativeof the mass properties with respect to α k is analytically calculated,
we can efficiently solve the optimization problem with respect tothe weight density ratios {α k } using the L-BFGS method [Liu andNocedal 1989] (a gradient-based optimization algorithm) to obtain
the optimal density distribution. After obtaining the continuous
valued result, the system determines whether a metal ball shouldbe placed or not for each cell by simple thresholding.

Mass Cost

It is sometimes desirable to reduce the burden on the user and thematerial cost. Thus, we incorporate a cost term that tries to reducethe mass m of the controller C to the optimization. This mass costis represented as
Em = m.

5.5(6)

Optimization

To obtain an optimal controller design, the system solves the following minimization problem:C ∗ = arg min{w S E S + wm Em }C

(7)

under the size constraint (i.e., all the vertices are in Ω), where w S
and wm are the weights for controlling the effects of these costfunctions, for which the user can freely provide the values.
To solve the above design optimization, our system utilizes techniques similar to those presented by Bächer et al. [Bächer et al.
2014], which consists of (1) cage deformation, (2) weight placement,and (3) inner carving. Cage deformation generates an initial solution that satisfies the size constraint, weight placement minimizes
the costs with respect to the placement of additional weights, andinner carving further refines the optimization with respect to theplacements of holes.
5.5.1 Setting an Initial Solution by Cage Deformation. To get
an initial solution that satisfies the size constraint, our systemdeforms the target shape. As a deformation strategy, we use boundedbiharmonic weights [Jacobson et al. 2011]. Specifically, our systemgenerates a set of control points around the controller mesh, andthen moves these points into the user-specified bound Ω. In thisdeformation process, the system ensures that the mesh vertices of7

VRST ’17, November 8–10, 2017, Gothenburg, Sweden

Fujinawa et al.
Table 2: The length and width of the test visual shapes Sitest .

Margin from the outline (~ 5 mm)

Length [mm]Width [mm]6

S 1testS 2testS 3testS 4testS 5test

640
180

480
380

480
180

320
380

320
180

RESULTSWe designed several hand-held VR controllers using our systemand conducted a user study to validate that the designed controllers
present the target shape perception.
Hollowing for weight placement(9-mm diameter)

6.1

Hollowing for inner carving(9 mm by 9 mm in maximum)

Designed Controllers

Figure 10 shows VR controllers designed with our system. All ourmodels were fabricated using a laser cutter with an MDF and 9-mmdiameter lead weights are placed on them. Our system can produce
a controller that have a large perceived length and a small perceived
width. On the Sword model, weights placement is concentrated on
the tip (Figure 10, Left). Our system can also produce a controllerthat have a large perceived width. On the Tennis model, weights
placement is widely distributed to represent the wide shape of theoriginal model (Figure 10, Center). Our system can allows user toarbitrarily specify the handle position. On the Guitar model, thehandle is specified on the neck and the body is transformed while
the head is preserved (Figure 10, Right). The length of each designed
controller is almost half size in comparison with its original length(Sword: 57.65%, Tennis racket: 51.61%, Guitar: 55.0%).

Figure 9: Illustration of the 2-dimensional regular grid on
the controller model used for optimization steps. The sizeof each grid cell is 10 mm × 10 mm and has a certain margin from the outline mesh. During optimization, cells arehollowed either for the weight placement or for the innercarving. For weight placement, a cell is carved with a 9-mmdiameter circle. For inner carving, a cell is carved with a 9mm × 9 mm rectangle at maximum.

6.2

5.5.3 Optimization by Inner Carving. Although the weight placement step already provides a good optimization solution (Equation 7), it is possible to further refine the solution and achieve asmaller cost function value with the inner-carving step. To refine
the optimization, the system adjusts the density distribution bymodifying the controller’s inner structure. While there are sophisticated methods for modifying inner structures (e.g., [Bächer et al.
2014; Schumacher et al. 2015]), here we adopt a simple method;
the whole shape is divided using a 2-dimensional regular grid asused in the previous subsection and each grid cell is carved in accordance with its carving ratio. To each grid cell, we set carving
ratios βk ∈ [0, 1] (k = 1, . . . , #cells). The carving ratio indicates
how much the system carves the grid cell when the controller isfabricated. A carving ratio of 1 means that the corresponding gridcell is completely empty, and 0 means it is not carved at all. Thereby
the mass s ′ of the whole mesh can be denoted by the mass of theoutline mesh s, the carving ratios βk , and the mass of each cell sk :P
s ′ = s − k βk sk . The derivative of the mass properties with respectto {βk } can be analytically calculated, so we solve the optimization
with respect to the carving ratios {βk } using the L-BFGS method[Liu and Nocedal 1989] again. After obtaining the optimal carving
ratios, the system carves each cell by a square such that the massof the carved cell is proportional to the obtained carving ratio (seeFigure 9, Bottom left). We set an upper bound of the carving ratio
for the optimization to be 0.81 so that the the maximum hole sizefor each grid cell is a 9 mm × 9 mm square.

User Study

We conducted a user study to evaluate how successfully the controllers designed with our system can present the intended shapeperception to the users in a VR environment.
6.2.1 Test Controllers. We prepared five VR controllers (testcontrollers Citest (i = 1, . . . , 5)) designed for different visual shapes(test visual shapes Sitest (i = 1, . . . , 5)) using our system. Figure 11
shows the test controllers and the test visual shapes, and Table 2
shows the details of the test visual shapes. The test controllers
were fabricated using laser-cut 5.5-mm-thick MDF frames and 9mm-diameter lead weights. The outer shapes were the same forall the controllers, with a handle of 80 mm × 30 mm and frame
outline of 260 mm × 200 mm. A curing tape was wrapped aroundthe handle so that the participants could not guess its material
from the surface texture. Because the prediction errors of the shapeperception function are about 11 cm in length and about 10 cm inwidth (Table 1), the designed perceived shapes differed at least by
10 cm in either the width or the length.
6.2.2 VR System Setup. The system used for the experiment isalmost the same as that used in the data aggregation experiment. In
addition, we placed a table containing several controllers, in frontof the chair on which participants sit, which was also displayed in
the virtual environment as well.
6.2.3 Procedure. Each of the participant sat on a chair and wore
an HMD. Tracking markers were attached to all the test controllers,8Computational Design of Hand-Held VR Controllers Using Haptic Shape Illusion VRST ’17, November 8–10, 2017, Gothenburg, Sweden

Figure 10: VR controllers designed with our system: Sword (Left), Tennis racket (Center), and Guitar (Right). For each pair, thetarget virtual object is shown in the left, and the designed controller is shown in the right.
Table 3: Each row shows ratios of the test controllers Citest
answered for a test visual shape Sitest . The cells with boldcharacters shows a pair where the perceived shape of thetest controller matches the test visual shape.

380 mm
640 mm
1

2

4

5

3

S 1testS 2testS 3testS 4testS 5test

340 mm
200 mmFigure 11: Test controllers designed for the user study (left)
and test visual shapes seen in the virtual environment (right,
with reduced scale compared to the controller). The red numbers on the controller show the index i for Citest and thenumbers next to the visual shape show the index i for Sitest .
The pair of a controller and its assumed perceived shape areplaced in the corresponding positions. 9-mm diameter leadweights are placed on the test controllers. The white Velcro
is used to attach tracking markers.

C 1testC 2testC 3testC 4testC 5test

40%
20%
5%
0%
0%

30%
65%
10%
30%
5%

30%
0%
70%
5%
35%

0%
15%
15%
65%
5%

0%
0%
0%
0%
55%

with our system successfully presented the intended shape perception to the users.
Most of the misjudgments were made to the controllers that havethe same length or width as the intended one or to the controllers
that have similar scales (e.g., C 1test , C 2test and C 3test for S 1test , C 2test
and C 4test for S 4test , and C 3test and C 5test for S 5test ). For the test virtualshape S 1test , both C 2test and C 3test were selected many times while
C 1test remains only 40 %. Note that C 1test has a a little weak structure
due to the inner carving and it could slightly bend when participants
swing it strongly, which might have affected the shape perception.

which were arranged in a random order on the table so that theparticipants were unaware of their placement. We selected one ofthe perceived shapes and superimposed its appearance on all thecontrollers. The participant picked up the controllers with their
dominant hand in any order, compared his perception with thatof the current visual shape, and selected the one that matched
best. Before the test starts, participants are instructed to move thecontrollers in various directions for the comparison. This evaluation
was carried out four times for each visual shape (twenty times intotal) in a random order. Participants consist of 5 right handed malepeople in their 20’s.

7

LIMITATIONS AND FUTURE WORK

We proceeded with experiments with a simplified shape perception
model as the first step in exploring our concept. To extend thiswork and handle more complex shape perception, certain technicalchallenges should be addressed.
Simplification to Symmetric Planar Objects. In this work, we
specifically focused on the perception of symmetric planar objects.
Of course, there are objects that does not fit in this scope. As ourprincipal concept described in Equation 1 is general and not specific to symmetric planar objects, we believe that more complex
shapes could be handled by extending the parametrization of ourperception model. For example, perception of non-symmetric objects could be parametrized by the distance of the center of mass(CoM) from the handle axis in addition to the width and length.
Similarly, full 3D objects would be modeled within our concept
by considering additional dimensions such as thickness and the

6.2.4 Result and Discussion. Table 3 shows the percentage ofthe test controllers Citest answered for the test visual shapes Sitest .
Each row shows the rate at which each test controller was selected
when the test visual shape was displayed. For each test visual shape,
the corresponding test controller was selected the most frequently
among all the controllers. It suggests that the controllers designed
9

VRST ’17, November 8–10, 2017, Gothenburg, Sweden

Fujinawa et al.

position of CoM. Shape perception models for these cases could becomputed from additional data sets aggregated in a similar way as
our experiment. However, adequate parametrizations and effective
sampling strategies for larger dimensions (e.g., active learning) are
non-trivial, which we would leave as future work.

Alec Jacobson, Ilya Baran, Jovan Popović, and Olga Sorkine. 2011. Bounded Biharmonic
Weights for Real-time Deformation. ACM Trans. Graph. 30, 4, Article 78 (July 2011),
8 pages. https://doi.org/10.1145/2010324.1964973
Idsart Kingma, Rolf van de Langenberg, and Peter J Beek. 2004. Which mechanicalinvariants are associated with the perception of length and heaviness of a nonvisible
handheld rod? Testing the inertia tensor hypothesis. Journal of Experimental
Psychology: Human Perception and Performance 30, 2 (2004), 346–54. https://doi.
org/10.1037/0096-1523.30.2.346
Manfred Lau, Kapil Dev, Weiqi Shi, Julie Dorsey, and Holly Rushmeier. 2016. Tactile
Mesh Saliency. ACM Trans. Graph. 35, 4, Article 52 (July 2016), 11 pages. https://doi.org/10.1145/2897824.2925927
D. C. Liu and J. Nocedal. 1989. On the Limited Memory BFGS Method for Large Scale
Optimization. Math. Program. 45, 3 (Dec. 1989), 503–528. https://doi.org/10.1007/
BF01589116
Lin Lu, Andrei Sharf, Haisen Zhao, Yuan Wei, Qingnan Fan, Xuelin Chen, Yann Savoye,
Changhe Tu, Daniel Cohen-Or, and Baoquan Chen. 2014. Build-to-last: Strength to
Weight 3D Printed Objects. ACM Trans. Graph. 33, 4, Article 97 (July 2014), 10 pages.
https://doi.org/10.1145/2601097.2601168
Thomas H Massie and J Kenneth Salisbury. 1994. The phantom haptic interface: A
device for probing virtual objects. In Proceedings of the ASME winter annual meeting,
symposium on haptic interfaces for virtual environment and teleoperator systems,
Vol. 55. Chicago, IL, 295–300.
Kouta Minamizawa, Souichiro Fukamachi, Hiroyuki Kajimoto, Naoki Kawakami, andSusumu Tachi. 2007. Gravity Grabber: Wearable Haptic Display to Present Virtual
Mass Sensation. In ACM SIGGRAPH 2007 Emerging Technologies. Article 8. https://doi.org/10.1145/1278280.1278289
S. Nakagawara, H. Kajimoto, N. Kawakami, S. Tachi, and I. Kawabuchi. 2005. An
Encounter-Type Multi-Fingered Master Hand Using Circuitous Joints. In Proc. ICRA
’05. 2667–2672. https://doi.org/10.1109/ROBOT.2005.1570516
H. Omosako, A. Kimura, F. Shibata, and H. Tamura. 2012. Shape-COG Illusion:
Psychophysical influence on center-of-gravity perception by mixed-reality visual stimulation. In 2012 IEEE Virtual Reality Workshops (VRW). 65–66. https://doi.org/10.1109/VR.2012.6180884
Christopher C. Pagano, Paula Fitzpatrick, and M. T. Turvey. 1993. Tensorial basis to theconstancy of perceived object extent over variations of dynamic touch. Perception& Psychophysics 54, 1 (01 Jan 1993), 43–54. https://doi.org/10.3758/BF03206936
Michal Piovarči, David I. W. Levin, Jason Rebello, Desai Chen, Roman Ďurikovič,
Hanspeter Pfister, Wojciech Matusik, and Piotr Didyk. 2016. An Interaction-aware,
Perceptual Model for Non-linear Elastic Objects. ACM Trans. Graph. 35, 4, Article
55 (July 2016), 13 pages. https://doi.org/10.1145/2897824.2925885
Romain Prévost, Emily Whiting, Sylvain Lefebvre, and Olga Sorkine-Hornung. 2013.
Make It Stand: Balancing Shapes for 3D Fabrication. ACM Trans. Graph. 32, 4,
Article 81 (July 2013), 10 pages. https://doi.org/10.1145/2461912.2461957
Carl Edward Rasmussen and Christopher K. I. Williams. 2006. Gaussian Processes for
Machine Learning. MIT Press.
Jun Rekimoto. 2013. Traxion: A Tactile Interaction Device with Virtual Force Sensation.
In Proc. UIST ’13. 427–432. https://doi.org/10.1145/2501988.2502044
Christian Schumacher, Bernd Bickel, Jan Rys, Steve Marschner, Chiara Daraio, andMarkus Gross. 2015. Microstructures to Control Elasticity in 3D Printing. ACM
Trans. Graph. 34, 4, Article 136 (July 2015), 13 pages. https://doi.org/10.1145/2766926
Ondrej Stava, Juraj Vanek, Bedrich Benes, Nathan Carr, and Radomír Měch. 2012. Stress
Relief: Improving Structural Strength of 3D Printable Objects. ACM Trans. Graph.
31, 4, Article 48 (July 2012), 11 pages. https://doi.org/10.1145/2185520.2185544
M. T. Turvey. 1996. Dynamic Touch. American Psychologist 51, 11 (November 1996),
1134–1152. https://doi.org/10.1037/0003-066X.51.11.1134
M. T. Turvey, Gregory Burton, Eric L. Amazeen, Matthew Butwill, and Claudia Carello.
1998. Perceiving the width and height of a hand-held object by dynamic touch.
Journal of Experimental Psychology: Human Perception and Performance 24, 1 (1998),
35–48. https://doi.org/10.1037/0096-1523.24.1.35
Nobuyuki Umetani, Yuki Koyama, Ryan Schmidt, and Takeo Igarashi. 2014. Pteromys:
Interactive Design and Optimization of Free-formed Free-flight Model Airplanes.
ACM Trans. Graph. 33, 4, Article 65 (July 2014), 10 pages. https://doi.org/10.1145/
2601097.2601129
André Zenner and Antonio Krüger. 2017. Shifty: A Weight-Shifting Dynamic Passive
Haptic Proxy to Enhance Object Perception in Virtual Reality. IEEE Transactions
on Visualization and Computer Graphics 23, 4 (2017), 1285–1294. https://doi.org/10.
1109/TVCG.2017.2656978

Air Resistance and Deflection. We assumed that shape perception
is solely correlated with objects’ mass properties of objects andignored the influence of air resistance or deflection. Such effects
were negligible for small or mesh-structured objects. However, insome cases, air resistance and deflection could greatly influence theimpression of the perceived shape. For example, it is easy to imagine
that the feeling of wielding a paper fan is affected by air resistanceand deflection. To represent the impression of wielding such objects,
we must incorporate such effects into the shape perception model.
Dynamic Shape Perception. Our design tool focuses on a static
hand-held controller with a single haptic shape perception; thecontroller never changes its shape. However, our perceptual model
can be applied to dynamic perceptual shape modification. For example, it is possible to dynamically change the mass parametersand shape perception accordingly, by moving the controller’s innerweights. By employing such mechanisms on a controller, we canrender different shape perceptions without changing the controller.

8

CONCLUSION

We presented a novel computational design method for hand-held
VR controllers that uses a haptic shape illusion. Inspired by the studies in ecological psychology, we modeled a mapping from the massproperties of the controller to its perceived shape by a data-driven
approach. We prepared controllers with a variety of shapes andmass properties and conducted an experiment in which participants
reported the perceived shape of each controller. We constructeda shape perception model using regression techniques on the aggregated data and implemented a design system that can computationally design hand-held VR controllers that could reproduce
desired shape perceptions. We conducted a user study with thedesigned controllers and showed that many participants illusorily
perceived the intended shapes irrespective of the controller’s actual
appearance.

REFERENCES
Tomohiro Amemiya, Hideyuki Ando, and Taro Maeda. 2008. Lead-me Interface for a
Pulling Sensation from Hand-held Devices. ACM Trans. Appl. Percept. 5, 3, Article
15 (Sept. 2008), 17 pages. https://doi.org/10.1145/1402236.1402239
Mahdi Azmandian, Mark Hancock, Hrvoje Benko, Eyal Ofek, and Andrew D. Wilson.
2016. Haptic Retargeting: Dynamic Repurposing of Passive Haptics for Enhanced
Virtual Reality Experiences. In Proc. CHI ’16. 1968–1979. https://doi.org/10.1145/
2858036.2858226
Moritz Bächer, Emily Whiting, Bernd Bickel, and Olga Sorkine-Hornung. 2014. Spin-it:
Optimizing Moment of Inertia for Spinnable Objects. ACM Trans. Graph. 33, 4(2014), 96:1–96:10. https://doi.org/10.1145/2601097.2601157
Hrvoje Benko, Christian Holz, Mike Sinclair, and Eyal Ofek. 2016. NormalTouchand TextureTouch: High-fidelity 3D Haptic Shape Rendering on Handheld Virtual
Reality Controllers. In Proc. UIST ’16. 717–728. https://doi.org/10.1145/2984511.
2984526
Gregory Burton, M. T. Turvey, and H. Yosef Solomon. 1990. Can shape be perceived
by dynamic touch? Perception & Psychophysics 48, 5 (01 Sep 1990), 477–487. https://doi.org/10.3758/BF03211592
James Jerome Gibson. 1966. The Senses Considered as Perceptual Systems. Houghton
Mifflin.
10