Journal of Robotics, Networking and Artificial Life
Vol. 8(4); March (2022), pp. 263–268
DOI: https://doi.org/10.2991/jrnal.k.211108.007; ISSN 2405-9021; eISSN 2352-6386
https://www.atlantis-press.com/journals/jrnal

Research Article

HaptWarp: Implementation of Bend, Twist, and Press
Manipulations for Intuitive Interaction in between the
Virtual and Physical WorldLee Jen Tun1,*, R.P.C. Janaka Rajapakse2, Kazunori Miyata1
School of Knowledge Science, Japan Advanced Institute of Science and Technology, 1 Chome-1 Asahidai, Nomi, Ishikawa 923-1211, Japan
Graduate Institute of Animation and Film Art, Tainan National University of the Arts, No. 66, Daqi, Guantian Dist., Tainan City 72045, Taiwan

1
2

ARTICLE INFOABSTRACT

Article History

In virtual reality applications, such as games and training, the use of two-handed controllers to interact with virtual objects is
usually supported. To reproduce the interactive sensation of holding objects of various shapes and behaviors with both hands,
previous researchers have used mechanical connections or set various peripheral brakes between controllers to simulate physicalchanges. However, these external devices are hard to quickly adapt to for the simulation of dynamic objects, nor can they beremoved to support free manipulations. This research introduces Deformation Response virtual reality Glove, which is a pair ofsensor gloves. There is no physical link and users can stretch, bend, or twist flexible materials and display physical deformations
on virtual objects, allowing users to perceive the difference between haptic sensation and physical sensation simply by using
their hands.

Received 25 November 2020
Accepted 11 September 2021
Keywords
Interaction
game controllerinteractive bendinteractive twist
intuitive interaction© 2021 The Authors. Published by Atlantis Press International B.V.
This is an open access article distributed under the CC BY-NC 4.0 license (http://creativecommons.org/licenses/by-nc/4.0/).

1. INTRODUCTION
There is a consensus that haptic controllers have the ability to provide highly realistic immersive Virtual Reality (VR) experiences[1–3]. Although most of the haptic research has focused on building a haptic feedback with motors, Electric Muscle Stimuli (EMS),
and pneumatic devices [4–6] the realistic feedback should be generated by the object which is manipulated by the hands of the user.
In VR applications, there are many existing research of bimanual
coordinated input technology [1,7,8], but only a few examples of
haptic controllers.
Most related works describe grounded (and therefore fixed inspace) equipment [9]. However, since a physical locking mechanism between the two controllers is required to present a sense of
stiffness between the hands, the user’s speed, and differential freedom of moving the controller are limited.
For example, we tried to push, pull, bend, or twist objects which
only resulted in them being irregularly shaped. This limitation
makes it impossible to render highly dynamic gestures with manydegrees of freedom and the deformed state of objects. However,when the left and right hands perform different actions, these arevery often seen in real activities.
This article introduces the Deformation Response VR Glove, which
is a pair of sensing gloves. Unlike setting a mechanical or motor
constraints on the controller, there is no mechanical connection.
Corresponding author. E-mail: bmpsst511@gmail.com*

Based on Guiard’s conception of asymmetric division of labor inhuman skilled bimanual action [10], using both hands to operate
different flexible objects can still create the tactile illusion. Previous
research has shown that our perception of softness relies more on
touch than vision [11]. Based on this, simulating the deformation
caused by the user’s manipulation on the object is better than simulating the haptic feedback through other mechanisms or motor
constraints.
For example, various applications such as the fitness ring of theNintendo Switch [12] and GamesBond [8] have shown that thismethod effectively brings intuitive and immersive operation andhaptic feedback. Our VR Glove acquires the data on flexible materials (by bending, twisting, or pressing) through inertia, pressureand flex sensors set on the glove to render the deformed state of thevirtual object. Most of the commercial VR controllers are limited toprovide common manipulation with input device such as button,
touch pad, keyboard, and mouse. However, to provide intuitiveand immersive experiences in the virtual environment, delivering
hand manipulation is a key function, which can be observed fromHTC Vive, Oculus and Leapmotion. These pilot VR companies
are all targeting this path in the industry. In our proposed methodof Haptwarp application, is to implement bend, twist, and press
manipulations for intuitive interaction in between the virtual andphysical world.
The structure of the rest of this paper is as follows: (1) We introduce the underlying mechanism of the Deformation Response
VR Glove, explain the calculation and rendering principle of flexible deformation. (2) We show the design, implementation, andL.J. Tun et al. / Journal of Robotics, Networking and Artificial Life 8(4) 263–268

264

technical evaluation of the Deformation Response VR Glove. (3)Transform the vertex coordinates through a series of linear equations, which are used in virtual scenes to simulate the deformationof objects being bent, twisted, and pressed.

pins including the A0 pin on Microcontroller Unit (MCU), and one
customized Printed Circuit Board (PCB) with five 10k Ohms SMD
resistors to minimize the box size as shown in Figure 2. Finally, we
acquired the hand space position data through HTC vive tracker(HTC Corporation, Taiwan).

2. RELATED WORK

A clear circuit layout as shown in Figure 3. The schema display that
there’s an analog to digital signal converter connect with Wemos
D1 mini and five analog sensors (force sensor or flex sensor). To
minimize the circuit size, we adopt the 0805 SMD resistor to connect between sensor signal pin and Wemos D1 mini digital pin.

This research builds on previous work on creating illusions oftactile sense, psychophysical studies on the human perception ofsoftness and motor functions, and techniques that create the illusion of softness and provide intuitive manipulation using various
types of sensors. Also relevant are haptic feedback devices andtechniques that create movement or deformation feedback for VR.

4. PRESSED DEFORMATION
IMPLEMENTATION
To test the application of gloves to a variety of different shapes, we
simulate the pressed deformation in Unity.

2.1. Data Glove
There are many types as shown in the literature of data glove operating devices such as rotary potentiometer [13], piezoelectric sensors [14], inertial motion sensors [15], and conductive fibers [16].
However, these types require high expense or lead a bulky sizewith additional sensors. Compared with these works, we select theflex sensor and pressure sensor, which is included in the glove, forsensing.

2.2. Haptic Feedback
In the haptic research field, many research describe how they
have delivered those tactile illusions into a virtual environment toenhance the immersive experience such as TouchVR [17], PuPoP[18]. However, we believe that having the player hold an object
while we simulate the haptic feedback in the virtual environment ismore realistic compared to simulating it through motors, pneumatics, and Electric Muscle Stimuli (EMS) device.

4.1. Force Direction
The experience we are trying to simulate is the deformation of thevirtual object being pressed or poked by the user. This requires
pushing the vertices near the collision point into the surface. In
addition, we observed that the directionality of an applied force isdiffusely applied to all directions. This will cause the vertices of thefacade to be pushed away instead of moving inward.
For example, the poked point is (1,1,0), which means there is no Zaxis depth, that is of course, because we can only touch the surface

3. HARDWARE IMPLEMENTATIONThe Deformation Response VR Glove can be divided into twoparts: glove, and data box. Pressure sensors are pasted up on thefive fingertips to read the force data generated by users. Flex sensors are seamed on the glove and firmly connected to the top of thefinger to read the bending angle of each finger as shown in Figure 1.
The Data box comprises of one Wemos D1 mini which is equippedwith a WIFI module that allows us to implement the wireless datatransmission, one ads1115 which can extend analog pin up to five

Figure 2 | Inside data box.

Figure 1 | Data glove.

Figure 3 | Data Glove PCB schema.

L.J. Tun et al. / Journal of Robotics, Networking and Artificial Life 8(4) 263–268

of the object, but the force direction can be obtained through thenormal point (0,0,−1). So when we add them together, it is (1,1,0) +(0,0,−1) = (1,1,−1). That is, if the depth change is multiplied by a force
offset at the force spreading range, it can simulate the displacement
range of each vertex after being pressed as shown in Figure 4.
Where Pd represents point with depth, Ppoked represents poked point
and Np represents point along the normal line:
		 Pd = Ppoked + (N p ´ offset) (1)Then, we calculate each vertex with new depth as shown below,
where i represents the total amount of vertex.
Pd (i) = Pd (i - 1) + (Ppoked + (N p ´ offset)) 

Where i represents the amount of mesh vertices, D(i) represents
the distance between the mesh point and hit point, XmV(i), YmV(i),ZmV(i) represent that mesh vertex on three axis, Pdx(i), Pdy(i), Pdz(i)represent Pd(i) value in three axis.
(3)

And we understand that the force effect will attenuate until zerowith spread distance, and where we can adopt the inverse-square
law as Equation (4). We add one plus the square D(i) to guaranteethat the force is at full strength when the distance is zero.
force
1 + D(i)2(4)

Now that we have the attenuated force, we can convert it into avelocity delta. We know the Newton’s second law of motion F =Mass × Acceleration. And the deforming speed can be derived from
uniform accelerated motion formula V = V0 + Acceleration × Time,
where V0 is zero before object is poked. So, we can get Equation (5),where Tstep represents the frame time per second.
		 V = Fa (i) ´ Tstep (5)mass
Then, we calculate each vertex velocity as shown below. Where
Vmesh(i) represents the velocity of each mesh vertex and i represents
the total amount of vertex.
A

Then, a vertex is updated to its deformed position, throughEquation (7).
Pnew (i) = Pnew (i - 1) + (Vmesh (i) ´ Tstep ) (7)

4.3. Stay in Shape
We have applied the attenuated force for each vertex. However, thischange will not stop, and object has lost its original shape. We needto let the object bounce back to its original shape.

4.3.1. Spring effect

The mesh is deformed by an applied force. Over time, the deformation all changes vertices position as explained in the paragraphbelow. First, we need to know the distance between the poked point
and mesh vertex.

		 Fa (i) =

(6)

(2)

4.2. Add Force into Vertex Change

D(i) = ( XmV (i) - Pdx (i), YmV (i) - Pdy (i), Z mV (i) - Pdz (i)) Vmesh (i) = Vmesh (i - 1) + (D(i) ´ V ) 

265

BFigure 4 | Force direction offset. (A) This represents the force direction
before giving offset value. (B) This represents the force direction after
giving offset value.

First, we need to get the distance between displaced state and original state. Where D(i) represents the distance, and Po(i) represents
the original point.
		 d(i) = Pnew (i) - PO (i) (8)Then, we assign Vmesh(i) a value that represents the mesh reflection
speed of each point. Vfreq is used to represent the value of reflection
frequency.
Vmesh (i, j) = Vmesh (i - 1) - (d(i) ´ VfreqTstep ) (9)4.3.2. Damping
However, the reflection will continue and will not stop, so we needto give a constant CDamp as a damping property for the object tomake sure the speed will decrease until it is stopped.
Vmesh (i, j) = Vmesh (i - 1) ´ (1 - CDampTstep ) (10)Then, we can feed the new Vmesh(i) back to Equation (7) to get therealistic deformation in a virtual environment.

5. BEND AND TWIST DEFORMATION
IMPLEMENTATION
Object deformation not only has the pressed deformation, butalso have the bend and twist deformation. We will explain how to
implement them in this section.

5.1. Bend Deformation
In the beginning, we need to decide this object’s starting deformation point with lerp equation. However, the length and width in thevirtual environment is symmetrical, so we can modify the equation
as shown below. Where Pformed represents the start position, lwidth
represents the mesh width and Pfrom is an input value from 0 to 1.
		 Pformed = lwidth + Pfrom 2(11)

L.J. Tun et al. / Journal of Robotics, Networking and Artificial Life 8(4) 263–268

266

Then, we need to acquire and store each vertex from original vertex
z(i) to Pformed. Here we use the value Pz(i) to represent the original
mesh vertex point in the z axis alone.
		 Dz (i) = Pz (i) + Pformed æ Dz (i) ö

çè l
÷ø

B

C

(12)

Once we have the Dz(i), we can calculate each vertex angle of θ(i).
Where θBend represents the input bending angle.
		 q (i) = q Bend ´

A

(13)

Figure 5 | Rounded cube deformation: (A) pressed deformation,(B) twisted deformation, (C) bend deformation.

width

A

Now, we have each bending angle of each vertex, then we need toconvert into radius to get the actual displacement value Dz(i).
		 Dz (i) = Dz (i − 1) − 2 ysin(q (i)) (14)

We can update the Y coordinate value by adding the displacementof Dz(i) to get the correct y position.
		 PnewY (i) = PY (i) + Dz (i) × sin(q (i)) 

B

(15)

We can also update the Z coordinate value by subtracting the Pformed
to get the latest position after it is bent.
		 PnewZ (i) = Dz (i) × cos(q (i)) − Pformed (16)

In the end, the latest updated position will be like as below:öæ PnewX (i) ö æ
PX (i)
ç P (i) ÷ = ç P (i) + D (i) ´ sin(q (i)) ÷z÷÷ çç newY ÷÷ çç Y
è PnewZ (i) ø è Dz (i) ´ cos(q (i)) - Pformed øC(17)

5.2. Twist Deformation
Twist deformation is a typical deformation in physical objects.
We will now explain how to implement this in a virtual object. In
Equation (17), we can convert the input twist angle into an angle
value. Where, j represent the scale value which related to the sizeof virtual object.
æ P (i) ö		 q (i) = ç z
(18)
÷÷ 
ç j ´qTwist øè
We can then update the x position value by adding the correlation
value of y and further update the y position by subtracting the correlation value of x to simulate the twist deformation on the virtual
object. Since the twisting object will not affect the length of z, thez position is kept the same as the original value.
æ PnewX (i) ö æ PY (i)sin(q (i)) + PX (i)cos(q (i)) öç P (i) ÷ = ç P (i)sin(q (i)) + P (i)cos(q (i)) ÷ X÷÷çç newY ÷÷ çç Y
PZ (i)
øè PnewZ (i) ø è(19)Figure 6 | Demonstration of illusion deformation in between virtual andphysical world. (A) The illusion of pressing, (B) the illusion of bending,(C) the illusion of twisting.

grouped them into three categories: (1) the illusion of pressing, (2) the illusion of bending, (3) the illusion of twisting. The
application best demonstrates the VR glove’s unique capabilityof intuitive hand manipulation and rendering the haptic sensation of a highly dynamic and deformable object. As shownin Figure 6, the user can experience a variety of deformation
illusions such as sponge and rubber with hand manipulation.

7. LIMITATIONS AND CONCLUSION
We see numerous areas of possible improvements to this VR Glove.

6. APPLICATION

The main limitations are related to the precision of force detection
and variety of object deformations in a virtual environment. Using
five pressure sensors to render the deformation of a handheld object
is a simple solution that comes with several trade-offs. The min–max ranges of detected force are limited, and in the case of pressing,
the force is uneven depending on the direction of the fingertip.

To further illustrate the design space of the Data Glove, wedeveloped three VR applications using Unity 3D, and loosely

The flex sensor is quite useful to detect the finger bending angle,
but as we know, the finger consists of three bones, and the sensor

Finally, we can reveal these three deformation bases on the deformation algorithm as shown in Figure 5.

L.J. Tun et al. / Journal of Robotics, Networking and Artificial Life 8(4) 263–268

can only detect the angle for the end bone, forcing us to stimulate the other two finger bone movements using math. In practice,
more complex object deformation cannot be realistically rendered,and a polyhedron could make the deformation algorithm ineffective. This could be solved by, for example, using soft printable PCB
technique which pastes force sensors and flex sensors into each key
place on hand.
Furthermore, we expect that by adopting free-form surface deformation [19] and customized per-vertex stiffness map [20], we will
be able to simulate deformation for a polyhedron in a virtual environment. Finally, the finger calibration method can occasionallygenerate value, creating an unrealistic sensation. Future work will
explore various tasks that fit to everyday life as well as exploring theeffect of different hand-held objects. Theoretically, the VR glove
could be including multiple sensors to generate haptic feedback.
In conclusion, in this paper we introduced the VR glove, capable ofcreating the illusion of object deformation. We presented the overall hardware, PCB design and described how press, bend, and twist
deformations can be simulated.

CONFLICTS OF INTEREST
The authors declare they have no conflicts of interest.

ACKNOWLEDGMENTS
The authors thank the Ministry of Science and Technology for
financially and technically supporting this research in part underGrant MOST (IV) MOST 109-2218-E-002-038- and the projectfrom National Taiwan University.

REFERENCES
[1]

[2]

[3]

[4]

[5]

E. Strasnick, C. Holz, E. Ofek, M. Sinclair, H. Benko, Haptic Links:
Bimanual Haptics for Virtual Reality Using Variable Stiffness
Actuation, Proceedings of the 2018 CHI Conference on Human
Factors in Computing Systems, Association for Computing
Machinery, New York, NY, USA, 2018, pp. 1–12.
N. Ryu, W. Lee, M.J. Kim, A. Bianchi, ElaStick: A Handheld
Variable Stiffness Display for Rendering Dynamic Haptic
Response of Flexible Object, Proceedings of the 33rd Annual
ACM Symposium on User Interface Software and Technology,
Association for Computing Machinery, New York, NY, USA,2020, pp. 1035–1045.
Y. Tokuyama, R.P.C. Janaka Rajapakse, Y. Nakazawa, K. Konno,
Torque display method for free-form deformation with haptic
device, 2009 ICCAS-SICE, IEEE, Fukuoka, Japan, 2009, pp.
3803–3808.
I. Choi, E. Ofek, H. Benko, M. Sinclair, C. Holz, CLAW: A
Multifunctional Handheld Haptic Controller for Grasping,
Touching, and Triggering in Virtual Reality, Proceedings of the2018 CHI Conference on Human Factors in Computing Systems,
Association for Computing Machinery, New York, NY, USA,2018, pp. 1–13.
A. Takahashi, K. Tanabe, H. Kajimoto, Haptic interface using
tendon electrical stimulation with consideration of multimodal
presentation, Virtual Reality Intell. Hardware 1 (2019), 163–175.

267[6] K. Song, S.H. Kim, S. Jin, S. Kim, S. Lee, J.S. Kim, et al., Pneumatic
actuator and flexible piezoelectric sensor for soft virtual reality
glove system, Sci. Rep. 9 (2019), 8988.
[7] E.J. Gonzalez, S. Follmer, Investigating the Detection of
Bimanual Haptic Retargeting in Virtual Reality, 25th ACM
Symposium on Virtual Reality Software and Technology(VRST), Association for Computing Machinery, New York, NY,
USA, 2019, pp. 1–5.
[8] N. Ryu, H.Y. Jo, M. Pahud, M. Sinclair, A. Bianchi, GamesBond:
Bimanual Haptic Illusion of Physically Connected Objects for
Immersive VR Using Grip Deformation, Proceedings of the 2021
CHI Conference on Human Factors in Computing Systems,
Association for Computing Machinery, New York, NY, USA,2021, pp. 1–10.
[9] T.Y. Wei, H.R. Tsai, Y.S. Liao, C. Tsai, Y.S. Chen, C. Wang,et al., ElastiLinks: Force Feedback between VR Controllers with
Dynamic Points of Application of Force, Proceedings of the
33rd Annual ACM Symposium on User Interface Software andTechnology, Association for Computing Machinery, New York,NY, USA, 2020, pp. 1023–1034.
[10] Y. Guiard, Asymmetric division of labor in human skilled bimanual action, J. Mot. Behav. 19 (2013), 486–517.
[11] M.T. Fairhurst, E. Travers, V. Hayward, O. Deroy, Confidence ishigher in touch than in vision in cases of perceptual ambiguity,
Sci. Rep. 8 (2018), 15604.
[12] Y.I. Mari Shirakawa, Ring Fit Adventure - Nintendo, 10 (2019).
[13] A. Othman, N. Hamzah, Z. Hussain, R. Baharudin, A.D. Rosli,
A.I.C. Ani, Design and development of an adjustable angle
sensor based on rotary potentiometer for measuring finger flexion, 2016 6th IEEE International Conference on Control System,
Computing and Engineering (ICCSCE), IEEE, Penang, Malaysia,
2016, pp. 569–574.
[14] Y. Cha, J. Seo, J.S. Kim, J.M. Park, Human–computer interface
glove using flexible piezoelectric sensors, Smart Mater. Struct. 26(2017), 057002.
[15] P.C. Hsiao, S.Y. Yang, B.S. Lin, I.J. Lee, W. Chou, Data glove
embedded with 9-Axis IMU and force sensing sensors forevaluation of hand function, 2015 37th Annual International
Conference of the IEEE Engineering in Medicine and Biology
Society (EMBC), IEEE, Milan, Italy, 2015, pp. 4631–4634.
[16] R. Takada, J. Kadomoto, B. Shizuki, A Sensing Technique for
Data Glove using Conductive Fiber, Extended Abstracts of the2019 CHI Conference on Human Factors in Computing Systems(CHI EA), Association for Computing Machinery, New York,NY, USA, 2019, pp. 1–4.
[17] D. Trinitatova, D. Tsetserukou, TouchVR: a Wearable Haptic
Interface for VR Aimed at Delivering Multi-modal Stimuli at
the User’s Palm, SIGGRAPH Asia 2019 XR (SA), Association for
Computing Machinery, New York, NY, USA, 2019, pp. 42–43.
[18] S.Y. Teng, T.S. Kuo, C. Wang, C.h. Chiang, D.Y. Huang, L. Chan,et al., Pupop: Pop-up Prop on Palm for Virtual Reality, Proceedingsof the 31st Annual ACM Symposium on User Interface Software
and Technology (UIST), Association for Computing Machinery,
New York, NY, USA, 2018, pp. 5–17.
[19] J.C. Léon, J.P. Pernot, S. Guillet, F. Giannini, B. Falcidieno,
Aesthetic design of shapes using fully free form deformation
features, Proceedings of the TMCE 2004, Lausanne, Switzerland,
2004, pp. 1–12.
[20] R.P.C. Janaka Rajapakse, Y. Tokuyama, C. Sonehara, T. Hattori,
The influence of texture on perceiving haptic stiffness, Kansei
Eng. Int. J. 11 (2012), 199–206.

268

L.J. Tun et al. / Journal of Robotics, Networking and Artificial Life 8(4) 263–268

AUTHORS INTRODUCTION
Mr. Lee Jen Tun

Dr. Kazunori Miyata

He received his master’s degree from the
Department of Mechanical Design Engineering,
National Formosa University, Taiwan in
2019. He is currently a Doctoral course student in Japan Advanced Institute of Scienceand Technology, Japan. His research interest
includes human computer and robot interaction (HCI/HRI),
Mixed Reality applications.

He is a Professor at School of Knowledge
Science in Japan Advanced Institute of
Science and Technology. He received a PhD
degree (Engineering) from Tokyo Institute ofTechnology, Japan, in 1996. His research interest includes computer graphics, media art, andmultimedia applications. He is the 4th president of the Society
for Art and Science, and a member of ACM, IEEE, IPSJ, IEICE,
and others.

Dr. Janaka Rajapakse
He is an Associate Professor at the Graduate
Institute of Animation and Film Art, Tainan
National University of the Arts, Taiwan. And
he is also a visiting scholar in the Departmentof Media and Image Technology at the Facultyof Engineering, Tokyo Polytechnic University,
Japan. He was a Postdoctoral Researcher at
the Center for Hyper Media Research, Graduate School ofEngineering, Tokyo Polytechnic University. He received his
PhD in Knowledge System Science from Japan Advanced
Institute of Science and Technology in 2008. His researchinterests include computer animation, motion capture, VR/
AR/MR, haptic interfaces, AI, computer graphics, and Kansei
Engineering. He is a member of the Motion Capture Society,
The Society for Art and Science, ASIAGRAPH, and SIG- Design
Creativity.