Submitted to the Fifth Annual Symposium on Haptic Interface for Virtual Environments and Teleoperated Systems(With the Winter Annual Meeting of the ASME, November 1996)

HAPTIC AND VISUAL FEEDBACK FOR MANIPULATION AID
IN A VIRTUAL ENVIRONMENT
Haruo NOMA†

Yoshifumi KITAMURA††

Tsutomu MIYASATO†

Fumio KISHINO†††† ATR Media Integration and Communications Research Laboratories
2-2 Hikaridai, Seikacho, Kyoto 619-02, Japan†† CANON INC. Media Technology Laboratory
890-12 Kashimada, Saiwai-ku, Kawasaki-shi, Kanagawa 211, Japan††† Graduate School of Engineering, Osaka University
2-1 Yamadaoka Shuita, Osaka 565, Japan
main contact to : noma@mic.atr.co.jp
ABSTRACT:This paper proposes a new method of object manipulation in a
Virtual Environment that uses a visual and haptic feedback based
manipulation aid. If the displayed position of the object is constrainedon a face of another object in the virtual environment, a user can place
the virtual object at a precise position as easily as in a real environment.
The visual feedback based manipulation aid constrains the motion of thevirtual object as if it were moved in a real environment. Furthermore, in
simultaneous correspondence with the visual constraints, the haptic
feedback based manipulation aid constrains the motion of the user’s handin the real environment. The proposed method introduces a magnetic
metaphor: the user has the sensation of manipulating a virtual object on
a rigid surface without using a large and powerful robot arm and thetime consuming simulation of detailed physical phenomena. The
properties of this manipulation aid based on two kinds of feedback are
discussed. Experimental results show the effectiveness of our proposed
method.
Keywords: Virtual Environment, object manipulation, manipulationaid, haptic feedback, visual feedback

behavior in VE. This is one of the main reasons why the simplest VR is
not commonly used in daily work as is a 3D CAD.
Here we are interested in object manipulation in the VE. A usefulway to provide as natural sensation for precise and natural manipulation
as in RE is to restrict the excessive degrees of freedom (DOF). That is,
the system creates a virtual ruler in the VE.
There are many approaches to creating a virtual ruler. Most of themcan be classified into haptic and visual approaches. The first method,
named “Haptic feedback-based manipulation aid”, physically restricts
the motion of the user’s hand in the RE by using a special mastermanipulator called force display. In the previous example, the systemprevents the user’s hand from sticking onto the table physically in RE.
Using an ideal force display, there is no difference from manipulation in
RE. The other approach, named “Visual feedback-based manipulationaid”, visually restricts the object’s motion in the VE. Here, when thehandled object collides with another one, the motion of the handled object
is retouched to prevent geometrical contradiction in VE. The simulatorof the VE does not deal with complex and time consuming accurate
physics of the object, so it can be put to practical use in recent VR
application.
In this paper, we propose a combination of visual and haptic feedback
based approaches. In a haptic feedback based method, an Ideal force
display has to have a powerful enough reaction force, low mechanicalfriction and more than six DOF motion (three for translation and otherthree for rotation). This means that the higher-quality force display
becomes more complicated. On the other hand, the visual based system
ignores a part of the user’s motion, so there is a discrepancy in operation.
Moreover, the user depends on the object’s visual reaction to recognize
the status of manipulation. This often causes incorrect operation.
The proposed method used each approach to compensate for the
disadvantages of the other. The method introduces a magnetic metaphor
in visual and haptic feedback to restrict the motion of the handled object
and the user’s hand. This trial system reduces the contradictory effects
of using direct and involuntary feedback, so it allows the user to penetrate
an object on a table dexterously without high level of skill, the same asin the RE.
In the following section, the properties of these manipulation aids

1. INTRODUCTION
In the ultimate virtual reality system that generates perfect stimulus
for all of the user’s senses, a user could manipulate a virtual object aswell as s/he could in the real environment (RE). To simulatecomprehensive physical rules in the virtual environment (VE), the system
requires costly calculations and simulations: the avoidance of intersection
among virtual objects, the fall of a virtual object caused by gravity andfriction between virtual objects. Additionally, generating wholestimulation requires special display devices for visual, auditory and haptic
sensations. Therefore, it is difficult to realize a perfect VE.
The simplest VE using a typical VR system (a HMD and a glovelike device) requires a user to exercise skillful operation for even a simple
pick and place task. When s/he penetrates a block on a table in VE by thesimple VR system, it cannot present feelings of collision that are presented
to the user as direct feedback of the involuntary hand motion. Thus, s/he
has to pay much attention to controlling all motions of a block inmanipulation. Of course, some motions are naturally restricted in theRE, but the system needs a lot of resources for such realistic physical
1/8

dynamics of the objects. This means that a geometrical contradiction
still remains, and that the user has to monitor the aided condition in someway.
2.3 Haptic feedback based manipulation aid
The haptic feedback based approach applies a physical limitationon the user’s hand by a force display (Brooks 1990, Koutoku 1994, Ishii
1993, Iwata 1990, Noma 1993). When the user puts an object on a tablein the VE, the system detects a collision between the object and the tablein the simulation and then the force display presents a suitable reaction
force to the user’s hand.
Previously such haptic feedback systems have been studied inrobotics as a master-slave system. Their goal has been to realize an exact
physical simulation of a slave arm on a master manipulator in a RE.
Such a system requires that the robot arm suppress the human muscles,
so its design has been large and powerful. Additionally, a physical model
introduces spring and damper based physics, so it needs a high-speed
feedback controller to avoid vibration in simulating a rigid surface.
As mentioned in the first section, the force display has to follow theuser’s motion in the entire working area and generate enough reaction
force to restrict the user’s motion to our purpose. Therefore, as with themaster-slave system, the force display has to be designed as a large andpowerful device with enough DOF and movable volume. However, it ishard to design such an ideal force display because of the physicallimitations.
The force display shown in Fig. 2 is used in a trial system in a later
section. Although we have designed it as a desktop force display and it isadequate for this purpose, it is insufficient for more general application.
This force display has three actuated joints to generate a reaction force at
the grip on the end of the display, where it generate maximum 10 N
reaction force. It is insufficient to present a highly stiff surface for moregeneral purposes. Moreover, the grip is connected through unactuatedthree axis universal joints, so it can follow and measure a six DOF motionof a user’s hand but cannot physically restrict rotational motion.
2.4 Combination of the visual and haptic approaches.
As mentioned above, each of these two methods has its own
disadvantage. The proposal method can complement each method's
disadvantages with the other's by using both visual and haptic feedback
based manipulation aids. Such an approach allows a user to manipulate aare discussed and a trial configuration is introduced. Experimental results
show the effectiveness of our proposed method, which uses both visualfeedback and haptic feedback based manipulation aids.
2. MANIPULATION AID USING VISUAL AND HAPTICFEEDBACK
2.1 Assistance for object manipulation
When an object has no intersection with other objects in a RE, it
can be freely moved and rotated in any direction by the user. When theobject is placed on the surface of another object, its motion is physically
restricted. To illustrate these fundamental properties, consider the simple
task of aligning cubes on a table. A cube has six planar sides that areperpendicularly connected. Ignoring rolling off a table, the DOF of acube placed on the table is reduced to three (two translation and one
rotation.)
In a VE, the motion of a user’s hand is usually reflected in the
motion of the handled cube. As mentioned in first section, ideally it isnecessary to reduce the DOF of both the user’s hand and the cube toprecisely place the cube on the desk in the VE. From this point of view,
the properties of visual and haptic feedback based manipulation aids are
first explained in full. Then we will propose an effective combined
method.
2.2 Visual feedback based manipulation aid
The simplest approach to support the accurate positioning of objects
in a VE is to use visual constraints among objects. This restricts theobject’s motion but does not restrict the user’s hand motion, so there isdiscrepancy between the actual user’s hand and the displayed object’s
position and orientation. An illustration of this approach is shown in
Fig. 1-a. The original DOF of the virtual object is six, but DOF is restricted
to three when the object is placed on a table. The actual sensor positionand orientation of the object are controlled by the user’s hand input. The
displayed object position is modified after constraints are applied. Fig.
1-b shows a sample manipulation using this method. Although the imageof the statue is stood straight on the desk, the line drawing statue indicatesthe real position of the hand.
Much literature is devoted to visual feedback based approaches(Baraff 1989, Chanezon 1993, Fernando 1993, Kijima 1995, Snyder
1995), but a user may have a sense of incompatibility caused by thedifference between visual feedback and kinesthetic sensation. The key
solution in this approach concerns how the manipulation aid modifies
object position to provide intuitive and compatible assistance to the user
while the handled object interacts with other ones.
The drawback of incompatibility is reduced by using a ‘magnetic
metaphor’ to express the state transition of objects (Kitamura 1995).
The handled object visually behaves as if it is attracted by a pseudo magnet
on the face of an object. When exiting from the aided condition, the user
feels as if s/he detaches the pseudo magnet. Although the magnetic
metaphor solves the above drawback, it only visually simulates thedisplayed position

sensor position

locus of the hand

locus of displayed
cube(a) Concept of the visual constraint(b) Example in the Virtual Environment

Figure 1: Constraints of object's motion by the visual feedback based manipulation aid
2/8

manipulated
cube

rotationmanipulated
cube

B
C
A

translationtranslation(a) one-face constraint(b) two-face constraint

manipulated
cube(c) three-face constraint
Figure 3: Constraints among faces for object manipulation

Figure 2: The force display with TOCUS
virtual object precisely and naturally with these limited resources.
As the visual approach introduces a magnetic metaphor, the reaction
force acts not as a rigid surface simulator but as a direct indicator of themanipulation mode to haptic sensation.
When the handled object is restricted on one face, the reaction force
is proportional to the distance of the object from the restricted face. In
short, the user’s hand is kept around the face without the user’s conscious
motion control. This simulates an elastic surface, so the highly stiff force
display is not necessary.
Naturally, the direction of the constrained object is determined fromgeometrical conditions, so the user controls only unrestricted rotation of
the object on the face. This means that the force display does not needactuators to restrict the hand’s rotation.
When the user intends to pull the handled object from the face, s/he
has to intentionally pull it against the reaction force. Moreover, thedisplayed object is stood straight on the surface visually, so this method
is expected to reduce discrepancy between the haptic sensation and thedisplayed object's motion so that the user feels that s/he is manipulatinga virtual object on a rigid surface. The point is that these feedback methods
provides highly direct manipulation, so the user can focus attention on
only manipulating the object's unrestricted DOF.
The mechanism is described in greater detail in a later section.

constraints among faces to an assembly task that uses multiple objects
with complicated shapes, it is necessary to dynamically detect constraining
faces according to object motion rather than to limit detection topredefined faces in advance. For this purpose, the method has to detectcollisions or interferences among objects. In this paper, we use a methodof real-time colliding face detection for polyhedral objects withcomplicated shapes (Smith 1995). This method can detect colliding pairsof faces within 70 milliseconds when the objects have less than 4000
faces.
3.3 Visual feedback based method
The basic flow of visual feedback based manipulation aid is shownin Fig. 4, and only the essential points are discussed in this subsection.
3.3.1 State transition to constraint
An object with a complicated shape may have a number of colliding
face pairs detected by the above collision detection stage. By examining
the geometry between collision pairs and the speed of interaction, theface to be constrained can be dynamically selected. If a new constraint
face pair is found, the displayed object position is modified to reflect the
new aided position. Translation and rotations are applied to the object tomove the selected face in a parallel manner onto the constraining face.
The constrained faces for both objects change color.
The motion of the manipulated object is constrained on the faces.
In the one-face constraint mode, the constrained translation is determinedby projecting the change in the translational hand motion vector (sensor
data) onto the planar of the target face. The constrained rotation angle isdetermined by using the change in angles of hand motion (sensor data)
and the direction of rotation. In the two-face constraint mode, theconstrained translation is along a vector orthogonal to both constrained
face normals.
3.3.2 Release from constraint
Since the visual approach uses an intuitive magnetic attraction to
constrain an object, the method for releasing the object from a constraint

3. METHOD OF MANIPULATION AID ON A TRIAL SYSTEM
3.1 Overview
The trial system is described in this section. In this VE, all objects
are modeled as polyhedra (boundary representations), are rigid (nondeformable), and can be concave or convex. We do not calculate andsimulate the fall of virtual objects caused by gravity and friction between
objects.
The proposed method is a technique that restricts the motions of
the user’s hand and the handled object. For simple objects that havefaces perpendicularly connected, we consider three kinds of constraints
as shown in Fig. 3. In the (a) one-face constraint mode , the object has
one constrained face and the motion is constrained to three DOF: twotranslations on the planar of the target face and one rotation around thenormal of that face. In the (b) two-face constraint mode, the object has
two constrained faces and motion is constrained to one DOF translation
along the two faces. In the (c) three-face constraint mode, the object has
three aided faces; thus, there is no DOF when the object remains
constrained to all three faces.
3.2 Real-time colliding face detection.
In order to flexibly apply a manipulation aid method that uses thecollision detection
find best face pairsnap object to surface
constrain object motion
on surface

Figure 4: Flow of visual feedback based manipulation aid
3/8

reaction
force

distance_thresholddistance_threshold

Actionapproach

snap

constraint
on faceunsnappush

neutralpull

max pullfreeManipulation Aided Phase(a) Reaction force with one-face constraint
Figure 5: State transition and presentation of haptic sensationin manipulation

reactionforce Fis also intuitive. The release action is like unsnapping something from amagnetic surface. There are two conditions under which a constrained
object may unsnap from a face: overlap ratio and distance from face.
Satisfying either of these conditions is sufficient for an unsnap.
[Overlap Ratio]
The overlap ratio is the ratio of overlap area of the area of the two
faces to the smaller face. If this value is less than the overlap_threshold,then the manipulated object unsnaps from the face.
[Distance from Face]
The distance-from-face condition uses the distance of the grasped
position to the constrained face of the target object. When this distancebecomes larger than the dist_threshold, the manipulated object unsnaps
from the constrained face. Here, dist_threshold is a dynamic thresholdthat varies with the overlap area given by
dist_threshold = kAdistance_thresholddistance fromsurface
Inside face

Free space(b) Distance versus reaction force
Figure 6: Relationship between distance from the constrained
face and generated force
not visually reflected while manipulation is aided, the reaction force acts
to reduce the geometrical inconsistency between the user’s hand and therestricted object, as shown in Fig. 6(a) and (b). It must be noted that
although the user’s hand roughly moves above and below the constrained
face, the user does not have to pay strict attention to keeping his or her
hand within the dist_threshold.
As mentioned above, when the inconsistency is larger thandist_threshold, the manipulated object is unsnapped from the constrained
face. The force display applies maximum attracting force at the
dist_threshold, and when the object is unsnapped this reaction force issuddenly cut off. This means that the user must consciously tear off theobject against the force when s/he wants to unsnap the object.
3.4.2 Force display with TOCUS
The force display shown in Fig 2 is actuated by custom-made torquecontrollable ultrasonic motors (TOCUS). The TOCUS is driven byfrictional force and is completely different from the typical electric
magnetic motor (EMM) used in many fields. TOCUS has many features,the most interesting of which is its ability to act as both a motor and abrake.
Fig. 7 shows the principles of the TOCUS. The TOCUS consists ofa rotor and a stator. A piezo-electric device and an elastic material aremounted on the surface of the stator. The rotor is pressed against thestator with a spring. In the power-off mode, as the surface of the elastic
body is glued to the rotor, the TOCUS acts as a brake (Fig. 7-a). Whentwo synchronized sine signals are supplied to the piezo-electric device,
the piezo-electric device and the elastic body vibrate together in thevertical direction. Under this condition, the sustaining torque is reduced(Fig. 7-b). The grade of deformation is in proportional to the voltage ofthe supplied signals. In our TOCUS, this control mode is used to control
the sustaining torque.

(1)

where A is the overlap area and k is a positive parameter. Therefore,
we assume that the greater the contact area, the greater the distancerequired to unsnap from the surface.
3.4 Haptic feedback based method
3.4.1 Haptic representation of state translation
The haptic feedback is designed to allow a user to perceive themode shift unconsciously with haptic sensation. Fig. 5 shows therelationship between the manipulation phases and the haptic
representation of each phase.
When the system detects no constraint, the force display releases
all joints and the user has no physical restrictions. When the phase isshifted to snap (from (a) to (b) in Fig. 5), the force display locks all jointof the force display in a short time and displays an impact on the hand tostop motion.
While the object is constrained on the face (Fig. 5 (c)), the force
display simulates a spring and damper system in the vertical direction of
the constrained face.
As mentioned above, the magnetic metaphor is also introduced in
haptic feedback. When a real magnet placed on metal is pulled off at a
certain point, it generates as much reaction force as applied force. Whenapplied force exceeds a threshold, the magnet is suddenly separated fromthe metal.
In our technique, the same behavior is utilized. One different is thatthe reaction force is proportional not to the applied force but to the distancefrom the face. As the user’s motion along to the restricted direction is
4/8

stereo viewer
with 6 DOF
trackerrotor

70-inchdisplayrotor

stator

1280x480
stereo3 DOF
force
display

speaker

TOCUS(a) Maxmum sustaining
torque mode in the poweroff state (no signals)

38400bps

elastic bodypiezo-electric
devicestatorMIDI(b) Low sustaining torquemode in the power-on
state(synchronized signals)

PC

Ethernet

Fastrack

19200bps

Graphics
workstation
SGI ONYX
R4400/150 MHz

Figure 8: Hardware setup for virtual object manipulation using
visual and haptic feedback based aid
vertical vibrationYtilted planar surface

Rotational Direction(c) Rotational torque modein the power-on state(unsynchronized signals)

Handled cube

ellipsoidal vibration

Z

Figure 7: Torque controllable ultrasonic motor

X
When two sine signals whose phases are shifted 90° apart aresupplied to the piezo-electric devices, the surface of the elastic bodyrotates elliptically and the traveling wave is raised on the stator (Fig. 7c) The traveling wave generates a frictional force. The frictional force
depends on the velocity of the traveling waves, so it can be controlled by
the distance the two signals’ φ are phase shifted apart.
Additionally, it has a light and compact structure. It can generate
twice as much reaction force per unit weight as a typical EMM. It works
most effectively and silently at low speed. This gives the TOCUS an
advantage over the EMM as an actuator for a force display. Furthermore,
as TOCUS is driven by a frictional force, it almost never generates
magnetic noise. This means that the TOCUS never disturbs theelectromagnetic field of the magnetic positioning sensor that is widelyused in VR-systems.

Figure 9: Experimental setup measuring distance difference
during object manipulation on a planar surface.
are snapped together.
4.2 Mode definition of manipulation aid
In the following two experiments, virtual object manipulation andplacement, each of the following four modes are compared.
Mode V: With visual feedback based manipulation aid, but nohaptic feedback based manipulation aid.
Mode F: With haptic feedback based manipulation aid, but novisual feedback based manipulation aid.
Mode VF: With both haptic feedback based manipulation aid andvisual feedback based manipulation aid.
Mode NO: No manipulation aid. This is the same as the simplest
VR configuration with 3D glasses and a glove-like device.
This mode is used for reference.
4.3 Exp.1: Object manipulation on planar surfaces
An experimental method to measure the discrepancy during object
manipulation on a planar surface is described. The purpose of thisexperiment was to confirm the effectiveness of the proposal manipulation
aid while the handled object is restricted. Subjects were asked to move a
7 cm cube on a tilted planar surface for about 20 seconds as fast andprecisely as possible under the one-face constraint as shown in Fig. 9.
All users were tested on four tilted target surfaces with four manipulation
modes. Here, the discrepancy between the position of the hand (sensor
position) and the constraining surface is measured.
4.4 Exp.2: Object positioning task

4. EXPERIMENTAL METHOD
Two experiments were conducted to examine the features of thehaptic and visual feedback based manipulation aid and to determine theeffectiveness of the proposed method.
4.1 System for virtual object manipulation
Fig. 8 shows the hardware configuration of the experimental system.
All input and output devices and sensors were controlled by the SGIONYX™ workstation. Two 70-inch CRT projectors were used to present
position-tracked stereoscopic images. User eye position was derived fromthe Fastrack™. Accordingly, the system could present nondistorted
images with depth sensations and motion parallax. A user can grasp andmanipulate objects with his or her hand by using the Fastrack™. At thesame time, the user feels the reaction force provided by the force display
to simulate magnetic attraction. In addition to visual and haptic feedback,a MIDI sound generator makes a wooden knocking sound when objects
5/8Locus of the hand along vertical direction of face (cm)Mode V

70mm
35mm
140mm
35mm

(a) Task A: one-face constraint

70mm

35mm
140mm
35mm

(b) Task B: two-face constraint

70mm

10

20

Mode F

1020

Mode VF

1020Mode NO

35mm
35mm

0

140mm

10

20

Time sequence (seconds)(c) Task C: three-face constraint

Figure 11: Typical locus of the hand along vertical directionof constrained face during object manipulation on planar
surface

Figure 10: Experimental tasks for accuracy evaluation
An experimental method was used to examine and compare theaccuracy of object placement in each of the four modes with/without
haptic and visual feedback based manipulation aid.
4.4.1 Task definition
Three tasks were designed for testing one-, two- and three-faceconstraint. Fig. 10 shows the task configurations.
Task A: One-face Constraint — consists of two 7 cm cubes thatwere initially separated in the horizontal and forward directions. The
task is to place the front cube on top of the other cube, aligning all four
corners of the faces, as shown in Fig. 10(a).
Task B: Two-face Constraint — consists of three 7 cm cubes; twoof the cubes share an edge and form a right angle surface, and the third
cube is initially separated from the lower cube as in the above task. The
task is to place the third cube into the space between the right angle
surfaces formed by the other two cubes, aligning all six vertices, as shownin Fig. 10(b).
Task C: Three-face Constraint — consists of four 7 cm cubes; thesame configuration as for task B but with a fourth cube having sharededges with the previous two cubes, forming a right angle corner. The
task is to place the forth cube into this corner, aligning all seven vertices,
as shown in Fig. 10(c).
4.4.2. Experiment
The purpose of this experiment was to compare the accuracy ofobject placement in each of the four modes previously described. Subjects
were asked to complete tasks A, B and C as accurately and quickly as
possible. Ten trials were done for each mode and task. In this experiment,
three measurements were taken: task completion time, distance accuracy,
and angular accuracy. The task completion time was the time between

object grasp and release, as measured by a computer real-time clock.
Each task consisted of one object grasp and one release; accordingly theuser could not readjust the object after it had been released. The distanceaccuracy was the sum of the distances among the four, six or seven verticesof the cubes in tasks A, B and C, respectively. The angular accuracy was
the sum of the three angle errors (azimuth, elevation and roll) from thetarget position.
5. Results and discussion
This section describes experimental results and considerations of
two experiments.
5.1 Results of exp.1: manipulation on a constrained faceFig. 11 is a time series graph that indicates the distance between theactual user’s hand and the target planner surface while the cube is moved
on it. The cross-dot line at zero indicates that the cube is just on the face.
The result depends on whether the haptic feedback based method aid isused.
In the group without the haptic approach, when the cube is restrictedon the constrained face in the V mode, the users did not have to pay somuch attention to operation. On the other hand, in the NO mode, thesubjects had to always adjust the whole cube’s motion, as shown in the
result. Consequently, the width of the distance error in the V mode islarger then in the NO mode.
The results with F and FV showed less distance error and less widthof vibration than with the V and NO modes. This means that the reaction
force allowed the subjects to move the cube near the surface. The loci of
F and FV, however, look almost identical in the result. In the
6/8

Table 1: Standard error of distance between actual hand
position and constrained face during object manipulation onplanar surfaces

Table 2: Averages of position and angular error for tasks A, B
and C from all subjects
T

Plane surface

T
A

B

C

D

V

2.3

3.2

3.0

1.5

F

0.5

0.9

0.5

0.5

VF

0.7

0.9

0.5

0.7

NO

1.2

1.8

1.3

1.3
(cm)

configuration, both modes used the same haptic feedback model, so that
there is no difference in the loci. The difference in them depend onwhether the cube was visually restricted in the surface; therefore, thehandled cube was swinging on the surface as it directly followed theswinging user’s hand motion in mode V. Furthermore, this haptic
feedback cannot reduce the offset in the mode. We assumed that this isbecause a small force was absorbed by the inertia and the viscosity of
the force display. Therefore a small reaction force is always used as aclue for manipulation. Even so, the cube in the mode VF is stood always
straight on the surface.
Although the graph is a typical result from one subject, all subjects
indicate almost the same tendency, that is other subjects showed nearly
the same results. Table 1 indicates the standard errors of the distancefrom the results of all subjects for four surfaces (A, B, C, D). It was
found that the haptic feedback based method (F and FV) could reduce
vibration during manipulation. As for modes V and NO, the amount oferror in mode V is larger than in mode NO. Therefore, we assumed thatthe user could manipulate an object easier in mode V than in mode NO,
however, as mentioned above, this may lead to unsatisfactory operationsuch as sudden unsnapping of the object. Finally, it was concluded that

A

V
F
VF
NO

B

V
F
VF
NO

C

V
F
VF
NO

Angular error (deg)Positional error (mm)mode FV has the advantages of both the visual and haptic feedback based
manipulation aids.
In this experiment, we focused on the distance discrepancy of theuser’s motion to evaluate the proposed method. We should focus on
subjective and physiological accepts in future work.
5.2 Results of exp.2 : the object arrangement
5.2.1 Accuracy of the arrangement
We used five subjects for this experiment, each of whom had
sufficient training before the experiment. Fig.12 (a) and (b) show final
position and angular error versus task completion time for task A (oneface constraint) of one subject. Results show that positional error andtask completion time depended on whether visual feedback based method
is applied. The subject could complete a task in mode F faster and more
precisely than in mode NO. However, their performances were worse
than with modes V and VF. This is because that the subjects feel only asmall reaction force around the goal in these configurations. Furthermore,
the force display could not restrict rotational motion, so the angular error

Task completion time (seconds)

Task completion time (seconds)

Figure 12: Positional and angular error versus task completion time for task A of one subject
7/8

sensation has many unique features and can indicate the condition of
impact and the material of the object. Therefor it is expected to increase
the intuitiveness of the VE.

Distance Difference (mm)References
Baraff, David. Analytical methods for dynamical simulation of nonpenetrating rigid bodies, Computer Graphics, Vol. 23, No. 3, pp. 223232, 1989.
Bouma, W. J. and Vanecek, G. Jr. Modeling contacts in a physically
based simulation, proc. of Symposium on Solid Modeling andApplications, pp. 409-418. ACM, 1993.
Brooks, F. P., Ough-Young, M., and Batter, J. Project GROPE-Haptic
Display for Scientific Visualization, ACM Computer Graphics, Vol. 24,
No. 4, pp.177-185.1990
Task A

Task B

Chanezon, A., Takemura, H., Kitamura, Y., and Kishino, F. A study ofan operator assistant for virtual space, proc. of Virtual Reality Annual
International Symposium, pp. 492-498. IEEE, 1993.

Task C

Figure 13: Average and standard errors of distance differenceof tasks A, B and C for all subjects

Fernando, F. and Dew, P.M. Interactive constraint-based solid modelingusing allowable motion. In Symposium on Solid Modelling andApplications, pp. 243-252. ACM/SIGGRAPH, 1993.

in F mode is no different from the error in NO mode. These problems
were solved by the visual feedback based method. In other words, thesubject could achieve higher precision in a shorter time with the visualfeedback based method. Comparing modes V and FV, results show thattask completion time in mode FV was shorter than in V, but there wassignificant difference in the errors.
Table 2 shows the average positional and angular errors for all tasks
from all subjects. In every task, each subject could get more precise
results in V and VF modes than in F and NO modes.
5.2.2 Distance and Angular difference
Next, we focused on the difference between V and VF modes. Fig.
13 shows averages and sampling errors of the positional difference
between the actual hand position and the retouched object position intasks A, B and C of all subjects. In all manipulation modes, the positional
difference in mode VF is smaller then in mode V. This means that themethod with both visual feedback and haptic feedback can allow a user
to precisely manipulate an object without much discrepancy.
Angular difference was measured in the same way. Since the systemdoes not restrict rotation, there was no meaningful difference in the results.

Kijima, R. and Hirose, M. The impetus method for the object
manipulation in VE without force feedback, proc. of Symbiosis of Humanand Artifact, pp. 479-484, 1995.
Kitamura, Y., Yee, A., and Kishino, F. A sophisticated manipulation
aid in a VE based on the dynamic constraints among object faces, proc.
of International Conference on Systems, Man and Cybernetics, pp. 46654672. IEEE, 1995.
Koutoku, T., Takamune, K., and Tanie, K. A VE display with constraint
feeling based on position/force control switching, proc. of International
Workshop on Robot and Human Communication, pp. 255-260. IEEE,
1994.
Ishii, M. and Sato, M. A 3D interface device with force feedback: avirtual work space for pick-and-place tasks, proc. of Virtual Reality
Annual International Symposium, pp. 331-335. IEEE, 1993.
Iwata, Hiroo. Artificial reality with force-feedback: development ofdesktop virtual space with compact master manipulator, Computer
Graphics, Vol. 24, No. 4, pp.165-170, 1990.
Noma, H. and Iwata, H. Presentation of multiple dimensional data by 6
DOF force display, proc. of IEEE/RSJ Intelligent Robots and Systems,
Yokohama, Japan: pp.1495-1500, 1993.

6. Conclusion
This paper proposed a new method of object manipulation in a VE
using visual and haptic feedback based manipulation aids. Although thetrial system employed a six-DOF force display, only three of them wereactuated for translation. This means that the force display did not restrictrotation of the user’s hand. The visual feedback based manipulation aid
can support this insufficient DOF. On the other hand, the haptic feedback
based manipulation aid can reduce the discrepancy between the actualuser’s hand and displayed object position. Using the proposed method,
the user’s hand could always follow the constrained object motion.
Experimental results confirmed that the system allows the user to precisely
and intuitively manipulate the object in the VE.
It is expected that our method will make the user believe that s/he ismanipulating a virtual object on a rigid surface without the use of a large
master arm and the time consuming simulation of detailed physicalphenomena. This method can be easily applied to the many fields suchas a 3D-CAD and 3D simulation. In future work, we should confirm
which factor is most effective in this configuration. Also, we should
introduce another sensing channel. In this configuration, the system
always used the same sound to indicate collision between objects. Sonic

Sayers, C. P. and Paul, R. P. An operator interface for teleprogramming
employing synthetic fixtures, PRESENCE, Vol. 3, No. 4, pp. 309-320,
1994.
Smith, A., Kitamura, Y., Takemura, H., and Kishino, F. A simple andefficient method for accurate collision detection among deformablepolyhedral objects in arbitrary motion, proc. of Virtual Reality Annual
International Symposium, pp. 136-145, IEEE, 1995.
Snyder, John M. An interactive tool for placing curved surfaces without
interpenetration, proc. of Computer Graphics, Annual Conference Series,
pp. 209-218. ACM, 1995.
Venolia, Dan., Facile 3D direct manipulation, proc. of INTERCHI, pp.
31-36. ACM, 1993.

8/8