MVA2000 lAPR Workshop on Machine Vision Applications. Nov. 28-30, 2000. The University of Tokyo. JapanA Unique Recovery of 3D Shape and Size from Motion
Akira Aillano *Tsuyoshi Migita *Department of Intelligent Syste~nsHiroshima City Ulliversit,yAbstract
Fact,orization is one of the most practical lnethocl
to recover 3D shape arid motion simultaneously fro111
2D images with stable and fast computation. However, there still remain two crucial problems to besolved in real situations; one is to determine the true
shape from a pair of visually equivalent candidates
and the other is to measure the actual size of theobject. This paper presents a method to solve theshape and size problems by the factorization withaction; that is, 3D recovery is performed from animage sequence with a known trajectory of a single feature point given by the compiiter-controlled
robot hand, arid we determine the shape and size by
evaluating the co~isistencybetween computed shapeand the given trajectory. Experi~nentsresults performed in sir nu la ti or^ study and in real world haveshown the effectiveness of our niethocl.

1Introduction

Naoki A.sada *paradigm Active Recognition in which vision problems are solved with physical information obtained
actively by touching or pushing the objects. In thispaper, we present an active recognitio~iapproach toovercome the limitation of the conventional vision
paradigm.
The factorization is one of the most practicalmethod to recover 3D shape and motion simultaneously from 2D images with stable and fast computation. However, there still remain two crucial
problems t o be solved for the succeeding action task;that is.
Shape uncertainty:
While the factorization yields a unique solution mathematically based on the linear algebra, there are two candidates of 3D shape physically. This uncertainty comes from that the axisof depth is reversible in the orthographic projection. This is know11 as a Necker's reversal or
depth reversal problem.
Size uncertainty:

A major concern in computer visiori is to recover
3D shape from 2D images. Several methods havebeen developed so far to estimate depth information, from a set of images taken by varying a lensparameter in depth from focus[l], from a pair of images in shape from stereo[2], from a set of images
taken under a structured illurninatio~lin photometric stereo[3], from a image sequence taken by a nioving camera in factorization[4] or in active vision[5],and SO on.
In all vision studies, problems are solved tincler
an intrinsic condition; namely, nothing but the visual information is available. In other words, theobserver is not permitted to alter tlie scene configuration but allowed to watch the scene.
In practical situations, however, the output of vision task such as 3D shape and position of an object
is expectccl to be an input of action task such thatthe object is pushed, picked up or moved. This concatenation of successive two tasks suggests a new
Address: 3-4-1 Ozuka-Higashi, Asa-hlina~ni,Hiroshima
73 1-3194, J~ ~ , - ~~ , ~ {a-amanoi l ~:~ ,migita,,asada) Qcv. its.
hiroshima-cu. ac jp.

The factorization tells no infor~nationabout theactual size of the object. This is why the arbitrary scale factor is involved in the projective
transformation.
This paper proposes a method to solve theseshape and size problems by the fa.ctorization withaction; that is, 3D recovery is performed from animage sequence with a known trajectory of a single feature point given by the computer-controlled
robot hand, and we determine the shape and size by
evaluating thc consistency between computed shapeand the given trajectory. In a situation where aremote robot having camera and nia~iipulatorthat
is controlled by a human operator, such a trajectory is once obtained by pushing an object througha human-aided operation, the robot can con~putetheprecise shape and size of the object so that an action task such that the object is picked up would beperformed.
In the followilig section, we first discuss the shapeand size problems in the formulation of the factorization, then present the method to reach the unique' image p~

axobjectFigure 1: Image and object-centered coordinate systems.
solution by using a known trajectory. Finally, experimental results carried out in simulation as wellas in real world will be shown to demonstrate theeffectiveness of our method.

22.1

Solution of Shape and Size Problems
Shape and size uncertainties

Suppose a situation where an image sequence ofa moving object is taken by a static camera and allfeature points on the object are tracked throughout
the image sequence.
Using the following notations, (Fig. 1)

sP = (xp, yp, z ~ ) 3D~ :object-centered coordinates of the pth feature point, where 1 5 p 5 P.

mf and n f : 3D vectors representing the x andy axis of the image plane, respectively, where1lfSF.
(xfp, yfp): 2D image coordinates of the pth feature point in the f t h frame.

[a cyf IT: 2D translation vector of the object
center in the iniage coordinates.

Figure 2: Depth reversal problem. Is vertex A convex or concave?
depth reversal problem; that is, the object is convex
or concave around the point A in Fig.2. The scale
factor z is necessary to represent the actual size ofthe object, and there is no way to determine z in theprojective transformation.

2.2

Factorization with a known trajectory

To obtain a unique solution of 3D shape and size,
we solve Eq.(2) by the factorization with a known
trajectory of the object. Such trajectory is given by
a computer-controlled robot hand whose coordinates
is calibrated with respect to the camera coordinate
system. The object is pushed by the end of the robot
hand, and the object motion is recorded in a image
sequence together with the accurate 3D position of
the robot hand. Thus, we have many 2D trajectoriesof the feature points detected from the images andone 2D trajectory as a projection of the known 3Dtrajectory. We define the ( P 1)th feature pointby the given 3D trajectory which is represented as avector v .

+For each q E (1, -1) in matrix Q, 3D coordinates(XfPq,Yfpq,Zfpq) of the pth feature point in the f t hframe are calculated from Eq.(4),

we have the relation between 3D and 2D coordinates,
as follows,

For the physical interpretation, it is necessary tointroduce a matrix Q = diag(1, 1, q), where q E(-1, l ) , and a scale parameter z E 72,thus we obtain,(2)
The matrix Q represents a pair of visually equivalentcandidates of 3D shape which corresponds to the(4)where 1 denotes the focal length of the camera, andz, is a scale factor for each q E (1, -1).
Using all 2D trajectories including the 2D,projection of ( P 1)th feature point, 3D trajectories arecalculated by the factorization with Eq.(4), and wehave the computed 3D trajectory of the ( P 1)thfeature point which is denoted by a vector u,,++#optical axis
view point

Figure 3: Relation between give11 trajectory v andcomputed trajectories u*l.

Figure 4: Experimental environment.

From the relation zquq = v, the scale factor z, foreach q are deterniined bv the least squares method.
iq= (u;v)/(u;uq).

(6)

Figure 3 illustrates the relation between v and u,.
Now, we can determine that the true shape either ofq = 1 or q = -1 by evaluating the remainder ofllzqu, - d l 2 .

3
3.1

(7)

Experiments(a) T h e first frame..
(b) The last frame

Simulation study

We used a cube(100 x 100 x 100mm) for the simulation study. The motion was given in 30 degrees rotation and 200nnn horizontal translation from rightto left. The object is viewed from a camera placed a t950mm distant and 345mm above from the object.
From sixty one 2D trajectories of the featurepoints in fifty frames and one known 3D trajectoryof the top right vertex of the cube, the shape andsize were recovered by the proposed method. We
obtained two candidates of the scale factor, i.e. zl
and z- 1, by using Eq.(6). The remainders of Eq.(7)corresponding to q = 1 and q = -1 were 1.40 x lo2
aricl 4.12 x lo4, respectively. Thus, we can determinethat the shape and size of q = 1 is true.
Using the scale factor zl , the estimated size of theobject was 103.6 x 104.4 x 104.4mm. While we have
13% volumetric error, it is important that we canestimate the real size of the object which is not obtained in the conventional factorization. Note thatthe error of the shape and size mainly conies fromthe linear approximatio~lof the perspective projection.

3.2.

Experiment in real environment

Figure 4 illustrates our experimental environment
in remote operation environment. The remote operator first moves the object manually by the robot

Figure 5: An image sequence of a block pushed byrobot hand.

hand, then the unique shape and size of the object
is recovered by the proposed method.
We used a cork block whose size was 43 x 49 x
86mm, and the object was pushed by a robot hand(MITSUBISHI RV-E3-ST) which is controlled by
the remote operator through a computer network.
The object is viewed by a camera (SONY EVI-G20)
a t llOOmm distant and 300mm above. The hand-eye
calibration were performed in this environment.
Fifty seven feature points detected on the object are tracked by the CMU method[6] throughout
sixty five images while the object tra.vels along 60mmtranslation pushed by the robot hand. The first andlast frames are shown in Fig. 5. Figure 6 shows thefeature points in the first image.
The remainders of Eq.7 for q = 1 and q = -1
were 4.8 x lo2 and 2.3 x lo3, respectively. Thus
we have found that q = 1 gives the true shape andsize of the object. Figure 7 shows the trajectoriesof u1 and u-1 with the known v trajectory drawn
in the bird-eye view. It is clear that u l shows goodconsistency with v and u-1 does not.
Using the scale factor z1 obtained by Eq.(6), theestimated size of the cork block was 41 x 49 x 83mni.

Figure 6: Detected feature points in tlie first frame.

(a) q = 1(b) q = -1

Figure 8: Side view of the recovered object.

Figure 7: Consistency evaluation between v andu*1.

The volumetric error was a t most 8.0%, and such agood result shows that our method works successfully in real environment.
The side views of the recovered object corresponding to q = 1 and q = -1 are shown in Fig.
8. The horizontal lines show the surface of the table. Since q = 1 gives correct shape in this case,the recovered pose is stable as shown in Fig. 8 (a).
Note that q = -1 gives a skewed shape with unstable pose. Figure 9 is a snapshot where the object was
grabbed by the robot hand successfully as a resultof 3D shape and size recovery.

4Conclusion

This paper proposes an active recognitioil approach to solve the shape and size problems in 3D
recovery. We first discussed that two paranieters
representing shape candidates and scale factor arenecessarily involved i11 the formrllatio~lof the yrojective transformation, then described the procedure toobtain the unique solution by the factorization witha known 3D trajectory.
Experiments were performecl in simulation st,udy
as well as in real environment, and both results have

Figure 9: A snapshot of the object grabbed by robot
hand.

shown the potential usefrllness of our method. Infuture, we will examine the robustness and stabilityof the proposed method in more realistic situations.

References[I] N. Asada, H. Fujiwara, T. Matsuyama: "Edgeand depth from focus," IJCV, Vo1.26, No.2, pp.153163, 1998.
[2] S.T. Barnard, M.A. Fischler: "ComputationalStereo," Computing Surveys, Vol. 14, No. 4, pp.553572, 1982.
[3] R.J. Woodlialn: "Analyzing images of curved
surfaces," Artificial Intelligence, vol. 17(1-3), pp.
117-140, 1981.
[4] C. Toinasi, T . Kanade: "Shape and motion fromimage streams under orthography: A factorization method," IJCV, 9-2, pp. 137-154, 1992.
[5] Y. Aloimonos, A. Bandopadhay, I. Weiss: "Active Vision," Proc. 1st ICCV, pp.35-54, 1987.
[6] C. Tomasi, T. Kanade: "Detection and Trackingof Point Features", CMU-CS-91-132, 1991.