JAIST Repository
https://dspace.jaist.ac.jp/

Title

On the Generality and Application of Mason’s
Voting Theorem to Center of Mass Estimation forPure Translational Motion

Author(s)Gao, Ziyan; Elibol, Armagan; Chong, Nak Young

Citation

IEEE Transactions on Robotics: 1-16

Issue Date

2024-04-22

Type

Journal Article

Text version

author

URL

http://hdl.handle.net/10119/18863

RightsThis is the author's version of the work. Copyright(C) 2024 IEEE. IEEE Transactions on Robotics,
2024, 1-16. DOI:
https://doi.org/10.1109/TRO.2024.3392080.
Personal use of this material is permitted.
Permission from IEEE must be obtained for allother uses, in any current or future media, including
reprinting/republishing this material for advertisingor promotional purposes, creating new collective
works, for resale or redistribution to servers or lists,or reuse of any copyrighted component of this workin other works.

Description

Japan Advanced Institute of Science and TechnologyOn the Generality and Application of Mason’s
Voting Theorem to Center of Mass Estimation forPure Translational Motion
Ziyan Gao

, Armagan Elibol, and Nak Young Chong

Abstract—Object rearrangement is widely demanded in manyof the manipulation tasks performed by industrial and service
robots. Rearranging an object through planar pushing is deemedenergy efficient and safer compared with the pick-and-place
operation. However, due to the unknown physical properties ofthe object, re-arranging an object toward the target position
is difficult to accomplish. Even though robots can benefit frommulti-modal sensory data for estimating novel object dynamics,
the exact estimation error bound is still unknown. In this work,
firstly, we demonstrate a way to obtain an error bound on thecenter of mass (CoM) estimation for the novel object only using a
position-controlled robot arm and a vision sensor. Specifically, we
extend Mason’s Voting Theorem (MVT) to object CoM estimationin the absence of accurate information on friction and object
shape. The probable CoM locations are monotonously narrowed
down to a convex region, and the Extended Voting Theorems(EVT’s) guarantee that the convex region contains the CoM
ground truth in the presence of contact normal estimation error
and pushing execution error. For the object translation task,
existing methods generally assume that the pusher-object system’s physical properties and full-state feedback are available, or
utilize iterative pushing executions, which limits the applicationof planar pushing to real-world settings. In this work, assuming
a nominal friction coefficient between the pusher and object
through contact normal error bound analysis, we leverage theestimated convex region and the Zero Moment Two Edge Pushing(ZMTEP) method [1] to select the contact configurations forobject pure translation. It is ensured that the selected contact
configurations are capable of tolerating the CoM estimation error.
The experimental results show that the object can be accuratelytranslated to the target position with only two controlled pushes
at most.
Index Terms—Object Physical Property Estimation, Planar
Pushing, Non-Prehensile Manipulation.

I. INTRODUCTIONPUSHING is one of the well-known non-prehensile manipulation skills that can be deployed in different typesof robotic platforms. Compared with grasping, pushing is
especially useful when the object is ungraspable becauseof its geometrical shape, dimension, or physical properties.
Therefore, pushing has been used in various tasks such as
This work was supported in part by JSPS KAKENHI Grant Number
JP23K03756, and in part by the Asian Office of Aerospace Research andDevelopment under Grant/Cooperative Agreement Award No. FA2386-22-14042.
Z. Gao and N. Y. Chong are with the School of Information Science, Japan
Advanced Institute of Science and Technology, Nomi, Ishikawa 923-1292,
Japan {ziyan-g, nakyoung}@jaist.ac.jp
A. Elibol is with IAS-8: Data Analytics and Machine Learning, Institutefor Advanced Simulation, Forschungszentrum Jülich, 52428 Jülich, Germany

a.elibol@fz-juelich.de, Senior Member, IEEE

object rearrangement [2], target acquisition [3], singulation [4],bin picking [5], in-hand re-grasping [6], and similar others. In
this work, we are looking at ways to translate the novel object
to the target position accurately and efficiently without using astate feedback controller. Here, the meaning of “novel” has two
aspects; the physical properties are unknown, and the shapeinformation is not perfectly known. Despite its simplicity,pushing introduces challenges around object rearrangement.
Our method ensures that the pushed object undergoes puretranslation to the target position with a minimum number ofpushes.
The primary emphasis of this work is placed on the estimation of the CoM of novel objects as it plays an important rolein pushing. To translate the object toward the target, the net
force composited by contact forces must pass through the CoM
and parallel to the target direction. In general, object inertial
parameters identification leverages force and tactile sensing
or multi-modal sensing as summarized in [7]. However, little
research has addressed the estimation error bound, which isimportant to guarantee that the selected contact configurationis robust enough to tolerate the CoM estimation error. Furthermore, if the force measurement is unavailable and the object
shape is partially unknown, the CoM estimation becomes moredifficult. The proposed method, only collecting visual data of
the pushed object, is easily applicable to the case where theobject is neither attached at the end-effector nor graspable.
Inspired by the MVT, if the object is pushed along thecontact normal and the object’s sense of rotation is known,
the CoM lies inside a convex region specified by the halfplane defined by the ray of the pusher motion. However, theMVT only determines the instant sense of rotation given aknown contact normal. If the contact normal is imperfectly
known and the pusher slides along the object, the object’s
sense of rotation may not follow the MVT, leading to thewrong CoM estimation of objects. In this work, the MVT
is extended to facilitate the estimation of the CoM of novel
objects in the presence of contact normal estimation error andcontinuous push in a straight line. We present the contact
normal estimation error bound and a CoM region update rule,ensuring that the estimated CoM region encloses the CoM
ground truth given a series of pushes. Then, to deal with thesliding between the pusher and the object, we give a sufficientcondition for determining the sense of rotation of polygonal
objects pushed continuously in a straight line.
For selecting a nominal friction coefficient between thepusher and object given a contact normal error bound, wemodify the ZMTEP [1] for object pure translation. The
sampled contact configurations are capable of tolerating theCoM estimation error and guaranteed to resist any rotations
during translation. When pushing the object using the sampledcontact configurations, sticking contact between the pusher andthe object is ensured. Therefore, no object state feedback isneeded, as the object pose relative to the pusher remains thesame. When the target position cannot be achieved by any of
the contact configurations, we utilize two contact configurations whose pushing directions can positively span the target
direction. In addition, our proposed method is insensitive tothe pushing speed. We show through real experiments that theobject can be translated to the target position with two pushes
at most no matter how far the distance is from the target tothe object, with a wide range of pushing speed which accounts
for both the quasi-static and dynamic pushing.
In summary, this paper introduces the following theoreticalcontributions:• Generalize and apply the MVT to CoM estimation ofnovel objects under unknown friction parameters andimperfect contact and shape information.
• A sufficient condition for pure translation of novel objects
with the CoM estimation uncertainty.
• A sufficient condition for determining the sense of rotation of novel polygonal objects.
The assumptions are made as follows:• The object is flat, and it does not tilt or flip during andafter being pushed.
• The pusher, the object, and the support plane are rigid.
• Coulomb’s law of friction applies.
The rest of the paper is organized as follows. In Section II,
we review the related work in object inertial parameter estimation and planar pushing. In Section III, we present ourCoM estimation method. In Section IV, based on the CoM
estimation method, we explain the modification of ZMTEP
for object pure translation. In Section V, we clarify theexperimental setting for conducting object CoM estimationand pure translation. In Section. VI, we offer a comprehensive
explanation of the implementation process for both the proposed method and the baselines. In Section. VII, we describe
the experiments conducted to validate the modified ZMTEP
method for purely translating objects. Finally, conclusions are
drawn in Section VIII.
II. R ELATED W ORK
A. Object Physical Property Estimation
A comprehensive survey on the estimation of objects’
inertial parameters was performed in [7]. Our proposed method
belongs to “exploratory methods” characterized by the interaction between the robot and the object. Here, we further
categorize the exploratory methods as tactile, vision, andmulti-modal sensor-based methods by the type of sensors used.
Among tactile sensor-based methods [8], [9], [10], [11],[12], [13], for ungraspable objects, Yu et al. [8] defined
Gravity Equi-Effect Plane which contains the center of gravity. The object CoM was determined by the intersection ofmultiple Gravity Equi-Effect Planes found by multiple tip

operations. The assumptions made were the rigid polyhedronshaped object and sticking line contact between the object andthe ground. For planar objects, Yu et al. [9] used two fingertip
tactile sensors to measure the reactive forces during the interaction with the box, and formulated a least-squares problem
based on the equation of motion and the measurement. However, this method assumed a known object shape and identicalcontact normals between two fingers. For graspable objects,
the dynamics of the grasped object was estimated based on
the measured forces from the fingertips [10]. Specifically, theestimation of friction between the fingertip and the object
was done through a precise control scheme, measuring theforces when slippage occurs to estimate the boundary of thefriction cone. The estimation of object inertial parameters
was formulated as a factor graph taking the measurement
uncertainty into account. Wang et al. [11] used a vision-based
tactile sensor and encoded the object’s inertial parametersand friction into a latent space using the deformation at thecontact during object tilting and shaking. In [12], the CoMof a quadrilaterally-faced hexahedron object was identified
through the cross-correlation between the force signals of two
fingertips. Zhao et al. [13] detected the relative rotation angle
due to slippage when the object was lifted and measured
the gravitational force, and estimated the CoM using static
equilibrium.
For vision sensor-based methods, one earlier work [14]uniformly sampled a set of hypothesized support points inside
the convex region of the object projected onto the plane,
and the friction parameters were estimated using a linear
programming formulation. However, this method requires a
known object shape and precise object pose tracking. Morerecently, in [15], the mass density was estimated using theimpulse response to a striking motion. However, this method
requires a high-rate camera and can not apply to fragile
objects. Song et al. [16] proposed to learn the coupled massfriction parameters using a differentiable simulator. However,the contact normal direction can not be well estimated as theobject is approximated by rigidly connected 2D small grids.
Li et al. [17] used recurrent neural networks to estimate theobject CoM given a history of pushing interactions. Kumar
et al. [18] employed a policy network to interact with anarticulated object, and a predictor network to predict the mass
distribution of an articulated object. Xu et al. [19] proposed alearning framework to encode the object’s physical properties
implicitly using high-speed pushing and colliding. In ourprevious work [1], [20], the object CoM was estimated basedon the MVT, assisted by a prediction model of the pushed
object motion [21].
Among multi-modal sensor-based methods, Tanaka etal. [22] estimated the visual center of the object and foundpushing forces that can purely translate it. The estimation of
the object mass was based on the measured contact forces
and a known friction coefficient. In [23], the friction coefficients of the unknown objects were inferred from thevisual color feature and partial haptic measurements. Vereset al. [24] proposed to learn the CoM implicitly and predictthe grasp affordance in an end-to-end fashion. Specifically,
several grasping trials were executed to collect the support set

for generating the grasp affordance.
In this work, we extend MVT to estimate the CoM of novel
objects with inaccurate object shape and unknown friction.
We show that the estimation of the CoM can be carried out
accurately only using a vision sensor.
B. Object Rearrangement by Pushing
The previous studies [25], [26] focused on the use of openloop pushing to reposition objects with known geometry andinertial properties using a flat pushing fence. However, thisstudy aims to address the challenge of manipulating objects
with inaccurate geometry and unknown inertial properties. Under the assumption of quasi-static pushing, an ellipsoid model
was derived in [27] to approximate the limit surface [28],[29] which relates the applied force with object velocity. The
ellipsoid model was then used for pushing controller design
as in [30], [31], [32]. However, the ellipsoid approximationneeds a known CoM and friction between the pusher andthe slider and it under-fits the real object motion. Recently,
in [33], instead of using an ellipsoid model, the GaussianProcess (GP) was used to model the dynamics of the object
from experimental data. The authors leveraged GP and model
predictive control to repose the object. The methods proposedin [30], [31], [32], [33] require accurate full-state feedback,which is expensive to obtain in the common setting.
The object rearrangement task can also be solved by multistep pushing. In [17], [34], end-to-end learning models weretrained to predict object motion. Then, a series of pushingactions were planned to achieve the goal pose. The authorsin [35] showed the exact step bound for a given object
rearrangement task and a set of known line pushes with known
pushing effects. However, these methods lack efficiency, as thenumber of pushes is proportional to the difference between
the initial and target object poses. In contrast, our proposed
method for object translation only requires at most two pushes.
On the other hand, without estimating the object dynamics
or prior knowledge of pushing effects, Lloyd and Lepora [36]showed that the novel object can be rearranged using tactile
and vision sensor feedback. This method is time-consuming
and computationally intensive as the contact normal needs to
be inferred online through a convolutional neural network. We
show that our method can translate the object efficiently andaccurately with a variable speed range. In [37], the authors
optimized the contact configurations to achieve minimum
effort when pushing a known uniform-density object along therequired trajectory using several mobile robots. In our work,
the contact configuration is chosen to eliminate the uncertaintyof the CoM. In [38], [39], physics engines were leveraged toplan the pushing action for uniformly distributed objects.
N OMENCLATUREp

A sampled contact position on the object boundary in
the image frame.
nContact normal direction represented by a unit twodimensional vector.
ñ, ñl , ñr Estimated contact normal direction (into the object)and its left and right limits.

R f l , R f r Rays delimiting the left and right boundaries of thefriction cone.
Rp
Rays specifying the pusher motion.
Rn , Rñ , Rñl , Rñr Rays starting from p and along n, ñ, ñl , ñr .
R pl , R pr Rays specifying the left or right bound of the difference between the commanded pushing and the realpushing.
Cñ
Estimated contact normal error cone (bound), specified by two rays Rñl and Rñr delimiting the left andright boundaries of the bound.
Cp
Pushing execution error cone (bound), specified by
two rays R pl and R pr delimiting the left and rightboundaries for pushing execution error.
+C
Enlarged Cñ with Rñl and Rñr shifted to the left andñright along the object boundary to minimally wrap C p .
The shifted Rñl and Rñr are represented by + Rñl and
+R
ñr
Cr
Rotation uncertainty cone.
ACoM Possible CoM locations represented by a convex region inside the convex hull of the object mask.
AñlArea of intersection between the half plane (on which+ R has positive moment) and A
ñl
CoM .
Añr
Area of intersection between the half plane (on which+ R has negative moment) and A
ñr
CoM .
A pl
Area of intersection between the half plane (on which
Rpl has positive moment) and ACoM .
A prArea of intersection between the half plane (on which
Rpr has negative moment) and ACoM .
pCoM CoM ground truth.
p̃CoM Estimated CoM, or the centroid of ACoM .
αHalf angle of the friction cone.
δ
Half angle of the contact normal error bound Cñ .
θeHalf angle of pushing execution error cone C p .
deMax contact position error between the commanded
and real contact position.
III. C O M E STIMATIONThe objective of this study is to estimate the CoM of an
object using vision-guided planar pushing in the presence of
inaccurate shape information and unknown friction. This section begins with briefly examining the possibility of applying
the MVT to the estimation of the CoM of an object. Under
realistic conditions of inaccurate contact normal and friction
cone between the pusher and the object, we introduce theconcept of rotation indeterminacy cone with the estimation
error bound on the contact normal. We then present thefirst EVT, which specifies the conditions for eliminating theerror caused by inaccurate contact normal. Additionally, we
introduce the second EVT, which specifies the conditions foreliminating the error introduced by the relative sliding between
the pusher and the object. Next, we provide the update rule for
the CoM region estimation and the selection method for the
pusher motion. Finally, we summarize the proposed methodin Alg. 1.
A. Applying MVT to Object CoM Estimation
In this work, the coordinate system is right-handed withthe positive z-axis pointing out of the page. Given a pusher-Fig. 1. Probable CoM region given R p , R f l , R f r , and the object sense ofrotation. In this figure, R p solely determines the rotation sense.

object system, the CoM of the object is denoted as pCoM .
The two rays delimiting the left and right boundaries of thefriction cone are represented as R f l and R f r , respectively. R p
specifies the ray of the pusher motion. MVT states that the
spatial relationship among three rays R f l , R f r , R p , and thepCoM vote for the slider object’s sense of rotation. For instance,
the object rotates clockwise if at least two rays have negative
moments about pCoM as illustrated in Fig. 1.
Projecting the object to the support plane, the pCoM lies
inside the convex hull of the object boundary. If R p , R f l , andR f r are known and the object’s instant sense of rotation isobserved, according to the MVT, pCoM is located in a convex
region enclosed by the convex hull of the object and the middleof the three rays R p , R f l , and R f r . We refer to this convex
region as the CoM region denoted by ACoM . By imparting aseries of pusher motions to the object such that the middle
ray of each of the motions passes ACoM , the region can bemonotonously narrowed down. The smaller the region it gets,the more approximate its centroid is to the CoM.
If R p is inside the friction cone as an a priori knowledge,
it can determine the CoM region using the sense of rotationof the object, regardless of the directions of two other rays
or friction cone limits. Under the condition that the friction
between the pusher and the object is unknown but accuratecontact normal is available, the R p can be kept aligned withthe contact normal. Since the contact normal evenly splits thefriction cone, a minimum of two rays including R p always votefor the same rotation sense. Therefore, R p becomes the right orleft border of the CoM region, depending on the rotation sense,as shown in Fig. 1. Furthermore, if the object is pushed along
the contact normal passing through the centroid of the CoM
region and rotated, one-half of the CoM region is reduced. If
the object is purely translated, the CoM region degenerates toa line segment specified by the intersection between the CoM
region and R p . If we use all pixels inside the CoM region toapproximate the CoM region, ideally, the CoM ground truth
can be bounded within one pixel by pushing the object log2 (n)times, where n is the number of pixels occupying the CoM
region.
In practice, both the accurate contact normal and friction
cone at the contact point are unknown. To apply the MVT
to narrow down the CoM region, at least the contact normal
has to be identified. However, contact normal estimation isFig. 2. The object mask is refined using RGB image and segmented object
mask, and the contact normal error bound is estimated based on the refinedmask.

inaccurate due to sensor resolution or noise. R p outside thefriction cone might cause the CoM region update process tofail, and the real CoM would fall outside the CoM region.
In addition, if real-time pose tracking of the pushed object isunavailable, sliding between the pusher and the object may
lead to successive changes in the object’s sense of rotation,e.g., the object rotates counterclockwise and then clockwise
so that the rotation sense determined by the MVT may bedifferent from the real observation. All of these imperfections
introduce challenges around the MVT. In order to estimate the
CoM accurately in the presence of such challenges, we present
extensions on the MVT by modeling and taking into accountthe contact normal estimation error bounds.
B. Contact Normal Estimation Error Bounds
To estimate the contact normal direction, we employ avision sensor to capture images of the object to find thenormal to the lateral surface using a set of pixels on theobject boundary. To better extract the object boundaries andreduce the estimation error introduced by the sensor noise,
we employ a high-resolution mask segmentation model called
CascadePSP [40], and obtain the refined object mask using
the raw RGB image and segmented object mask as shown in
Fig. 2. Then, the surface normal can be estimated by analyzing
the covariance matrix of a local image patch around a contact
point. The eigenvector associated with the smallest eigenvalueof the covariance matrix is regarded as the surface normal [41].
The estimated contact normal, however, is inaccurate due tothe sensor noise and errors introduced during conversion fromthe real world to the discretized image frame. To deal with thisproblem, we introduce an error bound on the contact normal
estimation specified by the left and right limits ñl and ñr ,respectively.
We formulate the contact normal error bound estimation
problem as an optimization problem which will hereinafter be
described. The superscript i denotes the index of a discretized
contact point on the boundary of an object. We estimate
the normal direction ñi at the contact point pi using a set
k−1
of k adjacent contact points {p(i+ j) }, j ∈ (− k−1
2 , 2 ) on

Fig. 3. Finding contact normal error bound by solving the optimization
problem in Eq. 1, where ñl and ñr are the left and right boundaries of Cñ .
Local contour pixels are represented by cuboids. The circular dots represent
the optimization results indicating the local curvatures. Ellipses are drawn
using the dots, and the minor axes of the ellipses indicate the estimated contact
normal error bounds.

a two-dimensional object boundary, where j is an integer
and k is an odd number of neighborhood window size.
To find the left limit of the contact normal error boundñl , we create k two-dimensional variables represented as(i+ j)
k−1{pv }, j ∈ (− k−1
2 , 2 ). The ñl can be found by minimizing
the magnitude of the cross product of ñil and ñi given by Eq. 1.
min ñil × ñiñil

1 i+ k−1
s.t. Qv = Σ 2 k−1 (pvj − p̄v )(pvj − p̄v )⊤k j=i− 2i(1)
Qv ñl = λ ñil
11
p j − ≤ pvj ≤ p j +2
2
1
pvj ≤ (≥) (pvj+1 + pvj−1 )2
Qv is the covariance matrix calculated based on piv , and ñil isthe eigenvector of the Qv . p̄v is the mean of k two-dimensionalvariables. piv is bounded by the sub-pixel accuracy of pi .
In addition, the local curvature is assumed either convex orconcave as stated in the last constraint via ≤ (≥). In practice,
we exclude the estimation of neighboring pixels that do notexhibit convex or concave behavior. Similarly, the right boundis found by minimizing the cross product of ñ and ñr . Fig. 3
shows an example of finding ñl and ñr .
After obtaining ñl and ñr , the estimated contact normal ñ
is recalculated to be aligned with the halfway between ñl andñr . i.e., ñ = ñl +2 ñr . Provided that the contact normal estimation
uncertainty can be calculated, in the following, we present
how to utilize this uncertainty within our context of CoM
estimation.
C. Instant Sense of Rotation Indeterminacy Analysis with
Contact Normal Error Bounds
Using ñl and ñr at a contact point, the contact normal
error cone Cñ can be constructed. We assume that the contact
normal ground truth can be positively spanned by Cñ . If therobot pushes an object along the estimated contact normal,

Fig. 4. Illustration of rotation indeterminacy cone Cr . If object CoM lies
inside Cr as shown in (a) and the object is pushed along ñ, the rotation sense
may be opposite to the case pushed along the contact normal ground truth.
In (b), the presence of a larger friction cone eliminates rotation uncertainty.

which lies outside the actual friction cone, and updates the
CoM region based on the sense of rotation of the object, theCoM region and the CoM ground truth are mutually exclusive.
In such a case, the ray of pushing direction does not cast avote on the object’s sense of rotation. Therefore, to ensure thatthe estimated CoM region encompasses the CoM ground truth,
the contact normal estimation error bound must be considered.
In this study, we present the concept of the rotation indeterminacy cone Cr as a means to address the issue of exclusivity.
This cone refers to the uncertainty bounds on the rotation
sense, meaning that an update on the CoM region could gowrong if the CoM ground truth is inside this cone and theobject is pushed along the estimated contact normal. If the Cr
is accurately identified, the exclusivity case can be resolved byincorporating the region that is shared by Cr and the current
CoM region, regardless of the sense of rotation of the object
in updating the CoM region.
Assuming that the friction coefficient between the pusher
and the object is known, the Cr can be determined via thefollowing steps:• Draw contact normal error bounds specified by ñl and ñr
at the contact point.
• Construct friction cones by regarding ñl and ñr as thecontact normal direction.
• The left boundary of Cr is the right boundary of thefriction cone which regards ñl as the contact normal.
• The right boundary of Cr is the left boundary of thefriction cone which regards ñr as the contact normal.
Cr is illustrated in Fig. 4 (a). To enhance the expression of
the idea, let us assume that the true contact normal is alignedwith ñl . In this case, ñ is outside the friction cone. If the CoM
ground truth lies between the right boundary of the friction
cone and ñ, the update on the CoM region goes wrong. Thisis because ñ votes for counterclockwise rotation, however, theobject will rotate clockwise according to the MVT.
The definition of the half angle of Cr is given by δ − α,where δ specifies the half-angle of the contact normal error
bound, and α represents the friction angle. δ −α is meaningful
only if it is greater than zero. The instant object sense ofrotation error only occurs when the CoM ground truth is insideerror, the intersection region between the current CoM region
and pushing execution error cone (ACoM ∩ C p ) must lie in the
updated CoM region.
To incorporate both the contact normal estimation error
and pushing execution error in the process of object CoM
estimation, we enlarge the contact normal estimation error
cone Cñ by shifting Rñl and Rñr to minimally enclose theregion ACoM ∩ C p as shown in Fig. 5, denoted as + Cñ .
D. Sliding between Pusher and Object
Fig. 5. Enlarged contact normal error bound + Cñ which is specified by + Rñland + Rñr , and pushing execution error bound specified by R pl and R pr .

the Cr . To eliminate this error, the rotation indeterminacy
region represented by Cr ∩ ACoM must lie in the updated CoM
region, no matter the object’s sense of rotation. This guarantees
that the inclusiveness between the CoM region and the CoM
ground truth.
However, if the friction angle α is greater than the halfangle of contact normal error bound δ , the rotation indeterminacy cone becomes equal to 0/ as illustrated in Fig. 4(b). In this case, the contact normal estimation error does
not introduce any error in the rotation sense. Therefore, theestimation error due to inaccurate contact normal estimation
can also be eliminated when the friction cone is larger thanthe estimated contact normal error bound.
Based on the aforementioned analysis, we introduce thefirst EVT stated in Theorem 1.
Extended Voting Theorem (EVT) 1: If the friction angle
is greater than the half-angle of contact normal error bound,and the object is pushed along the estimated contact normal,then the instant sense of rotation of the pushed object is solely
determined by R p .
Proof: If the friction angle is greater than the half-angle ofthe contact normal error bound, the contact normal error boundis fully encompassed by the friction cone. Thus, any ray ofpushing inside the contact normal error bound is also inside thefriction cone. As a result, the estimated contact normal which
evenly splits the contact normal error bound also is inside thefriction cone. Based on the MVT, the ray of pushing R p is the
middle ray of (R f l , R p , R f r ) so that R p solely determines theobject sense of rotation.
In practice, controlling the pusher motion precisely is difficult due to hand-eye calibration and object pose tracking
errors. The execution error in pushing direction and contact
point are specified by θe and de , where θe defines the maximal difference between the commanded and actual pushingdirections, and de defines the maximal distance between thecommanded and actual contact points. Now, the pushingexecution error cone, denoted as C p , is specified by R pl andR pr , which depend on θe , de as shown in Fig. 5. Any actual
push action represented by R p can be bounded by C p . Herethe half angle of C p is smaller than that of Cñ , which isconsidered reasonable if the camera is well calibrated. To
avoid the estimation error introduced by the pushing execution

The preceding sections examined the possibility of using
the MVT to estimate the CoM of an object with the instantsense of rotation when the contact normal is inaccurate.
However, in real-world conditions, the instant sense ofrotation of an object may not always be available due tohardware limitations or lack of textures on the object forpose estimation. To address this issue, we approximate theinstant sense of rotation by performing a short straightline push. Based on the difference in object orientation
before and after a push, the instant sense of rotation canbe approximated. Nevertheless, the approximated sense ofrotation of the object may be inaccurate because of the pusher
sliding motion along the object boundary. The object could
initially rotate clockwise and then later counterclockwise
due to the alteration of the contact point. In this study, we
present a condition that is adequate for determining theobject’s sense of rotation using the initial spatial relationshipbetween R p and pCoM when the pusher slides along the object.
Extended Voting Theorem (EVT) 2: If the R p is initiallyinside the friction cone of a polygonal object and sliding
occurs continuously between the pusher and the object, theobject’s sense of rotation is uniquely determined by the initial
spatial relationship between R p and CoM location.
In other words, given a polygonal convex object, if the rayof the pusher motion R p lies inside the friction cone and thecontact point stays on the same edge, then the object’s senseof rotation remains unchanged regardless the sliding motion
between the pusher and the object.
Proof: We prove EVT 2 by contradiction. Let us assume thatR p is within the friction cone and indicates a clockwise senseof rotation for the object. The hypothesis is that R p eventuallychanges its indication due to the sliding motion. At first, theobject rotates clockwise based on the MVT, while R p rotatescounterclockwise with respect to the object frame. Since theobject has a polygonal shape, the directions of R f l and R f r
with respect to the object frame will not change regardlessof the direction of relative sliding motion between the pusher
and the object. Consequently, R p will move toward R f l . If the
hypothesis holds, the rotation of R p must rotate in a clockwise
direction w.r.t the object frame, leading it to move closer to
R f r and eventually surpass the boundaries of the friction cone.
However, this outcome is not feasible. Hence, this leads to acontradiction.
The proof of EVT 2 can also be done by enumerating the
spatial relationships among R p , R f l , and R f r during the sliding
between the pusher and the object as illustrated in Fig. 6.

Fig. 6. Proof of EVT 2. LS or RS means that the pusher slides to the left or right w.r.t. the object fixed frame depicted in red color. (a) shows the case thatthe pushing direction R p is initially outside the friction cone, while (b) shows the case that R p is initially inside the friction cone.

We attach an instant frame to the object whose origin is at
the contact point. One axis of the frame is along the contact
normal and the other axis is along the object boundary. Basedon the MVT, if the middle ray among three rays passes throughthe object CoM, the object translates along this ray. If the
middle ray is R p , the object is purely translated and no sliding
occurs between the pusher and the object. If the ray in the
middle is the right boundary of the friction cone as illustrated
in Fig. 6(a-i), the object is translated along R f r , meanwhile thepusher slides to the right with respect to the object-attachedframe. In the case of continuous straight-line pushing, theobject will then rotate counterclockwise. Therefore, successivechanges in the object’s sense of rotation might happen when
the ray in the middle is the R f l or R f r as shown in Fig. 6(a-i).
Fig. 6 (a) illustrates the case that the R p lies initiallyoutside the friction cone. A unique sense of rotation is notguaranteed due to the unknown sliding direction between thepusher and the object. Fig. 6(b) shows the case where the R p is
initially inside the friction cone. In this case, regardless of thesliding direction, the successive changes in the object’s senseof rotation will not occur. Therefore, if EVT 2 is satisfied,regardless of the pushing magnitude, the object’s sense ofrotation is initially determined and will not change.
E. CoM Region Update Rules
Here, we aim to answer the following question: Given apusher motion and the resultant sense of rotation of an object,
how can the object CoM region be maximally reduced in
the presence of contact normal estimation error and pushingexecution error?
If the friction cone is not guaranteed to be larger than thecontact normal error bounds, based on the aforementioned
instant object sense of rotation analysis, the intersection region
between the rotation indeterminacy cone Cr and the CoM
region must lie in the updated CoM region. However, therotation indeterminacy cone cannot be found precisely if thefriction is unknown. But the rotation indeterminacy cone isupper bounded by the contact normal error bound. Therefore,
instead of finding an exact Cr , the CoM region is updated insuch a way that it contains the region formed by the contact
normal error bound Cñ and the current CoM region to ensure
the inclusion of the CoM ground truth.
In the presence of both the pushing execution error andcontact normal estimation error, Cñ is replaced by + Cñ toavoid the potential estimation error caused by the pushing execution error. Therefore, if the object rotates counterclockwise,
the region of (+ Cñ ∩ ACoM ) ∪ Añl will be the updated CoM
region. Similarly, if the object rotates clockwise, the region of(+ Cñ ∩ ACoM ) ∪ Añr will form the updated CoM region. If theobject is purely translated, meaning that R p passes throughthe CoM, the region of C p ∩ ACoM is considered the updated
CoM region.
On the other hand, if the friction cone is larger than the contact normal error bound, following EVT 1, the contact normal
estimation error is eliminated. In this case, only the estimation
error introduced by the pushing execution error is taken into
account. Therefore, if the object rotates counterclockwise, theregion A pl ∪ ACoM is selected as the updated CoM region.
Similarly, the region A pr ∪ ACoM becomes the updated CoM
region if the object rotates clockwise. The CoM region isC p ∪ ACoM if the object is purely translated. In summary, theupdating rule is illustrated in Fig. 7.
F. Pusher Motion Selection
We now select the pusher motion, maximizing the reductionof the area of the CoM region using the CoM region update
rule. Specifically, the scoring functions (sn , s p , sd ) are designed
to quantify the area reduction. We aim to minimize the CoM
region and avoid elongated shapes. In the case that the friction
cone is not guaranteed to be larger than the contact normal
estimation error cone, Eq. 2 is designed.
sn (i) =2Aiñl Aiñr(Aiñl )2 + (Aiñr )2 + (ACoM ∩ + Ciñ )2(2)

Fig. 7. CoM region updating rule: (a) when the friction cone is not guaranteed to be larger than the contact normal error bound, (b) when the contact normal
estimation error is eliminated and only the pushing execution error is considered.

Fig. 8. Areas used for pusher motion selection. VCoM1 and VCoM2 are theprincipal axes calculated based on the current CoM region ACoM . The areas
are used to select a pusher motion (a) when both contact normal estimation
error and pushing execution error are present, and (b) when the friction coneat the contact is larger than the estimated contact normal error bound.

where the subscript n infers the presence of contact normal
error bound. i represents the index of the sample within theset of sampled pusher motions. Specifically, the boundaries+ R and + R+ñr of enlarged contact normal error bound Cññl
are used to split the current CoM region into at most three
regions: Añl , Añr and + Cñ ∩ ACoM , as shown in Fig. 8. It
aims to maximize the product of the size of Añl and Añr andminimize ACoM ∩ + Ciñ , since ACoM ∩ + Ciñ is always includedin the updated CoM region regardless of the object senseof rotation. Consequently, the pusher motion of the highest
score is determined by Aiñl and Aiñr of the same size and theminimum intersection between ACoM and + Ciñ .
In the case that the friction cone is larger than the estimatedcontact normal error cone, Eq. 3 is used where the subscript
p infers the presence of pushing execution error bound, as
the contact normal estimation error does not cause a CoM
estimation error.
2Aipl Aipr(3)s p (i) = i 2(A pl ) + (Aipr )2 + (ACoM ∩ Cip )2
Apart from the area of the CoM region, the shape of the CoM
region, which depicts the uncertainty in estimation, should
also be optimized to reduce the estimation error. In the casewhere the CoM region is elongated, there is a possibility ofsignificant estimation errors even if the ground truth CoM isguaranteed to be within the CoM region. This is particularly
true when the ground truth CoM is far away from the centroid

Fig. 9. CoM estimation process. In each figure, the green area is the CoM
region, the red arrow shows the pushing direction, and dashed lines are theleft and right limits of + Cñ . The CoM ground truth is depicted as a green
dot, and the estimated CoM is depicted as a red dot.

of the CoM region. By controlling the shape of the CoM
region, it is possible to decrease the estimation error in theworst-case scenario. Therefore, Eq. 4 evaluates the geometricisotropy of the CoM region based on the direction of pushingmotion indicated by the subscript d. Specifically, VCoM2 is thesecond principal eigenvector of the current CoM region. This
cost function evaluates how well the selected pushing action
reduces the uncertainty along the first principal direction of
ACoM . We therefore combine Eq. 2 or Eq. 3 with Eq. 4 in aweighted-sum fashion.
sd (ñi , VCoM2 ) = ∥ñi⊤ VCoM2 ∥(4)

G. Pipeline of CoM Estimation
Based on the EVT 1 and EVT 2, we propose the CoM
region update rule and pusher motion selection strategy. By
repeatedly applying the pusher motion and CoM region update
rule, the CoM region can be narrowed down until no further
push can reduce the CoM region or a maximal number ofpushes is achieved. The pipeline of the CoM estimation is
illustrated in Alg. 1, also supported by an example in Fig. 9.
Algorithm 1: CoM Region Narrowing Down Process
Input: agent, {p}i , {ñ}i , {ñl }i , {ñr }i , θe , θPT , de , w, T/* θe , de are used to construct C p and calculate
A pl and A pr . T is the maximal time steps forinteraction. w is a two-dimensional weightvector. θPT is an angle thresholddetermining if the object is purelytranslated.

*/

Output: ACoM
1 Initialize ACoM /* The region inside the convex
hull of the object.
2/* Pusher motion selection/* opencv library
3
4
5

6

9
10
11
12
13
14
15
16

17
18
19
20
21
22
23
24
25

*/

∆θ ← agent.execute(pi , ñi )/* CoM region update

8

*/
*/

VCoM2 ← PCA(ACoM )if include uncertainty thenConstruct {+ Cñ }i , {C p }i , {Añl }i , {Añr }i basedon {p}i , {ñ}"i , {ñl }i , {ñr }i , θ#e , desn (i)
i=argmin w⊤
sd (ñi , VCoM2 )/* Pusher-object contact interaction

7*/

for t = 1 to T do*/

if ∥∆θ ∥ > θPT thenif ∆θ > 0 thenACoM ← (+ Ciñ ∩ ACoM ) ∪ Aiñr
else
ACoM ← (+ Ciñ ∩ ACoM ) ∪ Aiñlelse
ACoM ← (Cip ∩ ACoM )else
Construct {C p }i , {A pl }i , {A pr }i based on{p}i , {ñ}i , θe", de#s p (i)⊤i=argmin w
sd (ñi , VCoM2 )∆θ ← agent.execute(pi , ñi )if ∥∆θ ∥ > θPT thenif ∆θ > 0 thenACoM ← (Cip ∩ ACoM ) ∪ Aipr
else
ACoM ← (Cip ∩ ACoM ) ∪ Aipl
else
ACoM ← (Cip ∩ ACoM )contact normals and the pushing direction that passes throughthe CoM ground truth intersect at a single point. Specifically,
the pusher needs to touch the object at two different edges,
and the ZMTEP selects a two-edge-contact configuration withthe highest tolerance against the touch position and CoM
estimation errors. The larger the distance between two contact
points, the larger the tolerance on CoM estimation error.
If the exact contact normal and friction cone are knownat the contact point, for a specific two-edge-contact configuration, two contact forces are bounded by (Rif l , Rif r ) and(R jf l , R jf r ) at each contact point pi and p j , respectively. Givena target pushing direction and two-edge-contact configuration,
the CoM tolerance range can be found as follows:• draw line that is parallel to the target pushing direction

and is passing through the intersection of two left boundaries of friction cones at two contact positions represented
by Rif l ∩ R jf l .
• draw the other line that is parallel to the target pushingdirection and is passing through Rif r ∩ R jf r
• find the region encompassed by the intersection of twolines as well as the convex hull of the object.
It should be noted that the CoM tolerance range is only valid
when the target pushing directions can be positively spannedby both (Rif l , R jf l ) and (Rif r , R jf r ). Using the CoM region,
without the CoM ground truth, the following conditions should
be satisfied for pure translation of an object.
• The CoM tolerance range of the two-edge-contact con-figuration completely encompasses the CoM region.
• The target pushing direction can be positively spannedby contact forces at two contact positions.
These two conditions are sufficient conditions for object puretranslation with the estimated CoM region.
In this study, we modify the ZMTEP method to accommodate the situations where the contact normal, friction cone, andobject CoM are coarsely known. Given the target direction, estimated object CoM region, and estimated contact normal error
bounds, our goal is to efficiently find the two-edge-contact
configurations that ensure pure object translation toward thetarget position.
Theoretically, if the friction angle is two times greaterthan the half-angle of the contact normal error bound given
by Eq. 5, it is ensured that the friction cone includes theestimated contact normal error bound. In other words, any
contact force inside the contact normal error bound can bepositively spanned by the left and right boundaries of thefriction cone. We refer to Eq. 5 as the friction condition forpure translation when the object geometry is not perfectly
known. To make Eq. 5 fulfilled, we wrap the pusher withhigh-frictional materials. After that, the approximated CoM
tolerance range can be found by utilizing the left and right
limits of contact normal error bounds Rñl and Rñr .

IV. O BJECT T RANSLATION BASED ON C O M R EGIONα ≥ 2δ

Our recent work [1] has shown that the ZMTEP can purelytranslate novel objects, if two contact normal directions positively span the pushing direction, and the lines specified by two

The computational complexity for finding two-edge-contact
configurations is O(n2 ), where n represents the number ofdistinct points on the object contour, as we iterate through all(5)

Fig. 10. The approximated CoM tolerance range is depicted by the lightyellow colored region. The pushing direction is specified based on the centroidjof the CoM region and the intersection point between Riñ and Rñ . The plus
and minus signs in the hatched regions are the moment labels of the two-edge
contact configuration.

possible combinations of two contact points. To efficiently determine the configuration for any target directions, we sample
uniformly a set of two-edge-contact configurations. We thendelineate the pushing direction for each sampled configuration
using the centroid of CoM region and the intersection point
Riñ ∩ Rñj as shown in Fig. 10. We finally check if the sampled
configuration and its corresponding pushing direction satisfy
the sufficient conditions for pure translation or not. The steps
involved in the above pipeline are described below:• Paireach contour point with another contour
point on different edges, represented by a set of{pi , p j , ñi , ñ j , Ciñ , Cñj , nd }m with m examples. Here, nd
denotes the target pushing direction specified by p̃CoM
and Riñ ∩ Rñj .
jj• Remove examples whose (Riñl , Rñl ) and (Riñr , Rñr ) can
not positively span nd .
• Remove examples whose tolerance range cannot fully
cover the CoM region.
• Remove examples whose distance between two contact
points is larger than the maximum stroke of the gripper.
What remains of the initial m examples are all two-edgecontact configurations and their corresponding stable translation directions. As the remained contact configurations satisfy
the sufficient condition of pure translation, the tolerance range
formed by two parallel arrows always encloses the CoM region
and there must be no intersection between the two parallelarrows and the CoM region.
It is well established that the real vector space Rn can be
spanned positively by a minimum of n + 1 vectors. Hence, if
there exist three stable pushing directions that can positively
span R2 , any target direction can be achieved. An object canbe translated through a single push if the target directionis the same as one of the three stable pushing directions.
Alternatively, an object can be translated through a linear
combination of two stable pushes. To determine whether touse one or two pushes to translate an object, we set a small
threshold and find the set of two-edge-contact configurations
whose cosine distance between the corresponding pushingdirections and the target direction is below the specified

Fig. 11. Experimental setup for CoM estimation and pure translation of
objects with different shapes.

threshold. After that, we remove the configurations which do
not satisfy the sufficient conditions for pure translation fromthe set. Finally, if the set is still non-empty, we simply choose
the one whose stroke distance is the largest, because it has thelargest CoM tolerance range.
However, if the set after removal is empty, the target
direction cannot be achieved using only one push. In this case,
we use two configurations in a sequence whose corresponding
pushing directions (referred to as nid , ndj ) can positively span
the target direction. An optimal combination of two pushingdirections can be found by minimizing the sum of travel
distances. Let dtarget denote the pushing distance and ntarget
denote the pushing direction. The pushing distance of eachconfiguration can be found by solving Eq. 6.
minj

di + d jnid ,nds.t. [d i , d j ]⊤ = dtarget [nid , ndj ]−1 ntarget(6)

di ≥ 0
dj ≥ 0where d i and d j are the corresponding pushing distances toachieve the target position. To minimize the travel distance tothe target position, we choose two configurations with which
the sum of pushing distances is minimal.
V. E XPERIMENT S ETTING
The experimental setup is shown in Fig. 11, where an RGBD vision sensor is attached to the end link of the robot arm,
capturing the overhead image of the object. We conducted
two experiments; one was to validate the proposed method
for CoM estimation, and the other one was to purely translate
the object to the target position, leveraging the estimated CoM
region and the modified ZMTEP method.
The experimental setup includes both high-friction and lowfriction pushing. In high friction pushing, we use the pusherwith a rubber wrap, while in low friction pushing, we simply
remove the wrap. We 3D printed ten hollow objects in different
shapes categorized into four groups based on the convexity andedge curvature as shown in Fig. 11. The hollow space insideeach object is divided into multiple uniform grids where weinsert four lead blocks and change the CoM ground truth.
Fig. 14 illustrates an instance of lead block configurations
for each object. In the high friction setting, the frictionalcoefficient between the pusher and the objects is higher than
0.70. In the low friction setting, the frictional coefficient
between the pusher and the objects is lower than 0.18.
We attach a textured surface paper on top of each object
to monitor the change in pose of the pushed object with the
Scale Invariant Feature Transform (SIFT) [42]. Prior to push
initiation, the robotic arm adjusts its joint angles to ensure thata complete object shape is observable within the camera’s fieldof view. Then, we remove the background from the image
and register it that contains solely an object. The camera
captures the pushing execution at the rate of 16.7 Hz. For
each image frame obtained, we extract the keypoints and theirdescriptors. The projective transformation matrix is computed
based on the corresponding keypoints between the registeredimage and each image frame using a brute force matcher andLowe’s test ratio. The difference in object pose from the pose
before the object is pushed can be acquired by decomposing
the projective transformation matrix. We compute the pose
estimation error for object image, capturing multiple imagesof an object (whose pose remains unchanged with respect tothe robot base) from different robot poses (or camera angles).
We found that the position estimation error is negligible andthe orientation estimation error is less than one degree.
VI. C O M E STIMATION E XPERIMENTIn this section, we assess the accuracy of our proposed CoM
estimation method detailed in Alg. 1 and compare it with twobaseline methods proposed by Lynch [14] and Li [17]. In [14],the author used a linear programming formulation to estimate
the normalized magnitude of the friction forces between theobject and the ground.
We discovered that a quadratic objective function yields
more accurate results, and reformulated the problem as aquadratic programming problem. Therefore, we refer to themethod in [14] as quadratic programming (QP). Additionally,
we combine Alg. 1 and QP to improve the accuracy andeffectiveness of an estimation.
A. Proposed CoM Estimation ExperimentAlg. 1 starts with finding the contact normal and pushingerror bounds. This section explains the methods employed
to compute the contact normal and pushing error bound.
Furthermore, it outlines the process of determining the object
sense of rotation based on the pose change of the pushed
object, and steps for conducting experiments.
1) Contact Normal and Push Execution Error Bound:Firstly, we use the RGB-D image and deep CascadePSP model
to obtain the object mask and contour points. We then remove
the sharp points and their 10 adjacent pixels to reduce theestimation error. After that, we leverage 7 adjacent pixels foreach contour point to estimate the contact normal error bound.
To construct the pushing execution error bound, we movethe robot to each corner of the object and measure the position
error. After multiple measurements, we found that the robot

can move to each corner within position error 2mm, and thedifference between the command pushing direction and thereal pushing direction is less than 3° . Therefore, we set de to
2mm and θe to 3° in Alg. 1.
2) Object Sense of Rotation: In the proposed CoM estimation pipeline, we set θPT to 1° , i.e., if the accumulatedchange in orientation of the pushed object, denoted as ∆θ ,is less than 1° , the motion is considered pure translation. If∥∆θ ∥ > θPT , we check the sign of ∆θ to identify the object’s
sense of rotation. If we have a high frictional pusher and aconvex polygonal object, the sense of rotation is determinedaccording to the EVT 2 without the need to continuously trackthe object’s pose.
3) Experiment: Striking a balance between outputs of Eq. 2
and Eq. 4 (or similarly Eq. 3 and Eq. 4), we set the weightvector w to [0.5, 0.5]⊤ in Alg. 1. Therefore, the CoM region
does not tend to elongate. To evaluate the CoM estimation
accuracy, we assign each object three different lead block
configurations. For each configuration, we conducted threeexperiments in both high-friction and low-friction settings. In
each run of Alg. 1, the object was pushed a total of eighttimes, with each push being executed 5cm along the estimatedcontact normal direction. After each push, the CoM region
is updated, and the centroid of the updated CoM region isregarded as the estimated CoM. We refer to this method as
EVT-based CoM Estimation (EVT-CoM). The estimation
procedure may be terminated early if no push contributes tonarrowing the CoM region. We conducted 180 CoM estimation
experiments in total.
B. Quadratic Programming
In [14], a set of hypothesis contact points between theobject and the table should be determined. In our study, we
uniformly selected contact points within the object mask. To
ensure an accurate estimation of frictional distribution, we set
the number of hypothesis contact points to a value greater than
20 based on the findings in [14]. Each hypothesis contact point
has a corresponding non-negative weight, which is analogous
to the normalized magnitude of the frictional forces. If theweights can be accurately estimated, the CoM of the object
can be determined by summing the weighted positions of all
hypothetical contact points.
Moreover, the contact points between the pusher and theobject are uniformly selected in [14]. Following this idea, we
uniformly sample contact points around the object’s perimeter.
The robot pushes the object 5cm along the estimated contact
normal direction at each contact point. Based on the change inthe pose of the object, the motion of each hypothesis contact
point can be computed so that the direction of the frictional
force at the hypothesis contact point can be estimated.
After synchronizing the pusher and object motion, aquadratic programming method is employed to minimize themagnitude of the moment at the contact point between thepusher and the object. This is based on the principle thatthe sum of the moments of frictional forces at the contact
point between the pusher and the object should be zero under
the assumption of quasi-static interaction. Valid solutions on

weights of hypothesis contact points should be non-negative,
and the sum of weights is required to be one to avoid an allzero solution. As the exact contact normal and friction coneat the contact point between the pusher and the object are
unknown, there is no constraint on the direction of the resultant
of all frictional forces applied to the object from the ground.
To evaluate the influence of the number of pushes on
estimation accuracy, the robot pushes the object around its
perimeter 25 times. We then uniformly sample the pushinginstances, which is referred to as QP Uniform Sampling (QPuniform).
C. PushNet
PushNet [17] is a recurrent neural network trained in sucha way to capture the object dynamics based on the history ofpushing instances. Specifically, the input is the current object
mask, start and end pusher positions in the image frame, andthe output is two folds; the first one is the object CoM withrespect to the image frame and the other one is an encodedfeature vector for pushing action selection purpose. As we only
compare the CoM estimation accuracy, we use the first outputof the network. To make the pushing sequence for PushNet,
we re-utilize the pushing instances collected in Section VI-B.
D. Combining the CoM region with Quadratic Programming
This study investigates the potential benefits of incorporating our proposed framework into quadratic programming toenhance estimation accuracy and efficiency. Instead of uniform
sampling of the pushing instances, we select the pushing action
in a manner that narrows down the CoM region. Additionally,
we integrate the CoM region into the quadratic programming
formulation as linear constraints to find the optimal solution.
Specifically, the weighted sum of hypothesis positions must
fall within the CoM region. To ensure a fair comparison of
method performance, we utilize the previously collected datafrom Section VI-A. As each pushing motion is selected basedon the current CoM region in each step, we refer to this method
as QP EVT-based Sampling (QP-EVT).
E. CoM Estimation Result
The estimation error is illustrated in Fig. 12. In general, it
can be observed that the estimation accuracy of PushNet does
not improve as the number of pushes increases. One potential
explanation for this is that PushNet is trained on a simulation
dataset, which introduces a sim-to-reality gap that hinders its
generalization to our experimental settings. Conversely, theother methods demonstrate a reduction in estimation error
with an increasing number of pushes. EVT-CoM consistently
achieves a lower estimation error after each push, whencompared to the QP-uniform, and ultimately attains a lower
estimation error. However, it is worth noting that QP-EVT
achieves the lowest estimation error or equivalent performanceto the EVT-CoM. Furthermore, our findings suggest that QPEVT can achieve comparable or superior results with only two
pushes.
Fig. 13 illustrates the final estimation errors of all methods,
excluding PushNet, for different categorical objects in both

high and low friction scenarios. The mean and standard
deviation of PushNet are found to be 63.1mm and 26.5mm,
respectively, which is incomparably worse than the others.
According to Fig. 13, we found that EVT-CoM has similar
performance in different frictional settings. QP-EVT achieves
a lower estimation error in the high friction scenario. In
addition, comparing two quadratic programming-based methods, QP-EVT benefits significantly from our push action
selection strategy in reducing estimation errors, regardless ofthe frictional setting.
Fig. 14 illustrates the results of the CoM region identification for all objects with one location configuration of lead
blocks both in high and low friction settings. Each sub-figure
shows an image overlay of three runs of Algorithm 1. Due tothe pushing execution errors, the three CoM regions are notexactly identical. However, our proposed algorithm takes thecontact normal and pushing execution errors into account toensure that the CoM ground truth is inside the CoM region.
As a result, the CoM ground truth is also contained inside theintersection of three CoM regions. Notably, it is observed that
the CoM regions estimated in high frictional settings tend tobe smaller than those in low frictional settings. This is because
there is no need to deal with the uncertainty in contact normals
in high frictional settings.
The CoM regions for convex polygonal objects with theother two lead block location configurations in high frictionalsettings are depicted in Fig. 15. This is to confirm the validityof the EVT 2. It is noticeable that the CoM ground truth
remains within the intersection of three CoM regions eventhough the estimation is solely based on the difference between
the initial and final poses of the object.
VII. O BJECT T RANSLATION E XPERIMENTThis study conducts object translation experiments withhigh frictional pushing to verify the validity of Eq. 5. Following the completion of the CoM estimation experiments,
three CoM regions are obtained for each object in all threelead block configurations. The intersected CoM region andestimated contact normals are then utilized to select two-edgecontact pushing configurations. The stable pushing directions
can positively span R2 space visualized in Fig. 16, where thetwo-edge-contact configurations and corresponding pushingdirections are depicted in the same color. In order to assess
the effectiveness of object pure translation, the robot is tasked
to push each object in nine different directions. The angles
between the pushing direction and the horizontal directionin the image frame are evenly distributed within the rangeof (−180° , 180° ). To determine the factors that influence theaccuracy of the modified ZMTEP, the object is pushed withvarying distances (100mm and 180mm) and speeds (30mm/s
and 150mm/s). Throughout the pushing process, the rotations
and translations relative to the pusher are recorded. Overall,
the translation experiment is repeated a total of 1080 times.
The mean and standard deviations of translation and rotation
errors for pure translation experiments are presented in Table I.
It can be observed that the pose error is generally small forall object types, indicating that our ZMTEP method translatesFig. 12. Error in estimating the CoM of objects after each push. Each sub-figure displays the estimation error for objects within the same category. The mean
estimation error is visually represented by the height of each bar, while the solid black line at the middle of each bar represents the standard deviation.

Fig. 13. Estimation error after the last push under both high and low frictional settings.

Fig. 14. Illustrations of CoM regions for both high and low frictional pushing interactions. Each image shows one lead block configuration and threeoverlapping CoM regions.

Fig. 15. Illustrations of CoM regions of the convex polygonal objects with the other two lead-block configurations in the high frictional setting.

Fig. 16. Pushing directions that are available for each test object based on the intersected CoM region. Two-edge-contact pushing configurations are illustrated
on the boundary of the object. The corresponding directions are illustrated on the centroid of the CoM region, using the same color.

Fig. 17. Joint plot of translation and rotation error.

TABLE IT RANSLATION AND ROTATION E RRORSObjectCategory
Convex Polygonal
Convex Non-Polygonal
Concave Polygonal
Concave Non-Polygonal

Translation
Error(mm)
mean
std
0.72
0.84
0.72
0.79
0.58
0.41
1.11
1.02

RotationError(degree)
mean
std
0.52
0.77
0.48
0.57
0.48
0.62
0.76
0.85

novel objects without changing their orientations to the target
position using the CoM region and contact normal estimates.
However, for concave non-polygonal objects (in particular,
slender objects), the object pose error is relatively larger
compared with other object types. This can be attributed tothe potential slippage of the pusher with a small gripping
force when the target direction significantly deviates from thepusher’s contact normal directions.
Fig. 17 shows the joint distributions of translation androtation errors of the pushed object. It can be seen fromthe figure that the performance of the proposed method does
not depend on translation distance and/or pushing speed toa noticeable degree. To be more specific, the change inpushing distance and speed has a limited effect on the final
pose of the pushed object. In contrast to almost all previous
work relying on quasi-static pushing, our method enabled totranslate arbitrarily shaped objects with variable speeds up
to 150mm/s under our experimental setup. However, from apractical point of view, we found difficulties in controlling
two fingers of the parallel-jaw gripper to simultaneously touch
the object’s different edges. Therefore, a slight change in theobject’s pose occurs during the initial touching. We conjecturethat this asynchrony between two fingers of initial touching
could be the main factor that leads to the error.
VIII. C ONCLUSION
In this work, extending Mason’s Voting Theorem, we proposed a pusher motion selection strategy and CoM region
update rule to efficiently find a convex region containing theCoM ground truth given noisy object shape information. With
the nominal friction cone defined for novel objects whose
actual value is not known, we kept the pusher motion at thecontact point aligned with the contact normal. As the exact
contact normal was difficult to extract from the vision sensor,
we found a contact normal error bound acting as an upperbound of the rotation uncertainty cone and eliminated the CoM
estimation error due to the inaccurate contact normal.
We presented two Extended Voting Theorems widely applicable to real-world settings. The first theorem states that theuncertainty in the direction of contact normal will not introduce any error in the CoM estimation procedure if the friction
cone at the contact point is larger than the contact normal error
bound. The second theorem states that continuous pushing ina straight line will not introduce the CoM estimation error if
the object is a polygon and the pushing direction lies initiallyinside the friction cone at the contact point. We conducteda large number of the CoM estimation experiments using aparallel-jaw gripper as the pusher and ten CoM-controllable
objects in real settings. We compared our CoM estimation
method with two baseline methods. The experimental results
indicated that our method outperformed the baselines in termsof estimation accuracy. Furthermore, our estimation pipeline
combined with quadratic programming achieved the lowest
estimation error, significantly reducing the required numberof pushes. In addition, we have proven the validity of the
EVT 2 by showcasing the estimated CoM region for polygonal
convex objects in a high frictional setting. It is worth notingthat this estimation does not entail any implementing measuresfor object tracking or considering relative sliding between
the pusher and the object. Despite these simplifications, theestimated CoM region encompassed the true CoM position.
Based on the estimated CoM region and the modified
ZMTEP method, we sampled a set of two-edge contact
configurations that can tolerate the CoM estimation error.
In contrast to existing methods which usually make use offull state feedback or multiple short-length pushes, we showthat if the stable pushing directions positively span R2 , ourproposed method can efficiently translate the object using
at most two pushes regardless of the pushing direction andpushing magnitude.
Currently, the second EVT does not guarantee the uniquesense of rotation for non-polygonal objects. In the future,
toward extending our method to deal with the CoM estimationof non-polygonal objects, we will explore different ways ofpushing or integrating the pose estimation of the pushed object.
This work has focused on rearranging the position of theobject. Optimizing the contact configuration for rearranging
both the position and orientation of the object is also a
potential direction.
R EFERENCES[1] Z. Gao, A. Elibol, and N. Y. Chong, “Zero moment two edge pushingof novel objects with center of mass estimation,” IEEE Transactions on
Automation Science and Engineering, vol. 20, no. 3, pp. 1487–1499,
2023.
[2] C. Song and A. Boularias, “Object rearrangement with nested nonprehensile manipulation actions,” in 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2019, pp. 6578–6585.

[3] B. Huang, S. D. Han, A. Boularias, and J. Yu, “Dipn: Deep interaction
prediction network with application to clutter removal,” in 2021 IEEE
International Conference on Robotics and Automation (ICRA). IEEE,
2021, pp. 4694–4701.
[4] A. Eitel, N. Hauff, and W. Burgard, “Learning to singulate objects using
a push proposal network,” in Robotics Research: The 18th International
Symposium ISRR. Springer, 2020, pp. 405–419.
[5] M. Danielczuk, J. Mahler, C. Correa, and K. Goldberg, “Linear pushpolicies to increase grasp access for robot bin picking,” in IEEE
International Conference on Automation Science and Engineering, 2018,
pp. 1249–1256.
[6] N. Chavan-Dafle, R. Holladay, and A. Rodriguez, “Planar in-handmanipulation via motion cones,” The International Journal of Robotics
Research, vol. 39, no. 2-3, pp. 163–182, 2020. [Online]. Available:
https://doi.org/10.1177/0278364919880257[7] N. Mavrakis and R. Stolkin, “Estimation and exploitation of objects’
inertial parameters in robotic grasping and manipulation: A survey,”
Robotics and Autonomous Systems, vol. 124, no. 103374, 2020.
[8] Y. Yu, K. Fukuda, and S. Tsujio, “Estimation of mass and center ofmass of graspless and shape-unknown object,” in Proceedings 1999
IEEE International Conference on Robotics and Automation (Cat.
No.99CH36288C), vol. 4, 1999, pp. 2893–2898 vol.4.
[9] Y. Yu, T. Arima, and S. Tsujio, “Estimation of object inertia parameters
on robot pushing operation,” in IEEE International Conference on
Robotics and Automation, 2005, pp. 1657–1662.
[10] B. Sundaralingam and T. Hermans, “In-hand object-dynamics inference
using tactile fingertips,” IEEE Transactions on Robotics, vol. 37, no. 4,
pp. 1115–1126, 2021.
[11] C. Wang, S. Wang, B. Romero, F. Veiga, and E. Adelson, “Swingbot:
Learning physical features from in-hand tactile exploration for dynamic
swing-up manipulation,” in 2020 IEEE/RSJ International Conference onIntelligent Robots and Systems (IROS), 2020, pp. 5633–5640.
[12] K. Yao, M. Kaboli, and G. Cheng, “Tactile-based object center of mass
exploration and discrimination,” in 2017 IEEE-RAS 17th International
Conference on Humanoid Robotics (Humanoids), 2017, pp. 876–881.
[13] Z. Zhao, X. Li, C. Lu, and Y. Wang, “Center of mass and friction
coefficient exploration of unknown object for a robotic grasping manipulation,” in 2018 IEEE International Conference on Mechatronics andAutomation (ICMA), 2018, pp. 2352–2357.
[14] K. M. Lynch, “Estimating the friction parameters of pushed objects,” inIEEE/RSJ International Conference on Intelligent Robots and Systems,
1993, pp. 186–193.
[15] M. Artashes and D. Burschka, “Visual estimation of object density
distribution through observation of its impulse response,” in VISAPP,
2013.
[16] C. Song and A. Boularias, “A probabilistic model for planar slidingof objects with unknown material properties: Identification and robust
planning,” in IEEE/RSJ International Conference on Intelligent Robots
and Systems, 2020, pp. 5311–5318.
[17] J. K. Li, W. S. Lee, and D. Hsu, “Push-net: Deep planar pushing for
objects with unknown physical properties,” in Robotics: Science andSystems, 2018.
[18] K. N. Kumar, I. Essa, S. Ha, and C. K. Liu, “Estimating mass distributionof articulated objects using non-prehensile manipulation,” arXiv preprint
arXiv:1907.03964, 2019.
[19] Z. Xu, J. Wu, A. Zeng, J. B. Tenenbaum, and S. Song, “Densephysnet:
Learning dense physical object representations via multi-step dynamic
interactions,” in Robotics: Science and Systems, 2019. [Online].
Available: http://www.zhenjiaxu.com/DensePhysNet/[20] Z. Gao, A. Elibol, and N. Y. Chong, “Estimating the center of mass of an
unknown object for nonprehensile manipulation,” in IEEE International
Conference on Mechatronics and Automation, 2022, pp. 1755–1760.
[21] ——, “Planar pushing of unknown objects using a large-scale simulation
dataset and few-shot learning,” in IEEE International Conference on
Automation Science and Engineering, 2021, pp. 341–347.
[22] S. Tanaka, T. Tanigawa, Y. Abe, M. Uejo, and H. Tanaka, “Active massestimation with haptic vision,” in Proceedings of the 17th International
Conference on Pattern Recognition, 2004. ICPR 2004., vol. 3, 2004, pp.
256–261 Vol.3.
[23] T. N. Le, F. Verdoja, F. J. Abu-Dakka, and V. Kyrki, “Probabilistic
surface friction estimation based on visual and haptic measurements,”
IEEE Robotics and Automation Letters, vol. 6, no. 2, pp. 2838–2845,
2021.
[24] M. Veres, I. Cabral, and M. Moussa, “Incorporating object intrinsic
features within deep grasp affordance prediction,” IEEE Robotics andAutomation Letters, vol. 5, no. 4, pp. 6009–6016, 2020.

[25] S. Akella and M. T. Mason, “Posing polygonal objects in theplane by pushing,” The International Journal of Robotics Research,
vol. 17, no. 1, pp. 70–88, 1998. [Online]. Available: https://doi.org/10.1177/027836499801700107[26] Q. Li and S. Payandeh, “Manipulation of convex objects via
two-agent point-contact push,” The International Journal of Robotics
Research, vol. 26, no. 4, pp. 377–403, 2007. [Online]. Available:
https://doi.org/10.1177/0278364907076819[27] K. Lynch, H. Maekawa, and K. Tanie, “Manipulation and active sensing
by pushing using tactile feedback,” in IEEE/RSJ International Conference on Intelligent Robots and Systems, 1992, pp. 416–421.
[28] S. Goyal, A. Ruina, and J. Papadopoulos, “Planar sliding withdry friction part 1. limit surface and moment function,” Wear,vol. 143, no. 2, pp. 307–330, 1991. [Online]. Available: https://www.sciencedirect.com/science/article/pii/0043164891901043[29] ——, “Planar sliding with dry friction part 2. dynamics of motion,”
Wear, vol. 143, no. 2, pp. 331–352, 1991. [Online]. Available:
https://www.sciencedirect.com/science/article/pii/0043164891901054[30] F. Bertoncelli, F. Ruggiero, and L. Sabattini, “Linear time-varying mpc
for nonprehensile object manipulation with a nonholonomic mobile
robot,” in 2020 IEEE International Conference on Robotics and Automation (ICRA), 2020, pp. 11 032–11 038.
[31] J. Zhou, Y. Hou, and M. T. Mason, “Pushing revisited: Differential
flatness, trajectory planning, and stabilization,” International Journal of
Robotics Research, vol. 38, no. 12-13, pp. 1477–1489, 2019. [Online].
Available: https://doi.org/10.1177/0278364919872532[32] F. R. Hogan, E. R. Grau, and A. Rodriguez, “Reactive planar manipulation with convex hybrid mpc,” in 2018 IEEE International Conferenceon Robotics and Automation (ICRA), 2018, pp. 247–253.
[33] M. Bauza, F. R. Hogan, and A. Rodriguez, “A data-efficient approachto precise and controlled pushing,” in Conference on Robot Learning.
PMLR, 2018, pp. 336–345.
[34] Z. Gao, A. Elibol, and N. Y. Chong, “A few-shot learning frameworkfor planar pushing of unknown objects,” Intelligent Service Robotics,
vol. 15, no. 3, pp. 335–350, 2022.
[35] C.-Y. Chai, W.-H. Peng, and S.-L. Tsao, “Object rearrangement throughplanar pushing: A theoretical analysis and validation,” IEEE Transactions on Robotics, vol. 38, no. 5, pp. 2703–2719, 2022.
[36] J. Lloyd and N. F. Lepora, “Goal-driven robotic pushing using tactile
and proprioceptive feedback,” IEEE Transactions on Robotics, pp. 1–12,
2021.
[37] F. Bertoncelli, M. Selvaggio, F. Ruggiero, and L. Sabattini, “Taskoriented contact optimization for pushing manipulation with mobile
robots,” in 2022 IEEE/RSJ International Conference on Intelligent
Robots and Systems (IROS), 2022, pp. 1639–1646.
[38] A. Pasricha, Y.-S. Tung, B. Hayes, and A. Roncone, “Pokerrt: Poking
as a skill and failure recovery tactic for planar non-prehensile manipulation,” IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 4480–
4487, 2022.
[39] Z. Dong, S. Krishnan, S. Dolasia, A. Balakrishna, M. Danielczuk, and
K. Goldberg, “Automating planar object singulation by linear pushingwith single-point and multi-point contacts,” in 2019 IEEE 15th International Conference on Automation Science and Engineering (CASE).
IEEE, 2019, pp. 1429–1436.
[40] H. K. Cheng, J. Chung, Y.-W. Tai, and C.-K. Tang, “CascadePSP:
Toward class-agnostic and very high-resolution segmentation via global
and local refinement,” in CVPR, 2020.
[41] H. Hoppe, T. DeRose, T. Duchamp, J. McDonald, and W. Stuetzle,“Surface reconstruction from unorganized points,” in Proceedings ofthe 19th annual conference on computer graphics and interactive
techniques, 1992, pp. 71–78.
[42] D. G. Lowe, “Object recognition from local scale-invariant features,” inProceedings of the seventh IEEE international conference on computer
vision, vol. 2. Ieee, 1999, pp. 1150–1157.

Ziyan Gao received the B.S. in mechanical engineering from the Qingdao University of Science andTechnology, Qingdao, China, in 2016. He received
the M.S. and Ph.D. degrees in information sciencefrom the Japan Advanced Institute of Science andTechnology, Ishikawa, Japan, in 2019 and 2022, respectively. He is currently a post-doctoral researcher
with the Japan Advanced Institute of Science andTechnology. His research interests include object
physical parameter estimation, non-prehensile manipulation, robotic grasping, and deep neural networks.

Armagan Elibol is a senior researcher at
the Institute for Advanced Simulation, IAS-8,
Forschungszentrum Juelich in Germany. He obtained
a Ph.D. in Computer Science (2011) from the Computer Vision and Robotics Group (VICOROB) of
the University of Girona, Spain, and both a B.Sc.
and M.Sc. in Mathematical Engineering from Yildiz
Technical University in Turkiye, respectively 2002
and 2004. He has worked in different universities
in Japan, South Korea, and Turkiye. His researchinterests include the wide area of Computer Vision
and Robotics.

Nak Young Chong received the B.S., M.S.,
and Ph.D. degrees from Hanyang University,
Seoul, Korea, in 1987, 1989, and 1994, respectively. From 1994 to 2003, he was with Daewoo
Heavy Industries, Korea Institute of Science andTechnology, Mechanical Engineering LaboratoryAIST/MITI, and Intelligent Systems Institute-AIST.
In 2003, he joined the faculty of JAIST, where hecurrently is a professor of information science andserved as a councilor, director of the Center for Intelligent Robotics, and chair professor of Intelligent
Robotics Group. He was a visiting scholar at Northwestern University, Georgia
Institute of Technology, University of Genoa, and Carnegie Mellon University,and served as an associate faculty at the University of Nevada Las Vegas andKyung Hee University. He serves/served as editor of the IEEE Robotics andAutomation Letters, Intelligent Service Robotics, and International Journal ofAdvanced Robotic Systems, and associate editor of the IEEE Transactions on
Robotics. He served as program (co)-chair for JCK Robotics 2009, ICAM
2010, IEEE Ro-Man 2011/2013/2022, IEEE CASE 2012, URAI 2013/2014,DARS 2014, ICCAS 2016, IEEE ARM 2019. He was a general (co)-chairof URAI 2017 and UR 2020. He also served as co-chair for IEEE RAS
Networked Robots TC from 2004 to 2006, and Fujitsu Scientific System WG
from 2004 to 2008.