2023 32nd IEEE International Conference on Robot and Human Interactive Communication (RO-MAN) | 979-8-3503-3670-2/23/$31.00 ©2023 IEEE | DOI: 10.1109/RO-MAN57019.2023.10309337

2023 32nd IEEE International Conference on Robot andHuman Interactive Communication (RO-MAN)
August 28-31, 2023. Paradise Hotel, Busan, Korea

Depth image-based deformation estimation of deformable objects forcollaborative mobile transportation
Giorgio Nicola1 , Stefano Mutti1 , Enrico Villagrossi1 , Nicola Pedrocchi1
Abstract— Human-Robot collaborative transportation is apromising technology that combines the strength of humans androbots. The most common approaches rely on methodologies
that exploit force-sensing. However, the drawbacks are multiple.
First, the magnitude of force applied might be limited to avoid
damages. Then, force measurements might be unidirectional
according to the material properties; e.g., compression forces
are not measurable for fabrics. This paper proposes an approach based on the estimation of the deformation state of
the manipulated object from depth images. Specifically, thesegmented depth image of the manipulated object are fed toa Convolutional Neural Network (CNN) model to estimate
the current deformation status. Compared with the desired
deformation, the current deformation status is used to generate
the robot’s twist command. The methodology is proved ina mobile robot application, where carbon-fiber fabrics aretransported. A comparison with the state-of-the-art is reported
proving that the proposed method is more accurate and more
repeatable.

I. I NTRODUCTION
Human-robot collaborative transportation is increasingly
investigated for industrial applications, mainly when applied to large objects requiring multiple people to handle.
However, multiple challenges need to be solved still. First,
the robot should be able to infer the human objective [1],and based on that, the human and robot should be able
to switch the leader and follower roles during the physicalinteraction [2]. Second, the robot should minimize humaneffort while ensuring safety [3]. Then, human control actionin physical human-robot interaction can be approximated as adelayed linear model because of the human reaction time [4].
Such a reaction time varies significantly between people and,
if not addressed, can even lead to instability.
A peculiar case of human-robot collaborative transport iswhen the manipulated object is deformable, such as fabricor cables. The robot should avoid excessive deformation of
the manipulated material and aid the human simultaneously.
Standard control techniques, based on sensing the force
applied by the human, e.g., impedance control, struggle dueto the low forces that can be applied before damaging theobject. Thus, control strategies based on visual feedback of
the comanipulated object are viable alternatives.
Building on our previous work [5], we propose a visionbased strategy to perform collaborative transportation of
deformable objects with an industrial mobile manipulator(IMM). We use an eye-in-hand RGB-D camera to capture
Giorgio Nicola et al. are with the Institute of Intelligent Industrial
Technologies and Systems for Advanced Manufacturing, National Research Council of Italy, Via A. Corti 12, 20133, Milan, Italy; giorgio.nicola@stiima.cnr.it

979-8-3503-3670-2/23/$31.00 ©2023 IEEE

Fig. 1: Developed setup for collaborative transportation of
deformable objects. A sheet of carbon fiber fabric is manipulated by a human and a mobile robot. An RGB-D camera
is used to sense the deformation of the carbon fiber sheet.

the depth image of the manipulated object. The segmented
image of the manipulated object is fed to a Convolutional
Neural Network (CNN) to estimate its current deformation
status, defined as the current human-robot relative distance.
The difference between the current deformation status and apredefined rest configuration status is finally converted intoa twist command and fed to the robotic controller.
A. Related Work
The manipulation of deformable objects has been investigated [6], [7] focusing on two main classes of deformableobjects: cables [8], [9] and cloth-like [10], [11]. However,a few works study the problem of the collaborative humanrobot manipulation of such objects, and they typically focus
on cloth folding [12], [13]. One of the main challenges istracking the deformable object and estimating its current
shape, in other words, its deformation. Two main approaches
have been developed to estimate the material deformationin human-robot manipulation: direct or indirect via motion
capture of the human. The direct approach uses visual features that are subsequently converted into robot commands.
However, multiple visual features have been developed in theliterature; therefore, a standardized control architecture has
yet to be established. Indeed in [14], fabric folds combinedwith force measurements are used. In [15], [16], the authors
use Histograms of Oriented Wrinkles (HOWs) computed byapplying Gabor filters to RGB images. The indirect approachinstead tracks the position of the human grasping points on

2658

Authorized licensed use limited to: CNR Biblioteca Centrale. Downloaded on November 16,2023 at 07:37:13 UTC from IEEE Xplore. Restrictions apply.

(a)

(b)

Fig. 2: Problem formulation of the human-robot collaborative transportation problem. a) Shows the top view highlighting
the definition of the human grasping point Hgp . b) Shows a lateral view highlighting the parameters that compose TH
R,
TH
R,des , and ∆T. For the sake of simplicity, rotations have been neglected.

the material via a motion capture system based either on IMU
sensors [17], combined with force/torque measurement for
stiffer materials, or a camera [18], [19]. However, the generalassumption is that the contact point between the human andthe manipulated material is known a priori or detectable.
A different approach is in [20]. A motion tracker detects
handcrafted coded gestures that, combined with torque force
measures, are used directly to compute robot end effector
speed without estimating the material deformation.
Deep Neural Networks have recently been used as feature
extractors via autoencoder networks [21], [22], [23], andthe robot movement is planned according to such features.
However, those methods have yet to be applied to humanrobot manipulation.
Finally, only a few works study the human-robot collaborative transportation with mobile manipulators [20], [17],while many works investigate non-collaborative scenarioswith multiple mobile manipulators [24], [25].
Collaborative transportation has been investigated mostly
applied to rigid objects, such as in [26], [27]. The generalapproach is based on force sensing combined with impedanceor admittance control. However, force measurements with
deformable materials like fabrics are unreliable due to i)
low forces sustained by the object before damage; ii) force
measurement might be unidirectional, specifically only traction forces; iii) translation-rotation ambiguity. Thus suchcontrol architecture cannot be applied to deformable objects
without the aid of vision-based deformation estimation suchas in [17].
B. Contribution
The approach proposed in this paper relies on learning a deformation model of the manipulated objects from
depth images and converting it into robot twist commands.
Compared with methods in the literature, it has multiple
advantages. Firstly, the deformation estimation is direct yet
not based on handcrafted visual features; instead, features arelearned and can be general with a sufficiently large dataset.
Furthermore, indirect deformation estimation based on a
motion capture system has various drawbacks, with IMUbased skeleton tracking being affected by drift, body sensors
needing to be worn by the operator with uncomfortablestraps or specific suits, and camera-based skeleton tracking
having a limited field of vision. Vision-based methods arealso generally computationally expensive, requiring high-end
hardware to reach 30 Hz (standard RGB-D cameras’ frame
rate upper limit). Second, the proposed control architecture
is very straightforward compared to other works using a
combination of force and vision-based control architectures.
II. M ETHOD
A. Problem formulation
The problem of human-robot collaborative manipulation of
deformable materials is composed of two agents handling thedeformable material simultaneously, as shown in Figure 1.
One agent is the human that leads the activity, and the second
one is the robot that should follow the human movement. The
objective is to manipulate the desired object while minimizing the deformations from a rest configuration that guarantees
no damage to the material. The human movements during thecollaborative transport deform the material; thus, the robot
compensating for the deformations follows the human.
The problem of collaborative transportation can be decomposed into two separate problems. First is the definition andestimation of the deformation status. Second is computing
the robot commands to minimize the deformation from therest desired position.
We briefly report the definition of deformation status first
proposed in [5] and expanded in [28]. Let us consider adeformable material handled by two agents, a human, anda robot. Denote hr and hl as the human’s right and left
hand grasping positions and Rcur as the robot’s current
grasping point. The object shape can be described as thetuple of relative roto-translations between the robot grasping

2659
Authorized licensed use limited to: CNR Biblioteca Centrale. Downloaded on November 16,2023 at 07:37:13 UTC from IEEE Xplore. Restrictions apply.

Fig. 3: Architecture for the collaborative transportation problem. After preprocessing, the depth image acquired is segmented
to obtain the segmented depth image of the comanipulated object. The segmented depth is fed to a deformation model that
estimates the current deformation status as defined in Section II-A. The current delta deformation between the estimatedand desired setpoint is converted into a twist command fed to the robot controllers that output the IMM joints’ speed.

position and the human holding positions, except for the local
deformation around the hands. This model fully describes the
problem of collaborative shape servoing but is unnecessarily
accurate for the objective of collaborative transportation. It
also models the deformation in the corners depending on
where the human grasps the object.
We also assume that the human does not deform theobject between the hands, considering the material segment
between the two hand-grasping points as rigid, for example,
spreading the arms. Such movements do not have a purpose
in a collaborative transportation application; therefore, we
assume that if performed are involuntary and neglectable.
Given these assumptions, we define an arbitrary point on theobject between the hands invariant to the different grasping
positions. The relative roto-translations between the robot
grasping position and this point is a robust proxy for theobject deformation, except for the local deformation aroundthe hands and the geometry of the corners, which areirrelevant in a collaborative transportation scenario. Finally,
given the frame reference in Figure 2b, we also assume thatthe x-axis rotation is always zero since it causes only local
deformations of the object around the robot grasping point.
Referring to Figure 2a, Hgp is the human grasping point
Hframe, TH
R and TR,des are the actual roto-translation matrix
between the robot’s current pose Rcur or desired Rdes
H
and Hgp . TH
R and TR,des are respectively described byH
the parameters dR and dH
R,des , three translations and threerotations following the X-Y-Z extrinsic Euler conventions.
Given this formulation, the problem consists of (i) imposH
ing the target TH
R,des and (ii) estimating TR the collaborativetransportation. The robot should be controlled based on such
estimations to minimize the distance from the target’s desiredpose, i.e., ∆T → I, where I is the identity matrix. Given theformulation above, we define the robot controller as follows:q̇ = f (∆T)

∆T →
− I

where q̇ denotes the robot joint command speed.

(1)B. Proposed solution
We solve the collaborative transportation with the architecture shown in Figure 3. An RGB-D camera is mounted on the
robot end effector to acquire depth images of the manipulated
object; see Figure 1. The background and the human partnerin the depth images are segmented with a specifically trained
segmentation model, implemented using Unet [29] with theencoder-decoder pre-trained on the ImageNet dataset. The
segmented depth image of the object is then fed to the deformation model, based on DenseNet121 [30] pretrained on
ImageNet, which estimates roto-translation TH
R parameters.
Both the segmentation and deformation models are trainedon a single dataset specifically acquired of deformed plies.
Subsequently, the delta deformation between the deformaHtion setpoint TH
R,des and the estimated deformation TR isconverted into a twist command which is then handled by
the robot control algorithm.
C. Datasets acquisition
As described above, we developed a procedure to acquirea single dataset to train the deformation and segmentation
models. The dataset acquisition is divided into two steps.
First, the deformed object dataset is acquired, called thedeformation dataset. Second, the dataset for the segmentation
model, called the segmentation dataset, is generated startingfrom the deformation dataset. This approach allows for minimizing the required time to acquire datasets and minimizing
the necessary human intervention as much as possible.
The deformation dataset should comprise many depth
images of the deformed material with different humanrobot relative distances and human grasping positions. We
substituted the human with a frame that holds the deformablematerial, as in Figure 4, to avoid inaccuracies introduced bya human operator. Then rather than deforming the object
moving the human or frame, the robot moves relative to theframe whose position is estimated through a pair of fiducial
markers AprilTags. At the same time, the frame allows
higher accuracy and repeatability, simulating different human

2660
Authorized licensed use limited to: CNR Biblioteca Centrale. Downloaded on November 16,2023 at 07:37:13 UTC from IEEE Xplore. Restrictions apply.

D. Mobile manipulator control
Consider (1). The mobile manipulator controller should
compute robot joints speed q̇ taking as input the deformation
error, i.e., ∆T . First, we compute twist commands fromdeformation, as in Figure 3, then we convert such commands
into joint speed commands.
The deformation module outputs the parameters describing
H
TH∈ R6 , as shown in Section II-B.
R denoted as dR
Following the same rotation convention, we compute THR,desparameters, dH
R,des . The delta deformation ∆d turns in:H∆d = dH
R − dR,des .

Fig. 4: The setup to acquire the dataset of a carbon fiber
object. It comprises an RGB-D camera, Azure Kinect, an
IMM composed of a Universal Robot UR10 and a custommade mobile platform, an aluminum frame to mimic thehuman, and a pair of fiducial markers, Apriltags, to localize
the frame.

grasping positions on the material, and minimizes humanintervention. Finally, at each robot position, a set of RGB-D
images are taken to account for the camera’s noisy output andare autonomously labeled with TRH parameters described in
Section II-A. The deformation dataset acquisition is almost
fully autonomous since the only human intervention required
is to modify the grasping position on the frame.
Finally, the deformation estimation model should be agnostic to anything from the object shape, such as the background or human partner. Thus, the depth images are segmented with a fully autonomous segmentation routine. The
segmentation routine depicted in Figure 5 uses thresholdingof the depth, static mask, and a mask based on Apriltags’
position in the RGB image. The Apriltags define a rectangular box in the depth image, and all pixels outside are set tozero. Each image is cropped and resized to the resolution of
224 × 224. During the deformation model training, randomtranslation and random rotations are applied to the segmented
depth image as a form of data augmentation.
Such a segmentation routine cannot be unlikely applied incollaborative transportation since the operator must wear apair of fiducial markers. Therefore, the segmentation model
should be able to segment the object independently by the
background and the human operator. Thus, instead of acquiring and labeling a new dataset sufficiently large to generalize
over different backgrounds and humans, we generated asynthetic dataset starting from the segmented depth imagesof the deformation dataset. In detail, the synthetic dataset
is created by combining a minimal dataset of human-depth
images, 100 images per 5 random people, from the same
point of view, in random positions not overlapping withthe segmented object, as shown in Figure 5. Finally, fakebackgrounds with the depth value thresholded are added.

(2)

Twist command, denoted as ẋtool , is computed with the same
conventions with a simple gain K ∈ R6 .
ẋtool = K ⊙ ∆d(3)

Based on the formulation in Section II-A, ∆T is defined in
the frame Rcur , therefore also ẋtool , which is then converted
to the IMM base frame Hmr , and referred to as ẋ for thesake of simplicity.
Considering an n degrees of freedom IMM, with joint
velocity q̇ ∈ Rn , formed by the robotic arm joint velocities
and by the generalized velocities of the AMR (Autonomous
Mobile Robot), and task space velocity ẋ ∈ Rm , theirnominal relation follows+q̇ = J(q) ẋ

(4)+

where J(q) ∈ Rn∗m is the pseudo-inverse of the task
Jacobian matrix referred to the IMM base Hmr .
The task Jacobian J(q) is formed by the robotic arm
Jacobian J(q)R , referred to the Hmr frame, followed by
the trivial AMR Jacobian asJ(q)R




1
0
0
0
0
0

0
1
0
0
0
0


0
0

0

0

0
1

(5)

Being the system redundancy (m < n), we enforce additional
constraints to keep the robotic arm as close as possible to a
resting given position qrest during the control, as:q̇err = Kp (qrest − q)
+

(6)
+

q̇ = J(q) ẋ + (I − J(q) J(q))q̇err(7)

and Kp is a proportional gain that affects the null movement’s magnitude. The AMR compensates for displacements
and aids the robotic arm in keeping joint positions close to
the qrest , preventing the reaching of joint limits.
III. E XPERIMENTS AND RESULTS
A. Experimental setup
The experimental setup shown in Figure 1 comprises a
Universal Robot UR10 mounted on top of a custom-made2661
Authorized licensed use limited to: CNR Biblioteca Centrale. Downloaded on November 16,2023 at 07:37:13 UTC from IEEE Xplore. Restrictions apply.

Fig. 5: Developed pipeline to generate data for training the segmentation and deformation estimation models. The segmented
depth is generated by combining masks, thresholding, and cropping. The segmented depth images are combined with humans’
depth images to train the segmentation model to segment the object from humans not included in the dataset.

omnidirectional mobile platform and an Azure Kinect RGBD camera. A roughly rectangular carbon fiber ply withsize 130 × 38 [cm] was used. The acquired deformation
dataset comprises 134550 depth images of the ply deformed
along the three translations and the rotation along the zaxis. Five images are obtained for each robot pose, i.e.,
human-robot relative distance, and the dataset includes two
different human grasping positions. Given the desired resting
deformation of [0, 1.05, 0, 0] the studied deformation range
was:−0.12 ≤ x ≤ 0.12 [m],−0.255 ≤ y ≤ 0.255 [m],• −0.12 ≤ z ≤ 0.12 [m],◦◦• −30 ≤ yaw ≤ 30 .
••The maximum resolution of the deformation range for translations is 3 [cm], and for rotations is 6◦ . The total required
time for the dataset acquisition was about 3.5 hours. The
segmentation and the deformation models are deployed ona laptop with Intel Core i7-7700HQ 2.8 GHz CPU, 16 GB
RAM, and NVIDIA GeForce GTX 1050 running Ubuntu
20.04. The deformation estimation routine, including preprocessing segmentation and estimation, runs approximately at
25 Hz, which is close to the hardware bottleneck of the Azure
Kinect frame rate. Mobile platform control and manipulator
control are deployed on an Intel NUC 10. All software has
been developed within the ROS2 framework.
B. Experiments
This Section reports the results of two sets of tests. The
first was designed to estimate the performance of our model’sobservational error. The second was designed to demonstratethat the approach is independent of the human operator andthat the human-robot interaction is natural.
1) Observational error: The standard approach in theliterature for estimating material deformation in human-robot
collaborative transportation relies on tracking the hands’
position with motion capture techniques. Among the others,[18] uses a vision-based skeletal tracker for the hands’
position and a physics simulator to reconstruct the 3D shape.
The simulator design and implementation are relevant for
performance achievement in such a family of methodologies,
and they change from one use case to another. A faircomparison of such methods with our method is not trivial
since implementing the simulator may bias the evaluation.
This set of experiments compares our method with ad hoc
method we deployed that integrates a motion tracker as thesource of input data. Specifically, such a method consists of1) estimating the deformation status defined in Section IIA using the hand’s key points from the official skeleton
tracker software provided by the Azure depth camera
used in the experiments.
2) The segment connecting the two hand key points iscomputed, and ∆T is computed between the robot
grasping point and the middle point of such segment.
This approach provides a fair comparison since, as in [18],the same input data are used and the rest is deterministic.
Therefore no other source of error is introduced except for
the input data.
We exploited the setup previously used for the dataset
acquisition to apply ground truth known deformations to theobject. The difference with the dataset setup acquisition isthat while the frame holds the ply, the human pretends tohold it without applying further deformations. Specifically,

2662
Authorized licensed use limited to: CNR Biblioteca Centrale. Downloaded on November 16,2023 at 07:37:13 UTC from IEEE Xplore. Restrictions apply.

Fig. 6: Comparison of the observational error between our and skeletal tracking methods. Top left estimation error on thex-axis. Top right estimation error on the y-axis. Bottom left estimation error on the z-axis. Bottom right estimation error
on the z-axis rotation.

the frame pose is acquired via the Apriltags; subsequently, aknown deformation is applied, moving the robot to a known
relative pose that will be used as ground truth.
Both methods are tested on ten random robot poses, andfor each pose, the deformation is estimated ten times; results
are shown in Figure 6 as boxplots.
The accuracy of skeletal tracking is generally lower according to the mean estimation error. This depends onthe fact that the detected hand key points are placed in
the palm’s center, not precisely where the ply is grasped.
Without a specific calibration, it is impossible to know apriori the distance between the key points and the grasping
point. It can be noticed that even the repeatability of themeasurement, described by the interquartile range, is far
worse for the skeletal tracking approach. Indeed the skeletal
tracking algorithm produces noisy estimations of the key
points, particularly on the edge of the depth image. The error
on rotation z is particularly affected. Indeed the ply is pretty
tight, and the rotation is computed as the orientation of thesegment connecting the hands’ key points that, in this case,is not particularly long. Thus a little noise from the hand’s
key points position can lead to high noise in the estimatedrotations. Finally, the skeletal tracker estimation was at alower rate of approximately 20 Hz.
2) Collaborative transportation: Two human operators,
not involved in the human-depth images dataset, were required to perform arbitrary movements in an arbitrary order.
The only instructions provided to the operators are: i) themobile manipulator will follow the human compensating
deviations from a predefined deformation status; ii) themobile manipulator will also try to keep the robotic arm
close to a rest configuration. Both the predefined deformation
status and the robotic arm’s rest configuration were known
to the human operators. The video of the experiments isavailable at the following link: https://doi.org/10.
5281/zenodo.7795589.
First, the segmentation model proved not to overfit thesmall number of individuals in the human depth image
dataset and was not affected by multiple people in the
background and different types of backgrounds. Second,
both operators could perform the collaborative transportation
naturally without being hindered by the mobile manipulator.
IV. C ONCLUSIONSThis work presented a data-driven vision-based approachto human-robot collaborative transportation of deformableobjects combined with the usage of a mobile manipulator.
The proposed approach estimates the deformation statusof the comanipulated deformable object via a deformation
model based on CNN that takes as input the segmented
depth images of it. Depth images are obtained from an
RGB-D camera mounted on the robot end effector pointed
toward the comanipulated object and subsequently are segmented with an encoder-decoder segmentation model. Boththe deformation and the segmentation models are trainedon a single dataset whose described acquisition procedure
is designed to minimize human intervention. The estimatedobject deformation is compared with a predefined rest deformation. The delta deformation is first converted into
robot twist commands and subsequently converted into joint
speed commands considering the system task redundancy.
The proposed approach is first compared with the techniquefor collaborative transportation commonly used in literature
based on skeletal tracking. Our approach proved to deliver
more reliable measurements in terms of both accuracy andrepeatability. Finally, a real case of collaborative transportation was tested with two human operators. The proposed
method proved to generalize well, and the operators that

2663
Authorized licensed use limited to: CNR Biblioteca Centrale. Downloaded on November 16,2023 at 07:37:13 UTC from IEEE Xplore. Restrictions apply.

had received minimal instruction and no specific training
achieved a natural collaboration with the mobile manipulator.
The main drawback of the approach is that a specificdataset for every co-manipulated object, based on shape andmechanical properties, needs to be acquired. Even though
the routine is almost fully autonomous, it is still a timeconsuming activity, highly limiting its applicability. In future
works, the authors plan to use synthetic datasets for training.
Furthermore, we will train multiple deformation models
concurrently, one for each studied object, using a common
CNN backbone to learn more common and general features.
Subsequently, applying transfer learning to new comanipulated objects by retraining only the last fully connected layers
should require a much lower amount of data.
ACKNOWLEDGEMENT
This work is partially supported by the European Union’s
Horizon 2020 research and innovation program under grant
agreement No 101006732, ”DrapeBot – A European Project
developing collaborative draping of carbon fiber parts.”
R EFERENCES[1] P. Franceschi, N. Pedrocchi, and M. Beschi, “Inverse optimal control
for the identification of human objective: a preparatory study forphysical human-robot interaction,” in 2022 IEEE 27th International
Conference on Emerging Technologies and Factory Automation(ETFA). IEEE, 2022, pp. 1–6.
[2] ——, “Adaptive impedance controller for human-robot arbitration
based on cooperative differential game theory,” in 2022 International
Conference on Robotics and Automation (ICRA). IEEE, 2022, pp.
7881–7887.
[3] L. Roveda, S. Haghshenas, M. Caimmi, N. Pedrocchi, and L. M.
Tosatti, “Assisting operators in heavy industrial tasks: On the design ofan optimized cooperative impedance fuzzy-controller with embedded
safety rules,” Frontiers in Robotics and AI, vol. 6, 2019.
[4] A. Scibilia, N. Pedrocchi, and L. Fortuna, “Modeling of control
delay in human-robot collaboration,” in IECON 2022–48th Annual
Conference of the IEEE Industrial Electronics Society. IEEE, 2022,
pp. 1–6.
[5] G. Nicola, E. Villagrossi, and N. Pedrocchi, “Human-robot comanipulation of soft materials: enable a robot manual guidance using
a depth map feedback,” in 2022 31st IEEE International Conferenceon Robot and Human Interactive Communication (RO-MAN), 2022,
pp. 498–504.
[6] J. Sanchez, J.-A. Corrales, B.-C. Bouzgarrou, and Y. Mezouar,“Robotic manipulation and sensing of deformable objects in domestic
and industrial applications: a survey,” The International Journal of
Robotics Research, vol. 37, no. 7, pp. 688–716, 2018.
[7] D. Andronas, Z. Arkouli, N. Zacharaki, G. Michalos, A. Sardelis,
G. Papanikolopoulos, and S. Makris, “On the perception and handlingof deformable objects – a robotic cell for white goods industry,”
Robotics and Computer-Integrated Manufacturing, vol. 77, p. 102358,
2022.
[8] Y. She, S. Wang, S. Dong, N. Sunil, A. Rodriguez, and E. H. Adelson,“Cable manipulation with a tactile-reactive gripper,” The International
Journal of Robotics Research, vol. 40, pp. 1385 – 1401, 2020.
[9] W. Wang and D. J. Balkcom, “Knot grasping, folding, and regrasping,” The International Journal of Robotics Research, vol. 37,
pp. 378 – 399, 2018.
[10] A. Verleysen, M. Biondina, and F. Wyffels, “Video dataset of human demonstrations of folding clothing for robotic folding,” TheInternational Journal of Robotics Research, vol. 39, pp. 1031 – 1036,
2020.
[11] D. Mcconachie, A. Dobson, M. Ruan, and D. Berenson, “Manipulating
deformable objects by interleaving prediction, planning, and control,”
The International Journal of Robotics Research, vol. 39, pp. 957 –
982, 2020.

[12] A. X. Lee, H. Lu, A. Gupta, S. Levine, and P. Abbeel, “Learning
force-based manipulation of deformable objects from multiple demonstrations,” in 2015 IEEE International Conference on Robotics andAutomation (ICRA), 2015, pp. 177–184.
[13] Y. Li, Y. Yue, D. Xu, E. Grinspun, and P. K. Allen, “Folding
deformable objects using predictive simulation and trajectory optimization,” in 2015 IEEE/RSJ International Conference on Intelligent
Robots and Systems (IROS), 2015, pp. 6000–6006.
[14] D. Kruse, R. J. Radke, and J. T. Wen, “Collaborative human-robot manipulation of highly deformable materials,” in 2015 IEEE International
Conference on Robotics and Automation (ICRA), 2015, pp. 3782–
3787.
[15] B. Jia, Z. Hu, J. Pan, and D. Manocha, “Manipulating highly deformable materials using a visual feedback dictionary,” in 2018 IEEE
International Conference on Robotics and Automation (ICRA), 2018,
pp. 239–246.
[16] B. Jia, Z. Pan, Z. Hu, J. Pan, and D. Manocha, “Cloth manipulation
using random-forest-based imitation learning,” IEEE Robotics andAutomation Letters, vol. 4, no. 2, pp. 2086–2093, 2019.
[17] D. Sirintuna, A. Giammarino, and A. Ajoudani, “Human-robot collaborative carrying of objects with unknown deformation characteristics,”
in 2022 IEEE/RSJ International Conference on Intelligent Robots andSystems (IROS), 2022, pp. 10 681–10 687.
[18] D. Andronas, E. Kampourakis, K. Bakopoulou, C. Gkournelos, P. Angelakis, and S. Makris, “Model-based robot control for human-robot
flexible material co-manipulation,” in 2021 26th IEEE International
Conference on Emerging Technologies and Factory Automation(ETFA ), 2021, pp. 1–8.
[19] S. Makris, E. Kampourakis, and D. Andronas, “On deformable object handling: Model-based motion planning for human-robot comanipulation,” CIRP Annals, vol. 71, no. 1, pp. 29–32, 2022.
[20] D. D. Schepper, G. Schouterden, K. Kellens, and E. Demeester,“Human-robot mobile co-manipulation of flexible objects by fusing
wrench and skeleton tracking data,” International Journal of Computer
Integrated Manufacturing, vol. 36, no. 1, pp. 30–50, 2023.
[21] P.-C. Yang, K. Sasaki, K. Suzuki, K. Kase, S. Sugano, and T. Ogata,“Repeatable folding task by humanoid robot worker using deeplearning,” IEEE Robotics and Automation Letters, vol. 2, no. 2, pp.
397–403, 2017.
[22] D. Tanaka, S. Arnold, and K. Yamazaki, “Emd net: An encode–manipulate–decode network for cloth manipulation,” IEEE
Robotics and Automation Letters, vol. 3, no. 3, pp. 1771–1778, 2018.
[23] Y. Tsurumine and T. Matsubara, “Variationally autoencoded dynamic
policy programming for robotic cloth manipulation planning based on
raw images,” in 2022 IEEE/SICE International Symposium on System
Integration (SII), 2022, pp. 416–421.
[24] R. Herguedas, G. López-Nicolás, R. Aragüés, and C. Sagüés, “Survey
on multi-robot manipulation of deformable objects,” in 2019 24th
IEEE International Conference on Emerging Technologies and Factory
Automation (ETFA), 2019, pp. 977–984.
[25] Y. Wang, D. Held, and Z. Erickson, “Visual haptic reasoning: Estimating contact forces by observing deformable object interactions,” IEEE
Robotics and Automation Letters, vol. 7, no. 4, pp. 11 426–11 433,
2022.
[26] X. Yu, B. Li, W. He, Y. Feng, L. Cheng, and C. Silvestre, “Adaptiveconstrained impedance control for human–robot co-transportation,”
IEEE Transactions on Cybernetics, vol. 52, pp. 13 237–13 249, 2021.
[27] D. J. Agravante, A. Cherubini, A. Sherikov, P.-B. Wieber,and A. Kheddar, “Human-humanoid collaborative carrying,” IEEE
Transactions on Robotics, vol. 35, no. 4, pp. 833–846, 2019.
[28] G. Nicola, E. Villagrossi, and N. Pedrocchi, “Co-manipulation ofsoft-materials estimating deformation from depth images,” SSRNElectronic Journal, 2023. [Online]. Available: https://doi.org/10.2139/
ssrn.4355722[29] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional
networks for biomedical image segmentation,” in Medical Image
Computing and Computer-Assisted Intervention – MICCAI 2015,
N. Navab, J. Hornegger, W. M. Wells, and A. F. Frangi, Eds. Cham:
Springer International Publishing, 2015, pp. 234–241.
[30] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely
connected convolutional networks,” in 2017 IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), 2017, pp. 2261–
2269.

2664
Authorized licensed use limited to: CNR Biblioteca Centrale. Downloaded on November 16,2023 at 07:37:13 UTC from IEEE Xplore. Restrictions apply.