[100] D. P. Bertsekas, *Constrained Optimization and Lagrange Multiplier Methods*. Athena Scientific, Boston, MA, 1996.

[101] M. Santos Pessoa de Melo, J. Gomes da Silva Neto, P. Jorge Lima da Silva, J. M. X. Natario Teixeira, and V. Teichrieb, "Analysis and comparison of robotics 3d simulators," in *2019 21st Symposium on Virtual and Augmented Reality (SVR)*, 2019, pp. 242–251.

[102] J. Tremblay, A. Prakash, D. Acuna, M. Brophy, V. Jampani, C. Anil, T. To, E. Cameracci, S. Boochoon, and S. Birchfield, "Training deep networks with synthetic data: Bridging the reality gap by domain randomization," in *2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)*, 2018, pp. 1082–10828.

[103] Y. Chebotar, A. Handa, V. Makoviychuk, M. Macklin, J. Issac, N. Ratliff, and D. Fox, "Closing the sim-to-real loop: Adapting simulation randomization with real world experience," in *2019 International Conference on Robotics and Automation (ICRA)*, 2019, pp. 8973–8979.

[104] M. Križmančić, "Franka_gazebo," https://github.com/mkrizmancic/franka_gazebo.

[105] S. Sidhik, "Panda_simulator: Gazebo simulator for franka emika panda robot supporting sim-to-real code transfer," https://github.com/justagist/panda_simulator, DOI: 10.5281/zenodo.3818280.

[106] ——, "Mujoco panda," https://github.com/justagist/mujoco_panda.

[107] E. Todorov, T. Erez, and Y. Tassa, "Mujoco: A physics engine for model-based control," in *2012 IEEE/RSJ International Conference on Intelligent Robots and Systems*, 2012, pp. 5026–5033.

[108] A. Trabelsi, J. Sandoval, M. Ghiss, and M. A. Laribi, "Development of a franka emika cobot simulator platform (csp) dedicated to medical applications," in *Advances in Service and Industrial Robotics*, S. Zeghloul, M. A. Laribi, and J. Sandoval, Eds. Cham: Springer International Publishing, 2021, pp. 95–103.

[109] L. Pitonakova, M. Giuliani, A. Pipe, and A. Winfield, "Feature and performance comparison of the v-rep, gazebo and argos robot simulators," in *Towards Autonomous Robotic Systems*, M. Giuliani, T. Assaf, and M. E.
Giannaccini, Eds. Cham: Springer International Publishing, 2018, pp. 357–368.

[110] J. Collins, D. Howard, and J. Leitner, “Quantifying the reality gap in robotic manipulation tasks,” in *2019 International Conference on Robotics and Automation (ICRA)*, 2019, pp. 6706–6712.

[111] P. Corke, “The machine vision toolbox: a matlab toolbox for vision and vision-based control,” *IEEE Robotics Automation Magazine*, vol. 12, no. 4, pp. 16–25, 2005.

[112] P. M. Khiabani, B. S. Aghdam, J. Ramezanzadeh, and H. D. Taghirad, “Visual servoing simulator by using ros and gazebo,” in *2016 4th International Conference on Robotics and Mechatronics (ICROM)*, 2016, pp. 308–312.

[113] R. Smits, “KDL: Kinematics and Dynamics Library,” http://www.orocos.org/kdl.

[114] P. Corke, “Robotics, Vision & Control,” Springer 2017, ISBN 978-3-319-54413-7.

[115] A. De Luca and L. Ferrajoli, “A modified newton-euler method for dynamic computations in robot fault detection and control,” in *2009 IEEE International Conference on Robotics and Automation*, 2009, pp. 3359–3364.

[116] A. De Luca, A. Albu-Schäffer, S. Haddadin, and G. Hirzinger, “Collision detection and safe reaction with the DLR-III lightweight robot arm,” in *Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems*, 2006, pp. 1623–1630.

[117] W. Chung, L.-C. Fu, and S.-H. Hsu, *Motion Control*. Berlin, Heidelberg: Springer Berlin Heidelberg, 2008, pp. 133–159. [Online]. Available: https://doi.org/10.1007/978-3-540-30301-5_7

[118] P. Schmidt, J. Artigas, M. De Stefano, R. Balachandran, and C. Ott, “Increasing the performance of torque-based visual servoing by applying time domain passivity,” *IFAC-PapersOnLine*, vol. 48, no. 19, pp. 13–18, 2015, 11th IFAC Symposium on Robot Control SYROCO 2015. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S2405896315026270
[119] D. Ma, S. Dong, and A. Rodriguez, "Extrinsic contact sensing with relative-motion tracking from distributed tactile measurements," in *2021 IEEE International Conference on Robotics and Automation (ICRA)*, 2021, pp. 11262–11268.

[120] H. Bruyninckx, S. Demey, S. Dutré, and J. D. Schutter, "Kinematic models for model-based compliant motion in the presence of uncertainty," *The International Journal of Robotics Research*, vol. 14, no. 5, pp. 465–482, 1995. [Online]. Available: https://doi.org/10.1177/027836499501400505

[121] S. Tabrik, M. Behroozi, L. Schlaffke, S. Heba, M. Lenz, S. Lissek, O. Güntürkün, H. R. Dinse, and M. Tegenthoff, "Visual and tactile sensory systems share common features in object recognition," *eNeuro*, vol. 8, no. 5, 2021. [Online]. Available: https://www.eneuro.org/content/8/5/ENEURO.0101-21.2021

[122] W. Yuan, S. Dong, and E. H. Adelson, "Gelsight: High-resolution robot tactile sensors for estimating geometry and force," *Sensors*, vol. 17, no. 12, 2017. [Online]. Available: https://www.mdpi.com/1424-8220/17/12/2762

[123] A. De Luca, A. Albu-Schaffer, S. Haddadin, and G. Hirzinger, "Collision detection and safe reaction with the dlr-iii lightweight manipulator arm," in *2006 IEEE/RSJ International Conference on Intelligent Robots and Systems*, 2006, pp. 1623–1630.

[124] S. Haddadin, A. Albu-Schaffer, A. De Luca, and G. Hirzinger, "Collision detection and reaction: A contribution to safe physical human-robot interaction," in *2008 IEEE/RSJ International Conference on Intelligent Robots and Systems*, 2008, pp. 3356–3363.

[125] L. Manuelli and R. Tedrake, "Localizing external contact using proprioceptive sensors: The contact particle filter," in *2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)*, 2016, pp. 5062–5069.

[126] J. Bimbo, C. Pacchierotti, N. G. Tsagarakis, and D. Prattichizzo, "Collision detection and isolation on a robot using joint torque sensing," in *2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)*, 2019, pp. 7604–7609.

[127] M. Gautier and W. Khalil, "Direct calculation of minimum set of inertial parameters of serial robots," *IEEE Trans. on Robotics and Automation*, vol. 6, no. 3, pp. 368–373, 1990.
**Titre :** Intégration de la commande visuel et de la force pour l'interaction physique avec des
manipulateurs robotiques

**Mot clés :** Asservissement visuel; Commande de la forces; commande de conformité et d'impédance; commande basé sur des capteurs; Manipulateurs

**Résumé :** Dispositions combinées vi-
sion/force deviennent de plus en plus cou-
rants dans les petites et moyennes entreprises
à mesure que les “Cobots” deviennent plus
abordables. Caméras peuvent fournir une
description détaillée de la scène, tandis qu’un
capteur de force fournis des informations lo-
cales mais très précises sur le contact. Ce-
pendant, en raison de la nature très différente
de ces modalités de détection, il n’est pas
simple d’obtenir une utilisation combinée effi-
cace. Dans cette thèse, nous traitons du cou-
plage de la vision et de la détection de la force
en tirant parti de leur complémentarité et en
permettant à un robot manipulateur de réali-

ser activement un mouvement conforme aux
directions de la tâche visuelle. Nous avons
développé des schémas de contrôle avancés
avec des estimateurs qui gèrent la différence
de taux de mise à jour entre les capteurs. De
plus, nous contribuons à la récupération des
paramètres du modèle dynamique pour les
manipulateurs, qui sont des ingrédients clés
à la fois dans les contrôleurs proposés, qui
prennent en compte la dynamique du mani-
pulateur, ainsi que pour effectuer des simula-
tions plus précises. Les résultats rapportés,
tant dans le simulateur développé que dans
les expériences réelles, montrent l'efficacité
des techniques proposées.

**Title:** Integration of vision and force control for physical interaction with robotic manipulators

**Keywords:** Visual-servoing; Force control; compliance and impedance control; Sensor-based control; Manipulators

**Abstract:** Combined vision/force arrange-
ments are becoming more and more com-
mon in small and medium sized businesses
as “Cobots” becomes more affordable. Cam-
eras can provide a detailed description of the
scene, whereas force sensing can provide lo-
cal but highly precise information about the
contact. However, due to the very different na-
ture of these sensing modalities , obtaining an
effective combined use is not straightforward.
In this thesis we deal with the coupling of vi-
sion and force sensing leveraging their com-
plementarity and allowing a robot manipula-
tor to actively achieve compliant motion along

the visual task directions. We have devel-
oped advanced control schemes with estima-
tors that cope with the update rate discrep-
ancy between the sensors. furthermore, we
contribute to the dynamic model parameter re-
trieval for manipulators, which are key ingredi-
ents both in the proposed controllers, whose
take into account the manipulator's dynamics,
as well as for performing more accurate sim-
ulations. The reported results, both in the de-
veloped simulator and with real experiments
show the effectiveness of the proposed tech-
niques.