Simple benchmarks for speed and accuracy of rigid body dynamic simulators

The Gazebo robotics simulator incorporates multiple open-source physics engines for rigid body dynamics: Open Dynamics Engine, Bullet, Simbody, and Dynamic Animation and Robotics Toolkit (DART) [1]. These physics engines offer multiple implementations of a variety of algorithms for multibody dynamics simulation. For example, a summary of the approaches to modeling contact, joint damping force and articulated body coordinates are summarized in Table 1 for Gazebo’s four physics engines. Consequently, each physics engine offers a different set of trade-offs between accuracy, parameter sensitivity, and computational speed. This presentation will compare the trade-offs of Gazebo’s physics engines using benchmarks based on simple physical tests and behavioral tests related to task level robotics simulation. Table 1: Physics engine feature support. Physics engine Contact Joint Damping Coordinates DART Rigid / Impulse Implicit Generalized Open Dynamics Engine Rigid / Impulse Explicit or Implicit Maximal Bullet Rigid / Impulse Explicit Maximal Simbody Rigid / Force Implicit Generalized Benchmarks are useful for making fair comparisons between physics simulators [2, 3, 4, 5, 6, 7]. Each benchmark consists of a three components: a scenario to simulate, a selection of solver parameters to vary, and performance metrics. The scenario consists of a dynamic model, initial conditions, disturbances, control inputs, and the expected behavior. For simple scenarios, the expected behavior may be an analytical solution of system states, while for complex scenarios it may be energy conservation. The second benchmark component, selection of solver parameters, may consist of parameters that are common to each physics engine (such as solver time step size) and parameters that are available for a subset of the physics engines. The final aspect of benchmarking are the performance metrics. After each scenario is simulated with a set of solver parameters, the simulated behavior can be compared with the expected behavior compute an accuracy metric. The computational time required to compute each simulation is recorded as well. By comparing performance across multiple simulations, the sensitivity of each physics engine to parameter selection can be evaluated as well. For example, a simple benchmark for multibody simulation consists of independent rigid bodies moving in a constant gravity field. The center-of-mass of each rigid body should follow a parabolic trajectory, and the angular momentum should be conserved in a global frame. Though seemingly trivial for a multibody benchmark since it does not have contact or body articulation constraints, it is a useful control for comparing the speed and accuracy of each solver. An example benchmark result is shown in Figure 1. Results will be drawn from these simple benchmarks and used to motivate benchmarks of higher complexity for physics engine comparison. References [1] S. Peters. Comparison of Rigid Body Dynamic Simulators for Task-Level Robotic Simulation. Proceedings ASME Conference on Multibody Systems, Nonlinear Dynamics, and Control, Buffalo, NY, USA, 2014. Figure 1: Comparison of angular momentum conservation accuracy with computational speed. Each data point indicates a different set of solver parameters. The relative location of the curves indicates the trade-off between speed and accuracy. [2] Y. Lu and J. Williams and Claude Lacoursiere and J.C. Trinkle. A framework for problem standardization and algorithm comparison in multibody system. ASME International Design and Engineering Technical Conferences and Computers and Information in Engineering, Buffalo, NY, USA, 2014. [3] T. Erez, Y. Tassa, E. Todorov. Simulation Tools for Model-based Robotics: Comparison of Bullet, Havok, MuJoCo, ODE and PhysX. IEEE International Conference on Robotics and Automation (ICRA), Seattle, WA, USA, May 2015. [4] M. Renouf and V. Acary and G. Dumont", 3D frictional contact and impact multibody dynamics. A comparison of algorithms suitable for real-time applications. ECCOMAS Thematic Conference Multibody Dynamics, 2005. [5] M. Gonzalez and F. Gonzalez and A. Luaces and J. Cuadrado. A collaborative benchmarking framework for multibody system dynamics. Eng. with Comput. , v. 26, n. 1, pp. 1–9, Jan 2010. [6] M. Gonzalez and D. Dopico and U. Lugris and J. Cuadrado. A benchmarking system for MBS simulation software: Problem standardization and performance measurement, Multibody System Dynamics, 2006. [7] V. Acary and T. Koziara and C. Lacoursiere and M. Bremond and F. Perignon. Frictional Contact library. https://gforge.inria. fr/projects/fclib/ (Date Last Accessed, Nov, 6, 2014).