NISTIR 8097 

Benchmarking Robot Force 

Control Capabilities: Experimental 

Results 

Joe Falco 

Jeremy Marvel 

Rick Norcross 

Karl Van Wyk 

This publication is available free of charge from:

http://dx.doi.org/10.6028/NIST.IR.8097
NISTIR 8097 

Benchmarking Robot Force 

Control Capabilities: Experimental 

Results 

Joe Falco 

Jeremy Marvel 

Rick Norcross 

Karl Van Wyk

Intelligent Systems Division 

Engineering Laboratory

This publication is available free of charge from:

http://dx.doi.org/10.6028/NIST.IR.8097 

January 2016

U.S. Department of Commerce

Penny Pritzker, Secretary

National Institute of Standards and Technology

Willie E. May, Under Secretary of Commerce for Standards and Technology and Director
i

Benchmarking Robot Force Control Capabilities: Experimental Results 

Joe Falco, Jeremy Marvel, Rick Norcross, Karl Van Wyk 

Table of Contents 

DISCLAIMER .................................................................................................................................... ii

I. INTRODUCTION..................................................................................................................... 1

II. Force Control Metrics .......................................................................................................... 1

III. Test Artifacts ........................................................................................................................ 2

IV. Test Methods and Experimental Results .............................................................................. 3

A. Settle Stability...................................................................................................................... 4

i.Test Method ....................................................................................................................... 4

ii. Performance Measures ..................................................................................................... 5

iii. Experimental Results ...................................................................................................... 5

B. Disturbance Handling .......................................................................................................... 8

i.Test Method ....................................................................................................................... 8

ii. Performance Measures ..................................................................................................... 9

iii. Experimental Results ...................................................................................................... 9

V. CONCLUSIONS .................................................................................................................... 10

VI. REFERENCES....................................................................................................................... 11
ii

Disclaimer 

Certain trade names and company products are mentioned in the text or identified in certain 

illustrations. In no case does such an identification imply recommendation or endorsement by the 

National Institute of Standards and Technology, nor does it imply that the products are necessarily 

the best available for the purpose. 
1

I. INTRODUCTION

Force-based robotic assembly systems exist within the robotics industry primarily in the form of 

end-of-arm force/torque sensors for extrinsic force sensing. These solutions are provided as 

packages by robot manufacturers and by third party integrators, often without performance 

specifications for anything other than the sensing capabilities. Some examples of emerging 

industrial robot arms, sometimes referred to as collaborative robots or “cobots,” use intrinsic torque 

sensing at the joint level to resolve force/torque at the tool center point (TCP). These robots with 

joint-level force sensing characteristics can be programmed to achieve force-based manufacturing 

tasks and also react to collisions with humans to prevent injury [1]. Metrics and test methods are 

needed to characterize the control capabilities of robots with both intrinsic and extrinsic force 

sensing. The availability of these benchmarks will motivate research product development, and 

provide a mechanism for reporting and evaluating systems for the application space. It is proposed 

that robot data, collected using these robot benchmarks as well as others in development [2-4], be 

used to aid and simplify the task of selecting a particular robot system for force-based applications 

(e.g., surface finishing or assembly [5]). 

This report presents a preliminary set of metrics and associated test methods with illustrative 

examples for assessing the performance of force-based robot control. Section II describes metrics 

for force-controlled robots as well as for force-based assembly operations. Section III presents a 

force measurement system using modular artifacts for independent measures of robot performance. 

Section IV presents the test methods and experimental results for settle stability and disturbance 

handling as described in Section II for two commercially available robot arm systems with force 

control capabilities. Section V discusses the validity of the test methods introduced as well as 

anticipated future efforts to benchmark force-controlled assembly operations. 

II. Force Control Metrics

In previous work we identified the best practices used in force-based control of robotic systems, 

developed a draft set of performance metrics, and proposed designs of supporting test artifacts [6]. 

The metrics, shown in Table 1 and Table 2, provide a basis for assessing the basic force control 

characteristics of a robot (Table 1), as well as assembly metrics to assess the functional performance 

of the automated assembly system (Table 2). 

Table 1: Force control metrics for collaborative assembly robots. 

Force Control Metrics Description

Settle Stability (see IV-A) A measure of the settling time, overshoot, and steady-state error when 

reaching a desired force contact with a surface.

Obstruction Stability A measure of the time, force reaction and compliance associated with 

an immovable obstruction placed in the path of a robot.

Control Switch Stability A measure of the time and efficiency to automatically transition 

between position and force control modes. 

Disturbance Handling (see IVB)A measure of the deviation from desired nominal force when moving 

along a surface profile.

Incurred Force Limitations A measure of the ability to limit forces applied to objects or humans in 

a workspace.
2

Table 2: Functional level, force-based assembly metrics.

Assembly Metric Description

Success Rate Number of times an assembly is successfully completed divided by the total number 

of attempts to perform the assembly

Assembly Time Time required to compete the assembly task.

Incurred Force The maximum and average force applied to an assembly during the assembly process.

The assembly metrics are provided here for reference only. The methodology described in this 

report focuses on the control problem rather than on applications of said control. 

III. Force Measurement System with Test Artifacts

While it is possible to use data, such as force and position, directly from the controller of the robot 

under test for calculating the performance metrics, these measurements would be based on the 

inherent capabilities of the system under test. Instead, extrinsic measurement systems must be 

developed as acting ground truths to facilitate unbiased, cross-system comparisons. 

Here we describe an independent force measurement system for assessing the performance of forcecontrolled robots using the metrics discussed in Section II. The system shown in Figure 1 

incorporates a 6-axis load cell fixture sandwiched between two plates. The bottom plate supports 

the overall system and secures the force/torque sensor to the testbed surface, while the top plate 

accepts modular artifacts. Each artifact supports one or more control or assembly test methods. 

Figure 1a is the artifact used for testing disturbance handling performance and Figure 1b is the 

artifact used for testing settle stability performance. More detail for use of the artifacts is given in 

the test method in Section IV. Figures 1c-1e are assembly artifacts for peg-in-hole, snap-fit, and 

gear-mesh, respectively. These latter artifacts are specific to characterizing assembly performance, 

and are not the subject of this paper and no additional detail is given. Realized artifacts with the 

load cell and supporting mounting plates are shown in Figure 2. 

Figure 1: Labeled CAD model of the modular artifacts (a) disturbance handling, (b) settle stability, (c) peg-in-hole, (d) 

snap-fit retaining ring, (e) gear insert and mesh 
3

Figure 2: Modular artifacts as produced using anodized aluminum parts and off-the-shelf gears and fasteners. 

IV. Test Methods and Experimental Results

Below are the metrics and test methods for settle stability and for disturbance handling with

accompanying experimental implementations and results. All experiments were conducted with

two commercially available robotic manipulators, Robot 1 and Robot 2 (Figure 3), using their

inherent force control capabilities. The purpose of these experiments is to demonstrate the use of

the defined test methods. Performance reported is for illustrative purposes only, and is not based

on optimal force control parameters. Data presented here should not be used as a selection criteria

for a particular brand.

Figure 3 Robot testbed used to validate performance measures. 
4

A. SETTLE STABILITY

Settle stability is the measure of the time to reach, the overshoot experienced during, and the steadystate error of a desired force contact with a surface. The settle stability of a force-controlled robot 

reflects the controller’s ability to detect and maintain contact with an object, and directly impacts 

the robot’s performance during force-based assembly and finishing applications. Peak overshoot, 

when initiated from a non-contact state, indicates the impact force of the robot when making initial 

contact with the working surface. The impact force is significant for predicting part and end-ofarm tooling damage, and can indicate performance characteristics that are more amenable for 

particular assembly operations. Settling time is the period required for the system to obtain a 

persistently repeating or “settled” control error. Settling time indicates the responsiveness of the 

controller to changing command forces. Steady-state error is a measure of the difference between 

the instantaneous exerted force and the final force achieved by the controller during the “settled”

control period. 

i. TEST METHOD

Using the settle stability artifact (Figure 1b), program a step input force to maintain an appropriate 

force level while contacting a surface according to the test directions indicated in Figure 4. For 

this test, the robot moves in the direction of each contact surface, and applies force normal to that 

surface. The test begins with the robot in a non-contact state. Specify a test at one of the following 

levels: 1) a minimum applied force that invokes a useful robot response1

, 2) a maximum that is

within the robot’s payload capability, 3) a middle point between the minimum and maximum force 

tested, or 4) a level defined by the intended robot application. Forces are recorded at the contact 

point throughout the test using an independent force measurement system. Each settle stability test 

is reported 32 times for each test direction chosen for the evaluation. 

Figure 4: Settle stability artifact showing possible test directions 

1

In this case, “useful” is defined in terms of eliciting motion from the robot. If the robot is commanded to 

apply too small of a force, the difference between the commanded and measured force may be insufficient to 

cause the robot to move from its initial position or in the direction of travel desired. 
5

ii. PERFORMANCE MEASURES

Evaluate the peak overshoot, settling time, and steady-state error as follows: 

1. Estimate the steady state force (Fs) as the average force over a specified time window where

all readings remain within a specified band.

2. Calculate overshoot (FO) as the difference between peak force (FP) and the steady state force

(FS).

3. Calculate settling time (TS) as the elapsed time from the start of control until the contact force

reaches and stays within the specified band for Fs.

4. Calculate steady-state error by calculating the Root Mean Squared Error (RMSE) between

the measured force (F) and desired force (FD) during the steady-state region.

These measures are identified and illustrated in a force response plot shown in Figure 5. 

Figure 5: Depicted force response plot showing peak overshoot, settling time, and associated variables. 

iii. EXPERIMENTAL RESULTS

1. TEST SETUP

The settle stability experiment used a cylindrical bullnose tool held by an industrial, pneumatic 

gripper attached to the robot tool flange (see Figure 6). The settle stability artifact was mounted to 

a 6-axis load cell as described in Section III and shown in Figure 1. The robot executed a step force 

command along a specified axis from a non-contact state. The experiment collected data from two 

robots at three step input magnitudes: 1) a reliable minimum that invoked a useful robot response, 

2) a reliable maximum that was within the robot’s payload capability, but did not incur appreciable

twisting about the robot’s wrist during contact, and 3) a middle point. The robots repeated the step

force from non-contact 32 times in the directions of the forces Fx, Fy, and Fz in Figure 4.
6

Figure 6: Robot test setup showing the independent force measurement system with the settle stability artifact and the 

tool held by a pneumatic gripper. 

2. DATA

Settle stability performance measures include peak overshoot, settling time, and steady-state error. 

The three measures were calculated as specified in the test method above. The steady state force 

(Fs) was calculated over a one second window where all readings remain within a 1 % band. 

The overshoot results are shown in Table 3 below. The lowest peak overshoots for both Robots 1 

and 2 occurred in the X and Y axes, where they remained well under 10 N. However the peak 

overshoot along the Z axis was typically well over 100 N. We speculate that since the Z axis was 

aligned with the direction of gravity, the larger peak overshoots are due to the greater influence of 

gravity on the robot’s control. 

Table 4 presents the settling times of both robots under the three scenarios. Both robots consistently 

yield settling times of less than 2 s (except for one instance of 2.13 s for Robot 1), with repeatability 

of less than 0.5 s. 

Table 5 reports steady-state RMSE performance of both robots. The mean RMSE indicates the 

sustained average difference between the desired force and the actual imparted force. Overall, 

Robot 1’s force tracking accuracy was within 5 N, while Robot 2’s accuracy was within 20 N. Both 

exhibited comparable repeatability in this category with the standard deviation below or near 1 N. 

Table 3: Overshoot. 

Platform Direction

(Figure 4)

Step Input 

Magnitude (N)

Mean (N) STD (N) Max (N) Min (N)

Robot 1 X 1 6.84 0.25 7.73 6.47

35 -0.06 0.13 0.45 -0.19

70 -0.66 0.09 -0.22 -0.70

Y 1 5.17 0.18 5.64 4.83

25 3.53 0.24 3.92 2.86

50 0.32 0.37 0.66 -0.39

Z 1 52.13 2.74 58.42 48.83

35 130.38 19.07 135.74 32.65

70 164.79 1.42 170.89 162.51

Robot 2 X 35 -0.08 0.04 -0.02 -0.20

50 0.21 0.21 0.77 -0.12

75 0.27 0.40 0.90 -0.59

Y 25 7.04 1.37 8.85 1.87

50 0.34 0.38 1.18 -1.13

75 -0.28 0.32 0.60 -0.62
7

Z 25 41.69 7.63 51.57 15.81

50 121.21 3.03 126.26 113.07

75 248.33 2.89 251.37 234.90

Table 4: Settling time. 

Platform Direction Step Input 

Magnitude (N)

Mean (s) STD (s) Max (s) Min (s)

Robot 1 X 1 1.70 0.07 1.79 1.55

35 0.80 0.11 1.32 0.67

70 0.71 0.02 0.81 0.70

Y 1 1.73 0.02 1.75 1.67

25 1.01 0.02 1.08 1.00

50 0.91 0.09 1.00 0.76

Z 1 2.13 0.04 2.18 2.03

35 0.99 0.01 1.00 0.95

70 0.74 0.02 0.79 0.73

Robot 2 X 35 1.19 0.11 1.41 1.00

50 0.95 0.10 1.44 0.70

75 0.84 0.10 0.93 0.61

Y 25 1.89 0.48 2.96 1.00

50 0.98 0.17 1.43 0.56

75 0.71 0.06 0.96 0.63

Z 25 1.18 0.35 2.01 0.97

50 1.02 0.12 1.68 1.00

75 1.42 0.00 1.43 1.42

Table 5: Steady-state error. 

Platform Direction Step Input 

Magnitude (N)

Mean (N) STD (N) Max (N) Min (N)

Robot 1 X 1 1.30 0.14 1.69 0.91

35 1.28 0.15 1.79 1.07

70 1.91 0.16 2.10 1.40

Y 1 5.18 0.17 5.49 4.87

25 2.08 0.09 2.28 1.90

50 1.84 0.05 1.92 1.74

Z 1 3.29 0.35 3.98 2.87

35 3.21 0.33 3.69 2.27

70 0.93 0.05 1.03 0.85

Robot 2 X 35 18.97 0.78 20.76 17.91

50 17.19 0.77 18.39 16.00

75 17.00 0.76 17.89 15.10

Y 25 17.23 0.69 18.61 15.88

50 20.55 0.40 21.11 19.55

75 22.40 1.11 24.02 20.63

Z 25 4.03 0.46 5.01 2.93

50 1.52 0.94 4.08 0.23

75 1.31 0.55 2.80 0.25
8

B. DISTURBANCE HANDLING

Disturbance handling is a measure of the deviation from desired nominal force when moving along 

a surface profile. This measure, over a range of operational speeds, can indicate a robot’s 

capabilities to perform continuous contact tasks such as surface finishing and quality control. 

Stable systems respond quickly to changes in the surface and maintain a relatively constant applied 

force, while less responsive systems skip (i.e., leave) or gouge (i.e., excessive pressure on) the 

surface. 

i. TEST METHOD

Using the disturbance handling artifact (Figure 1a), program the robot to perform linear, Cartesian 

moves along specified segments on the angled planar surfaces as shown in Figure 7. Start with 

the robot in a contact state applying a force along the chosen test segment. For example, a force 

vector can be chosen along the tool Z axis and the tool pose can be controlled to maintain a normal 

force on each surface (Figure 8a). Similarly, the tool can maintain axis orientation and force with 

respect to the artifact Z direction (Figure 8b). Specify the force magnitude at one of the following 

levels: 1) a reliable minimum that invokes a useful robot response, 2) a reliable maximum that is

within the robot’s payload capability, 3) a middle point between the minimum and maximum force 

tested, and 4) a level defined by the intended robot application. Choose an end-effector transversal 

speed. Faster transversal speeds capture the effects of more aggressive maneuvers while slower 

speeds reflect less aggressive maneuvers. Measure contact forces throughout the test using an 

independent force measurement system. Perform the disturbance handling test 32 times for each 

test segment (path, load, and speed) chosen. 

Figure 7: Disturbance handling artifact showing possible test directions (left) and angled planar surfaces (right) 

Figure 8: Example force vectors for disturbance handling showing the force vector (a) normal to test artifact surface 

and (b) along the artifact Z axis (right) 
9

ii. PERFORMANCE MEASURES

Evaluate the deviation from desired nominal force when moving along a surface profile as follows: 

1. Calculate the total RMSE as the difference between the measured force (F) and the desired

force (FD). The total RMSE indicates the collective force tracking accuracy which is a

function of the system and its internal force feedback.

2. Calculate the controller RMSE as the difference between the measured force and the mean

measured force from the entire operation (FM). The controller RMSE indicates the level of

disturbance rejection when navigating the variably sloped artifact.

These measures are identified in a force response plot shown in Figure 9. 

iii. EXPERIMENTAL RESULTS

1. TEST SETUP

Robots 1 and 2 were commanded to interact with the test artifact as shown in Figure 10 at a 35 N 

and 25 N force, respectively, in the artifact Z axis direction as depicted in Figure 8b. The 35 N 

force for Robot 1 was chosen as a middle point defined as the average of the minimum and 

maximum forces, while the 25 N for Robot 2 was chosen as a reliable minimum that invoked a 

useful robot response. During continuous operation, each robot made initial tool contact with the 

artifact’s center under force control in the Z axis, and then moved their end-effectors back and forth 

along segments AB and CD. Both line segments were traced 32 times. This test was repeated at 

various end-effector speeds, and the results are shown in Table 6. 

Figure 9: Depicted force response plot showing the evaluation parameters associated with the disturbance handling 

metric. 
10

Figure 10. Robot test setup showing the independent force measurement system with the disturbance handling artifact 

and the tool held by a pneumatic gripper. 

2. DATA

As shown in Table 6, both the total and controller RMSE increase with increased end-effector 

speed. Furthermore, the proximity of the total and controller RMSE for Robot 1 indicates that most 

of the error is due to the controller itself, and very little due to the internal force feedback. Overall, 

the imparted force for Robot 1 is within 2 N of the desired value. As shown in Table 6, Robot 2 

experienced a more perceptible difference between its total and controller RMSE. On average, 

Robot 2’s force control tracking performance was within 20 N of the desired force. We visually 

observed and verified using load cell data brief periods of skipping, where the robot end tool lost 

contact with the artifact. This was due to the fact that performance errors where close to the step 

input force magnitude of 25 N. The skipping of Robot 2 from the test artifact happened more 

frequently at the faster speeds, and at the onset of ramp descends. 

Table 6: Mean, standard deviation, maximum, and minimum steady-state RMSE performance for Robot 1 and 

Robot 2 as calculated from 32 runs. 

Platform Step Input 

Magnitude (N)

Speed (mm/s) Total RMSE (N) Controller RMSE 

(N)

Robot 1 35 5 0.81 0.79

50 1.03 1.03

100 1.56 1.56

Robot 2 25 5 15.56 10.91

50 15.93 12.66

100 17.76 14.93

V. CONCLUSIONS

A set of performance metrics to characterize robots with intrinsic and extrinsic force sensing and

control capabilities is reviewed. Two test methods, one to evaluate settle stability and the other to

evaluate disturbance handling, are selected for further discussion and characterization. The test

methods are validated using two robots with force control capabilities and an independent force

measurement system with modular test artifacts. In the case of the settle stability metric, we present

measurement results for settling time, overshoot, and steady-state error to reach a desired force

contact with a surface. In the case of the disturbance handling metric we present measurement

results for the deviation from desired nominal force when moving along a surface profile. The
11

purpose of the experiments in this document is to exemplify the use of the test methods and the 

results are presented in a generic fashion with no indication of the brand or type of force control 

for each of the tested systems used. It is the responsibility of the end-users of the test methods to 

evaluate the systems under test whether for development or application analysis. Future efforts will 

focus on the defined force control metrics for the collaborative operation of these robots, as well as 

the metrics associated with robot assembly operations. 

VI. REFERENCES

[1] ANSI/RIA/ISO 10218-1-2011, ”Robots and Robotic Devices -- Safety Requirements for

Industrial Robots” and ISO/DTS 15066, “Robots and Robotic Devices – Safety Requirements

for Industrial Robots – Collaborative Operation”, www.iso.org.

[2] S. Patel, T. Sobh, “Manipulator Performance Measures – A Comprehensive Literature

Survey”, Journal of Intelligent Robot Systems, Vol. 77, pg. 547-570, 2015.

[3] J. Falco, K. Van Wyk, S. Liu, S. Carpin, “Robotic Grasping: Facilitating Replicable

Performance Measures via Benchmarking and Standardized Methodologies”, Robotics and

Automation Magazine, Dec. 2015.

[4] J. Marvel, J. Falco, I. Marstio, “Characterizing Task-Based Human-Robot Collaboration

Safety in Manufacturing”, IEEE Transactions on Systems, Man, and Cybernetics: Systems,

Vol. 45, No. 2, pg. 260-275, 2015.

[5] M. Shneier, E. Messina, C. Schlenoff, F. Proctor, T. Kramer, J. Falco, “Measuring and

Representing the Performance of Manufacturing Assembly Robots.” NIST IR 8090, 2015.

[6] J. Marvel, J. Falco. "Best Practices and Performance Metrics Using Force Control for Robotic

Assembly." NIST IR 7901, 2012.