1

Minutes: ASTM F45 Committee Meeting

NIST Gaithersburg, MD, July 10-11, 2017

Recording Secretary: Adam Norton, UMass Lowell

Notes also provided by Roger Bostelman, NIST

Attendees

1. Roger Bostelman, NIST

2. Mitchel Weiss, Seegrid

3. Malcolm Roberts, Guidance Automation

4. Ryan Gariepy, Clearpath Robotics

5. Bob Holmberg, X (formerly Google X)

6. Elena Messina, NIST

7. Omar Aboul-Enein, NIST

8. SooCheol Yoon, NIST

9. Adam Norton, UMass Lowell

10. Chris Merther, Industrial Truck Standards Development Foundation

11. Joe Bencel, JBT

12. Matt LaFary, Omron Adept

13. Mary Ellen Sparrow, NextShift

14. Jason Komorowski, Intel

15. Ron Brown, EWI

16. Pat Picariello, ASTM F45 Staff Manager (current)

17. Karen Murphy, ASTM F45 Staff Manager (future)

Monday, July 10, 2017 Notes

F45.90 Execute Subcommittee Meeting

Agenda review

Administrative review

F45 committee status

• 51 F45 members. 
2

Member Positions Status

• Dave Lewis, F45.01 Subcommittee Chairman is stepping down as chair

• F45 is looking for a replacement 

• Adam Norton was voted in as Recording Secretary

Karen Murphy, the new ASTM F45 Standards Advisor

• replaced Julia Moynihan

Ballot review

• Ballot 17-02 closed June 5 and was discussed within each subcommittee meeting. Reports are shown 

there. 

Voting members review

• Roger has sent an email to M. Hammerling informing them of non-voting status and to request their 

expected future participation in F45. Roger has also spoken with J. Rowley about his non-voting status. 

• Karen Murphy will be sending emails to J. Fox, M. Longacre, P. Schilke informing them of non-voting 

status and to request their expected future participation in F45.

F45 document status review

Current ASTM F45 standards and work items:

- ASTM F3218-17 Standard Practice For Recording Environmental Effects for Utilization with A-UGV Test 

Methods

- ASTM F3244-17 Standard Test Method For Navigation: Defined Area

- ASTM F3265-17 Test Method for Grid-Video Obstacle Measurement

- ASTM F3200-17 Standard Terminology for Driverless Automatic Guided Industrial Vehicles 

Continued ASTM F45 work item:

- F45.04 - WK54431 Standard Practice for Testing Data Communications Interruption for A-UGVs (being 

reworked to simply standard) 

Current/New ASTM F45 work items:

- F45.02 – WK57000 Test Method for Measuring Docking of A-UGVs

- F45.03 - Standard practice for capturing A-UGV positions using Grid-Video techniques. Lead: M. Weiss

- F45.03 - Standard practice for implementing representative obstacles for utilization with A-UGV test 

methods. Lead: A. Norton

- F45.91 - Practice for Recording the A-UGV Configuration

Proposed F45 test methods (“roadmap” built over previous meetings)

New business discussion, overview of proposed standards list

1. Environment effects, boundaries: used for both measurement and navigation, but maybe it’s not totally 

clear in how we currently talk about them

2. Generalized grid-video method; which sub-committee does it live in? F45.03?

o Current F45.03 grid-video standard doesn’t capture vehicle rotation (it’s only 1 dimensional), but

generalized method will need to

o Some work in ISO about deviation from a path, but they don’t have a tool for it. So this could be 

useful for them

o Measuring Obstacle Avoidance: Do we have a definition for what an obstacle is? May need to 

separate between obstacles and occlusions

o Could use generalized grid-video for obstacle avoidance measurement. So, need to focus on 

getting that developed

o Need to clarify some aspects about frequency of recording

3. Localization test methods:

o Very varied approaches, so may be too hard to generalize for a test method

o Maybe it’s just characterizing the testing; like placing A-UGV in a map and determining where it is
3

o Initial localization is the hardest problem; could be useful to have a standard way for customers 

to test localization in their environment; could characterize the customer’s environment by 

percentage of change? 

o Way to describe a user’s localization so a developer can understand the intended environment of 

the A-UGV

o How robust is the localization/navigation system as the environment changes?

4. Doorways/pass-throughs:

o Maybe this is just another example of an obstacle type that could be rendered or navigation

challenges; not necessarily its own test method

o We’re not a safety committee; keep in mind for measurement tools we develop

o Going through doorways for a short period of time, might not be totally relevant for our domain

o Is the information of going through a doorway useful for end users? Or is it just an F45.02 

defined area with an overhead dimension element? 

o Does it matter if it’s the thickness of a doorway or the length of an F45.02 single lane defined 

area? What happens when you block part of the area with walls and overheads (blocking 

localization sensors, obstacle avoidance sensors, not being able to see beyond into the next area)

o Sounds like it would be most relevant for the F45.02 defined area standard, adding a third 

dimension for a ceiling/overhead boundary

5. Positive / negative obstacles

o Do negative obstacles include an abstraction of the edge of a loading dock? Relevant, although 

may be categorized as a negative boundary rather than an obstacle (depends on how we define 

each); 

o Sewer grates, loading docks, pot holes, removable floor plates; useful for detection of false 

positives

6. Human form obstacles

o Maybe a different standard for human-form detection; is it A-UGV specific? Could be in its own 

committee outside of F45

o But, B56.5 deals with static obstacles that are meant to replicate parts of humans, so maybe 

dynamic obstacles with A-UGV could be more relevant to F45

o Some companies more interested in throwing humans (virtual or physical) into testing for deep 

learning; probably out of scope for F45

o Are we talking about “a person detector” or just an obstacle detector? The more relevant aspect 

might be dynamic/moving obstacles. Maybe doing so with B56.5 test pieces

o From Bob Holmberg (e-mail):

§ ISO 18646-2 "Robotics - Performance criteria and related test methods for service 

robots - Part 2: Navigation" and was out for a vote in ISO/TC 299 in May 2017.

§ NOTE: This is a Service Robots standard and so is not meant to cover Industrial Mobile 

Robots – i.e., A-UGVs. They are working on issues including "Path Accuracy 

measurement" and "Behavior to moving obstacles." The text in the latest ballot was 

quite rough.

7. A-UGV Configuration Practice

o Should include configuration in all test methods: mass, dimension envelope

o Software components? Need enough information to re-run the test and provide context for the 

end user of the data

o Are there high level categories we can put them into? Can’t get too detailed, has to be broadly 

applicable

Future WebEx Meetings review

Future WebEx Meetings: 

• F45.01 - 3rd Friday/month, 10 AM – 12 Noon next: 18 Aug

• F45.02 - 2nd Wednesday/month from 1-3 PM next: 9 Aug (note time change)

• F45.03 - 3rd Wednesday/month from 11-1 PM next: 16 Aug

• F45.04 - 1st Wednesday/month, 1-3 PM next: 2 Aug
4

• F45.91 - 1st Monday/month from 11-1 PM next: 7 Aug

Next in-person Bi-Annual Meeting:

Dates: Wednesday, December 6th 2017 - Thursday, December 7th 2017

Location: Sheraton New Orleans; New Orleans, LA US

Nominating committee

• All the same, no further nominations

F45.90 ACTION ITEMS:

• New standard proposals to pursue:

o Generalized grid-video

o Standard obstacles (positive/negative)

o A-UGV configuration practice

F45.01 Subcommittee Meeting on Environmental Effects

Ballot results

• Bob’s affirmative with comments; suggesting to remove references to standards that are either outdated 

or don’t contribute anything; going through the document to remove them

• Also, to add options for lighting levels and spectrum to include sunlight/white-light

• Ed Walker: Suggestions to remove references to draft standards. But, that is not true. We can reference in 

as a work item, no problem. However, two of them are approved standards which can be updated, and 

the work item standard can remain referenced to.

• Need to make sure we refer to references in the text, otherwise they cannot be included. Thus far, we 

have referred to all of them in the text, but we should do a final check on any other documents.

Reference standards added to the document

• Relative humidity level; inserted ref for existing test methods

• Ground surface; elevation changes, deformability, undulation; discussion of more options to describe, and 

some relevant standards

o Changing language to not refer to the vehicle characteristic (e.g., wheel width), but instead 

changing it to just step width, describe step profile, same for gaps, and sketches

• Deformability: test method for how much a post sinks into the surface; the test method only needs to be 

developed if we deem it relevant; but what is the resulting metric? Deformability WRT weight of artifact 

and depth into ground from test

o For now? Need to determine if we find it relevant

• Grade (ramp)

• Undulation: adding references to ASTM E1155M, ASTM E1486M

o Do those test methods provide useful data to fit into the categories that we’ve named? Not sure.

o Is there something in building codes? Like ADA 2010

o Relevant for service robot companies

• Coefficient of friction: adding references to ASME B46.01 Surface Texture; maybe too complex for our 

purposes

o Suggestion: ASTM E30393 

• How much analysis do we need? Does it need to be this complex? Sounds like maybe not.

o Might be more relevant for the outdoor performance, so we could separate the indoor from the 

outdoor environmental effects

o Make sure that the references we include are simplest ways to measure the characteristic we 

talk about (they’re optional either way); some of them right now are not

o Comes down to how it’s worded; “could” use these other test methods

• Floor surface roughness measurement: Surtronic Duo handheld device?

F45.01 ACTION ITEMS:
5

• Update F45.01 standard document and report form as follows:

o Change all mentions of “exposure” to “consistency”

o Change “Spectrum” options for Lighting to be fill-ins for color, color temperature, wavelength

o Remove “0 – 0” as a range, just call it 0

o Change all “LUX” to “lux”

o Change any references to F45 standard drafts to reference the work item or final standard

o Update figures 1 a, b, and c such that “front” label is outside of A-UGV body

o Add language to document/report form to describe how a user should mark relevant 

environmental effect measurements on the drawing of the test space (e.g., lux values for each 

light in the space)

o Include language in document/report form to describe where/how each environmental effect 

was measured (e.g., lux value measured at “A-UGV level”, or “1 meter away” from light source)

o Develop high-level categories of environmental conditions and put at beginning of each section 

as options for user; ask Mitchell to provide list of categories to start

o Add field for each section on form to mark the specific brand/name that renders the 

environmental effect; for instance, the brand of halogen bulb, etc.

o At end of each section, put “If more specificity of measurement is required, the following 

standards may be used...” and insert other standard test method references

o Generate example tests with corresponding report forms to demonstrate how to use the test 

method; could be coupled with standard document as “eLearning” material (like an instructional 

video)

o Make simple and complex (with ref’s) change to structure – i.e., simple as currently structured, 

complex with references if practice-user wants more information/test methods

o Higher level deployment environmental categories

§ Facility, outdoors, ...

Environmental Effects Demo

o Overhead bulbs - should it be direct light, exposed bulb?

§ Overhead fluorescent lights with x lux. Add as general note.

o Lights turn on and off – transitional

o 200 lx start, 80 lx at intersection, 90 lx at goal – measurements of light during demo using a light 

meter

§ Two forms for marking start and goal

§ Mark lux level on drawing

o Measure directed light (halogen) at three locations – beginning, middle, end

§ Dependent upon the course distance

o Beside the NIST reaction wall is much different than the middle of the room

o Take photos of the system to include in report

o Mark the actual bulb if you know what it is. i.e., power of the bulb

o Key points to record - perhaps a list of what are key points to replicate the test

o Include in the document, what is the facility, outdoors/indoors, etc. in the body of the document

§ high level unique environments

o Include two examples: minimal form filling, maximum form filling

§ Can also use the ASTM website to show a training video and/or photos

§ Do a pristine test set and a more natural environment test

Navigation: Defined Area and Docking Demos 

o Reports docking error, however add it to the measurement of ground truth.

o Be sure to include physical barriers being tall enough such that the A-UGV does not get 

erroneous information due to environmental effects, such as bumps in the ground that make the 

sensor see above the short walls
6

§ May need to be specified in further iterations of document

Adjunct files

Need to talk with ASTM more about how to host ancillary files for a standard (such as an Excel file, 

additional PDF forms, training videos, etc.), but should be possible

Other:

• Questions about precision and bias: needs to be the same vehicle model/make, not the specific vehicle for 

doing the 5 year interlaboratory study component; or we can avoid it with a pass/fail, people setting their 

own criteria, so let’s stick with that

• This is for repeatability/precision; maybe can use this for self-reported errors of the robot? Up to the user 

to do this, but the standard is purely about the measurement technique

• It’s a holistic test, doesn’t tell you if it's a localization problem or a maneuverability problem

F45.02 Docking Subcommittee Meeting

Document history issues review

• Rather than define every single possible combination of approach angles, orientations, etc., define the 

dimension and orientation variables and show a few examples

• Need to update use of “operating space” in the document

• Terms specific to the document: docking infrastructure, location, approach; A-FLV (fork lift vehicle), 

fiducial marks, task performance margin (TPM), TPM: dX dY dZ, Cartesian route distances (Lx Ly), A-UGV 

interaction point, nominal traversal speed

• Metrics: route, space required by vehicle, route distance, time to complete route

• Navigational improvements for vehicle, space, and docking station recorded: sensor system(s), additional 

markers used for location, features used for location, additional marks on docking apparatus

o Hard for natural feature localization to be specific about what is used, but just describing it 

generally as such should be fine

o All of these factors are important to vehicle configuration specs in general

Ellipse vs. square grid for TPM

• We’re describing elliptical performance, so using a square grid may muddy with the confidence level that 

we need to achieve (Mitchell to look into this)

• Could use the square grid method to record the values, and then calculate the ellipses in post

Discussion

• Where is the origin of the X and Y of the TPM? It’s relative to the frame of the TPM, but need to properly 

describe that in the document

• This also factors into how the Lx and Ly is described, as it is the distance from the start and end vehicle 

position, or the start position of the vehicle and the center of the TPM? 

• Lots of discussion over how to properly describe the distances and orientations of the vehicle and the 

TPM relative to each other

• Sounds like the conclusion is to remove measures of “rate” for speed, due to potential inaccuracies with 

how information is recorded

• Fiducial markings: do they have to be on the A-UGV itself, or on the payload that you’re carrying?

o Could be part of the configuration practice to say that adding a payload is part of the A-UGV 

system (i.e., any part of the vehicle that moves)

• Probably don’t need to provide examples of what can happen after the vehicle stops (e.g., a human 

picking something off of it, interfacing with another piece of equipment)

• Is calling it docking making it hard to explain? It is ultimately a position performance measurement test, 

but call out some examples of docking; or stationary position performance measurement

o Our definition of docking in F45.91 does fit how we use it in this test method though

o Could replace “docking” with “positioning”; might not be needed right now

• Is A-FLV needed as its own definition within the standard?
7

o Fork tines is very obvious key points for the vehicle; B56.5 uses “load engagement means”

o So maybe get rid of A-FLV as its own term

• Fiducial marks definition: do we need to include the example for simulated mechanisms? Muddying the 

waters, so remove

o Should the fiducial marks be on the fork tines, if the tines are moveable? Probably not. But, if 

they are on something that is moveable/loose, then the TPM should be bigger

o But there may be issues with where you put the fiducials; either way, it needs to be recorded on 

the form where the fiducial marks were placed so that it is repeatable

o So! We remove the definition of “docking infrastructure” and change all references from 

“docking infrastructure” to “docking location”

• Ellipses vs. squares

o Heading tolerance: not captured by ellipses? But the statistics are two dimensional and are 

therefore elliptical

o It’s up to the test requestor to define how large the TPM; most likely going to be an ellipse

o Ultimately it is going to be a region within the TPM grid

• Does the start position need to be exactly the same each time, as it is in F45.03?

o Include another TPM at the start (or under another name, like a SPM), within a margin that is 

defined by the test requestor

• When does the task end?

o When the vehicle stops moving; when zero velocity is reached

F45.02 DOCKING ACTION ITEMS:

• Edits discussed during session were live edited by Malcolm, so they are not all captured here

• Origin for X and Y of TPM needs to be described in document

• Same goes for Lx and Ly; from what to what?

• General consensus to remove mentions of a “rate” metric

• Specify in document when the task repetition begins (starts moving) and when it ends (stops moving)

• More investigation into ellipses vs. squares for TPM; Mitchell to discuss with some of his team

• More investigation into defining a start position TPM, probably under another name like SPM (Start

Position Margin)
8

Tuesday, July 11, 2017 Notes

Obstacles Demo

• Positive Obstacles

o Ensure proper definitions of the environment and the obstacles

o Overhead thick/thin

o Materials/surfaces captured

o Stepwise: box -> desk -> table -> overhead

o Transient obstacles not part of the map (by obstacle avoidance we mean transient)

o Check B56.5 language for “deliberately placed” obstacles, something along those lines, “hazards”

• Negative Obstacles

o Variable floor types, removable panels

o Cracks (false positives), grates, reflective, mirrors, transparent

o Test for desired performance: false positives, missed negatives

o Keep in mind: not a safety test

o Surfaces that don’t get return from the sensors

o Larger drops (need to capture depth of the drop)

• Boundaries

o Capture what type of boundaries are used for a given test

o A-UGV configuration: dependencies (extrinsic/intrinsic localization)

o Test set up representative of target environment

• Grid Video

o A way to measure vehicle response/performance; good for measurement of 

distance/movement, could be pass/fail based on success criteria

o Can be applied to other test methods like around obstacles

• Temporary obstacles; appears for one pass of the environment, then moves away

• Dynamic obstacles

Communications Interruption Demo

• Where in the connection does the interruption happen? Demo showed before access point, between host 

and before access point; could be implemented right at the server if applicable

• Single A-UGV interruption, looking forward to fleet-level, multiple vehicles and multiple access points

o Multiple access points first due to intermittence of connection across a facility, channel-jumping

• Starting with a single connection; simulating the symptoms not the causes

• Could future of this test be used for testing of an E-stop?

• Need to clarify the purpose; we want to simulate interruptions to a network, but are agnostic to why they 

have occurred

• Demo was for wi-fi, but can apply to other connection methods

• “Storm” of data on the same connection frequency

• Current state is for data loss, next step could be interference/overloading

o “Data integrity” could be another avenue

F45.03 Obstacle Detection and Protection Meeting

Ballot review

• Comments

o Table 1 is missing

o Some editorial comments from Holmberg; one regarding whether or not measuring the stop 

distance width is something that needs to be known/presented in the procedure; 

Weiss/Holmberg to hack out the changes, will make no effect on the actual test

o Weiss made most of the simple editorial changes already
9

o Measurement uncertainty: mechanical issues and software issues are not part of the 

measurement error but are actually part of the A-UGV performance; not going to remove it as an 

editorial change for now

o Jeremy Marvel: 

§ Definitions for task, repetition and test are in the document, but have since been added 

to the terminology document. Fine to have them in both places

§ Summary of Test Method section is pretty empty, so he provided one. But, for editorial 

changes, let’s leave it out for now

§ Timer to be included in the apparatus materials, so has been added, as it is already 

described in the rest of the document

§ Comment about delays in light sensors, etc.; ignoring

• Negative ballot

o Ed Walker:

§ Questions about the scope of the test: is it reducing kinetic energy? As a combination of 

braking and impact energy? Misalignment between what we see as the scope and what 

the reviewer does

§ Could be due to how the scope describes the test method, vs. in the context of the 

introduction

§ Mitchell did write a concerted reply to Ed (two), but to no response

• Second response: Weiss 1 is in response to Ed general comment, Weiss 2 is Ed 

1, Weiss 3 is Ed 2

• Ed 3 – 10 is found to be non persuasive from Weiss

§ Has suggestions for adding a line to the scope 

§ We address each of Ed’s specific comments one by one with the group

o Motions to 

§ General Comments editorial change motion carries

§ 1 editorial change motion carries

§ 2 editorial change motion carries

§ 3 non-persuasive motion carries

§ 4 non-persuasive motion carries

§ 5 non-persuasive motion carries

§ 6 non-persuasive motion carries

§ 7 non-persuasive motion carries

§ 8 non-persuasive motion carries

§ 9 non-persuasive motion carries

§ 10 non-persuasive motion carries

Note: After the committee meeting, Ed Walker rescinded his negative vote and the document, therefore passes 

ballot. Hence, the new document is titled:

• ASTM F3265-17 Test Method for Grid-Video Obstacle Measurement

Next steps

• Mitchell to start first document on generalized grid-video

• Adam to lead task group on rendering obstacles

• Next F45.03 WebEx should have a scope for both proposed standards

• Mitchell’s pitch for sensor tests are being tabled, but could get covered by the new developments

• Standard Practice for Implementing Representative Obstacles for Utilization with A-UGV Test Methods

o Should include objects from the ground plane and up

o Use as an addition to other test methods with no a priori knowledge by the A-UGV with the 

obstacles

• Standard Practice for Implementing Representative Travel Surfaces for Utilization with A-UGV Test 

Methods
10

• Standard Practice for Capturing A-UGV Positions Using Grid-Video Techniques

o Grid artifacts on a plane, fiducials on the A-UGV, cameras to watch the grid artifacts, camera 

parallax, FOV, frame rates

o Poor man’s ground truth

F45.03 ACTION ITEMS:

• Mitchell to start generalized grid-video method document

o Standard Practice for Capturing A-UGV Positions Using Grid-Video Techniques

• Adam to start rendering positive and negative obstacles document(s)

o Standard Practice for Implementing Representative Obstacles for Utilization with A-UGV Test 

Methods

o Standard Practice for Implementing Representative Travel Surfaces for Utilization with A-UGV 

Test Methods (could live in F45.03 or F45.01, to be determined)

• Next F45.03 WebEx meeting (August 16, 2017) will review scope for proposed standards

F45.04 Communication and Integration Meeting

Review of what the communication interruption document is (essentially, it’s scope)

Review of code examples used during today’s demos (packet delay, loss, rate-limited)

• Is the code included as part of the standard? Maybe as an example in an adjunct, or in an appendix. Either 

way, the descriptions of how each condition is supposed to function

• Is there a standard for network emulation in another committee? If so, is it relevant for our committee? 

Probably not.

Scope discussion for subcommittee (starting with original from 2015)

• Practice vs. test method? Practices can have test methods, so probably appropriate to have its own test 

method for baseline testing, then the practice is using it with another test method

• Does not involve setting standards for communication protocols (e.g., between a robot and an elevator)

• Started live editing the subcommittee scope; need to be clear about what we are doing with 

communication and integration

• Integration

o Communications vs. mechanical integration with other things in the environment

o Mechanical such as interlock systems

o Impairment testing of mechanical integration

o Suggestions to remove mentions of mechanical integration 

F45.04 ACTION ITEMS:

• Bob to continue updating F45.04 document based on new scope and discussions from today

• Need to change technical contact to Bob Holmberg

F45 Main Committee Meeting

Upholding the voting of F45.03 non-persuasive (same voting on all issues voted on earlier)

F45.91 - A-UGV System (A-UGVS) – not currently defined, although used several times in F3200-17 Terminology

• To discuss in the next F45.91 meeting

• No definition in B56.5.

Review/writing of scopes for all subcommittees:

• F45.01 Environmental Effects

o The scope of the Subcommittee is the development of a set of practices and test methods that 

identify, render, and characterize the environment of the A-UGV during operation.

• F45.02 Docking and Navigation

o The scope of the Subcommittee is the development of a set of practices and test methods that 

evaluates the performance of the navigation and docking performance of the A-UGV.
11

• F45.03 Obstacle Detection and Protection

o The scope of the Subcommittee is the development of a set of practices and test methods that 

evaluates the performance of the A-UGV when presented with obstacles.

• F45.04 Communication and Integration

o The scope of the Subcommittee is the development of a set of practices and test methods that 

evaluates the performance of the A-UGV, and its control interactions with other automated 

systems, under varied communications conditions.

• F45.91 Terminology

o The scope of the Subcommittee is the development of terminology commonly used for A-UGV 

performance testing.

• All scopes approved by vote

Are humans carried by A-UGVs in industrial environments covered by F45?

• Not explicitly, but F45 standards still applicable to those vehicles

• Worthy of discussion? If so, may need to tailor committee scope

• For now, remain silent

New Orleans facility tour for December meeting?

• Mitchell to check

Need to change technical contact for F45.04 document to Bob Holmberg

F45 MAIN COMMITTEE ACTION ITEMS:

• Submit scopes for each subcommittee to ASTM

• Work out logistics/agenda for December meeting and potential facility visit in New Orleans