1

ASTM F45 Committee Meeting, December 2017 Notes

Recording Secretary: Adam Norton, UMass Lowell

Attendees

• Roger Bostelman, NIST

• Adam Norton, UMass Lowell

• Malcolm Roberts, Guidance Automation

• Mitchel Weiss, Piaggio Fast Forward

• Karen Murphy, ASTM F45 Staff Manager

• Mary Ellen Sparrow, NextShift

• Bob Holmberg, X

• Matt LaFary, Omron Adept

• Joe Bencel, JBT

• Ron Brown, EWI

Wednesday, December 6, 2017 Notes

F45.90 Execute Subcommittee

Agenda review

Membership and attendees review

•

• Key roles to fill: Membership Secretary and F45.01 Chairman

• Questions regarding pending members for rosters: no option to provide more information about the 

member/their company, so it can be hard for subcommittee chairman to make decisions

• One option would be to hold a specific membership meeting, quarterly or bi-annual (like at these inperson meetings)

F45 Status

Published standards:

• F3200-17 Terminology for Driverless Automatic Guided Industrial Vehicles

• F3218-17 Standard Practice for Recording Environmental Effects for Utilization with A-UGV Test Methods

• F3244-17 Standard Test Method for Navigation: Defined Area

• F3265-17 Standard Test Method for Grid-Video Obstacle Measurement

Work Items:

• F45.01 - WK54576 Standard Practice for Recording Environmental Effects for Utilization with A-UGV Test 

Methods (original document being updated with: test method references, boundaries)

• F45.02 - WK57000 Standard Test Method for Docking Driverless Automatic Guided Industrial Vehicles

• F45.03 - WK54662?? Test Method for Grid-Video Obstacle Measurement 

• F45.03 - WK60390 Implementing Representative Obstacles for Utilization with A-UGV Test Methods

• F45.04 - WK54431 Standard Practice for Testing Data Communications Interruption for A-UGVs

• F45.91 - WK60974 Recording the A-UGV Test Configuration Other Status?

Update from ASTM
2

• New developments: roster search and organization tools, emerging professionals training program, 

update on public resources litigation (earlier this year ruled in favor of ASTM)

New NIST staff added to work on F45 efforts and new NIST set-up for test method exercising

Test of test methods; 5 year “grace period” for data to be included with test methods to validate them. Minimum 

of 6 labs to collect data to build the precision and bias statement (lenient about the quantity)

• ASTM funds and supports inter laboratory studies programs (members and non-members can 

participate). There’s NIST, NERVE, and the companies that can are on the committee.

• What is considered an appropriate lab for collecting data on a test method?

• How to perform the ILS: verifying reproducibility of test results under certain conditions, or demonstrating 

variability of conditions across the same performance standard?

Singapore now participating in F45

UTC Aerospace looking to evaluate AGVs, turning to F45, and has joined committee as a member

Promotional video, looking for B-roll of A-UGV footage (not just testing)

• Bob maybe can get something out of a Stanford lab he’s working with

• Adam can get footage from new NERVE test spaceShould get footage of A-UGVs is real environments, 

testing footage, etc.; show the correlations between the two

o Joe Bencil, JBT to ask their marketing people to provide footage

Other business discussion

• “Independent measurement system error”

o Add a paragraph to test methods to document how the measurement tools were calibrated?

o State in the report any error that is known about the independent measurement system

• Roger writing paper on A-UGV intelligence levels – Autonomous-UGVs are still a large, unclassified area of 

mobile robots. Test methods specific to level of autonomy will be required.

F45.90 ACTION ITEMS:

• Add membership discussion to agenda for next in-person F45.90 meeting, using ASTM roster tools

o Get an update from ASTM regarding the updated rosters (Karen?)

• Hold WebEx meeting regarding ILS (Inter Laboratory Studies) (Karen?)

o Can this instead occur at the next in-person F45 meeting at NIST?

• Have edits to promotional press release ready COB 12/15

• Request from all members who are able to provide some footage of their A-UGVs for the promotional 

video (Karen?)

• Discuss addition of a statement to all test method standards stating “any known independent 

measurement system error, fluctuations, how the measurement system is mounted, etc.”

• Any comments regarding A-UGV intelligence levels for Roger’s report, message him

• Get information regarding Emerging Professional program

F45.91 Terminology (Brief)

New/revised terms:

• Natural features – leaving it as is, using the word “navigation”, NOT “performance.”

• A-UGV system, A-UGVS – discussion about including the word “payload” or not. Definition as it is has 

some other unnecessary wording that we can remove.

• Obstacle – discussion about where the obstacles are (on ground, wall, or ceiling)

Didn’t get to talk much about the configuration practice

F45.91 ACTION ITEMS:

• Come up with new term for “improved” vs. “unimproved” facilities, meaning features that are added to 

an environment, like markers, beacons, etc.; unnatural features? Improvements? Additions? 

• Revised A-UGV system, A-UGVs definition: A-unmanned ground vehicle and all associated components, 

equipment, software, and communications necessary to make a fully functional system

• Revised Obstacle definition, remove parenthetical and discussion, so now should read: static or moving 

object or feature that obstructs the intended movement.
3

• A-UGV System and revised Obstacle definitions will be balloted. 

F45.01 Environmental Effects / F45.02 Navigation

Review test method footage from NIST

• Issue with A-UGV getting caught on slippery tape fixing letter A to the ground; need to discuss this in the 

document

o Need language in the document that refers to apparatus features not interfering with the test 

performance (borrow language from 5.2.1 in F45.03 Grid Video standard)

o Is the tape on the floor an environmental effect or apparatus failure? It is affecting the 

performance

o What happens in the event of an apparatus failure?

• Reviewing filled out test report forms 

o Potential issues with red/green icons for pass/fail

o Issues with filling out the actual success criteria; should’ve noted how many successful 

repetitions were needed for completion

o F45.02 form: how to measure path length? Do we need to? Is the average traversal speed 

important?

o Are the classifiers of continuous and transitional causing confusion when it comes to comparing 

measurements throughout a test method? Maybe just at a high-level?

Boundaries

• Reviewing added text to latest rev of document

• Which boundaries do you record? Only those that are “used” in the test?

• Revision of virtual walls description

F45.01 / F45.02 NAVIGATION ACTION ITEMS:

• Looking for F45.01 Chairman

• Borrow language from 5.2.1 in F45.03 Grid Video standard to develop generic language about 

measurement systems/markers not interfering with the test method performance; add this language to 

all test method documents

• Add second page to F45.02 Navigation test report form for drawing out the space (or reference a file)

• F45.02 document (and all documents that refer to Environment on the test report form) need to 

reference the F45.01 standard for recording environmental effects

• F45.01

o Add “Other” for lighting type on report form

o Need to add area for marking where the measurements were obtained (e.g., standoff distance 

from light) on report form

o Remove continuous or transitional on each individual environmental effect; add as just a highlevel characteristic to describe the intent of the test

o Need boxes for “None” air velocity

o Need boxes for “Other” boundaries

o Need to clarify floor gap width/length; which is which?

o How to mark multiple environmental effects? On multiple forms, in the drawing?

o How to mark other effects that are not categorized?

F45.02 Docking

Measuring ability to repeatedly fall within a TPM, not accuracy of the docking capability (i.e., pass/fail criteria)

Need to state at the beginning of the test how many repetitions are intended to be attempted

Is the grid on the vertical TPM confusing?

• Might want to remove the grid lines, or any mention of distance from goal/ideal docking location, so as to 

not suggest that measures within the TPM are being recorded

Review of purpose language

• Change to “confirm” evaluation of A-UGV docking performance
4

Discussion of Lx and Ly dimensions

• Should there be two measurements, one for each TPM?

• Text in the document right now might imply the reference frame for these dimensions, but it is not clear

Current document is largely concerned with positioning of ground vehicles to a location, within an area of error 

(TPM). Future document might be concerned with additional degrees of freedom and adapting to changes for 

docking/manipulation upon positioning

ASTM requirement to have “WTO TBT” statement in document (in all documents?)

F45.02 DOCKING ACTION ITEMS:

• Add Success Criteria back to form, add spot for “reps” (and on all other forms)

• Adjust 1.1 language to define purpose to evaluate by “confirming” a docking performance

• Possible adjustment of standard title to add “Confirming” or “Validating” docking performance; also 

changing “Driverless Automatic Guided Industrial Vehicles” to “A-UGVs”

• Be explicit about reference frame/point of origin in the document for Lx and Ly

• Need to define delta/omega for where the angle measurement’s origin is

• Adjust language in 5.5 so test does not fail if the repetitions begins outside of the TPM

• Add language about test requestor defining confidence levels and number of reps from 5.5 into section 

4.1. Then reference 4.1 language in 5.5.

• Be sure that language and examples in document include non-forklift vehicles

• Remove graticule references and from TPM drawings

• Change description and diagrams in figures 3 and 4 to show TPM pairs

• Remove orange circles around docking location goal in all diagrams

• Remove section 5.4 for “Calculating Task Performance Margin”

• Record measurement/distance between docking location goals / center of TPM ellipses

F45.03 Obstacle Detection and Protection

Membership review (still have room for 5 more producers)

Generic Grid-Video Method

• Discussion of single camera view on A-UGV moving, rather than multiple cameras in the ceiling

• Mitchell and Matt compiling content for sensor test method document

Implementing Standard Obstacles Method

•

F45.03 ACTION ITEMS:

Generic Grid-Video Method

• Need to move work item for general grid-video method to its own collaboration space (currently in 

collaboration space for accepted standard for grid-video obstacle measurement)

Implementing Standard Obstacles Method

• Everything in the document comments

Thursday, December 7, 2017 Notes

F45.04 Communication and Integration

Member balance review (9 producer votes available)

Implementation has kind of “cook book” instructions on an example of how to perform this using a script

• Should the test method not be as prescriptive and instead just state the options for how to interrupt? 

Duration, frequency, etc.

• Pulling the cable out, writing a script, using off-the-shelf hardware/software, removing/damaging 

antennas; input into document all as examples

• Many different methods for interrupting communications, so long as they are recorded/documented 

properly for a test that was performed
5

Need to clarify how/where to note success criteria for the test method that this practice is being applied to

• For instance, an F45.02 navigation test method where the intended result is that upon interruption a 

human has to intervene/restart the vehicle. That might be a successful interruption test, but using the 

wording in F45.02 navigation it would be a failure

• Need to ensure that any success criteria stated when using this practice can augment some of the “shall” 

statements in the test methods it is being applied to

• Can we separate out the performance of a F45.04 practice and the test you apply it to?

• Instead, use target test method form to record the relevant characteristics of the task being performed, 

but use F45.04 report form to record pass/fail according to the success criteria

Procedure should involve picking the interruption settings, picking the test method, and documenting where/how 

the interruption will be applied

• Shouldn’t be redundant with procedure steps in other test methods (such as recording environmental 

effects)

Need to ensure language where human intervention is allowed per F45.04 success criteria but doesn’t violate 

F45.02 success criteria

• Ignore success criteria of target test method; set up tasks as per the target test method (e.g., F45.02 

Navigation: Defined Area), record whether it successfully completed the specified task, timing, human 

intervention was involved, etc.

• Clauses/fault conditions for target test method are still in play, but the success criteria of the F45.04 

application is paramount

Is language in human intervention clauses in other standards too stiff?

• In this document, good to be clear about how interventions can be carried out; for resuming operations 

not for preventing errors

Are we setting out a pass/fail test based on what it does for the communications interruption? Or just providing a 

method to implement communications interruptions?

• What about the language in the test methods you’re applying it to with respect to whether or not a test 

can be continued after the rep is failed, due to communications interruptions? 

• Just use the task set up, procedure, apparatus, etc., from the target test method

What are the options for the results of communications interruption?

• Stopped, stopped and resumed, stopped and resumed with human intervention, failed repetition per 

application test method, successful repetition per application test method

• Can you only have human intervention in the event of an automatic stop, to resume operation, NOT to 

prevent errors/preemptively intervene?

Could note on the top of the form the checkboxes that are supposed to be checked off for 30 reps to be successful

Could provide note for saying you can run a baseline without communications interruptions (merely a 

recommendation)

F45.04 ACTION ITEMS:

• Add examples of how to cause interruptions (pulling the cable, breaking antenna, running a script to 

pause, etc.); like in an appendix

• Record how the interruption was caused, for how long, frequency, etc., where/when in a test method

• F45.04 report form would just be pass/fail repetition checkboxes and a place to define success criteria

• Make spot on forms to designate which test methods being applied to / which form is the 

“parent”/primary test method for performance

• Updated form to include checkboxes for performance during each repetition, with “row 0” up top that 

denotes what the success criteria is

• Add note for recommendation on how to perform comparative testing (baseline without comms 

interruptions, then with)

F45 Main Committee Meeting

Karen reviewed My Collaboration Areas on the MyASTM website and upcoming ASTM Compass service for viewing 

standards
6

Review of action items for other subcommittees

Review deadline dates between bi-annual in-person meetings (meeting minutes, items for subcommittee ballot, 

items for main committee ballot)

• Can we fit a second main committee ballot before the July meeting? Looks like we have time

Going to be new website for F45 committee on NIST website 

F45 MAIN COMMITTEE ACTION ITEMS:

• Hold WebEx for My Collaboration Areas, Registering Work Items, Registering on a Subcommittee

• No WebEx call next week for F45.02

• All technical contacts for work items need to get their collaboration areas set up, and add all 

subcommittee members and work items

• All intended subcommittee members need to sign up for subcommittees through the ASTM website

• For next in-person meeting, decide on schedule for when test method data will be captured in order to 

satisfy data collection within 5 years 

• Plan next in-person F45 meeting at NIST in late July 2018

• Need updated deadline dates for ballots; add second main committee ballot date before July in-person 

meeting

• What about changing the title of the entire F45 committee to involve the term A-UGVs?