1

ASTM F45 Committee Meeting, July 2018 Notes

NIST, Gaithersburg, MD – July 31 – August 2, 2018

Recording Secretary: Adam Norton, UMass Lowell

Attendance: 

Tuesday, July 31, 2018 Notes

F45.90 Executive Subcommittee (partially held within the demonstration area)

Distinguished guests:

ASTM

• Kathie Morgan President

• Jeff Grove Vice-President, Global Policy and Industry Affairs

• Phillip Godorov Director, Interlaboratory Studies

• Caitlin Farrell Project Leader, Interlaboratory Studies Team

NIST

• Howard Harary Director, Engineering Laboratory

• Al Wavering Chief, Intelligent Systems Division

• Kevin Jurrens Deputy Chief, Intelligent Systems Division

• Elena Messina Leader, ISD Manipulation and Mobility Systems Group

• Gordon Gillerman Director, Standards Coordination Office

Awards:

• ASTM’s James A. Thomas President's Leadership Award – presented to Roger Bostelman by Kathie 

Morgan, ASTM President

• ASTM Awards for Outstanding F45 Participation – presented to Adam Norton and Matt LaFary by Roger 

Bostelman, F45 Chairman
2

Demos: F45.02 Docking

Comparison of grid TPMs vs. circular TPMs

Setting multiple margins throughout the test, at one point in the test, at another point in the test; positioning vs. 

docking

Better methods for how to attach lasers for accurate measurement

TPM relevant to infrastructure, movable infrastructure relative to environment

Discussion of alignment tolerances for grabbing carts

How to determine what dimensions to use as success criteria for the TPMs; would be good to provide a method for 

how to use appropriate measurements, but still allow to be stringent as necessary

Need to stick with grid TPM to allow for measurement of heading/orientation

Using laser sights to grid vs. camera views to grid; parallax issues, etc.

• Could put the laser along the same plane/line as the fiducial; seems like laser is maybe the best method

Docking demonstrations – forklift, unit loader, mobile robot/cart

Demos: F45.01 Environmental Effects

On ramps, 10 degrees and 5 degrees, from slick concrete to slick steel, to wood

Differentiating between environmental effect and obstacle

Demonstrations of obstacle detections of ramp surface (ascend) and ground surface (descend)

Experiments with running standard tests by non-standards developers, running into issues filling out the forms; 

good red teaming exercise to determine updates we need to make

Need to capture details on maps made to run a test, etc.; make sure this info is included in A-UGV configuration 

standard

How do we articulate gaps/steps within a flat floor? Check the standard

How to define and record boundaries
3

• Are boundaries immovable? Can they only be static features?

• Obstacles vs. environmental conditions

• How to define static vs. dynamic features? As relative to navigation technique?

Level of variability in an environment until the vehicle fails?

We want to stay out of how implementation to a certain degree

Ramp demonstrations - 50 and 100 ramps

F45.90 Executive Subcommittee

Agenda review

Membership review

Where to have next meeting? December of November 2018

• Possibly international (looking for something not covered in snow)

• Maybe overlap with F11 (Vacuum Cleaners), started subcommittee for robotic vacuum cleaners, Spring 

2019 occurring in Geneva, Switzerland

• Volunteered UMass Lowell NERVE Center (Lowell, MA) for December meeting; need to finalize logistics if 

that is the way to go, or possibly for the Spring meeting

• Maybe change cycle to Spring/Fall instead of Summer/Winter

• All TBD over the next couple of months

• Proposed next meetings in F45 Main Committee Meeting notes

Document status review

Voting status review; have some continuous no-return voters. Do we change their status to non-voting members?

Subcommittee Webex meetings: can we standardize on a time for Webex calls (i.e., same time, but different days, 

for each meeting)?

• Proposed WebEx meeting days/times in F45 Main Committee Meeting notes

Quick ILS (Interlaboratory Study) overview to generate questions for discussion tomorrow

F45.02 Docking

Review of document history, focusing on varying TPM implementation methods (bars, ellipses, grids, etc.)

• Why the ellipse on its own doesn’t work; must have a way of measuring the heading 

TPx, TPy, and TPtheta +/- are to be specified for the success criteria (TP = task performance)

For each repetition, the VPx (lateral error), VPy (forward error), VPtheta (orientation error) +/- is recorded (VP = 

vehicle position)

• Need to define which direction (x or y) is forward, etc., relative to the world frame

The resolution of the graticule will influence the measurements and the statistical analysis

Need to ensure that the method by which we are calculating the measurements is a proper representation of what 

it is we are trying to measure

• At the moment, the method is correct if the TPMs are in the same horizontal plane as the fiducial marks

• But if they are not in the same plane (or are off enough), then we must capture that in the document
4

• For example, do we need to measure the levelness of the TPMs and the fiducials?

• This factor becomes very detrimental to fork tines that are high up in the air, say with lasers attached 

perpendicularly pointed at TPMs on the floor, and minor tilts in the forks will greatly affect the 

measurements

• Do we just need to point out that there will be errors in measurement? Or do we need to have a way to 

measure the errors in the measurement?

May need to define location of TPMs in a local/relative sense (say, to the object or environment) rather than in a 

global sense

The key is that the TPMs and fiducial marks need to be near each other (not far away, due to increasing errors)

• Not specifying how close they are together, but just that they should be close and that the distance 

should be recorded

TPz and TProll +/- for vertical TPMs

Good to delineate between TPM1 and TPM2 (and to update variable names based on this)

Do we need to include likelihood of failure statistics as part of our justification for number of repetitions?

Terminology review

• Untangling docking infrastructure, goal points, goal position, and goal location definitions

• May need to define “stationary” in F45.91 general terminology

Global vs. relative measurements / coordinates: it’s always relative

Goal points -> fiducial marks -> goal locations

Start points -> fiducial marks -> start locations

ACTION ITEMS:

• Questions on statistical significance of pass/fail test results vs. variable measurements, how it relates to 

Precision and Bias, etc.

• Need clarity on how resolution of graticule will influence statistical analysis and non-independent 

measurements; review existing statistical analysis phrasing from other standards

• Define x and y directions as those of the vehicle; no world/global frame

• To reduce errors in measurement methods:

o Specify that TPMs and fiducials must be in the same plane (within some range? How to measure 

that range?)

o Specify that fiducials and TPMs should be close to one another at the goal points and that the 

distance should be recorded

• Update report form to capture all variables, with newly determined names (TPM1, TPM2, TPM1x, TPM1y, 

etc.)

• Need to ensure the use of a single goal point (single TPM) is possible in the standard, such as for tugger 

vehicles (i.e., 1 TPM = no heading measurement, 2 TPMs = heading measurement, etc.)

• Cleaning up goal points vs. goal location vs. fiducials, etc., definitions; need to ensure not circular and 

clear

• Ensure introduction presents what the test method is supposed to be used for clearly; that it is for 

ensuring 1 or more points are reached

• Update “goal line” diagram to be used for TPMroll

• Continued discussion of terminology, lots more to untangle; Malcolm and Adam working on new diagrams 

and cleaned up definitions to propose

F45.91 Terminology

Reviewing of ballot 18-01 affirmatives with comments

ACTION ITEMS:

• Add “(s)” to end of “A-unmanned ground vehicle(s)” in definition of “A-UGV system”

• Three action items from other meetings that pertain to F45.91:
5

o F45.91 Terminology: We need to redefine the “docking” term to actually be “positioning,” and 

then redefine “docking” to actually mean “docking”

o Define “stationary” (for how it’s used in the F45.03 describing stationary obstacles standard)

o Add “communication impairment” to F45.91 terminology standard on next revision; TBD

F45.01 Environmental Effects

Reviewing of ballot 18-02 comments

Ramps

• Treated as an obstacle or environmental effect

• Grade vs. ramp

• Need to articulate the difference between obstacle and environmental effect in the document; we do not 

treat a feature as an obstacle based on how the vehicle interprets it, only based on its relationship to the 

environment, etc.

• Obstacle is a physical impediment that is not on the map... maybe this definition doesn’t work

• An environmental effect is... need to articulate

Negative obstacles

• Gaps and thresholds

• Is an environmental effect something that can be traversed when manually driven?

• Do we need its own standard for defining negative obstacles?

• Does it live in F45.01 or F45.03?

• Does it get added to the existing stationary obstacle practice?

Are we missing anything for outdoor environmental conditions?

Boundaries

• How to better record what the boundaries are

• Key thing is what information you need to replicate the test?

• If the boundaries are not being used for navigation, then they probably don’t need to be recorded

• Maybe just need to ensure that the boundaries that need to be recorded are articulated by the A-UGV 

developer, manufacturer, etc.

ACTION ITEMS:

• Include language in recording environmental effects standard to note that human readable versions of 

maps may be simplified from actual, reduced in detail compared to what’s used by the A-UGV, etc. 

• Need to develop standard definition to separate what is an obstacle and what is an environmental effect

o Maybe good example using air particles with smoke that will cause issues for lasers (may end up 

being treated as an obstruction/obstacle from the perspective of the A-UGV, but it is an 

environmental condition from the perspective of the human)
6

Wednesday, August 1, 2018 Notes

Demos: F45.03 Obstacles

Can use smallest obstacles method for just sensor tests, or full A-UGV system tests?

Can we develop a set of ~dozen unique obstacle configurations that can cover all bases for obstacles? Or for at 

least 90% of common obstacles?

How to define the baseline test method for detecting an obstacle? Is this just a sensor test?

As part of the building blocks, how to describe the placement of an obstacle in the environment?

Mock chair and smallest obstacle concepts

Demos: F45.04 Communication Impairments

Used a (freeware) software communication impairment with mobile base docking with a cart.

To view results of impairment, may need to time the overall task without (baseline) impairment and with 

impairment.

Simple way to implement network errors: have multiple equipment on overlapping/conflicting frequencies, etc., so 

they step on each other

Wireless Communications for controllers, mobile robot, and cart
7

ASTM International Interlaboratory Study Program (ILS)

They cannot conduct testing, but can assist with administrative aspects; data report forms, facilitation, 

confidentiality, etc.

Confidentiality vs. identified A-UGVs for when providing data

• Providing configuration data will be inherently identified, so need to be sensitive to 

anonymized/identified organizations that participate in providing data to support a standard test method

What to do regarding shipping of equipment?

• Do we ship around the same vehicle between labs? (“Round-robin testing”) 

• Or do we ship multiples of the same vehicle simultaneously to different labs?

What facilities can participate? Anybody. Manufacturers, research labs, etc.

What ILS can be run? Has to be tied to a work item/standard, committee has to agree that it will happen, has to 

lead to the development of a standard

Need at least single lab precision to get initial standard designation; within 5 years after, need full study for at least 

6 facilities; technically have up to 8 years, can be withdrawn, etc.

• ONLY if the test result is quantitative, not qualitative, then you need to get the data

• But, a pass/fail test may still be worthy of ILS given the variability in apparatus fabrication, running 

procedures incorrectly, etc.

• Probably still worth doing ILS, even on our pass/fail tests, to make a stronger standard (TBD)

There will be deviations from the 6 labs

• Precision: repeatability within a lab, reproducibility between labs; so each lab runs tests multiple times

• If there is a bunch of variability, it might not necessarily be bad, it will just end up being stated in the 

precision and bias statement

Standard deviation of repeatability (within labs) and standard deviation of replicability (between labs) with limits 

are reported; to 95% confidence

ASTM has some funding available to support ILS; shipping of large AGVs can be very expensive

• Useful to have buy-in from industry; such as industry chips in X, so ASTM chips in X, etc.

• Request funding from ASTM in advance! Not as reimbursement

• Do not pay to move people, like travel funds 

• Maximize placement of A-UGVs in labs to take advantage; run multiple tests to develop multiple precision 

and bias statements

If there is high variability, is that evidence to update/edit the standard?

• The precision & bias statement that is provided can then be balloted; on us to determine when to ballot it

• Will ASTM reject high variabilities? No, it’s on the committee to come to consensus if they agree that it is 

acceptable

• But, probably best to publish the high variability as a revision to the existing standard (for transparency), 

and then work on the revision. You don’t want to hide the variability of the existing standard

When you receive a precision & bias statement you can edit the wording, but not the numbers

If it’s important enough to have a standard, then it’s important enough to have a study to show variability, 

regardless of cost, timeliness, etc.

Can reference data be included in a test method to support ILS replicability?

• Yes. If it’s not mandatory, then it goes into an appendix

We need to figure out the right level of variability needed for the ILS as related to different types of vehicles, 

characteristics of test settings, A-UGV configurations, etc.

What is the typical output from each lab? A spreadsheet full of data? Some type of checks and balances that can 

be included to ensure everyone exercised the test method accurately?

• There will be a standard data report form

Do you design an ILS or just hand standards off?

• There will be a protocol that gets designed 

Exercising the test method to ensure it can be run properly with just the info in the standard (such as what has 

been run at NIST), should be run before the standard gets balloted

If there are issues with a standard, like really, fundamentally wrong, it can be withdrawn 
8

There are many standards that are published and do not have the minimal one lab test result

• If there isn’t single lab precision, you should vote negative on it

• It is mandatory for the standard, but it is on the committee to police it

There should be a research report associated with a precision & bias statement

• Research report is not included with the standard, but will be referenced; it can be accessed otherwise

The precision & bias section is mandatory, but the reference to a research report is not (likely would be caught by 

an editor, but we can come to consensus that a reference to the report is not necessary)

Is there guidance on how to set the right granularity for measurements?

• If it pertains to the procedure, then it’s on us to set them (but should be in the reporting section)

• If it pertains to the results of the test, should be determined by the ILS

Sample size? It’s determined by us, but the ASTM ILS team will work with us

Why isn’t a precision and bias statement required for pass/fail standards?

• The committee has not come up with a good way to determine the precision and bias for pass/fail data

• What about repetitions? Well, we do not allow errors (such as 20 succeeds and 10 fails). The moment 

there is a failed repetition, the entire test fails

You cannot use multiple repetitions of a test as multiple test results; the 30 repetitions will produce one test result 

(the average of the 30, pass/fail, etc.)

To e-mail the ASTM ILS group: ILS@ASTM.org

ACTION ITEMS:

• Hold a separate meeting to discuss the design of ILS for our existing standards; is an ILS necessary, what 

variables to consider, etc.

F45.03 Object Detection & Protection

• Summarizing the 'Describing stationary obstacles’ document

o Discussion of negative obstacles, Decision: we'll put them in their own separate standard

o Discussion of 'flat black' and surface texture, and that we don't have a good way to measure it... 

o Discussion that no one in the room except for NIST can measure the reflectivity of the surfaces 

for a given sensor... and that we’ll say these should be described as how humans see them ‘as 

they appear to normal human visual processing’

o Discussion that angle of incidence and angle of viewing affect reflectivity (specific example of a 

black plastic pallet that viewed at a 45 degree angle will be invisible to sonar)

o Added sphere and cone to the shapes

o Noted that other standards (like B56.5) have test pieces that can be described using this example

o In addition to descriptions of constructed obstacles pictures should also be taken

o Obstacles can’t change in the middle of a run (if one is damaged and isn’t the same, it’d need to 

be a new run)

o Baseline tests discussion

▪ Should we take the baseline test out of the document, since it could quickly become a 

rabbit hole?

▪ Does Navigation test say you can’t have obstacles? Not really.... (it’s kind of in the 

diagrams and dimensioning)

o The title of the document talks about using it in another A-UGV Test Method... 

o Added a sentence to say that using genuine obstacles decreases reproducibility.... and discussed 

order of preference of genuine versus artifact.

ACTION ITEMS:

• Take some pictures of example constructed artifacts to make it clearer to new folks how to make them

• Put a sentence in the standard that people could do a baseline, then put in an appendix that gives a nonmoving A-UGV baseline and a moving A-UGV baseline (not a test method or practice, just a suggestion)
9

• Instruct users to include 3D models of obstacle designs in test report if available

• Design test report

• Move whole precision and bias section should be moved to procedure and a new one should be made

• Pre-ballot document, Webex to resolve comments, then ballot

• Draft method for finding minimum obstacle with Lego or similar

o Might be most relevant for ASTM E57 committee, not F45; can still go through NIST for 

development though

• Make a set of a different artifacts that are representative

• Negative obstacles; as its own standard/document, or supplemental to this standard, but to be dealt with 

after standardization of this document

F45.04 Communication & Integration

Is it worth including “communication impairment” as a term in F45.91 terminology?

Should conducting a site survey be mentioned in the standard?

Need a note, or means to quantify, the impact of the test system on the A-UGV prior to conducting the test?

• We recommend conducting a baseline in the standard

• If the impairment equipment is affecting performance (either through hardware or software), that effect 

needs to be quantified

• May not be worth it if you spend too much time determining if the impairment equipment is having an 

effect, rather than determining if the impairment is having an effect on performance

What if the A-UGV is not affected by the traffic controller? How can you tell?

• Do you run a baseline test using a condition that you know to cause issues? Like introduce a blackout on a 

connection that you know will produce an observable result (i.e., you can tell that the traffic controller is 

indeed working and affecting network communications)

How to perform evaluations of A-UGV fleets?

• Maybe shouldn’t need to be covered here, but as its own standard 

• What about testing communications in the presence of a fleet?

Does this standard cover physical obstructions to antennas, sensors, etc.? Not sure if it is repeatable

ACTION ITEMS:

• Mention how a site survey can be used to determine network issues 

• Include a note that introduction of the impairment equipment may have an impact on performance; 

perform another baseline test wired through the traffic controller, but without impairments active

• Add block to diagram for connections to the cloud; note that the diagram does not represent the only 

connections where impairments can be incurred (i.e., the connections to the cloud could happen at many 

points for many items, and we don’t want the diagram to get too muddy)

• Tell users to include a diagram of their network topology when conducting the test

• Check form and style for how to write test method within a practice correctly

• Development of fleet testing standard; put into the roadmap of what needs to be developed in the future

o Consider how tasks get re-assigned amongst the fleet when one A-UGV fails, etc.

• Update test report after standard is more fleshed out 

• Restructuring of content into the appropriate sections is needed

• Change “interruption” to “impairment”

F45 Main Committee Meeting

ASTM website -> Get Involved -> Membership -> Member Resources -> Virtual Classroom for Members; lots of 

training sessions for members on how to handle negative votes, roster maintenance, etc.

ASTM Officers’ Training Workshop, September 25-26, 2018 at ASTM Headquarters in West Conshohocken, PA

MYASTM -> My Committees -> My Tools -> Roster Maintenance/Report; many of the members that have applied
10

to the subcommittee only provide minimal information

• Maybe need to better police who is a voting member and who isn’t

Possibilities for next meeting, or changing the meeting schedule

ACTION ITEMS:

• Get more video of A-UGVs and test methods for promotional F45 video!

• Next meetings:

o Winter 2018: December 10th week: Omron –San Francisco, CA – Roger has a conf call next week 

to discuss with Eduardo’s group

o Spring 2019: possibly May 13th week - IEC HQ in Geneva, Switzerland (overlap with F11 meeting) 

– Karen to check on this

o No demos at IEC – unless we schedule elsewhere

o Fall 2019: Lowell, MA – Adam agreed to host

• Photos from Monday demonstrations and awards will be put on NIST F45 website: 

https://www.nist.gov/el/intelligent-systems-division-73500/unmanned-ground-vehicles-research-andstandard-test-methods

• Proposed days/times for new Webex meeting schedule (Karen will setup e-meeting schedule for sub’s):

• F45.02 - 1st Friday /month from 1-3 PM

• F45.03 - 2nd Friday /month from 1-3 PM

• F45.04 - 3rd Friday /month, 1-3 PM

• F45.91/.01 – 4th Friday /month from 1-3 PM
11

Thursday, August 2, 2018 Notes

Workshop 1: Guide to A-UGV Capability Levels

Draft paper from Roger that will soon be published

Idea being that is serves as a guide for classifying autonomous vehicles and possibly as a roadmap for the F45 

committee; do our test methods allow for each capability level to be demonstrated and evaluated?

There are also implications for safety standards within the capability levels

Are augmented modes are a part of this, like for automated cars? That involves human-in-the-loop, which this 

scale does not concern itself with

• Sort of fits into the lower levels (1 and 2) 

Where did the automatic, automated, autonomous term definitions come from? We developed them in previous 

meetings, but they did not pass as terminology in the standard

• Given that we have a number scale, do we need the separate terms? 

Review of each metric/capability and the definition of each level

• Discussion of the text used for each level; may be worth breaking out some of the capability levels further, 

maybe not as additional levels, but as alternatives/options for each level (i.e., more rows within the cell so 

that saying a vehicle is at a certain level can be further specified)

• Maybe remove sub-category ideas, 

• Is it worth breaking out each capability as its own standard?

Key to keep in mind: F45 is focused on performance evaluation, NOT on safety evaluation. It has implications on 

safety, but not specifically for validating safety or not 

Crucial to point out that the capability levels are additive

Why are the capability levels defined the way they are?

• Is it due to levels of abstraction? Should be explained as to why the levels are the way they are

• The capability passes from outside of the system to inside of the system

• To further clarify “outside” of the system; e.g., fleet manager vs. human operator

Should Table 2 column be named “METRIC” or “CAPAILITY”?

Good underlying concept: autonomy is always contextual; the context under which the classification occurs (e.g., 

“my navigation is level 5”), needs to be captured

• Need to appropriately limit the interpretation of these levels

Important to keep in mind that the manner in which the level is reached (i.e., what hardware/software techniques 

to use) is left to be agnostic

• Need to ensure the test methods provide observable metrics/demonstrations that show that the 

autonomy level was indeed reached

• If we can’t validate or prove that a level for each capability was reached, this effort may not prove as 

fruitful

How can this scale be used? Who consumes it? Is this information wanted?

• Stakeholders throughout the industry ask for this type of info, could be used for promotion, etc.

Is it worth putting capability levels on the report forms? If the info is known, maybe.

Another capability may need to be added for localization, perception (e.g., using reflectors to define where they 

are in space vs. using SLAM, etc.); but hard to order those in a way that is considered getting “better” as the levels 

increase?

Does the ordinal nature of the capability levels imply that a higher level performs better than a lower level? 

• This is not always the case; performance can be associated with each capability type 

• Maybe reduce from capability levels to capabilities

• Maybe alphabetical (A-G) instead of numerical (1-8). Can still imply that the higher letters include lower 

letters capabilities, but does not imply that it performs better

Need to ensure the capabilities are associated with a certain robot configuration

• Many robots have options that could put them into different capabilities; need to communicate what 

options are available

Is it indeed true that boxes to the right assume boxes to the left? May be cases where this isn’t true
12

• Such as a vehicle that can path plan around obstacles (higher level) but cannot navigate using a physical 

guidepath (lower level)

• May remove concept of including lower levels in the higher levels

The group reviews and edits some of the rows in the table, adding new ones, changing cell contents, etc. 

Can we link a test method to each capability? Good to keep the community thinking in this manner

Do we need a definition for each capability for the metric that delineates each capability type?

• We are establishing them as we fill in each cell, but should be explicitly said so the reader understands 

why

ACTION ITEMS:

• Good to include a note that safety concerns/standards are not explicitly covered here; focused on 

performance

• Ensure explanation for relationship between higher levels imply lower levels is clear; is it assumed that 

lower levels are assumed by the higher levels? I don’t think so anymore

• Explanation of the metric for the scale; need to establish what variable(s) characteristics for each 

capability are changing between levels

o Well put: “the capability passes from outside of the system to inside of the system”

• Drop automatic, automated, and autonomous labels from capability levels; good to also use unique 

number of levels for each capability

• Issues of guidepath terminology taken from B56.5 standards; needs to be cleaned up

• Hold future e-meetings within F45.91 for continued development of capability levels

• This document may be registered as a work item in F45.91 as a guide

Workshop 2: Combining A-UGV Standards 

Combining practices with standards; combining multiple standards, and maybe with practices

What is still considered a fault of a failure?

What changes will be needed to the existing standards to allow combining?

• Success criteria of singular test methods may no longer be relevant; may need to adjust language to say 

that the success criteria can be modified

• For example: F3244 (F45.02 Navigation: Defined Area): modifications needed for 

o 9.3 Test and Task Failure (human intervention may be allowed if a communication impairment is 

used)

o 9.4.9 for completing the standard test

o “...may be overridden when multiple standards are combined, as described in F####.” (the new 

standard)

o Combining multiple segments may be done in the Defined Area standard

o Issues with start and goal lines? Replace with start and goal positions? We lose timing 

measurements for planning, etc., when only crossing the lines are used. As the spaces become 

larger (i.e., connecting multiple segments), the planning time to start will be even longer

o “If they are being combined, the goal line may be modified by another test method.”

o Variables for X, Y, D1, D2, etc., will need to be expanded when using multiple defined areas

o Same considerations for minimum dimensions of aisle lengths “...only when used individually, 

when combined can be more variable.”

o Still checking off how many types of defined area segment shapes are used?

o Using multiple types of barriers throughout the set up

o Establish concept of “segments” for multiple defined areas?

• Standard Usage Guide for combining standards to run more complex tests

o Where does the actual “combination” parts go? Under Procedure? 

o What report forms do you need to use?
13

o Framework for how to combine standards; more “free-form” reporting method to define the 

criteria of the test that uses combined standards (such as columns of check boxes, like the 

Communication Impairments form, or just writing out a sentence)

o How is best to communicate the tests that were run? Naming convention? 

Does the standard guide template allow for procedures, set-up, etc.?

• Discussion section could describe how to combine them

Inputs and outputs from each standard

ACTION ITEMS:

• Update F3244 to start task repetition timing when the A-UGV receives the command to begin navigating 

(figure out the best way to phrase it)

• Future meetings: step through existing standards and work items and propose revisions to text that will 

allow for multiple defined areas to be chained together, and add new sections as needed to test with 

multiple segments (see notes above)

• Review text of standards and work items with proposed revisions for allowing combining standards:

o Roger: F3244-17 Standard Test Method for Navigation: Defined Area

o Malcolm: WK57000 Standard Test Method for Docking Driverless Automatic Guided Industrial 

Vehicles

o Adam: WK60390 Describing Stationary Obstacles Utilized within A-UGV Test Methods

o Bob: WK54431 Performance Testing of an A-UGV Under Varied Communication Conditions

o Mitchell?: F3265-17 Standard Test Method for Grid-Video Obstacle Measurement (not sure if 

applicable)

• Future meeting for combining standard usage guide boilerplate document