The main systems we examined in the survey are presented in Figure 4. Although this
listing does not include all of the systems that comprise the AGSS, it does cover those of greatest
interest to the end user, where the Trainee Station is central. In addition, systems not identified in
the figure may be captured within the context of larger applicable systems (e.g., aural cues (gun
sounds) within Weapons Simulation). With this schematic in mind, a brief description of each
major AGSS system follows.

Helmet ‘Visual System ‘Weapons Simulation

Instructor/Student Interaction

Instructor Operator
Station

Figure 4. Main System Schematic of the AGSS.
Helmet

The AGSS is equipped with three helmets for the left, right, and tail positions. The helmet
is a Kaiser Electro-Optics Sim EyeTM (see Figure 5). It is enclosed, lightweight (2 Ibs), and fully
adjustable with removable velcro pads of various sizes. Each helmet has a binocular HMD which
is attached to the helmet using plastic screws. In addition, each helmet is equipped with a
magnetic head-tracker system. The electronic and communication cables attach to the HMD and
extend past the shoulders and lead up to the electronic interface unit (EIU) in the center of the
AGSS. Each helmet also has headphones and a microphone.

Figure 5. Kaiser Electro-Optics Sim EyeTM AGSS Helmet.


track map page. Threats do appear on the ground track map and could be used to help navigation
calls, especially for the FEs.

External Environment

Comments. Several questions were designed to tap into the existence of potential
problems associated with locating the IOS outside the motion base of the AGSS (unlike other
simulators at the 58 TRSS). Question topics included noise level, visual distractions, and the
instructor level of task saturation due to the inability to see students and monitor screens for both
student performance and aircraft performance. The only question having a large proportion of
participants rating it as a problem “sometimes” or more was the level of instructor task saturation
experienced at the IOS. Task saturation was noted by participants whether there was one student
(5 of 8), two students (5 of 8), or three students (4 of 6) in the AGSS. |

The mean levels of instructor task saturation experienced with one, two, and three students
is shown in Figure 9. As can be seen, there is a linear relationship between the reported level of
task saturation and the number of students in the AGSS. Thus, as the number of students in the
AGSS goes up, so does reported level of task saturation. Although the comparisons between the
means were not statistically significant, participant comments suggested that the demands of flying
ownship, laying down threats, and monitoring students can wreak havoc on the level of training
that instructors are able to provide students in the AGSS. Figure 9 also shows that the severity of
reported task saturation increases when the one SME, who had a large role in the development of
AGSS IOS and over eight years experience as a simulator instructor/operator, is removed from the
calculations (i.e., the right bar graph of each pair in Figure 9).

Instructor Workload atthe AGSS 10S

Level of Reported Task Saturation

1 2 3
Num ber of Students in the AGSS

ED anios respondents J Minus 10S respondent with extensive 10S experience

j

Figure 9. Reported Levels of Instructor Task Saturation with One, Two, and Three Students in the
AGSS.



Recommendations. It is not surprising that the helmet adjustment process takes longer
during the first use, nor is it unexpected that readjustments are required during training sessions.
Nevertheless, AGSS curriculum developers need to be aware of these time requirements so that
tasks and objectives can be tailored to the allotted time periods. As mentioned previously, the
proper adjustment and alignment of the AGSS visual input system (e.g., CRTs) is critical. Here
again, proper alignment procedures are recognized as a primary factor for reducing readjustments
during AGSS flights. Two main recommendations are made in this regard. First, the 58 TRSS
should attempt to standardize and document the fitting and aligning process so that when
knowledgeable instructors and maintenance support relocate, their knowledge does not leave with
them. Second, the squadron should provide additional (or separate) time prior to training to
familiarize trainees with the helmet, VR technology and terminology, and alignment procedures.

Trainee Stations

Seven individual trainee station components (gun feel, gun set-up, mobility in station, VR
aircraft interior, VR other stations, crew seats and crew harnesses) were assessed in our survey.
Five are discussed below based on the Kolmogorov-Smirnov analysis and participant comments.

Gun Set-Up and Feel

Comments. Seven participants rated gun set-up as a “3” or less. The primary problems
identified concerned gun alignment and adjusting the gun barrels properly within the AGSS.
Similarly, six participants rated the virtual mapping of the surrounding gunner stations as a “3” or
less. The main criticism involved the virtual mapping of the other players in the aircraft
environment. Specifically, there is an FE “mapped” into the middle seat when one looks forward
in the AGSS virtual environment, but no other members of the cockpit nor the backend are
“mapped.” Several individuals commented on this selective mapping. Some participants
suggested mapping players into all crew positions (ie., model a pilot and copilot in the other
cockpit seats), while others advocated eliminating the mapping of virtual players altogether.

One participant provided two important insights regarding the general fidelity of the
trainee stations. First, he noted that he was more likely to downgrade the fidelity of the other
stations (vs his own) because he could more easily “nitpick” what should and should not be there
as he looked around. Secondly, he noted that if he was looking at the other stations and not out
his window, then he was not doing his job. The tradeoffs between selective fidelity and “time on
task” may need to be explored further as the AGSS becomes more fully integrated into the
training curriculum.

Recommendations. First, the difficulties with gun barrel alignment can be reduced by
standardizing and better conveying the general alignment process. In terms of mapping other
players, the two choices seem to be: (a) either map all the members of the cockpit in the AGSS
(which may not be possible given current IG modeling limitations), or (b) map none of them.
Here, again the tradeoff of selective fidelity should be explored. For example, mapping all of the
cockpit players may increase user-acceptance of the device which may in turn have a beneficial
effect on the training effectiveness of the device.

29



Visual alignment ratings, as previously reported, were unsatisfactory. In this case, the
assessment focused on the visual/virtual representation of the physical world, with six participants
rating it as problematic. Most of the comments identified small details of the virtual environment
that did not match the aircraft. One participant, however, said “that [the AGSS] was more two-
dimensional than [he] would like and that it was not as three-dimensional as VR technology
claims.”

Image color was rated as problematic by six participants. The main complaints were that
the colors looked “washed out” and that there were noticeable differences between new and old
CRTs. Importantly, AGSS image color has been improved since the survey, but the issue of CRT
breakage and decay still needs to be addressed.

Although rated significantly positive (see Table 10) by our survey Participants, we probed
the FOV feature further because of its importance in the fulfillment of AG/S tasks. We asked two
yes/no questions regarding FOV for daytime and NVG night conditions. Seven of 11 participants
reported that they did not achieve a full FOV for daytime operations, but only two of nine
participants reported being unable to achieve a full FOV for NVG night operations. This likely
reflects the fact that during NVG use, one’s peripheral vision is already reduced and the AGSS
visual system is able to mimic this. The primary concern during daytime operations was that even
with their understanding of the visual system’s constraints on FOV, participants were unable to see
fully forward or aft because they could not “get their heads out of the windows” as many
(particularly FEs) do in the aircraft.

Not surprisingly, seven survey participants gave low overall assessments of the visual
system. The specific comments seemed to reflect the visual system features participants were
most dissatisfied with, such as the CRTs, distance cues, visual contrast, etc. But despite their
apparent dissatisfaction, several comments suggested that participants could train with the device
as is, but there was room for “a lot of improvement.”

Recommendations. Participant dissatisfaction with the visual system was quite pervasive.
As a main component of the AGSS and given its criticality to AG/S tasks, a primary
recommendation is to explore the visual system in further detail, to gain a better understanding of
the many problems encountered by survey participants and which features are the most important
to the fulfillment of AG/S skills. While many of the comments focused on definite inadequacies
with the system (e.g., washed out color or faulty IG), others seemed to be related to
misconceptions about the capabilities of simulation (e.g., discrepancies between real-world image
and the virtual image). With a substantial percentage of the users (both instructors and trainees) of
this device being new to simulation, it may be that many of their critiques have more to do with a
lack of knowledge about simulation capabilities rather than inadequate device capabilities. This
would suggest that additional or separate familiarization time be provided where the device and its
capabilities are briefed. In addition, further exploration of the visual system would distinguish
areas for which there are real problems with the hardware/software from those which are user-
centered. Based on the survey and the skills the AGSS is designed to train, three visual system
features require immediate attention: object detail, color quality, and visual continuity (or update
rate).

38



Given the sophisticated computational and display capabilities associated with the AGSS,
this system clearly satisfies all the criteria for a Level 4, the highest level of immersion for VR
technology. While achieving immersion is a worthy goal from a technological standpoint, it
remains to be seen whether that sophistication translates into improved user performance or
increased training effectiveness.

Although there is now considerable commercial interest in VR, its application to combat
mission training is still in its infancy. For example, the Army Research Institute (ARI) is exploring
the use of VR technology to create virtual environments for training spatial knowledge with
dismounted soldiers (Bailey & Witmer, 1994). Researchers are finding evidence that participants
can learn to navigate through real-world places by training in a virtual environment, although such
training is not as effective as walking through an actual building. On the other hand, the
differences between VR and real-world navigation training in producing accurate spatial
knowledge may dissipate with greater amounts of practice.

The Defense Advanced Research Projects Agency (DARPA) and the Army Test and
Evaluation Command (TECOM) are conducting a joint project to determine the feasibility of
using VR technology to train automobile drivers to achieve high levels of performance on an
instrumented test track (Kuhl & Wargo, 1994). Initial transfer of training results are encouraging,
and through use of virtual prototyping simulation, the investigators are exploring the applicability
of VR to conduct tactical battlefield training in DIS-based exercises.

Not surprisingly, the USAF is sponsoring research to study the effectiveness of VR
technology in training situation awareness among fighter pilots (Hettinger, Nelson, & Hass, 1994;
Mowafy & Congdon, 1994). Researchers at AL/HRA are collecting data to identify those spatial
knowledge tasks whose performance appears to be aided by training in a virtual world. Some of
the areas where VR technology appears to hold promise involve providing pilots with multiple
perspectives (ie., inside the cockpit vs an outside or exocentric point of view) during an
engagement, navigating through the environment in three-dimensions to estimate energy
management performance, simultaneous comparison of flight trajectories flown by students and
experts, and joint use of icons and audio to give students immediate feedback concerning
performance in a simulated mission (Mowafy & Congdon, 1994).

While the potential for VR technology to positively impact combat mission training is
quite real, there are nevertheless some problem areas that have been highlighted in past
implementations. These problem areas must be resolved in any successful application and,
accordingly, will be assessed in our human factors survey.

There have been numerous reports, mostly anecdotal, of problems experienced by HMD
wearers in simulation settings. The most commonly reported problems inchide eye strain, blurred
vision, headaches, and occasionally, debilitating “flash backs” that occur several hours after the
simulation experience. In some cases, inadequate performance of the head-tracking mechanism
causes significant transport delays, producing system lags that result in scene blurring as the user
moves his/her gaze across the display. Other perceptual problems have been reported due to the
insufficient display resolution in many present-day HMD image sources (e.g., liquid crystal
displays or LCDs) for rendering detailed graphic scenes (DOD, 1994),



Miner, N.E. (1994, June). Using voice input and audio feedback to enhance the reality of a virtual
experience. Proceedings of the Image VII Conference. Tucson, AZ.

Mowafy, L. & Congdon, T. (1994). Air combat surveillance in a virtual world training system.
Proceedings of the Image VII Conference. Tucson, AZ.

Mowafy, L. & Miller, L. (1993, October). Visualizing spatial relationships: Training fighter
pilots in a virtual environment debrief interface. (AL/HR-TR-1993-0148, AD A273 972).
Mesa, AZ: Armstrong Laboratory, Human Resources Directorate, Aircrew Training Research
Division.

Norusis, M. J. (1993). SPSS professional statistics 6.1. Chicago: SPSS, Inc.

Nullmeyer, R.T., Bruce, P.D., Conquest, M.T., & Reed, E.T. (1992). The use of the MH-53J
weapon systems trainer for mission rehearsal: The mission preparation process and lessons
learned. Proceedings of the 14th Interservice/Industry Training Systems and Education
Conference. San Antonio, TX.

Nullmeyer, R.T., Bruce, P.D., & Spiker, V.A. (1994, April). Simulation-based mission rehearsal:
Some behavioral issues. Paper presented at the 14th Applied Behavioral Sciences Conference.
Colorado Springs, CO.

Nullmeyer, R.T., Spiker, V.A., & Reed, E.T. (1997, in press). Strategies for effective simulation-
based mission rehearsal. (AL/HR-TR-1996-0147). Mesa, AZ: Armstrong Laboratory,
Human Resources Directorate, Aircrew Training Research Division

Pimentel, K. & Teixeira, K. (1993). Virtual reality: Through the new looking glass. New York:
Intel/Windcrest/MacGraw Hill.

Reed, E.T. (1993). U.S. Air Force helicopter training high fidelity simulation providing advanced
mission training and rehearsal. Proceedings of the 15th Interservice/Industry Training Systems
and Education Conference. Orlando, FL.

Reed, E.T. (1996). The aerial gunner and scanner simulator: “Affordable virtual reality training
for aircrews.” Proceedings of the Royal Aeronautical Society Conference on Training -
Lowering the Cost Maintaining the Fidelity. London, UK.

Siegel, S. (1956). Nonparametric statistics for the behavioral sciences. New York: McGraw
Hill.

Thurman, R.A. & Mattoon, J.S. (1991). Virtual reality: Theoretical and practical implications.

Proceedings of the 13th Interservice/Industry T; raining Systems and Education Conference.
Orlando, FL.

48



scheduling of additional personnel (e.g., operating in the integrated mode or using an additional
AGSS instructor).

FUTURE RESEARCH

The next step in the human factors analysis of the AGSS is twofold: (1) user acceptance
and (2) training effectiveness (remember Figure 2). Both of these areas have begun to be
explored by the Kirtland AFB research team. In terms of user acceptance, the 58 TRSS has
created a series of questionnaires for instructors and students to determine their responses to the
device immediately following training sessions. The questions cover a wide array of topics
including, human factors aspects of the device, training provided, and device fidelity.

In regard to training effectiveness, we are looking at device impact on skill acquisition and
performance (largely guided by the skill ratings acquired from this survey, see Table 12) and
AG/S training hours in the aircraft. Without going into too much detail, we are: (a) observing
training for five scheduled Mission Qualification AGSS missions, (b) attaining skill ratings of
trainees by instructors for these missions, (c) obtaining objective performance pages when
possible for these missions, (d) securing ratings of skill performance from the flight line on
students receiving AGSS training, and (e) analyzing AG/S training folder data. It is through these
primary means that we hope to determine the impact of the AGSS to the AG training system.

Finally, because these various mechanisms are in place, we can also address the training
effects of system upgrades potentially resulting from this survey. For example, the AGSS has
currently been used solely as a stand-alone trainer because of delays in the integration process.
We will have several months of data on a significant number of students who received only AGSS
stand-alone training. As soon as the device becomes integrated, we will be able to compare those
students skills and device opinions to those who start to receive both stand-alone and integrated

AGSS training.



VISUAL SYSTEM - Field-of-View

1. Did you achieve a full FOV?

- If NO what items were you unable to see? (List here)

Did your FOV change when you moved your head Vertically?

Did your FOV change when you moved your head Horizontally?

Did your FOV change when you moved your head Forward?

Did your FOV change when you moved your head Backward?

Playa)

Were you able to execute a normal scan?

DEVICE CAPABILITY FOR TRAINING CERTAIN SKILLS

{9 =Poor 2 Min Req

SUSEnEn

SKILL

Rating

SKILL

Day Night

1. Voice Procedures? 22. Tactical knowledge?
2. Checklist Procedures? 23. Rules of Engagement?
24. Terrain/Obstacle avoid.?

3._ Gunner Cockpit Exchanges? 25. Waypoint identification?

4. _Left/Right/Tail call resp.?

5. _Left/Right/Tail interactions? 26. Light signals?

6. Radio Discipline? 27. Target acquisition?

7. Aircrew briefing? 28. Ammo conserve?

8. Passenger briefing? 29. Use limitations?

9. Other: 30. Emergency procedures?

10. Crew Coordination? 31. Gun maifunctions?

32. NVG failure?

11. Safety and judgment? 33. Aircraft taping and lighting?

12. Resource management? 34. NVG use and limitations?

13. Knowledge of directives? 35. Radar? XX

14. Communication system? 36. AAA Threats? xX

15. Taxi and hover calls? 37. Terrain masking?

16. Approaches and landings? 38. Threat breaks? xX

17. Go-arounds? 39. Countermeasures?

18. Scanning? 40. Overall Training Capability?

19. Search operations?

20. Remote operations?

21. Tactical approaches?

MISCELLANEOUS

1. Did the AGSS fly like the aircraft? Y N

2.__ Did you ever feel disoriented in the AGSS? Y. N_

If YES, under what circumstance(s)?

a. During Target Acquisition? YY
b. During TF/TA? Y
c. During Takeoff? Y
d. During Landings? Y
e. During Crashes? Y
f._During System Reset? _¥
g. When Instructor is flying with the space ball? 2,
h. When flying in Left Station? —¥
i. When flying in Right Station? YY.
J. When flying in Tail Station? YY

3.__If you ever felt disoriented, did a particular ‘system contribute to this perception? Y

4.__Were you satisfied with the motion system? Y

54



Additionally, throughout this report, we have made recommendations for improving those
AGSS components that trainees rated unfavorably and which were frequently criticized. In Table
15, we summarize the main problem areas and our recommendations for their improvement. The
table provides gross estimates of the cost and training “payoff associated with each

recommendation.

Table 15. Main AGSS Problem Areas, Recommendations for Improvement, Associated Cost
Estimates, and Potential Training Payoff Estimates.

. Training
Problem Area Recommendation Cost Payoff
CRT Failure Rate Tavest in higher quality CRTs. HIGH HIGH
CRT Discrepancies Purchase new CRTs more frequently. | MEDIUM HIGH
' Invest in higher quality, more recent
Slow Head-tracker Update Rate logy. HIGH MEDIUM
Introduce slower NVG scanning ’
rates early on in AGSS i NONE HIGH
Poor Pads Invest in higher quality pads. LOW HIGH
Length of Electronic and +
Cc nication Cables Invest in longer cables. MEDIUM MEDIUM
Provide a means of tethering cables
off the students? shoulders. Low MEDIUM
Virtual Mapping of Stations (re: * see
other players in other positions) Map players in all positions. Low* LOW
Do not map any players in other Low LOW
positions.
Second Instructor Position’s Provide capability to communicate
Lack of Communication directly with students in the GSS, | MEDIUM | VERY HIGH
Capability
Problems with Space Ball for Change software modeling of
Ownship Flying aer ‘cs. HIGH MEDIUM
Change controls to a joystick. HIGH HIGH
fone ‘use space ball for ownship NONE LOW
High Levels of Instructor Pa
Reported Task Saturation at IOS Operate only in integrated mode. HIGH MEDIUM
Limit number of students in AGSS. MEDIUM VERY LOW
Provide two instructors for training. MEDIUM HIGH
Change layout of instructor screens. MEDIUM HIGH



provided beneath each question so that the interviewer could write the details of subject
responses, including under what circumstances any breakage occurred, suggestions about
remedying noted problems, and ideas that participants had about improving the system. In the
cable subsection, specific yes/no questions were listed about whether the cables interfered with or
restricted task performance. The adjustments subsection provided space for specific time
estimates required for fitting the helmet properly and aligning the visuals (during the subject’s first
and subsequent exposures), and whether any readjustments were required during AGSS sessions.
At the end of the helmet section, space was provided for general comments.

The trainee station section included blanks for rating trainee station components, e.g.,
gun set-up, station mapping, etc. A 5-point Likert Scale 1 (poor) to 5 (exceptional) was
provided. The verbal anchors of the scale referred to both similarity of the station component to
the aircraft (the MH-53J or MH-60G as applicable) and the training capability of the component
(see Table 2). Space was again provided for comments and/or suggestions.

Table 2. Survey Scale for Rating Judged Similarity to Aircraft and Training Capability.

Rating Value Definition
1 Poor No similarity between simulator and aircraft. Provides negative
training and has major deficiencies.
2 Min . Little similarity between simulator and aircraft. Major
- Required modification is required to provide adequate training capability
3 Standard Acceptable similarity between simulator and aircraft. Adequate
training is provided.
4 . ‘The simulator and the aircraft similarity is nearly equal.
Outs B Training capability is similar to that provided in the aircraft.
. The simulator and the aircraft are virtually the same. Training
5 Exceptional capability is beyond expectations.

Like the helmet section, the IOS section was fairly detailed and inclusive. The first
subsection contained questions about the location of the IOS outside the AGSS and the level of
task saturation experienced by operators. The 5-point Never to Always scale was used, and space
was provided for comments. The main IOS subsection included specific components of the IOS
(e.g., the space ball) which were to be rated on their training effectiveness using a 1 (poor) to 5
(exceptional) point Likert Scale (see Table 3).

Table 3. Survey Scale for Rating Judged Training Effectiveness.

Rating Value Definition
1 Poor Cannot train required tasks; system has major deficiencies.
a a Minimal training can be provided; major modification i
2 Min. u 2 Pr 4 cation Is
Required tequired to provide adequate training capability.
3 Standard Training capability is acceptable.
q Most of the required tasks can be trained; few, if an:
4 , ifany,
Outstanding fifications ‘red.
: Training capability is above expectations, and tasks beyond
5
Exceptional those required can be trained.

16



Recommendations. There are at least two potential solutions for reducing the level of
instructor-reported task saturation: limit the number of students who are trained concurrently in
the AGSS or add a second instructor at the IOS (as already mentioned). We view the first
solution as inadequate. Although the instructor’s ability to provide feedback and observe training
is increased, the level of realism of training and the number of skills that can be practiced (e.g.,
Left, Right, Tail call responsibilities or crew communication) are significantly reduced. The
second solution is really the best, provided the second instructor position is enhanced to include
the ability to directly communicate with students in the AGSS. Two instructors at the AGSS
would allow increased realism, better feedback, more role playing, and better use of the available

AGSS capabilities.

Although administered as its own section during the interviews, questions concerning
instructor/student interaction and the communication system are logically related to the above
issues, so its results are noted here. Several yes/no questions about instructor and student ability
to hear one another were asked, None of the participants reported significant problems with
either hearing or understanding the students while at the IOS, nor were participants unable to hear
or understand one another or the instructor while in the AGSS.

Weapons Simulation

Based on the Kolmogorov-Smirnov analysis and participant comments, six individual gun
features are discussed below. The overall assessment of the weapons simulation is also discussed
due to the number of participants rating it as a “3” or less.

Modeling Fidelity

Comments. Two related items, the ballistic model and tracer fidelity, both had a large
number of individuals rating them as a “3” or less: eight individuals for tracer fidelity and six for
the ballistic model. Although the specific comments for each element were different, a common
problem was inadequate mapping of the projectile jump, or lead and lag, of the bullet’s path. In
particular, the problem arises from using F-16 (an aircraft with guns mounted on the nose)
software for modeling the bullet path. In the MH-60G or MH-53J, two of the guns are mounted
on the sides of the aircraft leading to different aerodynamics on the bullet. This requires different
aiming procedures since a bullet fired from the side window of a helicopter does not travel in a
straight line as it would if it were fired off the nose (or tail). Although most comments focused on
this issue, and how the AGSS would be better if this were incorporated, one individual said: “The
bullets go straight and this does not teach anything. You just can’t replicate real guns.” Other
critiques about the tracer path and the ballistic model concerned the number of tracers per rounds
fired, the different cues in the AGSS (compared to the aircraft), the inability to see tracers leaving
the gun, and the existence of jerky visuals during target engagement.

Recommendations. The primary recommendation regarding the ballistic model is to
improve it by incorporating lead and lag effects. Again, however, such a change requires
extensive modifications to software and additional AGSS funding. Alternatively, if this is not an
option, students may be instructed about the differences between the AGSS and the aircraft. The
AGSS could be used as a potent illustrator to contrast the bullet’s path in the aircraft versus the

34



TABLES

Page
Summary of Survey Participants’ AGSS and Flying Experience............. 15
Survey Scale for Rating Judged Similarity to Aircraft and
Training Capability ... 16

3 Survey Scale for Rating Judged Training Effectiveness 16
4 Survey Scale for Rating Judged Training Capability for Specific AG/S
. 17

5 Summary Assessments for Each Major Survey Section. 19
6 Helmet Rating Analysis............. 22
7 Trainee Stations Rating Analysis... 22
8 IOS Rating Analysis. 23
9 Weapons Simulation Rating Analysis... 24
10 Visual System Rating Analysis. 24
11 Survey Participant Verbatim Responses Regarding Helmet

Adjustments. 28
12 Specific Object Ratings and Representative Comments. 36
13 Mean Ratings (Descending Order) and Standard Deviations of AGSS

Training Capability for 39 AG/S SKiIIS.....0...cscsssssessssecsseecsseessseessseeensees 40
14 Main Findings from Each Section of the Survey: Problems

and POSitives.......eescescsssesssessssssseesseesssecsuesssnecsucsssecsssecssueessecssecensecreeeaeee 43

15 Main AGSS Problem Areas, Recommendations for Improvement,
Associated Cost Estimates, and Potential Training Payoff Estimates.



helmet slippage, several participants also suggested that the helmet pads were the primary culprit.
Indeed, one participant stated directly that the reason the helmet fit poorly was due to hard pads.
Despite the frequent mention of helmet pad problems, only two participants reported problems
with the comfort and fit of the helmet pads “sometimes” or more.

The second aspect of note was the report by four survey participants of hot spots when
using the AGSS helmet even after short (¢.g., 30 min) AGSS sessions. The helmet pads were
again cited as one reason for the hot spots. However, several respondents, including those who
did not rate hot spots as a problem, singled out the helmet’s method of use as a key factor. They
suggested that if the appropriate adjustments and sizing were made, hot spots could be avoided,
such that “with proper pads and proper adjustment these [hot spots] didn’t occur.”

Although only two participants reported major problems with the chin strap, it is worth
mentioning because of its relationship to poor helmet fit. Participants proclaimed: “it’s
uncomfortable,” “it cuts into your skin,” “no real padding,” or “needs a different kind of
padding.” They also suggested that the helmet’s method of use influenced their ratings. For
example, one participant said, “If the chinstrap is upside down with the fabric touching your face,
this is a problem, and this can occur if it [the helmet] is put on improperly or moves during

ining.”

Recommendations. Short of acquiring new helmets or integrating the CRTs and head-
tracker into the same helmets worn in the aircraft, all of the above complaints (helmet padding,
hot spots, and chin strap discomfort) could be remedied by improving the quality of the pads and
their method of attachment to the helmet. The helmet fitting and adjustment process also needs to
be monitored closely. Moreover, as the AGSS is incorporated into MQ and Annual Refresher
training, more standardized procedures should be established to help alleviate problems and/or
discomfort that arise from improper fit helmet.

Helmet Electronic and Communication Cables

Comments. Two questions concerning the helmet cables are notable because of the
comments associated with each. First, four participants reported that the current cable length
limited their amount of normal movement around the aircraft. Respondents qualified their
answers, stating that for the subset of tasks performed in the AGSS, their movements were not
hindered. One participant explained that “cable length is not too much of a problem because of
what is done in the AGSS, but amount of movement is less than in the aircraft. Tasks requiring
movement are things like switches, weapon malfunctions, and getting equipment.” Some cited
particular stations or configurations (methods of use) as more problematic than others. For
example, one participant reported that the range of motion in the tail position compared to the
aircraft was reduced in the AGSS as a result of the cable length. The MH-60G crewmember
interviewed also noted that when the AGSS was in the MH-60G configuration, the range of
motion from seat to door in the left and right positions was less than the aircraft.

Second, three participants reported qualified problems with the amount of pressure they
felt from the cables tugging at the backs of their head when performing tasks in the AGSS. Those
who encountered problems with this feature stated various caveats, such as, problems only

27



Trainee Stations

As mentioned, up to three students can be trained simultaneously in the AGSS. Each
trainee station provides a spatially and visually correct representation of the aircraft (either the
MH-53J or the MH-60G, depending on the designated configuration). A 7.62 mm or a .50 cal
weapon can be mounted in each station. The visual representation of the station is transferred
through the IG and contains aircraft-specific texturing and details. All window frames and edges
have the appropriate spatial relationship to the gunner. In addition, crew seats and harnesses are
visually and functionally realistic.

Instructor Operator Station

Instructors are located outside the device at an IOS with six screens (see Figure 6): one
for instructor input (lower left); one for information output (upper left); one for instructor
viewpoint selection, either pilot’s eye view, God’s eye view, or Target Zoom (lower middle); and
three that repeat each trainee’s visual scene (i.e., Left Station, upper middie; Right Station, upper
right; Tail Station, lower right). There are two positions at the IOS, each with a headset. The
primary instructor position is located in front of the instructor input and output screens. From
this position the instructor is capable of interacting with the students in the AGSS through the
intercom. The second position is in front of the student repeater screens. The instructor or
observer at this station can watch and listen to all of the training activities, but he/she cannot
interact directly with the students in the AGSS through the intercom. The IOS also has a space
ball—which enables the instructor to position targets, fly air-to-air models, and fly the AGSS
when it is used for stand-alone training—a color printer, and a work table. Pre-programmed
flight paths can be recorded and replayed for multiple practice sessions on the same flight path.
Performance monitoring pages allow trainers to print out gunnery performance scores (e.g.,
number of hits, type of ammunition, etc.) for individual gunners as well as gunner crews.

Figure 6. AGSS Instructor Operator Station.


Second, hierarchical cluster analysis was used to analyze the training capability ratings.
The primary goal of cluster analysis is to identify homogenous groups of objects, in this case
skills, based on distances among the objects. We used hierarchical clustering analysis to identify
distinct skill clusters. The main steps of this approach were: (a) transforming the rating values
provided by the subjects into z-scores, thereby appropriately weighting each participant’s use of
the scale; (b) calculating the distances between all possible pairs of skills; and (c) combining the
skills into clusters or groups. In hierarchical cluster analysis, all objects begin as separate clusters
where there are as many clusters as there are objects. At each iteration (or step) of the analysis,
an object is added to an individual cluster or two clusters are combined (Norusis, 1993). There
are several methods for combining clusters. We chose the between-group linkage approach which
ensures that at each step cases are combined so that the average distance between all cases in the
resulting cluster is as small as possible.

RESULTS
Overall Findings

Before discussing the specific results from each section of the survey, we present an
overview of our major findings (see Appendix B for Raw Rating Data). Table 5 summarizes the
results from the five major AGSS systems assessed and the overall assessment of the device’s
Training Capability. Besides the average overall rating which are generally acceptable, the table
gives our general assessment (inferred by the researchers from participants’ comments and
ratings) and representative comments for each system.

Table 5. Summary Assessments for Each Major Survey Section.

System Mew Ravn Assessment Representative Comments
Helms Good => When properly fit, everything should
“ 43 (when working) be okay.
Trainee Stations 38 Good => Much better than first design.
Overall, it is nice.
Instructor Operator 3.9* Good = L, it is nice. ;
i i => Ownship flying with space ball is
Station (room for improvement) ess.
it i => Not enough tracers per rounds fired.
Weapons Simulation 3.6 Needs Work => Lead and lag problem.
system. aulty = Can’t see targets until they are too
Views 32 F close even with instructor input.
ining C ne => Primary impacts will be on crew
* bility 38 (uiited) coordination, scanner calls, and target
acquisition if visual problems fixed.

*Note. Represents an inversion of the subject responses (i.c., never to always) to make it parallel to other
component ratings.

Starting with the helmet, we see that its overall rating was quite high, making it the highest
tated AGSS system. From these ratings, we gave it a general assessment of “good.” However,


survey participants had extensive instructor experience in their respective positions. In addition,
every participant had some level of experience with the AGSS. A summary of the participants’
AGSS and flying experience appears in Table 1.

Table 1. Summary of Survey Participants’ AGSS and Flying Experience.

Position
Flight Engineer (n=5) Aerial Gunner (n=6)
Mean (Maximum {Minimum Mean Maximum |Minimum
AGE 40 48 32 36 43 28
AGSSTOT 34.80 100.00 2.00 114.33 300.00 1.00
|AGSSBOX 5.70 12.00 +80 19.33 50.00 1.00
COMBAT 226.80 433.00 00 86.17 280.00 00
FLYTIME 4520.00] 6500.00] 1800.00} 1720.00] 2500.00 720.00
FLYTLM 8.00 20.00 00 19.17 40.00 00
FLYTNVG 1420.00} 2500.00 900.00} 1141.67] 2000.00 500.00

Note. AGSSTOT = estimated total number of hours experience with the AGSS, either during development or
training; AGSSBOX = estimated number of hours experience at an AGSS gunner station; COMBAT = estimated
number of combat hours; FLYTIME = estimated total number of flying hours; FLYTLM = estimated number of
flying hours in the month prior to the survey, and FLYTNVG = estimated number of flying hours using NVGs.

The Survey

The survey was designed to capture both quantitative and qualitative information about
the functionality and fidelity of the AGSS. A brief description of the data collection tool follows,
and an abbreviated version of the survey appears in Appendix A. The survey was divided into
nine sections: (1) background information, (2) helmet, (3) trainee stations, (4) IOS, (5) weapons
simulation, (6) instructor/student interaction, (7) visual system, (8) training capability, and (9)
miscellaneous.

The background information section included questions regarding the participant’s age,
current responsibilities, crew position, AGSS experience, flying experience, visual history, and
past experience with unusual physiological symptoms (e.g., dizziness or blurred vision) during
flight, after flight, and during everyday activities (e.g., reading).

Representing a primary component of the AGSS, the helmet section of the survey was
fairly detailed and thorough including subsections on helmet component breakage and
malfunctions, comfort and fit problems, the occurrence of problems with the electronic and
communication cables, and adjustment requirements. For both the helmet breakage and comfort
and fit subsections, specific helmet components (e.g., chinstrap or visor) were itemized for
ratings. A 5-point scale (never, rarely, sometimes, often, always) was provided. Space was also

15



occurred during extreme movements or that the cables were distracting but they would probably
get used to it. However, one participant did report that he felt like the cables “kept getting caught
on something,” and that he was forced to stop what he was doing in the AGSS and unhook them
from his back two or three times during training.

Recommendations. Due to the small number of respondents reporting cable length
problems associated with task performance or discomfort, there is some question as to whether
the cables really are a problem. Considering the participants’ comments, it seems that if only the
appropriate AGSS-supported tasks are performed, the cable length is satisfactory. However,
further consideration of cable length for the tail position seems warranted. Given some of the
specific comments, longer cables may be necessary for the tail position to perform his/her required
tasks. In addition, a few respondents did suggest that a means of tethering the various cables up
off their shoulders may be beneficial for the times when the cables did get caught, especially for
the MH-60G configuration, where the crew seats are used.

Helmet Donning and Adjustment

Comments. Included in the helmet subsection of the survey were questions that asked
participants to estimate the number of minutes required to put on and adjust the helmet during
their first and subsequent encounters with the AGSS. Participants also indicated whether
readjustments were necessary during training sessions. Average donning time for the first use was
11.1 min; the average for subsequent sessions was 3.2 min. The reduction in donning time was
Statistically significant (t = 4.15, p < .01, df = 9, with one participant excluded because he only
wore the helmet once). In addition, three participants reported the need to readjust the helmet
“sometimes” or more during AGSS sessions. (Five additional participants noted problems
“rarely” and some of their comments are included in the table below.) Three categories of
adjustment problems appeared in participants’ comments. These involved issues of alignment, the
air pump, and familiarity. Table 11 summarizes the participants verbatim responses.

Table 11. Survey Participant Verbatim Responses Regarding Helmet Adjustments.

Category Participant Responses

=> Had problems with the alignment process.

=> Figuring out what alignment means seems to be the biggest

Alignment hurdle when first setting up the helmet.

=> It seemed off, but I didn’t want to adjust it because I didn’t

want to screw with the alignment.

Miscellaneous problems with the air pump.

Adjusting pump. Normal head movements changes how

helmet feels, makes you adjust comfort level, often too tight.

. Found myself adjusting it every 5 minutes or so.

Air Pump = It was too tight and I had to release some air pressure.

=> Pumped it up too tight initially and let air out as flight

Progressed, ended up releasing most of the air.

Two hot spots in the back were relieved by letting out air.

Once familiar with its operation, it is okay. Operator error

Familiarity caused some initial problems.

=> Had to shift it a bit just to get it comfortable. If it is not set
up correctly, you will have serious problems.

vy

UYU

28



‘Training

Figure 2. Human Factors Survey Framework.

Below each arrow is a dotted Method of Use box indicating that a component’s use may
affect certain human factors assessments. For example, the IOS may be judged as highly effective
and “user friendly” when one student is being trained in an AGSS station, but this assessment may
change dramatically if there are two or three students being trained simultaneously in the AGSS.
Alternatively, the helmet may rarely be judged as problematic when the device is used for short
durations, but rated much lower when it is used for longer time periods. The Method of Use box
is dotted to indicate that these factors may or may not affect participants’ human factors
assessments. When describing the results for each device component, we will first discuss the
significant human factors findings, and then where applicable, the effects due to different methods
of use.

The survey also covers select overall device assessments, as represented by the two small
solid boxes, Physiological Symptoms and Training Capability. We assessed the occurrence of any
unusual physiological symptoms (e.g., dizziness, blurred vision, etc.) during or following the use
of the AGSS. In addition, as a lead-in to a subsequent planned study of AGSS training, we also
examined the overall training capability of the device by soliciting SME commentary on the
capability of the AGSS to train selected AG/S skills.

The next box in Figure 2 is Acceptance, which includes both instructor and student
acceptance of the AGSS, and the training it provides. Acceptance is an important contributor to
the training effectiveness of this (or any) device, and in many ways reflects the sum of the
Previous components of the model. That is, human factors judgments of the various components,

10



Participant Ratings: Raw Data

Component

Rating, frequency

Gun Recoil
Gun Sounds
53 Gun Position
60 Gun Position
Gun Weight and Feel
Gun Controls
Airstream Effects
Tracers
Ballistic Model
Gun Alignment
Gun Pivot Point
Target Acquisition
Overall Rating

Ho} of ty] tae] of =| of o| o] of Oo] o] of
mr] otro} opr] fal alfa] aloud
wo] a] =] of uo} =] ro] nf vol |] no] Hl

Visual Resolution: Object Det.
TAT

Hind-D

Jeep

Truck

SAMs

Trees

Roads

Runways
Explosions

Field of View
Brightness

Contrast

Visual Resolution: Scene Det.
Object Detectability
Distance Perception
Scene Clarity

Scene Completeness
Scene Continuity
Scene Depth

Image Stability
Image Clarity
Alignment

Image Color.
Overall Rating

ol of of of of} of of o}ty] ty] of of o} of of o] of =} o] ao} | ao} o} a
|r} ro} aff rof ro] —[ ool rol al =] eo] of of |] tu] ty] a] uo} a} vol a] oj ug
a af a] foo} ro] of orf afro] —] orf a} anf anf —| to} oof af ee] to] to] =] on] oof
as] of ea] ea] orf ea] ro] en] safes] a] us] en] af cof feof ro] of eof eof oo] ool

lolol H]r} eH] rl ololH]of ape] eH] llr] -]of—] of fr} =] ry)



instrument, data collection procedures, and the rationale behind the statistical methods used. The
results section presents the overall findings of the survey, followed by separate analyses of the
tatings and qualitative data. Finally, we summarize the major findings, offer some
recommendations for AGSS design and use, and briefly introduce our research on AGSS training
effectiveness and user acceptance.

Evolution of Training and Design Requirements for the AGSS

Since 1990, the 58 SOW has been vigorously developing a sophisticated set of simulation
and electronic training technologies, as the USAF undergoes massive changes to become combat
mission ready for the next century. Pursuing a path of innovative acquisition and an aggressive
strategy for system development, along with an integrated approach toward academic instruction,
flightline training, and simulation, the 58 SOW has acquired a “can-do” reputation for fielding
sophisticated, reliable simulators in a short time period and incorporating these technologies into
ongoing Mission Qualification (MQ) and combat mission training programs with minimal
disruptions. As of 1996, the impressive capabilities available at the 58 SOW included eight high
fidelity networked flight simulators; electronic classrooms for advanced academic instruction; a
comprehensive database generation facility to construct photo-enhanced, geospecific visual
databases for any location in the world; an integrated training and mission support system,
employing computer-aided instruction (CAI); a cadre of database modelers and programmers;
various hardware and software assets to support construction of photo-enhanced, geospecific
databases; and functional links to multiple nodes on the Department of Defense’s (DOD).
Distributed Interactive Simulation (DIS) constructive battlefield simulation network.

As recently as 1990, the flight simulators were only beginning to come on line at KAFB.
The first of these was the MH-53J WST—and its associated database generation system (DBGS)
and mission planning system. The USAF wanted to assess the suitability of the MH-53J WST for
supporting short suspense MR and training activities. An initial operational effectiveness
evaluation was conducted in 1991 in the context of a joint Air Force-Army training exercise at
White Sands, New Mexico (Nullmeyer, Bruce, Conquest, & Reed, 1992). Simulating a strike
mission to recover critical equipment from a hostile third-world nation, the concept of operation
called for five helicopter crews (3 MH-53Js, 2 MH-60Gs) to insert a 90-person Army team (assault
and support) into the objective area during two consecutive nights. Throughout the assessment,
AL/HRA scientists provided behavioral research support by directly observing the mission
Preparation process, interviewing selected mission participants, and administering formal, open-
ended questionnaires following both MR and mission execution (Nullmeyer, Bruce, & Spiker,
1994).

The overall assessment of the MH-53J WST was Positive, as evidenced by the high marks
given the system by all the MH-53J and MH-60G pilots and FEs who were queried. Indeed, crew
confidence in their mission plan was increased as a function of rehearsal in the WST (Nullmeyer,
Spiker, & Reed, 1996).

Although the aircrews strongly endorsed the MR capability of the system, critiques by the
AG/Ss contained some key points. Since the WST only simulates the cockpit for the two pilots
and one FE, the AG/Ss did not rehearse in the device. The AG/Ss suggested that the entire crew

2



Table 10. Concluded.

Object Resolution ul 239

Scene Clarity ll 057

Scene Completeness il 239

Scene Continuity ul 148

Scene Depth. ll 215

SPECIFIC OBJECTS:

| _747 9 -330 20

Explosions 23 A88 01
Hind-D 10 170
Jeep tr 10
Roads ll
Runways 10
SAMs 9
Trees ul
Truck 9

Visual Resolution ll

Visual System: Overall i .239

Problem Areas: Comments and Recommendations for Improvement

This section builds on the previous section by discussing the participant comments
associated with each of the features identified in the Kolmogorov-Smirnov analysis as having
either negative or “acceptable” ratings. We list the number of participants reporting problems
with a feature “sometimes” or more when the participants were required to use this scale. In
terms of our 5-point Likert scale, we report the number of participants (typically out of 11) who
rated a given component/feature as a “3” or less. We then report some of the criticisms cited for
each component discussed, followed by recommendations for improvement.

The best match between rating scores and the frequency and content of comments was
achieved by grouping “3” ratings with the more negative “I” and “2” ratings (as described) and
using a liberal Kolmogorov-Smimov criterion of p < .20. This was, therefore, adopted as our
decision rule for including topics in this section. The tendency to rate an area as a “3” (or
acceptable) and still provide negative comments may reflect the RFT status of the AGSS which
connotes that “standard” (or “3”) criterion levels have already been officially met. Additionally,
there were several survey questions associated with the helmet and the IOS which were not
amenable to the Kolmogorov-Smirnov analysis; the responses and statistics related to these
questions are also reported below.

Helmet

Seventeen aspects of the helmet were explored for either malfunctions or problems with
comfort and fit. We also asked several questions about problems associated with the helmet
communication and electronic cables and the required helmet donning time.

25



Table 6. Helmet Rating Analysis.

Component N Max. D Prob.
Chinstrap (maif.) i -670 01
Chinstrap Chafing 1 488 01
Chinstrap Choking i 579 01
Comm. Cable 1 379 05
CRTs i 215
Electronics Cable ll 397 05
Eye Piece Combiner iH 397 05
Eye Piece Misalign. ll -670 01
General Comfort ul 397 05
Head-tracker il 215
Helmet Pads il 488 01
Hot Spots at 306
Neck Fatigue 1 579 01
Helmet Fit ul 306
Slippage il 397 05
Unbalanced H 579 Al
Visor 133 A88 01

The four bolded, italicized components—CRTs, head-tracker, hot spots, and helmet fit—
failed to receive reliably positive ratings from the Kolmogorov-Smirnov test. As a result, we will
focus on these areas when discussing participant critiques and recommendations for the helmet in
the next section.

Trainee Stations

Of the six trainee station components shown in Table 7, only one (gun feel) received a
reliably positive rating. The others failed to achieve significance in either direction, and, as such,
achieved “standard” ratings (with positive trends). As the comments will illustrate, the current
development of the trainee stations is acceptable for training, but participants make many
insightful remarks regarding potential improvements.

Table 7. Trainee Stations Rating Analysis.

Component N Max.D Prob.
Gun Feel ul 667 01
Gun Set Up ul 242
Mobility in Station ll 242
VRA/C Interior u 303
‘VR Other Stations i 242
Station: Overall ul -302

Note. Crew seats and crew harnesses were not included in this analysis due to the few number of participants who
had used these components in the AGSS. They are, however, discussed in our qualitative summary.

22



Trainee Stations.
Gun Set-Up and Feel...
Mobility, Seats, and Harnesses..

Screen Layout...
Space Ball.....
Second Instructor Position.
Individual IOS Pages..
External Environment.

Weapons Simulation.

Motion....

Physiological Symptoms..........--sserssereereeeees

Training Capability

SUMMARY OF RESULTS AND RECOMMENDATIONS,

FUTURE RESEARCH.

REFERENCES.

APPENDICES

Appendix A: Aerial Gunner Scanner Simulator Human Factors Survey ....

Appendix B: Participant Ratings: Raw Data

29
29
30
30
30
31
32
32
33
34
34
35
35
36
39
39
40
43
46
47
50

50



FIGURES
Figure
No. Page
1 Typical Human Factors Conception of the User-Machine
Relationship.........-ssssssssssesssessssssseeesnnnseeesennnnrenggresanenensseeeee eee 8
2 Human Factors Survey Framework. 10
3 Outside View of the AGSS......... UW
4 Main System Schematic of the AGSS... 12
5 Kaiser Electro-Optics Sim EyeTM AGSS Helmet. 12
6 AGSS Instructor Operator Station. 13
7 Left Aerial Gunner Station with 7.62 mm Weapon Mounted
dn the Window......ssssssssssessseessssseesscnnsscennnsseseennnnsscgnnnrcnnanssnsees sess re0eee" 14
8 Recommended Improved Layout of IOS Screens. 31
9 Reported Levels of Instructor Task Saturation with One, Two,
and Three Students in the AGSS... 33
10 Reported Incidence of Any Simulator Sickness Symptoms
During AGSS Training. 39
11. Cluster Analysis Output of 39 AG/S Skills. 42



the experience of unusual physiological symptoms, and its training capability will undoubtedly be
summed and weighted in subjective ways by both AGSS students and instructors to influence their
acceptance of the device. We also realize, as depicted in the box below Acceptance, that Other
Factors (e.g., preconceptions about simulators, age, rumors about device-associated reductions in
flying hours, etc.) will affect device acceptance. Acceptance lies outside of the large shaded
dotted box because it is not explicitly addressed in this survey. However, some participants made
direct references to acceptance in their comments, and we cite these reports where appropriate in
the results section. We have included the Acceptance and Training Effectiveness boxes in our
model as these are the next two research areas we will pursue after this survey. Training
Effectiveness is shown to the far right since we expect that it will reflect all of the preceding
factors. The bi-directional arrow connecting the Acceptance and Training Effectiveness boxes
acknowledges the likelihood that these two items are highly interdependent.

The above description by no means exhausts all of the factors that might potentially be
included in an analysis of the utility and design of the AGSS. It does, however, establish the
dominance of human factors issues in determining the training effectiveness of the AGSS while
also providing an effective context for understanding the survey results.

METHOD
AGSS System Description

The AGSS (see Figure 3) is a ground-based trainer designed to provide both basic skills
and MR training to USAF AG/S and FE personnel. The AGSS is a reconfigurable (MH-53J and
MH-60G) device with a three-degree-of-freedom (pitch, roll, and heave) motion base. The device
simulates sound, recoil vibration, trajectory, and visual imagery for both the .50 caliber (cal) and
7.62 millimeter (mm) guns. Gun design mimics actual gun control, position, weight, feel, and
operation. All ship, gun, and air sounds are heard by the trainees, including: engines, rotors,
guns, landing gear, warnings, and airstream effects. The visual system provides both day and
NVG night scenes including: outside view, inside cabin, gun barrel, fixed and moving targets,
weapon effects, effects of hits or misses, and environmental conditions. The AGSS is designed to
operate as a stand-alone trainer (AGSS only), an integrated trainer (AGSS with a host cockpit,
either the MH-53J WST or the MH-60G WST), or networked (AGSS with a host and multiple
training devices).

Figure 3. Outside View of the AGSS.

uU



HUMAN FACTORS EVALUATION
OF THE AERIAL GUNNER SCANNER SIMULATOR

INTRODUCTION

This report describes a human factors evaluation of the Aerial Gunner Scanner Simulator
(AGSS) that was installed at Kirtland Air Force Base (KAFB), New Mexico and is being operated
by the 58 Special Operations Wing (SOW). The evaluation was performed during the period
January-August 1996, with survey data collected in March and April of the same year. The effort
was conducted as part of a cooperative research effort between the 58 Training Support Squadron
(TRSS) and the United States Air Force (USAF) Armstrong Laboratory, Human Resources
Directorate, Aircrew Training Research Division (AL/HRA).

Declared ready for training (RFT) in March 1996, the AGSS was developed to provide a
low cost, high fidelity virtual environment for training MH-53J Pave Low and M/HH-60G Pave
Hawk flight engineers (FEs) and aerial gunner/scanners (AG/Ss). The AGSS is a state-of-the-art
reconfigurable simulator that can train up to three crewmembers at a time. A dynamic virtual
environment is presented through a head-tracked, full-color, helmet-mounted display (HMD)
system that can support both daytime and night vision goggle (NV! G)-night simulation. Using a
three degree-of-freedom motion system, aural cueing, a powerful seven-channel image generator
(IG), and digital computation system, the AGSS is capable of simulating air, sea, and terrain
scenes; ground and airborne targets; and weapons effects, including tracer path and bullet impact
(Reed, 1996).

Because the AGSS represents the first use of virtual reality (VR) for aircrew training at the
58 SOW, an assessment of the human factors aspects of the system was requested by the 58 TRSS.
As one of the USAF Air Education and Training Command (AETC) training centers for
excellence, the 58 SOW deemed it necessary to conduct a thorough evaluation of the AGSS from a
human factors standpoint before incorporating the system into its impressive cadre of advanced
weapon system trainers (WSTs).

The evaluation was designed to achieve three major objectives. First, the information
collected during the evaluation was rapidly disseminated to key personnel within the 58 TRSS to
ensure that high payoff, incremental design improvements to the AGSS could be made as soon as
possible. Second, the effort sought to identify the skills most affected by use of the AGSS,
thereby laying the foundations for a subsequent assessment of the effectiveness of the AGSS as a
training device. Third, the research was designed to collect “lessons learned” information and
other insights from subject-matter experts (SMEs) that could be used in the design, development,
and implementation of other VR-based systems.

The remainder of the report is organized into four sections. We begin by providing some
background on the AGSS. This includes a description of the training and design requirements for
the system, a discussion of how VR technology is being applied to training and mission rehearsal
(MR), a consideration of applying human factors analyses to VR technology, and a delineation of
the scope of the human factors survey that we performed. In the next section we describe the
methods used to conduct the survey, including a description of the AGSS, the subjects, the survey

1



The weapons simulation section duplicated the format of the trainee station section, i.e.,
specific components were rated from 1 (poor) to 5 (exceptional), using the scale illustrated in
Table 2. Each component was rated separately by gun type (i.e., .50 cal and 7.62 mm) to account
for any judged differences between the two.

The instructor/student interaction section was very brief. It included six yes/no
questions about instructor ability to communicate with students from the IOS, and student ability
to communicate with instructors and each other while in the AGSS.

The visual system section had two main subsections. The first was a repeat of the
physiological symptoms portion of the background information section with the AGSS as the
context, e.g., Did any of the following symptoms (e.g., dizziness) occur during or after training in
the AGSS? This subsection was followed by a number of visual system elements and qualities
(e.g., visual contrast) with space provided for judged ratings and comments on both day and NVG
night training quality. Due to limited variability between the day ratings and the NVG night
ratings, we primarily used the day ratings. However, some exceptions (e.g., responses related to
day and night FOV) are noted in the text.

The training capability section included 39 required AG and FE skills that were selected
based on analysis and discussions with AG and AGSS SMEs. Space was provided for
participants’ 1 (poor) to 5 (exceptional) assessments (see Table 4) and comments on the training
capability of the AGSS for the skills under both day and NVG night conditions.

Table 4. Survey Scale for Rating Judged Training Capability for Specific AG/S Skills.

Rating Value Definition
Cannot train this skill in AGSS.
1 Poor
2 Min, Required Only minimal training on this task provided in the AGSS.
3 Standard Acceptable training provided on this task in AGSS.
4 Outstanding Training provided on this task is above expectations.
. Training on this task is beyond expectations due to capabilities of
5 Exceptional the AGSS not available in the aircraft.

The final section was a miscellaneous section. It included questions on the overall feel of
the AGSS, the motion system, and whether subjects experienced any disorientation in the AGSS.

Data Collection Procedures

The survey was conducted using one-on-one guided interviews. All of the interviews
were conducted by the primary author. The process began with both written and oral instructions
to the subjects, including information about the confidentiality and purpose of the survey. Each
subject read and signed a statement of informed consent. The participants were then asked to
make their ratings and comments based on their subject-matter expertise and their experiences

7


AL/HR-TR-1996-0146

UNITED STATES AIR FORCE
____ ARMSTRONG LABORATORY
ee Ah sneeeeienemenestmennsqeeneneecnneensess

HUMAN FACTORS EVALUATION: ‘OF THE
AERIAL ‘GUNNER SCANNER SIMULATOR

Denise R. Silverman

‘Operations.
soot south Power Road , Bldg 561
Mesa, AZ 85206-0904

Alan Spiker

ANACAPA SCIENCES, INC.
304 East Carillo

Santa Barbara, CA 93101

June 1997

AIR FORCE MATERIEL COMMAND

Brooks Air Force Base, TX 78235-5352



both reported “sometimes” or more by five participants. Interestingly, eyestrain has been, shown to
be the most commonly reported simulator sickness symptom in past research as well (e.g., Gower
& Fowlkes, 1989). Comments from survey participants primarily indicated that the AGSS visual
system was the main cause for the Teported eye strain and discomfort. Some of the comments
included: “seemed to be mostly due to poor helmet resolution and a mismatch between the two
separate tubes [CRTs];” “seemed to be due to monocle adjustment and small area of visual
acuity;” and “just a general hard time focusing on things in the box, trying to accommodate for
lack of visual resolution.” More research is tecommended to determine which conditions (e.g.,
with or without motion) or tasks (e.g., scanning) are the most likely candidates to invoke simulator
sickness symptoms in AGSS trainees, and what can be done to-alleviate or minimize their
occurrence.

Training Capability

Finally, we asked participants to rate from 1 (Poor) to 5 (exceptional) the training
capability of the AGSS on 39 required AG/S skills. For this assessment, we covered a broad array
of skills, including ones the AGSS was not explicitly built to help train. Specifically, we wanted
to assess: (a) whether the device was considered capable of training the skills it was built to help
train (¢.g., target acquisition); (b) whether there were skills that the AGSS was not explicitly
designed to train (e.g., gun malfunctions) that were considered “trainable” in the AGSS; and (c) to
what extent these skills were trainable in the AGSS. In order to extract this information, it was
important to maintain the integrity of the entire scale in our analysis, rather than collapsing the
scale into three levels as was done throughout the Kolmogorov-Smimov analyses. In addition, we
probed participants for training capability ratings for both day and NVG night conditions. Their
ratings did not differ between day and NVG and are, therefore, treated as overall ratings of training
capability for each skill.

Table 13 lists the AG/S skills assessed in rank order of their mean rating of AGSS training
capability. Their associated standard deviations are presented in the third column. This rank
ordering simply illustrates which skills participants feel the AGSS is best suited to train. Although
rank ordering alone is informative, we thought it would be worthwhile to group these skills into
categories using cluster analysis, as previously described.

Table 13. Mean Ratings (Descending Order) and Standard Deviations of AGSS Training
Capability for 39 AG/S Skills.

Gunner/Scanner Skill Rate Standard
Voice Procedures 4.36 67
Left, Right, Tail Call Responsibilities 4.18 60
Left, Right, Tail Interactions 4.09 54
Taxi and Hover Calls 4.00 63
Crew Coordination/Overall 4.00 63

40



methodological requirements of a comprehensive human factors evaluation, in the next
subsection, we discuss some of the basic concepts of VR technology as applied to combat mission
training and rehearsal.

VR Application Issues

From both a technological and societal perspective, VR has seen an unprecedented
explosion in popularity, interest, and debate over the last few years. While multiple definitions
abound, a common conception of VR is a type of interactive simulation that includes the user as a
necessary component. Recently, however, it is has been noted (e.g., Thurman & Mattoon, 1991)
that such conceptions are overly restrictive, since VR also encompasses the experience the user
has in the artificial environment. Within DOD, the prevailing conception of VR is as an
“Gnteractive, usually computer-mediated, experience characterized by a suspension of (or
inattention to) disbelief in the unreality of the experience” (DOD, 1994, p. 18).

In attempting to analyze the potential uses of VR in simulating a combat training
environment, it should be noted that DOD Directive 5000.59 defines simulation as a “method for
implementing a model over time (p. 4).” In turn, a model is viewed as a “physical, mathematical,
or otherwise logical representation of a system, entity, phenomenon, or process (p. 4).”

address how accurately the simulation Tepresents the objects of interest, how realistically the
modeled objects behave over time, and how efficiently the modeled environment is delivered to
the user.

Given the user’s large role in the VR experience, it stands to reason that an assessment of
the human factors aspects of the implementation is vital, as it will clearly influence how effective
the simulation will be. In this regard, it is important to keep in mind that VR technology is not a
single technology or medium, such as an HMD or a DataGloveTM. Rather, it is an integrated
human-computer system (Thurman & Mattoon, 1991). Consequently, when VR is incorporated
into a training device, all aspects of the human-computer system—including input, interface,
processing, and output—must be evaluated, both separately and in combination.

Among the various concepts used to explain VR, three are essential: virtual environment,
immersion, and presence. A virtual environment is a simulated space in which a viewer or user
may interact. Typically, a physical simulation of a vehicle, such as an aircraft or a tank, is the
interface between the individual in the simulation and the virtual environment. For an individual
to interact directly in a virtual environment, some or alll of the following conditions must be met
(Knerr, et al., 1993):

+ free motion of the eyepoint in space;

+ three-dimensional real-time interactive graphics, with stereopsis as needed;
+ multiple senses beyond visual (e.g., audition, touch);

+ direct manipulation of objects; and

+ multiple interacting, mutually visible humans.



be included in the MR. They also indicated that the scanners need a visual capability of their own,
and cited the need to communicate with the cockpit crew during the rehearsal to coordinate tactical
activities and maintain situation awareness.

Based on this feedback, it was recommended that a simulation capability be developed that
allows the AG/Ss to view the visual terrain database just as the pilots and FEs do, only from their
vantage point at the rear and sides of the aircraft (Nullmeyer, et al., 1992). Not only would such a
capability permit the training of crew-level tasks, as the entire crew would be present in the WST,
it would aid development of key gunnery and scanning skills while reducing demands on the
flightline to schedule sorties solely to accomplish AG/S-unique mission training events.

In parallel with the desire for full crew training, the 58 SOW explored developing a
gunnery simulator to serve as an electronic part-task trainer (Reed, 1996). While developing the
requirements for an AG/S training simulator, the 58 TRSS explored promising new technologies
for incorporation into their formal specification. These included low-cost, smal] motion platforms;
ruggedized, full-color HMDs; more accurate head-tracking systems; and smaller high fidelity IGs.

In May 1993, a contract was awarded to design the AGSS. At the outset, the AGSS was
expected to provide full crew training and MR. Not only would the AGSS provide the first true
MR capability for enlisted crewmembers, it would do so cost effectively by allowing squadron
commanders to trade high value training flight hours for simulator time (Reed, 1996).

During the design phase, four key factors would alter the physical configuration of the
device as well as shape the corresponding human factors evaluation. First, by using a modular
architecture, the AGSS was to be reconfigurable to support either MH-53J or MH-60G training.
This flexibility would not only increase the number and type of crewmembers to be trained, it also
expanded the range of training events to be covered.

Second, the small size of the simulator device necessitated that the AGSS instructor/
operator station (IOS) not be mounted on the motion platform, but instead, be placed on the
facility floor outside the MH-53J WST. This decision affects—and potentially limits—the ways
that instructors may interact with their trainees. The resulting effects on both instructor and trainee
must be captured in the human factors evaluation.

Third, to accommodate the heavy training schedule of the simulators at the 58 SOW, the
AGSS needed to be designed for operation in both an independent (stand-alone) mode and
integrated mode (networked with a cockpit simulator), enabling whole crew operations. The
different requirements for operation, and the added complexities of using the AGSS while
networked with other WSTs, must be reflected in a comprehensive human factors assessment.

Fourth, revolutionary advances in IG capability during the design phase created the
exciting possibility of incorporating full VR into routine training. By scanning digital
representations of the aircraft interior and displaying those on the AG/S’s HMD, coupled with
position-referenced aural cues and vibratory feedback from the guns, the AGSS has the potential
to offer a truly immersive training environment. These novel aspects must also be explored in a
human factors analysis of the AGSS. However, before further discussion of the general



APPENDIX B

Participant Ratings: Raw Data

Component Rating, frequen:
ie : eo oe |) ea : oo

. ‘Visor 0 0 2 0 9 11
Eye Piece Combiner 0 1 2 2 6 ll
Chinstrap ll 0 0 Y () ia
Head-tracker ) 0 5 3 3 il
Communication Cable 0 1 2 1 7 i
Electronics Cable 0 0 3 1 7 u
CRTs 0 2- 3 1 5 u
Helmet Fit 1 Lt} 3 4 3 u
Eye Piece Misalignment 0 o 0 3 8 ul
Slippage. 0 0 3 4 4 il
Neck Fatigue 0 0 1 0 10 i
Hot Spots 1 1 2 3 4 ll
Chin Strap Chafing 0 2 0 1 8 u
Chin Strap Choking 0 0 1 0 10 i
Unbalanced Weight Dist. 0 1 i} 1 9 ll
General Comfort 0 1 2 5 3 ll
1 1 0 5 4 ll
0 1 6 2 2 ll
0 0 0 6 5 iW
0 3 1 4 3 ll
0 1 5 3 2 ll
0 1 5 3 2 ll
0 1 2 1 0 4

1 0 3 ) 1

0 1 3 4 3

eee 2 Se

0 1 2 5 0 8
Instructor Screen Layout 0 0 2 5 1 8
Student Screen Content 0 1 2 4 I 8
Student Screen Layout 0 1 3 4 0 8
Station Layout 0 0 3 5 L} 8
Space Ball - Models 2 0 1 5 ) 8
Space Ball - Ownship 5 3 1 0 0 9
Second Instructor Position 1 3 2 1 i) 7
Ground Track Map Page 0 0 5 3 0 8
Gunner Performance Page oO C) 2 5 1 8.
. Crew Performance Page 0 0 1 7 0 8
Threat Status Page 0 0 4 2 1 7
Flare Monitoring Page 0 0 5 2 0 7
’ Overall Rating 0 it} 2 5 1 8



Visual System

Comments. As indicated by the Kolmogorov-Smirnov analysis, various features of the
visual system were rated “acceptable” or lower. We discuss several of these below, as well as any
previously unmentioned differences between NVG night simulation and day simulation that

participants’ noted.

Eight participants rated object detail and visual resolution as a “3” or less. Comments
highlighted the fact that the level of detail did not match the real world; this was particularly
obvious for targets. As a behavioral consequence of this lack of detail, these eight participants
also rated target or object detectability as a “3” or less. Comments focused on lack of detail and
the inability to make friend versus foe distinctions unless objects were at an unrealistically close
range. Particular object representations were also rated, including a 747, explosions, a Hind-D
helicopter, jeeps, roads, runways, surface-to-air missile (SAM) sites, trees, and trucks. Not
surprisingly, given the overall ratings of object resolution, seven of these objects were rated low
by a large proportion of survey participants. The primary reason for the low ratings in each case
was insufficient detail. Table 12 presents a summary of the data for each object. The shaded rows
(runways and explosions) indicate the objects with the fewest number of low ratings.

Table 12. Specific Object Ratings and Representative Comments.

. ber of
Object Rat ots Or Less Representative Participant Comment
747 5 of9 None noted.

Needs more detail, a higher texture .
Lacked clarity, couldn’t tell if it was airborne.

vul

Helicopter Sof ll
(Hind-D)

¥

As targets okay, but if you had to tell between good guys
Jeep 6 of 10 and bad guys, you couldn’t do it.

Was told it was there, but didn’t recognize it.
Couldn’t distinguish between roads and rivers.
Just brown lines.

YUly

Roads 8of 11

SS MERE EET
needs to be upgraded.

SAM Sites 6 of 9 => Could tell up close (400 ft), but farther away, problems.
=> Need work, a lot of variability.
Trees 9of 11 => Other trees in other visual systems are better. Trees are flat.
=> Not enough detail.
Truck 6 of 10 => Could have been a white blur.



Table 9. Weapons Simulation Rating Analysis.

Component N Prob.

53 Gun Position 10 1S
Air Stream Effects u

Ballistics 1

Gun Alignment u

Gun Controls ML 01
Gun Pivot Point H

Gun Recoil 10 20
Gun Sounds i 15
Gun Weight/Feel 1

Target Acquisition iL

Tracers 23 :
Guns: Overall U 15

Note. Gun Position 60 was not included in this analysis due to the few number of participants qualified to rate this
feature.

Visual System

Twenty-four specific elements of the visual system were rated for their training utility
under both day and NVG night conditions. The differences in participants’ ratings of the visual
system did not vary between day and NVG night use, and therefore, are reported as general
assessments of each feature.

The visual system was clearly the AGSS system that was associated with the greatest
amount of participant dissatisfaction. Three points from Table 10 endorse this conclusion. One,
ratings for object detection, a key requirement for numerous AG/S tasks (e.g., friend vs foe
distinctions), were reliably negative. Two, six other visual system features, although,

“acceptable,” show negative deviations. (These are indicated in the table by having gray-shaded
Max. D cells.) Three, only 5 of 24 features achieved significantly positive ratings. Of these, four
were at the most liberal probability level used in our survey analysis. The visual system was also a
source of many participant comments and critiques which are detailed in the next section.

Table 10. Visual System Rating Analysis.

Component N Max. D Prob.

Alignment u 148
| Brightness WW 330 20

Contrast ll 057
Distance Perception. a8 125
FOV u -330 20
Image Clarity i 215
Image Color H 148

Si 506

24.



Table 13. Concluded.

Threat Breaks 3.91 83

Scanning 3.91 94
Tactical Knowledge 3.82 387
Radar 3.82 98
Go-Arounds 3.82 87
AAA 3.82 98
Rules of Engagement 3.82 87
Defensive Countermeasures 3.80 63

Approaches and Landings 3.73 1.01
Gunner Cockpit Exchanges 3.73 1.10
Ammo Conservation 3.73 1.19
Tactical Approaches 3.64 1.03
Terrain Masking 3.55 69

Safety and Judgment 3.45 95

Terrain Obstacle Avoidance 3.45 82

Search Operations 3.36 1.21
Communication System 3.36 1.03
Aircrew Briefing 3.36 1.36
Remote Operations 3.27 1.19
Tactical Mission Navigation 3.18 1.38
Radio Discipline 3.09 1.38
Use Limitations 3.09 1.45
Waypoint Identification 3.09 1.04
Target Acquisition 2.91 1.04
Knowledge of Directives 2.64 81

Passenger Brief 2.18 1.54
Resource Management 1.91 -70
Checklist Procedures 1.55 69
NVG Use and Limitations 1.20 63

Light Signals 1.18 40
Emergency Procedures 1.18 40
Gun Malfunctions 1.09 30
NVG Failure 1.00 00
Aircraft Taping and Lighting 1.00 .00

When using cluster analysis there is often no single solution. Rather it is driven by

experimenter interpretation as to where the optimal solution is in terms of the objects clustered.

Figure 11, although still having 17 distinct clusters, was selected (forming at the 17th step) as the

best grouping of the skills rated. We chose this step primarily because of the four large clusters

that formed (although some of the two-skill clusters are also informative). These four large

clusters separate the skills into distinct AG/S skill areas. The first cluster is crew coordination

skills. The second is terminal area operation skills. The third is tactical skills. And, the fourth is

“skills the AGSS was not designed to train” (requiring actual guns or NVGs to train). This is not

. meant to imply that the AGSS was designed and built to train the remaining 32 skills, however this
category reflects the skills that the AGSS was never meant to train.

41



Ios

As shown in Table 8, the IOS runs the gambit of possible ratings. Several features were
rated significantly positive (instructor screen, layout; crew performance page; gunner performance
page; and IOS overall), one was rated significantly negative (the space ball for ownship flying),
and several features were rated “acceptable.” In addition, it should be noted that while all the
other components that were judged “acceptable” to this point have had positive maximum
deviations, the second instructor position of the IOS is the first acceptably judged component to
have a negative deviation. This may indicate that the second instructor position needs more
improvement than the other acceptably rated components. Later, we shall see that comments
from participant substantiate this interpretation.

Table 8. IOS Rating Analysis.

Component N Max.D Prob.
Instructor Screen, Content 8 295
Instructor Screen, Layout 8 -420 -10
Second Instructor Position 7 23:
5) Ball Models 8 295
Station Layout 8 333
Student Screens, Content 8 295
Student Screens, Layout 8 303
PAGES:
Crew Performance 8 545 10
Flare Monitoring 7 333
Ground Track Map 8 333
Gunner Performance 8 420 01
Threat Status 7 333
JOS: Overall 8 420 10
Weapons Simulation

Weapons simulation was assessed for both the .50 cal and the 7.62 mm guns.
Participants’ ratings did not vary across the gun types, so each feature is simply treated as an
assessment of the general weapons simulation of the AGSS. Twelve individual aspects of AGSS
weapons simulation as well as gun simulation overall were rated in terms of their fidelity and
training effectiveness.

As shown in Table 9, analyses of the weapons simulation revealed a wide range of feature
rating distributions: five subsystems were rated significantly positive (including guns overall),
three were rated acceptable with positive trends, and four were rated acceptable with negative
trends (indicated in the table by having gray-shaded Max. D cells). As discussed in the next
section, the latter four gun features (gun alignment, gun pivot point, target acquisition, and
tracers) were the primary focus of participant criticism in terms of weapons simulation.

23



PREFACE

This report documents the results of a human factors evaluation of the Aerial Gunner
Scanner Simulator which was conducted under Work Unit 1123-B3-01, Special
Operations Forces (SOF) Aircrew Training and Mission Preparation Research. The
Laboratory Work Unit Monitor is Dr. Robert T. Nullmeyer, Armstrong Laboratory,
Human Resources Directorate, Aircrew Training Research Division (AL/HRA). The
work was supported by Work Unit 1123-B2-06, Aircrew Training Research Support,
under Contract F41624-95-C-5011. The Laboratory Contractor Monitor is Mr. Daniel
Mudd; the Laboratory Technical Monitor is Dr. Robert T. Nullmeyer.

The authors would like to acknowledge several individuals who contributed
generously of their time and support to this report. We would first like to thank LTC Ed
Reed, Commander of the 58th Training Support Squadron (TRSS), who provided access
to the people and the state-of-the-art technology of the Aerial Gunner Scanner Simulator
(AGSS). We would also like to acknowledge Mr. John H. Fuller, Jr., the Hughes
Training, Inc. (HTI) program manager, and Mr. Steven J. Tourville, HTI’s Kirtland AFB
operations manager, for their encouragement throughout this research effort. In addition,
we specifically thank SSGT Steve Yeager and TSGT Tim Hadrych. SSGT Yeager’s
support throughout this research effort was vital. Without his help and critical insights
this project would not have succeeded. TSGT Tim Hadrych played a fundamental role in
providing the researchers with critical background information on AGSS design and
development. Finally, we offer a most sincere thanks to the crew members and
instructors of the 58 TRSS and the 551 Special Operations Squadron (SOS) for their
generous contribution of time, expertise, and insights to this project. These include in
alphabetical order:

Mr. Arthur Burkett, Lockheed Martin
TSGT Craig Coulomb, 58 TRSS
TSGT Marty Davis, 551 SOS

Mr. Clark Dawson, Lockheed Martin
Mr. Tom Green, Lockheed Martin
MSGT Scott Hedglin, 551 SOS

Mr. Steve Johnson, Lockheed Martin
TSGT Harry King, 58 TRSS

TSGT Robert Reilly, 551 SOS

vii



Weapons Simulation

The AGSS is capable of simulating sound, recoil, vibration, trajectory, and visual imagery
for both the .50 cal and the 7.62 mm guns (see Figure 7). The AGSS duplicates all gun controls,
positions, weight, feel, and operations of each gun. High fidelity airstream effects vary with
airspeed and gun orientation. These airstream effects are provided through individual control
loading mechanisms which support all three gun positions.

Figure 7. Left Aerial Gunner Station with 7.62 mm Weapon Mounted in the Window.

Visual System

The primary components of the visual system are: the HMD, the head-tracker system, and
the IG. The HMDs are full-color and support a 60° by 40° FOV with 1028 by 1024 resolution.
The head-tracker system has an eight-foot radius and provides head position and attitude
information to the IG and host computer. There is one transmitter with a receiver on each of the
three helmets. The SE 2000 IG is used when the AGSS is in stand-alone mode. When the AGSS
is in network mode, other IGs available at the 58 TRSS, like Compuscene V, may be used. The IG
provides photo-specific scenes to the HMDs. The visual simulation includes interior aircraft
features (e.g., cabin, weapon barrels, etc.) and exterior aircraft features (e.g., rotors). Further, the
out-of-the-window depiction includes: air, sea, terrain scenery, targets, and weapons effects, i.e.,
tracer path and bullet impact (Reed, 1996).

Subjects

Eleven experienced rotary-wing instructors volunteered to participate in this survey; five
FEs and six AGs. Ten were crewmembers from the MH-53J weapon system and one was from
the MH-60G. Seven were active duty USAF and four were Lockheed Martin instructors. All

14



INSTRUCT ‘OR OPERATING STATION ~ 10S (Continued)

“4 =Poor: 2-Min. Reg. 3-Standard __-4.- Outstanding _ 5.- Exceptional ve
FEATURES Rating? INSTRUCTOR STATION PAGES Rating |
. 4._ Instructor screens? 1. Ground track map pages?
2. Information content? 2. Gunner performance monitoring?
3._Instructor screens layout? 3. Threat status page?
. 4. Student screens information content? 4. Flare monitoring page?
5. Student screens layout? 5. Other:
6._ Station layout (position of screens)?
7. Space bail?
8. Other: 9. Overall IOS assessment? L. fe
INSTRUCTOR & STUDENT INTERACTION and COMMUNICATION SYSTEM
4. Did you have any problems hearing/understanding the instructor while in the AGSS? | Y. N
2. Did you have any problems hearing/understanding students while at the AGSS |OS? | ¥. N.
3._Did you have any problems hearing or understanding other crewmembers? ¥ 2 N
‘4, Was the instructor able to provide timely feedback while you were in the AGSS? Y. _N
5._Have you had any problems with communications in the heimet/device? ¥. CN
6. Was the sound quality and background noise comparable to the aircraft? ¥ N
- If YES, from a training perspective, is this good? ¥ N-
- If NO, from a training perspective, is this good? ¥. N
VISUAL SYSTEM
[ N-Never R-Rarely____S- Sometimes _O - Often A - Always :
While Flying in After Flying in
AGSS? AGSS?
1. Visual Discomfort? N ROS O AIN RS OA
2. Headache? NOR S O-AIN RS OLA
3._ Double Vision? N RS O AIN RS 0. Al
4. Blurred Vision? [NOR SO AIN R SUOA
5. Disorientation? NOR SO AIN RS OA
6._Eye Strain/Fatigue? NOR. S O AIN RS OA
7.__Neck Strain? N RS O- AIN RS OA
8. Stomach Discomfort? NR S O AJNR SOA
9. Nausea? NR S O AIN ROS OA
VISUAL SYSTEM - DAY and NIGHT OPERATIONS
-4-Poor. 2- Min. Req. 3~- Standard 4- Outstanding —-_5 - Exceptional
FEATURE Rating Representation of Specific Objects Rating
1. Visual resolution (object detail)? 1. 747 aircraft?
2. Visual scene clarity? 2.__ Airborne helicopter?
3.__ Visual scene completeness? 3. Jeeps?
4.__Visual scene continuity? 4. Trucks?
5.__Meaningful depth? 5._SAM sites?
6.__Image stability? 6. Trees?
7,__Image clarity? 7.__Roads?
8. Image alignment? 8. Runways?
9. Image color? 9. Explosions?
. 40. Visual system brightness? 410. Other?
11. Visual system contrast?
12. Object detectability?
. 13._Distance Perception?
14. Other?
15. Overall Visual System assessment?



that view is endorsed here. The proposed layout (see Figure 8) would enable instructors to better
divide their attention between IOS inputs and instruction, while also eliminating the instructor’s
disruptive requirement to get out of the seat to make inputs during training.

(amis | [masscoe | Yeon
KLE | KN

Instructor Output Instructor Input V cars
“ we

Figure 8. Recommended Improved Layout of IOS Screens.

Space Ball

Comments. The most unanimously agreed-upon poor feature of the IOS was the space
ball for flying ownship during stand-alone AGSS training. Nine out of nine participants rated the
space ball as a “3” or less, and it was singled out by the Kolmogorov-Smirnov analysis as having
significantly negative ratings. This result is in contrast to only three of nine who rated the space
ball as a “3” or less for flying moving models and the “acceptable” result of the Kolmogorov-
Smirnov analysis for space ball flying of moving models. This is a prime example of how the
method of use of a particular AGSS feature has a great influence on its judged effectiveness. The
central reason given for the overwhelming displeasure with the space ball for flying the ownship
was the poor software modeling of aircraft aerodynamics which manifests itself as a tremendous
difficulty for maintaining proper flight control using the space ball. The following excerpt is
representative of the space ball critiques we recorded, and is illustrative of the problems and
consequences of using the space ball for ownship flying during training:

The big thing with space ball ownship flying is the MH-53J WST code that is used
for the software modeling. This causes major problems if you are forced to fly with
the space ball during stand-alone training with students. It is impossible to interact
with students as you should. You [also] end up using the slew keys more often than
you should which gives students artificial cues. . . Do anything to stay away from
space ball flights. Get rid of space ball or make it fly more like a generic model.

Recommendations. With such unanimous displeasure associated with flying the ownship
using the space ball, we strongly recommend that it either be changed or its use limited during

31


training. Several participants suggested simply replacing the space ball with a joystick, enabling
quicker input adjustments and accommodations due to poor modeling of aircraft aerodynamics.
However, at the AGSS’s current level of development, this would require extensive and costly
changes to the software (e.g., changing drivers). Another less optimal suggestion is to conduct
AGSS training with minimal use of the space ball, either in the integrated mode—when the MH-
60G or the MH-53J is the host cockpit—or using the record-replay function to run through
prerecorded missions. This latter approach is, in fact, often employed during current AGSS
training. Among its advantages, this method greatly increases the instructor’s ability to: (a) focus
on the students in the AGSS, (b) provide timely feedback, (©) use the performance monitoring
pages, and (d) introduce moving models into the scenario. However, the interactive nature and |
realism of the flights are reduced because directional calls (e.g., break left) cannot be responded to
unless it has been prerecorded. Use of the integrated mode, on the other hand, will solve many of
the problems associated with the space ball for ownship flying. Yet this mode is still viewed as
less than optimal since it is not presently integrated and (three) stand-alone AGSS rides are still a
required part of the AG Mission Qualification training.

Second Instructor Position

Comments. There was extensive agreement among participants on lack of utility of the
second instructor position at the IOS. Six out of seven participants rated this IOS feature as “3” or
less. Comments focused on inability of instructors at this position to communicate directly with
students in the AGSS. Echoing the thoughts of the other five displeased participants, one
individual succinctly said, “You have the capability to hear, but you can’t talk. This is dumb.
Provide this capability.”

Recommendations. In order to enable instruction and accommodate some of our other
recommendations, the second instructor position must be provided the capability to communicate
directly with the students in the AGSS.

Individual IOS Pages

Comments. Other IOS features rated as “3” or less by a number of survey participants
were three IOS pages: flare monitoring, threat status, and the ground track map. Five of seven
participants rated the flare performance monitoring page as “3” or less and four out of seven
participants similarly rated the threat status page. For both pages, however, the comments
primarily addressed the fact that they “just weren’t used much,” suggesting a degree of skepticism
regarding content and utility of these pages. Five of eight participants, however, rated the ground

track map page as “3” or less. The primary criticism of the ground track map page was its lack of
waypoint representation.

Recommendations. A solution to the absence of waypoints on the ground track map
page is simply to provide them as in some of the other simulators at the 58 TRSS. However, the
AGSS lacks a navigation system, and without this capability, it is not possible to input waypoints.
The costs to implement a one-of-a-kind navigation imitator on the AGSS would undoubtedly
override any debatable benefits on a device such as the AGSS. Nevertheless, there is at least one
potential solution given the current AGSS design specifications. One survey participant offered
the clever suggestion of setting up inactive (or active) threats to mimic waypoints on the ground

32



REFERENCES

Bailey, J.G. & Witmer, B.G. (1994). Learning and transfer of spatial knowledge in a virtual
environment. Proceedings of the Human Factors and Ergonomics Society 38th
AnnualMeeting. Nashville, TN.

Department of Defense. (1994, October). Virtual reality and technologies for combat simulation.
Office of Technology Assessment.

Gower, D. W., Jr. & Fowlkes, J. (1989). Simulator sickness in the AH-1S (Cobra) flight
simulator. (USAARL Report No. 89-20). Fort Rucker, AL: Army Aeromedical Research
Laboratory. :

Grether, W.F. & Baker, C.A. (1972). Visual presentation of information. In H.P. Van Cott and
RG. Kinkade (Eds.). Human engineering guide to equipment design, 2nd Edition (pp. 123-
160). American Institutes for Research: Washington, D.C.

Hays, W. L. (1988). Statistics. New York: Holt, Rinehart and Winston, Inc.

Hettinger, L.J., Nelson, W.T., & Haas, M.W. (1994). Applying virtual environment technology to
the design of fighter aircraft cockpits: Pilot performance and situation awareness in a
simulated air combat task. Proceedings of the Human Factors and Ergonomics Society 38th
Annual Meeting. Nashville, TN.

Kantowitz, B.H. & Sorkin, R.D. (1983). Human factors: Understanding people-system
relationships. New York: John Wiley & Sons.

Kennedy, R. S., Lilienthal, M. G., Berbaum, K. S., Baltzley, D. R., & McCauley, M. E. (1989).
Aviation, Space, and Environmental Medicine, January, 10-16.

Knerr, B.W., Goldberg, S.L., Lampton, D.R., Witmer, B.G., Bliss, J.P., Moshell, J.M., & Blau,
B.S. (1993). Research in the use of virtual environment technology to train dismounted
soldiers. Proceedings of the 15th Interservice/Industry Training Systems and Education
Conference. Orlando, FL.

Kuhl, J.G. & Wargo, J. (1994). High fidelity virtual prototyping to support ground vehicle
acquisition. Proceedings of the 16th Interservice/Industry Training Systems and Education
Conference. Orlando, FL.

Lane, N.E., Kennedy, R.S., & Jones, M.B. (1994). Determination of design criteria for flight
simulators and other virtual reality systems. Proceedings of the Image VII Conference:
Tucson, AZ.

47



low ratings for certain essential parts of the helmet, as well as participant criticisms, required us to
include the “when working” caveat in our assessment. As discussed more completely below,
critical components of the helmet (e.g., the head-tracker and CRTs) are subject to frequent
breakage and/or malfunctions. Moreover, due to the newness of the device, the fitting and
aligning processes are fallible which can lead to additional helmet problems (¢.g., discomfort).

The mean rating for the helmet was determined in a different manner than the other
systems. For the trainee stations, IOS, weapons simulation, visual system, and training capability,
we asked participants to rate each system overall and averaged their responses to this question.
In order to provide the overall mean rating that appears in Table 5, we summed participant
responses to each of the helmet questions and divided by the number of questions. We believe
this measure provides an appropriate overall SME assessment of the helmet.

The trainee stations received a fairly high overall average rating. Indeed, participants’
comments were consistent with this high rating. It is perhaps a subtle distinction, but the majority
of the comments focused on issues that could make the trainee stations better, rather than flaws
about the design and functionality of the stations themselves, leading us to the unqualified
assessment of “good.”

The IOS had a slightly higher overall mean rating than the trainee stations. However, we
assessed it as “good” with a “room for improvement” qualifier because particular features
(discussed later) were given unacceptable ratings by participants. In addition, although some of
the comments were simple suggestions for improving the training capability of the IOS (as we
saw with the trainee station comments), many focused on changes that would enable training.

The overall “needs work” assessment of weapons simulation came primarily from the
frequently cited modeling deficiencies of the bullet path (described later). The fairly high overall
average rating of weapons simulation, despite this rather major problem, stems, we think, from
the fact that many of the gunnery-related training capabilities of the AGSS simply did not exist
prior to its introduction.

Survey participant comments and the somewhat lower ratings led to an overall assessment
of the visual system as “faulty.” Participants noted the tremendous importance of visual input for
many AG/S tasks, and given the current quality of the AGSS visual system, the level of training
that could be provided by the AGSS for many AG/S tasks was severely limited.

Finally, overall training capability of the AGSS was assessed as “good” with qualifications.
That is, participant comments and ratings reflected overwhelming agreement that the training
potential of the AGSS was tremendous, especially in the areas of crew coordination, target ident-
ification, and target acquisition. However, at least in terms of the latter two skills, the potential
could not be fully realized until (as mentioned) improvements were made to the visual system.

To further substantiate the average ratings and the general assessments of each AGSS
system, provided in Table 5, we have divided the rest of the survey results into two main sections:
a component rating analysis and a qualitative summary of participant comments. The component
rating analysis unveils both the positively and negatively rated features of the AGSS. In addition,

20



HELMET - Did you experience any breakage, EQUIPMENT - Did you experience any of the following
matfut or usabili lems in the fol 2. lequipment problems?
N-Never R- Rarely. S-Sometimes 6 - Often. A Always
1. Visor? NR SOA] 1. Poorheimet fit? NR S OCA
2. Eye Piece? INR SO AT? Eye-piece misalignment? INR S OA
3._Head-tracker? NOUR.S.OUAT3. Slippage?- N ROS OA
4. Comm. cable? NR US OA] 4. Neck fatigue? NURS 6A]
5._ Elect. cable? NUUR.S'O ATS. Hot spots? NR SOA
6._ Other: [N RUS O ATE. Chin straps chafing? N RS OCA
7. Chin straps choking? NOR SOA
8. Unbalanced weight? NURS OA
9._General discomfort? NR SS OUA
10._Other: NR SOTA
CABLES
1. Do the cables unduly tug at the back of your head? ¥ N
[2._Do the cables limit the amount of normal movement around the aircraft? ¥ N
3._Do the cables hinder normal head movements (e. |.. scanning ability)? ¥ N
4. Do the cables interfere with task performance (e.g., target acquisition)? ¥: oN
5. If YES, indicate which tasks:
HELMET ADJUSTMENTS
1. How long did it take to put on and adjust the helmet the first time?
2. How long did it usually take to put on and adjust the helmet?
3. Once the helmet was adjusted, did you require readjustments during the training Y oN
time? ma
AGSS TRAINEE STATIONS
1-Poor 2 Min. Reg: 3 - Standard ic et ‘5 - Exceptional:
TRAINEE STATION Rating | WEAPONS SIMULATION MiniGun -50 Cal
1. Overall station set-up? 1. Gun Recoil?
2._Gun set-up? 2.__Gun Sounds?
3._Gun feel? 3.__Gun Position?
4. Virtual mapping - acft. envmt.? 4. __Gun Weight/Feel?
5. Virtual mapping - other stations? 5. Gun Controls?
6. Mobility within station? 6. Airstream Effects?
7. Crew seats and hamesses? 7.__ Tracer Fidelity?
8. Other: 8. Ballistic Model?
1 9. Gun Alignment?
10. Target Acquisition?
11. Other:
INSTRUCTOR OPERATING STATION - 10S
1. Does the noise level of the Surrounding simulators affect your ability to train
at the AGSS IOS? x N
2. While at the 1OS, does your inability to see the student in the AGSS affect
your ability to instruct? Y No.
3. Do you become fully task saturated at the AGSS 10S?
With 1, 2, or 3 students? (Circle choice) Y N
4. Do the visual distractions in the simulator bay (e.g., the flashing red light of the
MH-53J WST) affect your ability to train at the AGSS 10S? Y Ne

52



Page
INTRODUCTION, 1
Evolution of Training and Design Requirements for the AGSS. 2
VR Application Issues... 4
Human Factors Analysis of VR Technology 7
Scope of the Human Factors Survey. 9
a
11
12
. 13
Instructor Operator Station. 13
Weapons Simulation... 14
14
14
15
17
18
19
Overall Findings... 19
Analysis of System Ratings. . 21
Helmet... 21
Trainee Stations. - 22
TOS... sseeeeseeee 23
Weapons Simulation. 23
Visual System... 24
Problem Areas: Comments and Recommendations for Improvement . 25

25
Helmet Components. 26
Helmet Comfort and Fit 26
Helmet Electronic and Communication Cables. 27
Helmet Donning and Adjustment 28

iii



We also recommend improving the FOV, but not by purchasing a new visual display. The
main criticism of the FOV was the inability to see fully forward or aft in the AGSS. The primary
reason for this was not the limited range of the HMD’s FOV, but instead, it was the trainees?
inability to get their VR helmets out the AGSS side windows and use head movements to expand
their FOV, as presently done in the aircraft. (However, there is some disagreement among SMEs
as to the actual frequency of this practice.) One solution would be to enlarge the AGSS windows
to accommodate the VR helmets. While this would reduce the AGSS’ physical fidelity with the
aircraft, its functional fidelity would be much enhanced.

Motion

The miscellaneous section of the survey contained several questions about the motion
system. Two questions and their responses offered some insight in this regard. First, six
participants reported that the AGSS did not fly like the aircraft. Yet, eight participants reported
being satisfied with the three-degree-of-freedom motion system. Combined, these two results led
to the tentative conclusion that the limited motion base was not the primary reason why
participants reported that the AGSS did not fly like the aircraft. Instead, they cited low vibration
level, lack of gravity, and lack of seat-of-the-pants feel.

Physiological Symptoms

Self-reports of the frequency and severity of nine different simulator sickness symptoms
(visual discomfort, headaches, double vision, blurred vision, disorientation, eye strain or fatigue,
neck strain, stomach discomfort, and nausea) during and after AGSS sessions were obtained. To
determine the number of subjects reporting symptoms, we first collapsed across all of the
symptoms. We found that 54% of the survey participants reported symptoms sometimes or rarely,
and 46% reported never having symptoms (both during and after AGSS flights). This is consistent
with the literature on simulator sickness, where the percentage of individuals reporting symptoms
across a selection of simulators has been observed between 10% and 60% (Kennedy, Lilienthal,
Berbaum, Baltzley, & McCauley, 1989). Figure 10 depicts the reported incidence of simulator
sickness symptoms during AGSS sessions.

Sometimes
36%

Rarely
18%

Figure 10. Reported Incidence of Any Simulator Sickness Symptoms During AGSS Training.

39



Similarly, we also asked about image resolution and detail level of the entire visual scene,
Six of 11 participants rated this feature as a “3” or less. Comments included: “could be more
stuff in there,” “lack of detail,” and “lack of realism.”

A critical element of any visual system is its representation of depth. We asked about this
feature in two ways. First, we asked for a general assessment of whether the visual scene
provided meaningful depth information. We also examined the behavioral manifestation of this in
terms of participants’ distance perception abilities. Five participants rated the portrayal of
meaningful depth in the visual scene as a “3” or less. Accordingly, participants’ judged distance
perception as poor, with six participants rating it as a “3” or less. Participants had different views
concerning the primary culprit. Several individuals cited the above problems with object
resolution as the key cause. Others blamed their perception difficulties on the “flat scene.” Still
others qualified their comments, saying that certain distances could not be judged but others
could, For example, one participant said that judging distance from the aircraft to the ground was
possible in the AGSS, but other aspects of distance judgment were difficult. The variability in
responses led us to believe that perhaps the main problem with this visual feature was certain
inevitable differences between the aircraft and the AGSS.

Related to both resolution and depth of the visual scene is visual contrast. Seven
participants rated this feature as a problem. Participants’ comments were fairly specific,
interpreting their inability to discern visual contrast as a result of the limited detail in the visual
scene and the lack of a three-dimensional scene representation. A particularly telling comment
was, “My focusing is impaired, constantly fighting to focus, trying to grab at sharp detail, but not
much there.” One participant, however, did mention that visual contrast was satisfactory during
the daytime scenes, but it was not sufficient during NVG night scenes.

Five of 11 participants rated visual brightness as a “3” or less. Like the above feature, and
in accord with our framework (Figure 2), there appeared to be some disparity due to method of
use. Specifically, NVG night scenes were the main source of the critical comments, e.g., “the
virtual NVG scene is grayer than the NVG scene in the aircraft and the color balance seems off.”

Visual scene clarity and image clarity were also judged as problematic by a number of the
participants (7 and 5, respectively), Participants claimed that although the system was good in its
present form, there was room for marked improvement.

The completeness of the visual scene was rated as a “3” or less by six participants, They
qualified their responses to reflect discrepancies between a properly versus a poorly functioning
IG. The central theme was that when the IG was functioning properly, the scene did not get
chopped or clipped off. But when it was not working, “it would do some weird stuff.”

Visual scene continuity was rated as a problem by seven participants, but the comments
were not qualified by the status of the IG. Participants were primarily concerned about the update
rate of the IG, maintaining that it seemed slow and that occasionally the scene had to catch up
with their scanning rates.

37



Another class of problems with VR technology entails user performance. For example,
there are frequent reports of users becoming disoriented while navigating in virtual environments
(e.g., Mowafy & Congdon, 1994). Since the user’s bearing is variable in a virtual world, it is
often difficult to maintain a consistent azimuthal relationship between one’s self and significant
objects (e.g., targets) in the environment. Consistent with this problem, there are many reports of
users walking into objects, both virtual and real world, while navigating through VR environments
(DOD, 1994).

Perhaps the largest potential problem concerns the apparently high incidence of simulator
sickness observed in a number of previous VR implementations. As in any simulation setting,
discrepancies between vection (perceived self-motion) and visual feedback from the VR
simulation can cause discomfort, a feeling of nausea, or outright sickness among many subjects
(Lane, Kennedy, & Jones, 1994). The percentage of VR users who experience some degree of
sickness varies, with one study (Bailey & Witmer, 1994) excusing only 16% of their subjects due
to illness, while Knerr et al. (1994) reported simulator sickness symptoms in most of their study

The true incidence and extent of simulator sickness will no doubt vary as a function of
many conditions. For example, there is some evidence to suggest that the amount of presence
experienced in the VR setting is negatively related to the severity of simulator sickness reported
for that environment (Knerr et al., 1994). Moreover, there are also data suggesting that simulator
sickness may adversely affect performance, even for those participants who do not experience
severe symptoms (Bailey & Witmer, 1994). Anecdotally, some believe that experiencing a VR
simulation while on a motion base (as opposed to a fixed-base) will alleviate many simulator
sickness problems (DOD, 1994). Whatever the mechanism(s), it is clear that definitive steps will
need to be taken to reduce the incidence of simulator-induced illness before VR technology can be
implemented as a training medium on a large scale.

Viewed in the long term, it is reasonable to expect that VR will prove effective for some
training tasks and not for others. As such, VR will not be the “silver bullet” that revolutionizes
the military’s approach to training nor radically alters how training is conducted at the squadron
level. Consequently, the job of the behavioral scientist is to identify the characteristics of the
user’s tasks whose performance are aided, hindered, or unaffected by VR so that the latter may be
judiciously and cost-effectively applied to other weapon training systems (Lane et al., 1994).
Such identification is the guiding focus of the human factors evaluation described in this report.

Human Factors Analysis of VR Technology

Human factors is a subdiscipline of psychology whose primary objective is to design
machines and/or work environments that accommodate the limits of the human user (Wickens,
1984). As typically practiced, a human factors analysis will conceptualize the human-machine
environment as a total system whose components and component relationships may then be
examined in more detail. Figure 1 depicts the user-machine environment as typically conceived in
a human factors analysis.



Van Cott, H.P. & Warrick, M.J. (1972). Man as a system component. In H.P. Van Cott and R.G.
Kinkade (Eds.) Human engineering guide to equipment design, 2nd Edition (pp. 17-40).
American Institutes for Research: Washington, D.C.

Wickens, C.D. (1984). Engineering psychology and human performance. Columbus, OH:
Charles Merrill Publishing Company.

49


Participant Ratings: Raw Data
Component Rating, frequency

Voice Procedures

Checklist Procedures
‘Gunner-Cockpit Exchanges
Left, Right, Tail Call Resp.
Left, Right, Tail Interactions
Radio Discipline

Aircrew Briefings
Passenger Briefings

‘Crew Coordination Overall
Safety and Judgment
Resource Management
Knowledge of Directives
Communication Systems
Taxi/Hover Clearance Calls
Approach/Landing Calls
Go-Arounds

Scanning

Search Operations

Remote Operations

Tactical Approaches
Tactical Knowledge
Radar/IR

AAA

Terrain Masking

Threat Breaks/Evas. Maneuvers
Countermeasures

Rules of Engagement
Tactical Navigation
Terrain/Obstacle Avoidance
Waypoint Identification
Light Signals

Emergency Procedures
Target Acquisition

Ammo Conservation

Use Limitations

Gun Malfunctions

NVG Failure

Aircraft Taping and Lighting

of S| SIS] yp] —] H] oe] eo] —] of a] o] a] a] Oo] of co} of of =} =] of o] Oo] Of Oo} =] HW] Of O} =] NO] Oo} =] BLO
=] ol o} =f] of ua] w]e] =] =|] =] of HL of 2] H] H] roftof =H] H] =] ro] o} to} v2] a] to} of 2] ] =| o] o] Oo] BO
ol ofo} ol —}us] vw] of of af uf) ry] wo] =] a} vo} oo] ropa] a] af ty] tof =] ro} on] afro alte} —] of a] =p ety el
o}olo}o}u} a] a] olor} uo] ul ala) a] a} a] alo) of vo] eo} on] af a] a] to} =] o] 8] a] oO} a] 9] 00] SP a] Of
lo} o} of vr] vo] of of of =] ro] A] ro] S] ro} =] ro] v9} to] to] =] ta] 2] 29] ty} V] 9] Of Of —] ty] dy} | ey] to} La} ta] Oo]

NVG Use and Limitations



are encountered, the proper solution(s) can be determined—modifying the device, providing
additional training to the user, or both.

As applied to the AGSS as a VR device, the human factors analysis looks at the users’
experiences of immersion, presence, and navigation through the virtual environment. It also
addresses potential anthropometric (ie., fit and feel) problems posed by wearing an HMD for
extended periods of time. The analysis also looks for reported instances of simulator sickness and
attempts to correlate those reports with the control-display conditions that were present at the
time of symptom onset.

Second, a human factors analysis should utilize a sufficiently broad array of data collection
methodologies so that all of the relevant aspects of human functionality are covered. That is, the
analysis should examine the visual requirements of the user, as well as his/her auditory, cognitive,
and haptic needs. Moreover, the analysis should include measures that distinguish between
operability of the device (i.e., is it as easy to use as practically possible) versus its acceptance by
the user (Wickens, 1984). We explicitly incorporated all of these features into our human factors
analysis of the AGSS, except for acceptance. Although we recognize the importance of
acceptance for the complete analysis of aircrew training devices, we only tangentially address this
issue within our survey. As discussed below, it is a research issue we will investigate more
explicitly and thoroughly in our subsequent AGSS research efforts.

In addition, a human factors analysis should survey a number of different users who have
had experience with the system. It should include a mix of survey, interview, and observational
measures to ensure that data on input, output, controls, displays, and processing deficiencies are
identified. The analysis should include measures that tap into both the device’s ease of use as well
as its effectiveness in achieving the users’ stated training objectives. Our analysis of the AGSS
incorporates all of these features.

Scope of the Human Factors Survey

The Human Factors Survey Framework depicted in Figure 2 is designed to capture the
logic underlying our presentation of human factors issues associated with the AGSS and its
various components. In addition, the framework characterizes our thinking regarding ensuing
research efforts related to the acceptance and training effectiveness of the AGSS. The topics
explicitly covered by the present survey are included within the shaded box. The items outside of
the box were not explicitly explored in this survey, but are nonetheless, relevant to the overall
assessment of the AGSS as a training device. A description of the survey framework follows.

Moving from left to right, we start with the assumption that the device was built according
to certain design specifications which met government standards since the device was approved as
RFT in March 1996. As shown in the figure, these design specifications are applicable to each
device component, i.e., helmet, trainee stations, IOS, weapons simulation, and visual system.
(The specific design specifications for the AGSS are not described in this report. We do,
however, provide a general description of the main systems of the AGSS in the Method Section.)
Associated with each device component will be a host of human factors issues, represented in the
Human Factors boxes to the right of each device component.


Fe A ved
REPORT DOCUMENTATION PAGE OMB Ne 0704-0188

Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering
‘and main the data needed, and completing and the collection of information. Send comments regarsing this burden estimate or any other aspect of this collection of
Elsa ae Oe emt ans soe ae Pa reed ter sean ee
1. AGENCY USE ONLY (Leave blank) | 2. REPORT DATE 3, REPORT TYPE AND DATES COVERED
June_1997 Final - October 1995 to July 1996
4, TITLE AND SUBTITLE 5, FUNDING NUMBERS
. . C_- F41624-95-C-5011
Human Factors Evaluation of the Aerial Gunner Scanner Simulator PE - 62202F
PR - 1123
Benise & Siverman wy - BS, BR
Alan Spiker - Ol,
Robert T. Nullmeyer
7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION
Hughes Training Inc Anacapa Sciences, Inc.
Training Operations 301 East Carillo
6001 South Power Road, Bldg 561 Santa Barbara CA 93101
Mesa AZ 85206-0904
9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSORINGIMON TORING
Armstrong Laboratory
Human Resources Directorate
Aircrew Training Research Division AL/HR-TR-1996-0146

6001 S. Power Road, Bldg 558
Mesa, AZ 85206-0904

11. SUPPLEMENTARY NOTES

Armstrong Laboratory Technical Monitor: Dr Robert T. Nullmeyer, (602) 988-6561

12a. DISTRIBUTION/AVAILABILITY STATEMENT 12b. DISTRIBUTION CODE

Approved for public release; distribution is unlimited.

13. ABSTRACT (Maximum 200 words)

This report describes a human factors evaluation of the Aerial Gunner Scanner Simulator (AGSS) that was installed at Kirtland
Air Force Base, New Mexico, and is being operated by the 58th Special Operations Wing (SOW). A brief review of the
literature concerning the levels of immersion associated with virtual reality (VR) is provided to help establish the nature of the
VR environment provided by the AGSS and to give some context for the issues that accompany using VR for training. The
evaluation was conducted using structured interviews with questions regarding the fidelity and/or ease of use of components
from each of the AGSS's main systems, (i.e., trainee stations, instructor/operator stations , visual, and weapons) and its training
capability for selected aerial gunner/scanner (AG/S) skills. Eleven experienced rotary-wing instructors volunteered to
participate. The resulting ratings and comments were quite favorable, indicating that most of the AGSS components were
acceptable. Two notable exceptions were the spaceball and object detection. Both areas were judged as being significantly
poor and received low ratings. We report comments and discuss recommendations associated with the components that were
judged acceptable or significantly poor. We conclude with a summary of the main findings and recommendations for
improvement with their associated cost estimates.

14, SUBJECTTERMS an . oe . 15, NUMBER OF PAGES

Aerial Gunner Scanner Simulator; AGSS; Flight simulation; Flight simulators; Flight 63

training; Human factors; Simulation-based training; Simulators; Training; Virtual 16. PRICE CODE

environments; VR :

17. SECURITY CLASSIFICATION | 18. SECURITY CLASSIFICATION | 19. SECURITY CLASSIFICATION

OF REPORT OF THIS PAGE OF ABSTRACT 20. LIMITATION ABSTRACT

Unclassified Unclassified Unclassified UL

NSN 7540-01-280-5500 ‘Standard Form 298 (Rev 2-89)

Prescribed by ANSI Std 2-39-18



Mobility, Seats, and Harnesses

Comments. Next, six participants rated their mobility (already briefly mentioned in
reference to the helmet cables) at the AGSS gunner stations as a “3” or less. The primary
criticism was that there is less mobility in the AGSS than in the aircraft. Several different reasons
were cited, including cable length, inhibitions about having an expensive helmet on one’s head,
and differences between the virtual and physical world. Regarding the latter, it appears as if one
can put one’s head out of the AGSS windows in the virtual aircraft although the simulator’s
structure does not allow this. However, most respondents reiterated that given the tasks
performed in the AGSS, reduced mobility was not that much of a problem.

Due to the limited use of the AGSS in the MH-60G configuration, only five participants
were qualified to rate the functionality and fidelity of the AGSS seats. Yet, four of them rated the
seats as a “3” or less. The primary problem was that while the seats met the design specifications
for the AGSS, they were still not comparable to the aircraft. As a result, participants expressed
concern about their durability.

The harnesses were also only rated by five participants, and again four of them rated the
harnesses as a “3” or less. Two main points arose from individual comments: First, the harnesses
seemed to be of poor quality. Second, they simply were not used much. Respondents noted that
while the harnesses were a part of the critical design review (CDR) requirements for the AGSS,
their purpose within the AGSS was unclear.

Recommendations. If the harnesses and seats are going to receive regular use, the
primary recommendation is to improve their quality to support AGSS training throughput.

Ios

Thirteen individual features of the IOS were rated on training utility. Six are discussed
below based on the Kolmogorov-Smimov analysis and participant comments.

Screen Layout

Comments. Four out of eight participants rated the student screen layout at the IOS as a
“3” or less, and three out of eight rated the general layout of the IOS as a “3” or less. The
primary critique was that, while the current layout was functional, better layouts were possible.
One individual noted that due to the height and location of the instructor information output
screen (see Figure 6), he had to get out of his seat to make some control inputs.

Recommendations. There are two main recommendations for improving the IOS layout
and making the IOS more “user friendly.” First, if the IOS layout remains as shown in Figure 6,
AGSS training would be greatly enhanced by using two instructors for the training sessions. One
instructor would be responsible for running the AGSS and the other for observing, instructing,
and role playing. (We make this same recommendation later, in response to instructor-reported
levels of task saturation at the IOS.) Second, if additional funds are available or the AGSS is
moved, a more ergonomic arrangement of the screens was suggested by a survey participant, and

30



with the AGSS. The interviewer stressed the importance of making candid device assessments
and the fundamental goal of improving the system and its use by students and instructors.

Each interview was conducted using the same format and following the same topical order,
ie., from the background information section through the miscellaneous section. However, some
sections and/or questions were skipped if the participant did not have experience with the device in
that capacity (e.g., as an instructor/operator or using the device under NVG conditions). The
interviewer prompted comments associated with extreme rating values. Responses were written
directly on the survey by the interviewer. Most interviews lasted 1-2 hrs (the minimum was 40
min and the maximum 117 min), depending on the subject’s amount of experience with the device.
Those with more device experience tended to take longer to interview.

’ Choice of Analysis

The survey contained three types of quantitative responses: (a) yes/no responses;
(b) never, rarely, sometimes, often, always scale ratings; and (c) 1 (poor) to 5 (exceptional) Likert
Scale ratings. In addition, each participant provided researchers with a substantial amount of
qualitative information, including their informed opinions about device components or features,
particular problems, and often suggested solutions. There were very few questions that simply
required a yes/no response, and for these we simply report frequency counts.

For the scaled responses, we were faced with more complex analytical issues. There were
two critical challenges that had to be met. One, the quantitative information had to be logically
linked to the qualitative information. Two, the selected analysis or analyses had to reflect both the
positive and negative comments and ratings concerning the AGSS. This was necessary to ensure
that the evaluation would yield a fair, impartial assessment of the device’s capacity and worth to
the training community.

In order to accomplish these goals, we selected two statistical techniques to analyze our
rating data, one for the AGSS component ratings and the other for the AGSS training capability
ratings. First, we chose the Kolmogorov-Smirnov, one-sample, goodness-of-fit test (Hays, 1988;
Siegel, 1956) to analyze AGSS system component ratings. The Kolmogorov-Smimov is a
nonparametric test of the extent to which an observed distribution of scores deviates from some
expected distribution. For each of our analyses, we used the Kolmogorov-Smirnov analysis to test
whether the obtained rating distribution was significantly different from a uniform rectangular
distribution. This analysis lets us determine whether the observed ratings for a given survey item
were significantly bunched toward the positive end of the scale. Importantly, the Kolmogorov-
Smirnov technique preserves the ordinal properties of the original rating scale, making it a more
powerful test than the traditional chi-square test. In addition, due to the nature of the scale use by
our participants, i.e., their limited use of extreme scale values, we chose to collapse the scale
responses into three categories. We combined the “Is” and “2s” into a single category and we
combined the “4s” and “5s” into a single category. The Kolmogorov-Smirnov analysis could still
be applied to these data without any loss of statistical power. Comparisons were done against a
fairly liberal alpha-level of .20.



NOTICES

The Office of Public Affairs has reviewed this report, and it is releasable
to the National Technical Information Service, where it will be available to the
general public, including foreign nationals.

This report has been reviewed and is approved for publication.

ROBERT T. NULLMEYER DEE H. ANDREWS
Project Scientist Technical Director

LYNN A. CARROLL, Colonel, USAF
Chief, Aircrew Training Research Division

Please notify ALIHRPP, 7909 Lindbergh Drive, Brooks AFB, TX 78236-5352, if your
address changes, or if you no longer want to receive our technical reports. You may
write or call the STINFO Office at DSN 240-3877 or commercial (210) 536-3877.



CREW COORDINATION SKILLS

‘Temain Obstacle Avoidance

i
i

TERMINAL AREA OPERATION
SKILLS

‘Waypoint Identification
Search: i
FACTICAL SKILLS
‘Tactical Mission Navigation
Tactical Knowledge
‘Threat Breaks ‘Radio Dis
Radar
AAA

UA

’

|
|

Figure 11. Cluster Analysis Output of 39 AG/S Skills.

Several interesting points arise from the rank orderings, standard deviations, and cluster
analysis combined. First, and perhaps most important, there was considerable agreement among
the raters (as evident by the small standard deviations listed in Table 13) concerning both the
highly trainable (first three large clusters) and totally untrainable skills (last large cluster) in the
AGSS. For the middle-range skill ratings (3s), there was more variability. Some of the comments
associated with these skills suggested uncertainty, not based on the device, but because
participants had not used the device in this capacity before. Other comments suggested that some
training in these skill areas was possible with a little instructor imagination, but that it would not
be as complete as in the aircraft. Combined, these factors led us to believe that there may be
considerable untapped training potential of the AGSS across these middle-range skill areas. Table
13 also highlights one notable discrepancy between one of the key skills the AGSS was explicitly
built to train, target acquisition, and its low mean training capability rating by participants. Three
specific reasons were cited for participants’ low ratings: poor visuals, inability to judge distances,
and inaccurate modeling of the bullet path.

42



Helmet Components

Comments. Seven helmet components were rated according to frequency (from “never
occurred” to “always occurred”) of breakage or malfunctions. Five of 11 survey participants
teported problems “sometimes” or more with only two helmet components: the head-tracker and
the CRTs.

With regard to the head-tracker, two of the main problems identified in participant
comments were that it malfunctioned frequently during AGSS sessions and that it did not keep up
with required scanning rates. One participant reported that certain locations in the AGSS cabin
were problematic, creating diagonal lines in the visual field.

The main problems noted were CRTs performance deterioration over time, CRT failure
rate, and perceptual mismatches from using old and new CRTs together. For example, one
participant reported that “sometimes one [CRT] is better than the other so your eyes see two
different things, and that this is more and more of a problem as time goes by.” Another individual
stated that “the new ones [CRTs] burn out quickly, creating distracting blotches and some
disparity between the two eye pieces.”

Recommendations. From our perspective, it is critical that the performance of the head-
tracker and CRTs should, at the very least, be monitored over time. In addition, the CRTs could
be replaced more frequently, and in pairs, to prevent large visual discrepancies between the two
eyes. For the head-tracker, more recent technology with faster update rates should perhaps be
explored. These recommendations, however, require substantial additional funds to be invested in
the AGSS. Until the addition of funding, a number of clever, innovative “work arounds” may
offer interim solutions. During our observations of training, for example, one particularly adept
instructor recommended that students use an “NVG scan,” which is a slower scan, to
accommodate the lag between the head-tracker and student scanning rates. In this way, students
can maintain “procedural fidelity” without creating their own simulator-specific “work arounds”
that could interfere with proper skill performance in the aircraft. Also, if the CRTs cannot be
replaced more frequently, students should be warned about possible visual discrepancies. From
our experiences in other training environments, students are much more receptive to new training
technologies when both the virtues and limitations are discussed up front.

Helmet Comfort and Fit

Comments. The frequency of problems associated with ten aspects of the helmet comfort
and fit were rated, again from “never” to “always.” Three aspects are worth discussing because
of the comments associated with each. They are the helmet pads, the occurrence of hot spots,
and the chinstrap. These are all related to the results from the Kolmogorov-Smirnov analysis
which indicated that helmet fit is only “acceptable.”

The helmet pads were frequently cited in participant comments as being inadequate, with
their method of attachment to the helmet singled out most often. Participants noted that the
velcro tape comes off inside the helmet, and that the pads unglue and rip. Others reported that
the pads are hard and uncomfortable. In responses to other queries about poor helmet fit or

26



simulator. Due to current IG modeling constraints, the limited number of tracers per rounds fired
may also require additional “coaching.”

Alignment

Comments. Another aspect of weapons simulation receiving almost unanimous
displeasure was the alignment of the virtual gun with the physical gun in the data base, with nine
participants rating it as a “3” or less. The two main criticisms were the (a) unfamiliar and
unstandardized alignment process and (b) apparent degradation of the alignment during training.

Similarly, the alignment of the gun on its pivot point was rated as a “3” or less by eight
participants. The main issues cited were that the gun “floats,” the alignment process itself is poor,
and it is necessary to make readjustments during flight. Many participants suggested that this
problem was strongly related (again) to lack of understanding and definition of the appropriate
procedures for aligning the gun position in the data base.

Recommendations. Again, as mentioned, we recommend improving and standardizing
the alignment procedures to alleviate many of these problems. We would also recommend
improving the modeling of the gun in the data base. (Note: Since the survey was conducted, the
software modeling of the gun position has, indeed, been improved and there is no longer a gap
between the gun and the pivot point [i.e., the “floating gun”]).

Targeting

Comments. Target acquisition was also surveyed as part of AGSS weapons simulation,
with seven participants rating it as a “3” or less. The primary comments were related to poor
characteristics of the visual system, as exemplified by: “really due to poor visuals,” “associated
with visuals,” or “can’t see targets until they are very close, does not equal reality, or even come
close.”

Four survey participants rated airstream effects as a “3” or less. The primary comments in
this area were related to aircraft and simulator discrepancies (e.g., control loading) during AGSS
development. (Note: Since the survey was conducted, this problem has also been remedied.)

Recommendations. The primary recommendation in terms of improving the target
acquisition capability of the AGSS is to improve the quality of the AGSS visual system. This will
be discussed in further detail in the next section.

Finally, seven participants rated the overall weapons simulation of the AGSS as a “3” or
less. However, the Kolmogorov-Smirnov analysis indicated that weapons simulation was rated
significantly positive. Participants’ comments support this apparent contradiction. The
overwhelming sentiment expressed in terms of weapons simulation was that participants were very
enamored with the new capabilities of the AGSS—indicated by high participant ratings. They
were, however, disappointed in several key features (the ballistic modeling, visual system, and
alignment process) and this is clearly indicated in their comments. In essence, this combination of
results is typical of any new technology assessment—participants recognized the AGSS’s training
potential, but were also well are of its current shortcomings.

35



this analysis reveals features of the AGSS that received “standard” or “acceptable” ratings (i.e.,
neither significantly positive nor negative) that in conjunction with the participant comments
(detailed in the next section) suggest several AGSS features which may require modifications.
Briefly, the ratings and qualitative analysis illustrate that: (a) most features of the AGSS are
acceptable for training, but may benefit from improvement; (b) quite a few AGSS features are
superb; and (c) the AGSS, overall, functions fairly well as an AG/S training medium.

Analysis of System Ratings

As explained in the Choice of Analysis section, we chose the Kolmogorov-Smirnov one-
sample test for goodness of fit (Hays, 1988; Siegel, 1956) to analyze the component rating data
for each system. Because this technique retains the ordinal nature of the data inherent in the
rating scale, it is more powerful than the traditional chi-square statistic. It also allows us to
combine data from adjacent cells (ie., the “1” and “2”, and “4” and “S” ratings) without
sacrificing statistical power. We have organized the component ratings results into five
subsections corresponding to each of the AGSS systems—Helmet, Trainee Stations, IOS,
Weapons Simulation, and Visual System.

Each subsection is organized around a table that depicts the results of the Kolmogorov-
Smirnov one-sample test. Within each table, we indicate the system component that is rated, the
number of participants who rated the component (N), the maximum deviation (Max. D) between
the expected and observed cumulative probability distributions, and the two-tailed probability
level for that deviation. When a probability level is not stated, this means that the ratings
associated with that component were neither significantly positive nor significantly negative. In
other words, participants provided “acceptable” ratings for these components. These
components, as well as the features receiving significantly negative ratings, are discussed further
in our qualitative analysis of participant comments.

It should be noted that the items associated with negative deviations (though still
statistically acceptable) are also identified through this analysis. Given the level of development of
the device (i.e., judged RFT for stand-alone training), it stands to reason that many components
are exceptional, while a majority of them are acceptable, but could be improved. Perhaps the
most remarkable finding, however, given the one-of-a-kind nature of the AGSS and the novel use
of VR technology in the AG/S training realm, is the fact that only two components of the AGSS
were rated significantly poor. We briefly describe the results of the rating analyses for each
system below.

Helmet

Collapsing across two subsections of the survey (helmet malfunctions and helmet comfort
and fit), 17 helmet characteristics were rated for frequency of problem occurrence. All but four
characteristics had low levels of reported problem occurrence, i.e., were significantly positive (see
Table 6).

21



Considering these criteria, the AGSS and its associated IG-generated visual database certainly
qualifies as providing a virtual environment for AG/S trainees.

Within the VR domain, immersion is achieved when one or more of the user’s senses
(typically vision and audition) are isolated from the surrounding environment and fed only
information from a computer simulation (Pimentel & Teixeira, 1993). An HMD or large-dome
display is typically required to achieve such immersion. However, true immersion is said to occur
when the user interacts with the virtual environment as if it were his/her actual physical
environment. By this definition, it would be difficult for an outside observer to attract the
attention of a totally “immersed” user. This situation might pose some potential drawbacks in a
training setting, where the instructor requires periodic, real-time contact with the trainee.
Interestingly, there appear to be individual differences among users in their susceptibility to
immersion, much as there is to hypnosis (Knerr, et al., 1993).

A related VR concept is presence, which is the subjective experience of being in one place
when one is physically in another (Bailey & Witmer, 1994), In a sense, presence can be viewed as
a weaker form of immersion, since the user’s senses do not need to be fully engaged.
Importantly, this may be a more realistic and practical goal for a VR-based training technology.
In this regard, there is evidence to suggest that users can still experience presence while modifying
the viewpoint in the virtual environment, thus improving their overall spatial knowledge of objects
in the virtual world (Mowafy & Miller, 1993).

As a technology, there has been a rapid expansion of VR capabilities since the early 1980s.
Depending on price and one’s task area of interest, it is possible to distinguish among four levels
of VR (DOD, 1994). Entry level VR includes a cathode ray tube (CRT) graphic display; desk top
computer; and a two-dimensional input device, such as a mouse, trackball, or joystick. Expansion
into a Level 2 or basic VR adds some basic interaction and display enhancements. These might
include a stereographic viewer and a more capable input device. The latter might be either a
DataGloveTM or a multidimensional (3- or 6-dimension) mouse or joystick.

Stepping up to Level 3 or advanced VR entails the addition of a rendering accelerator card
for faster display of animated graphics and possibly a sound card for true 3-D audio output. As
parallel processors have come down in price, it has become more feasible to render real-time
imaging video over a larger display area. Such a capability adds realism to the visual presentation
and typically enhances the user’s interest and enjoyment of the experience. Moreover, including
audio feedback has been shown to dramatically enrich the user’s VR experience (Miner, 1994).

One reaches the highest level of VR (Level 4) by adding some type of immersive display
system. At the present time, the choices are HMDs, multiple large project displays, or binocular
omni-orientation monitors (BOOMs). A BOOM device has the viewer look through a
stereoscopic display whose field of regard is swiveled around a fixed boom platform. In each
case, it is typically assumed that the user will have an “immersive experience” if the display can
provide at least 60° of horizontal field of view (FOV). An immersive system might also involve
some form of haptic interaction mechanism so that the user can feel virtual objects.


SUMMARY OF RESULTS AND RECOMMENDATIONS

All participants in the survey were quite positive about the training potential of the AGSS.
Indeed, the most frequent positive comment across the board, even with the number of participant
criticisms, was that the device provided unique training capabilities unavailable prior to the
introduction of the AGSS into the 58 TRSS. With this said, participants’ ratings and comments
were extremely insightful as to improvements that could make the device an outstanding training
medium for AG and FE training curricula. To summarize, the central findings, both positive and
negative, associated with each section of the survey are presented in Table 14.

Table 14. Main Findings from Each Section of the Survey: Problems and Positives.

same wey Main Problems Reported __ Positive Aspects Noted
=> CRT failure rate and discrepancies => Lightweight, less than those used in
=> Head-tracker: slow update rate the aircraft
=> Alignment and fitting problems
Helmet => Pads: type and method of attachment to helmet
=> Cords: too short for requirements of tail position
and they sometimes get caught during training
Trainee => Virtual mapping of stations and guns, a few noted => Mapping of outside of ownship
Stations problems
=> Second instructor position lacks the ability to talk => Threats on ground track map page
directly with students in the AGSS => Similarities with other 58 TRSS
=> Space ball: poor modeling of ownship OSs makes it easy to learn
aerodynamics, very difficult to use
IOs => Task saturation: monitoring students, running IOS,
flying moving models, loading threats, etc. becomes
quickly overwhelming
=> Layout of screens potentially a contributor to task
saturation and instructor inefficiency
=> Poor modeling of bullet path, no lead and lag => General capability never before
Weapons => Insufficient number of tracers viewed per rounds available
Simulation fired =>__ Feedback on gunner performance
=> Alignment procedures = Explosions, including secondaries
=> Alignment of virtual and physical environment
Visuals => Poor color quality
=> Visual continuity (slow update rate)
=>_ Limited FOV for daytime operations
= 50% of participants noted the occurrence of => Reported symptoms did not disrupt
Physiological simulator sickness symptoms during and after AGSS training
Symptoms training
=> Eye strain and visual discomfort were reported by a
significant proportion of survey participants
=> Visuals make it difficult to train certain tasks => Outstanding tool for training voice
Training procedures, terminology, and crew
Capability coordination
=> Rated training capability will be

higher when the AGSS is integrated

43



Table 15. Concluded.

Problem Area Recommendation Cost TM on’
Poor Modeling of Bullet’s Path |= Change modeling. HIGH VERY HIGH
=> Provide training on aircraft versus
AGSS discrepancies. Low MEDIUM
Low Number of Tracers Visible [=> Increase number of tracers per MED!
Per Rounds Fired rounds fired. Lows UM
=> Provide training on aircraft versus MEDIUM
AGSS discrepancies and reasons. Low

Reduced FOV for Daytime => Change size of windows to

Operations accommodate VR helmet. MEDIUM | _ MEDIUM

Poor Visual Quality => Improve visual systems. HIGH VERY HIGH
=> Improve other systems (e.g., CRTs) HIGH HIGH

“NOTE. Associated costs may be low, but due to current limitations in the IG, these suggestions may not be
possible.

The shaded rows in the table indicate areas in which improvements have already been
made to the AGSS since the completion of the survey. Alignment procedures and fitting
procedures have been greatly improved, and the current level of expertise is high among the
Lockheed Martin personnel maintaining and working with the AGSS. However, the means of
transferring this knowledge is still largely through demonstration and trial and error. This
suggests a need to impose greater documentation and dissemination of this process, such as by
writing out step-by-step instructions for distribution to all AG/S trainers. The modeling of the
gun and its pivot point has also been improved dramatically since the survey, and observed
Tesponses have been overwhelmingly positive in terms of greater student acceptance of the device.
In addition, the visual system and various aspects of the data bases used in the AGSS are in
constant flux. As a consequence, several visual features of the AGSS have also been improved
since the survey. In particular, the colors and the detail and texturing of many of the moving
models have been modified. These improvements have all been well received by AGSS students
and instructors.

As for the non-shaded items in Table 10, we have attempted to include recommendations
that have potentially significant training payoffs. The cost is based on the estimated dollar costs
for providing the hardware or software modifications, the costs of extra training time and/or

45



Figure 1. Typical Human Factors Conception of the User-Machine Relationship.

As seen in the figure, the limitations and capabilities of the human user can be studied at
several points in the system. The user’s abilities to make inputs to the machine should be
considered. Depending on the application, the user might make inputs to the machine using
components such as a keyboard, joystick, mouse, touch screen, or voice recognition system.
Each of these in turn might involve interaction from the user using various parts of the body, such
as fingers, arms, and vocal chords. The corresponding human capacities in such areas as strength,
dexterity, speed, or stamina need to be reviewed to determine their suitability for the machine’s
control mechanisms (Van Cott & Warrick, 1972).

As machine inputs are processed, their results are displayed back to the user in some form.
Display choices might be a CRT, HMD, large dome, and voice, to name just a few. Again, human
limitations in the sensory domain of the display (vision, audition, touch) must be considered to
ensure that the displayed information is sufficiently large, detectable, perceptible, legible, and so
forth (Grether & Baker, 1972).

The circles in the middle of the figure encompass what is rapidly becoming the most
critical area of human factors—information processing. Thus, regardless of the method of display
or the medium of control, substantial processing will occur by both the human-user and the
machine. Since the processing of both entities is hidden from direct observation, the challenges
facing the human factors analyst are greatest here (Kantowitz & Sorkin, 1983). On the machine
side, the focus will be to ensure that the system responds sufficiently soon after input so that the
user does not experience any lag or delay that might degrade his/her performance. Responding to
the user side, the goal is to design the system so that user’s cognitive processing capacities
(attention, short-term memory, long-term memory, etc.) are not overwhelmed.

With regard to understanding the impact of VR technology on these factors, thorough
human factors analyses are essential when new VR devices are introduced into training programs,
because limitations in the user-device interactions will undoubtedly establish upper limits on the
device’s training effectiveness. There are several key elements of a well-designed human factors
study. First, the analysis should attempt to distinguish between problems resulting from
_ limitations in the user’s basic sensory-cognitive-motor capacities versus those arising from
deficiencies in the machine. Such a distinction is necessary so that when problems in device use