GRAPHICAL USER INTERFACE SYSTEM FOR MANPORTABLE APPLICATIONS

Abstract: The present invention relates to a method and device for providing a graphical user interface between an individual soldier and a computer-based warfare system worn by him. The graphical user interface is made up of icon-based visual control and information screens that the soldier views on a display and manipulates with a computer mouse-like device. The interface allows the individual soldier to: view information gathered by a variety of sources, inccluding sensor components, radios, and satellites; process information using a variety of input devices and software utilities; and, act on that information using weapons, communications devices, and the like. The graphical user interface organizes information into three distinct "combat task" groups and provides the soldier with easily understood pictorial icons to select. By tailoring the information to a particular situation, the graphical user interface allows the soldier to view critical data without being overwhelmed by too much information.

Description: αSλPEIClOi USBK IUTKBJAt FOR NlUπKJRXλBLX AFPLI BxcKGBouiro or THE IN The present invention relates generally to the efficient management of computer information by a human user. More speci¬ fically, the present invention relates to a graphical user interface system for a warfare system which is worn by an individual soldier. The interface allows the individual soldier to receive, process, and possibly transmit information gathered by a variety of sources and processed by a central microprocessor in the warfare system. Modern technology, especially computers and electronics, have advanced rapidly in the recent past. It is only logical that these technological advances would be applied to the art of war, specifically to weapons and other equipment designed to make the modern soldier a more efficient fighting machine. One approach for applying the technological advances to improve the soldier's efficiency is to view the soldier as one part of a comprehensive warfare system. This warfare system approach must give a soldier the ability to devise real-time solutions to problem situations and keeps the soldier responsive and flexible enough to operate in an uncertain and frequently dangerous environment. The first major system to propose a system warfare approach was the soldier Integrated Protective Ensemble ("SIFE"), deve¬ loped experimentally by the Ό.S. Army. The SIPE attempted to integrate multiple electronics subsystems and related elements and provided the soldier-user with easy-to-use means to interface with those subsystems. The lessons learned from the SIPE experiment led to the development of a new generation warfare system known as the Land Warrior ("LW") , which is currently being pursued by a number of defense contractors. The W includes im- provements in communications, computing, night vision, weaponry, ballistic protection, and load carrying capability while providing the individual soldier with enhanced lethality, command and control, survivability, mobility, and austainment. The LW system is divided into various subsystems, each sub- system consisting of similar or related hardware and software which is dedicated to accomplishing a certain task or family of tasks. The LW system is composed of five such subsystems: (1) Computer/Radio Subsystem ("CRS") ; (2) Weapon Subsystem ("WS") ; (3) Integrated Helmet Assembly Subsystem ("IHAS") ; (4) Protective Clothing and Individual Equipment Subsystem ("PCIES") ; and, (5) LW Software Subsystem ("SS") . For a comprehensive system such aβ the LW system to be effective, it must do more than gather large amounts of data. The data obtained must be organized and relayed to the individual soldier in an easy-to-underεtand format. Prior to the SIPE experiment, a soldier's "systems" were completely isolated from each other. There was no central microprocessor, so any systems used were coordinated only by the soldier himself. For example, the use of a precision range finding device, such as a laser range finder, could not be electronically integrated with the soldier's weapon. The soldier's navigational "system" consisted of several maps carried in his backpack or pockets, and the dissemination of navigational information was confined to verbal communication. SIPE attempted to connect all of the soldier's systems and eliminate the isolated nature αf them using the system warfare approach. Unfortunately, the SIPE experiment did not go far enough. Informational output to the soldier (such as navigational and topographic information) was limited to text only. In other words, all the information which was routed to the soldier looked exactly the same, regardless of differences in magnitude of importance. Lacking the ability to quickly prioritize information and take action based on that information, the SIPE waβ lees than ideal, especially during chaotic and potentially life-threatening situations. BTnOUUtY OT THE IHVKHTION The Graphical user Interface ("GUI") of the present invention is designed to overcome the problems encountered with the SIPE experiment and what came before SIPE. The GUI is a software-based, hardware implemented system which serves as the primary interface between the soldier and the LW system. It provides the soldier with a simple, intuitive method of obser¬ ving, interacting with, and controlling the many features of the LW system. The GUI is in part made up of icon-based visual con¬ trol and information screens that the soldier views one of several video display options and manipulates by pointing on a screen with a mouse-like device. The large, pictorial iconβ generated by the GUI are easy to read and comprehend, even under adverse visual conditions and in stressful situations. The GUI includes access to tactical data including digitized battle information, electronic range cards, video imaging, radio control, electronic message generation and reception, and inventory status. It also provides utilities such as word processing, file management, and graphics manipulation. One major advantage of the GUI is its ability to provide the soldier with information specifically tailored for the combat task at hand. Combat tasks are organized into three major segments ("fight", "caution", and "plan") associated with soldier operational states to ensure that all of the soldier's needs are met for each situation encountered. In the "fight" combat state, the soldier's mission is to survive and destroy the enemy. Information provided by the GUI for this state is concise and to the point, and is highly focused on enemy actions, targets engagements, receipt and transmission of command and intelligence data. In the "caution" state, the soldier's concentration is on preparedness for combat. Information supplied by the GUI to the soldier mainly involves situation awareness, reconnaissance results, changes in the tactical situation, command and control inputs from squad and platoon leaders, and status of supplies an d ammunition. In the "plan" state, the soldier is concerned with reor ¬ ganization and consolidation of forces while awaiting combat, including tactical mission planning. The GUI provides informa¬ tion concerned with intelligence, operational readiness, and training. The invention itself, together with further objects an d attendant advantages, will be best understood by reference to the following detailed description, taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 is a block diagram of the LW system which incor- porates the graphical user interface ("GUI") of the present invention. Figure 2 is a block diagram of the input/output structure of the present invention. Figure 3 is a front perspective view of the Remote Point- ing/Indicating Device of the present invention. Figure 4 is a front perspective view of the hand held keyboard/display unit of the preβent invention. Figure 5 is a front perspective view of the Soldier Access Module of the present invention. Figure 6 is a front perspective view of a soldier-user of the graphical user interface of the present invention. Figure 7 is a side elevational view of the IHAS as it relates to the present invention. Figure 8 is a block diagram αf the LW system which incor- porates the graphical user interface ("GUI") of the present invention. Figure 8B is a typical screen layout as displayed by the present invention. Figure 8C is a block diagram of a sequence of screens displayed by the present invention. DETAILED DESCRIPTION OVE VlEW The Graphical User Interface ("GUI") 800 of the present invention provides a simple, intuitive method αf controlling the Land Warrior ("LW") system 100, shown as a block diagram in Figure 1. The LW system 100 includes the Computer/Radio Subsystem ("CRS") 200, the Software Subsystem ("SS") 300, the Integrated Helmet Assembly Subsystem ("IHAS") 400, the Weapon Subsystem ("WS") 500, and the Protective Clothing and Individual Equipment Subsystem ("PCIES") 600. The GUI 800 itself is con¬ sidered part of the SS 300. Together, these subsystems can amass a staggering amount information, with the help of the GUI 800, the soldier can efficiently retrieve, process, and transmit information supplied by the subsystems, giving the proper priorities to emergent conditions without being overwhelmed by too much information. I/O STRUCTURE The input/output ("I/O") structure of the Computer/Radio Subsystem 200 as it relates to the GUI 800 is shown in Figure 2. All I/O devices mentioned will be further described in detail below. There are at least four sources 825 for the soldier to input data into the CRS 200. A Remote Indicating /Pointing Device 250, the primary means of inputting data into the CRS 200, is a com¬ puter mouse-like device which is attached to the clothing of the soldier. A hand held keyboard 260a is an optional input device usually reserved for unit leaders. A Soldier Access Module 270 is a portable self-contained data storage device which plugs into the CRS 200 for data transfer. There are many external sources 835 of information which may input into the CRS 200. They may include: laser detectors 408- 411, which indicate when the soldier is spotted by an enemy laser; a laser range finder\digital compass assembly 530 for computing target ranges and providing navigational information; a video camera 535 which provides video images to the CRS 200; the soldier and squad radio antennas 640a and 640b, respectively. for receiving both voice and data transmissions; and, the GPS antenna 640c for supplying Global Positioning System information to the CRS 200. There are at least four output devices 845 which convey image and text information to the individual soldier. The Day Component 440 and Night Sensor/Display component 445 (which together make up the Sensor Display Assembly 432) handle most of the display work. However, signals may be routed to the eyepiece 526 of the Thermal Weapon Sight 525 or to the display 260b of the hand held display/ eyboard 260 if needed. Soldier inpyt The Remote Pointing and Indicating Device ("RIPD") 250 shown in Figure 3 is the primary input device for the soldier. The RIPD 250 is a computer mouse-like device for entering data into the CRS 200. The pointing surface 252 is a capacitive membrane which allows the soldier to move a cursor on the screen with his finger. The functions of buttons 254 are programmable, for example, to allow the soldier to select a certain display or to select a desired sensor component. The RIPD 250 contains a keyboard-like 8-bit controller (not shown) to send button pushes to the microprocessor (not shown) of the CRS 200. The remote CRS control 550 is a two-position membrane switch of conventional construction designed to provide a back-up control for the RIPD 250. The remote CRS control 550 is located just above the trigger of the soldier's weapon, as shown in Figure 6. From the remote CRS control 550, the soldier can select the video camera 535, the thermal weapon sight 525, or the laser range finder/digital compass assembly 530 or toggle between main battle display screens (all described further herein) . Certain military personnel, for example, unit leaders, may be provided with the separate hand-held keyboard/display unit 260 shown in Figure 4. The keyboard/display unit 260 is made up of two sections, the keyboard section 260a and the display section 260b, further described herein. The keyboard/display unit is a typical "clamshell" design, very much like a standard notebook computer. The keyboard section 260a is a standard off-the-shelf ("QWERTY") keyboard and is designed to be used during normal (non-combat) operations for text entry. The two sections 260a and 260b may be used together, as shown in Figure 4, or may be used separately. The Soldier Access Module ("SAM") 270, also known as the mission data module, shown in Figure 5 is a portable data storage device which allows the soldier to customize the LW system 100 for each mission undertaken. The SAM 280 is plugged into the CRS 200 prior to a mission. The data transferred to the CRS 200 from the SAM 270 may include a combat ID, codes, or specific mission information which allow the LW system 100 to support the in¬ dividual soldier in every way possible during the upcoming mission. External Ir-But One of the key features of the LW system 100 is its ability to gather large amounts of data from external sources and to input that data to the CRS 200. With the data stored and organized in the CRS 200, it is then available for the soldier to retrieve as necessary. External sources of information may include sensor-type components aa well as other sources such as radio or satellite signals. The first example of a sensor-type component is the four- unit set of laser detectors 408-411. These laser detectors 408- 411 are of conventional design and are mounted on the helmet 402 of the soldier, best shown in Figure 7. Once the laser detectors 408-411 sense an incoming laser, the soldier is alerted to the source of the laser via an Emergency Alert message on one of the displays, as further described herein. Another vital external source of information is the laser range finder/digital compass assembly ("LRF/DCA") 530 for comput¬ ing target ranges and providing navigational information. The LRF/DCA, shown generally in Figure 6, is an integration of two commercially available components: a Fibertek mini-laser range finder and Leica digital compass assembly. Mounted on the soldier's rifle 501 or other modular weapon (not shown), it can provide a wealth of information to the CRS 200. A video camera 535 is a third example of an external source which provides images to the CRS 200. The video camera 53 5 is also mounted to the soldier's rifle 501 or other modular weapon (not shown) similar to the LRF/DCA 530. The camera 535 is an off-the-shelf component manufactured by Sekai Electronics, model number RSC-700. The soldier and squad radio antennae 640a and 640b (shown only in Figure 2) are mounted on the load carrying equipment ("LCE") of the individual soldier (not shown) and receive radio signals from radios (not shown) carried by other members of a military contingent. The antennas 640a and 640b, which are o f conventional construction, are an important external source of information. They can receive data signals (much like a standard modem) at a rate of 64 kilobits per second as well as standard voice transmissions which provide the soldier with the most current developments during operations. The GPS antenna 640c is the final example of an external source of data which supplies Global Positioning System infor¬ mation to the CRS 200. GPS satellites (not shown) can identify a geographical position within one meter of any location on earth. Such locational information is crucial to the individual soldier attempting to report the movement of enemy troops, the location targetB, and the like. Ω1EEU£ The primary display of the Graphical User Interface 800 is the Sensor Diβplay Assembly ("SDA") 432 of the IHAS 400, best shown in Figure 1. The SDA 432 is a two-piece monocular device having an LCD display which provides the soldier with display capabilities from various components throughout the subsystems. The SDA 432 includes a Day Component 440 and a Night Sensor/Dis¬ play Component 445. The SDA 432 is wired through the DC/NSDC helmet mount 441 to the Display Electronics Module 425 via cable 420. The cable 420 rune along the lower edge of the outer surface of the helmet shell 402 but is hidden beneath the cloth cover 412. The Display Electronics Module 425 is wired through cable to 430 to junction block 925 and through wires (not shown) contained in the soldier's Load carrying Equipment (not shown) to the CRS 200. The Thermal Weapon Sight ("TWS") 525, shown generally in Figure 6, is a thermal sighting system designed by Hughes Aircraft for use by the U.S. Army (military designator AM/PAS13) . Besides performing its design function, the display 526 of the TWS 525 also provides an alternate display for the Sensor Display Assembly 432 of the IHAS 400. The display section 260b αf the hand held keyboard/display unit 260 is the complement of the keyboard 260a, previously described herein. The display 260b is a conventional LED design which can be used by the individual soldier as an alternate display to the Sensor Display Assembly 432. OPERATTON OF THE GUT As Figure 8A illustrates, the GUI 800 organizes the enormous amount fo available information and functione into easily understandable groups so that the individual soldier can access it quickly and ef iciently. As soldier input sources 825 and external information sources 835 supply information to the CRS 200, the GUI 800 organizes the information into data groups 280. There are currently sixteen data groups divided into either a functional set 298 or a utilities set 299. The functional set includes communications, location, fire control, soldier preferences, weapon, system control, imaging, timing, messages, inventory, user support, and startup data groupβ. The utilities set includes word processor, draw package, file manager, help, computer-based training driver, and radio transmission utility data groups. Each data group 281-286 is represented by an easily understandable pictorial icon 291-296. The icons 290 are routed to one of a variety of "screens" depending upon whether each data group 281-286 corresponding to each icon 291-296 is classified as functional or utility. The screens, dβcribed further herein, are a composite arrangement of icons, video images, and text dis- played on an output device 845. For example, information related to communications being input into the CRS 200 from an external source 835 would be clas- sified as a part of the functional set 298, may be organized into a data group 281, which is represented by the icon 291. A typical icon for communications related information may be shape d like a radio. When the icon 291 is selected by the sol d ier (using a soldier input device 825) , the communications informa ¬ tion would then appear on an output device 845. Similarly, if the soldier needed to draw a sketch, he would use a soldier input device 825 to select an icon 296 (which may be shaped like a pen) to access the proper data group 286, which would be draw package information. The individual soldier physically interacts with the Graphical User Interface ("GUI") 800 by observing screens on one of its displays, usually the Sensor Display Assembly 432 of the IHAS 400 (shown in Figure 7) , and selectively choosing on-screen options. The GUI BOO allows smooth transition between screens and modes of operation. A typical screen layout 801 is shown in Figure SB. Area 802 is located along the upper edge of the screen 801 and is reserved for a title bar (not shown) . Areas 803 are located in the upper corners of the screen 801 and are reserved for hot buttons (not shown) . Icon bars (not shown) may be located along the right and left edges of the screen 801 in areas 805. Pictorial icons (not shown) are located in the icon bar area 805 and represent the various data groups of information available to be selected. Emergency alerts or other information (not shown) may appear in the lower center portion 807 of the screen 801, also as a pictorial icon; the lower left corner 806 and the lower right corner 808 may also contain relevant data such as weapon inventory and the correct time, respectively. A message (not shown) may appear directly under the emergency alert area 807 in scroll bar area 810. Finally the remaining portion of the screen 809 displays a selected image (not shown) , for example, a video camera image, topographical map, etc. The block diagram of Figure 8C shows the screen sequence as seen by the individual soldier as he operates the GUT 800. The GUI 800 screens are configured for two basic modes of operation: Battle Mode and General Operations Mode. The default mode of operation, the Battle Mode, provides the soldier with only that information which increases his lethality and improves his sur- vivability. The two screens available in the Battle Mode are the Core Battle Screen 805 and the Tactical Display Screen 810. When placed in the Battle Mode, the GUI 800 defaults to the Core Battle Screen 805. The Core Battle Screen 805 displays a visual or sensor target view (not shown) in the middle center portion 809 of screen 801 overlaid by a target reticle (not shown) , optional time/weapon status icons (not shown) in lower left corner 806 and lower right corner 808, respectively, and emergency alerts (not shown) in the emergency alert area 807 when required. The soldier can toggle back and forth between the Core Battle Screen 805 and the Tactical Display Screen 810 using weapon-mounted remote CRS control 550, shown in Figure 6. The Tactical Display Screen 810 provides immediate tactical position¬ ing information. Both screens 805 and 810 feature an Emergency Alert area 807. The LW system 100 inserts pictographs (not shown) in this space 807 to warn the soldier of neurological, biological, or chemical ("NBC") threats, imminent system failure, or other critical conditions. Unit leaders may also insert emergency text messages (not shown) in this area 807 to be viewed by the individual soldier. The GUI 800 provides the soldier with one-touch control of critical operations within the data tiers via hot buttons (not shown) located in screen portions 803. The function of these buttons can be fixed or programmed by the unit leader. For example, a hot button (not shown) on the Tactical Display Screen 810 could be used to instantly call up a topographical map screen 815 when needed. The soldier may place the system into the General operations Mode from the Battle Mode by selecting the tools icon (not shown) located in area 804. The operations screen 820 appears first and includes a Functional Toolchest set αf icons (not shown) in the icon bar area 805. The GUI 800 returns to the Battle Mode upon the command of the soldier or his leader. The Functional Toolchest contains icons (not shown) representing classes of data and operations that the soldier may select. A readily understandable icon represents each class of data and functions. For example, a "radio-shaped" icon (not shown) represents communications, an "artillery-shaped" icon (not shown) represents a call for a fire mission. Choosing an icon may also cause the GUI 800 to display further screens. For example, by selecting a "rifle-shaped" icon (not shown) , a electronic range card screen 825 is accessed for target identifi ¬ cation. The Utility Toolchest programs such as word processing and file management are also represented by icons (not shown) . For example, by selecting a "finger-pushing-a-button-shaped" icon (not shown) , a Soldier Preferences Utility screen 830 is accessed; by selecting a "winged-radio-shaped" icon, a Trans¬ mission Utility Screen 835 is accessed to allow the soldier to transmit a message. The key to the GUI 800 is the organization and presentation of each screen in each mode. The soldier is provided with all of the relevant information needed for a specific mission and nothing else. By tailoring each presentation of information to a particular situation, the GUI 800 provides information to the soldier quickly and in a way that is easily understood, thus increasing his overall efficiency. Of course, it should be understood that a wide range of changes and modifications can be made to the preferred embodiment described above. It is therefore intended that the foregoing detailed description be regarded as illustrative rather than limiting and that it be understood that it is the following claims, including all equivalents, which are intended to define the scope of the invention.

Claims: IBM IS CLAIMED IS; 1. In a multiple function warfare system worn by an individual soldier, a device for interfacing said individual soldier with said warfare system, said device comprising: a processing section, said processing section being carried by said individual soldier; an input section; said input section being electrically connected to said processing section; an output section, said output section being electri- cally connected to said processing section. 2. The device according to Claim 1 wherein said processing section comprises a microprocessor. 3. The device according to Claim 1 wherein said input section comprises a remote pointing and indicating device. 4. The device according to Claim l wherein said input section comprises a keyboard. 5. The device according to Claim 1 wherein said input section comprises a portable data storage device. 6. The device according to Claim l wherein said output section comprises a display. 7. The device according to Claim 6 wherein said display comprises a thermal weapon sight. 8. The device according to Claim 6 wherein said display comprises a day component of a sensor display assembly. 9. The device according to Claim 6 wherein said display comprises a night component of a sensor display assembly. 10. In a multiple function warfare system worn by an individual soldier, a method of managing information, said method comprising the steps of: gathering said information; routing said information to a central microprocessor; organizing said information into a plurality of data groups; creating a pictorial icon to represent each of said plurality of data groups; creating a first screen, said first screen being com¬ prised of a first plurality of icons. 11. The method according to Claim 10 where each of said first plurality of icons represent data groups containing related information. 12. The method according to Claim 11 wherein said infor¬ mation contained in said data groups represented by said first plurality of icons is related to military battle. 13. The method according to Claim 12 further comprising the step of displaying said first screen on an output device. 14. The method according to Claim 10 further comprising the step of creating a second screen, said second screen being com¬ prised of a second plurality of icons. 15. The method according to Claim 14 where each of said second plurality of icons represent data groups containing related information. 16. The method according to Claim 15 wherein said infor¬ mation contained in said data groups represented by said second plurality of icons is related to military operations. 17. The method according to Claim 16 further comprising the step of displaying said second screen on an output device. 18. In a multiple function warfare system worn by an individual soldier, a method of managing information, said metho d comprising the steps of: gathering said information; routing said information to a central microprocessor; organizing said information into a plurality of data groups; creating a plurality of pictorial icons, each one of said plurality of icons representing each one of said plurality of data groups; creating a plurality of screens, said first screen being comprised of a said plurality of pictorial icons. 19. The method according to Claim 18 wherein the image of one of said plurality of icons is related in function to said data group represetned by said one of said plurallity of icons. 20. The method according to Claim 18 further comprising the step of dividing said plurality of screens into a first mode and a second mode. 21. The method according to Claim 20 wherein said screens divided into said first mode contain said pictorial icons related to battle functions. 22. The method according to Claim 20 wherein said screens divided into said second mode contain said pictorial icons relating to operations functions. 23. The method according to Claim 18 wherein each of said plurality of screens comprises a composite of said pictorial icons, video images and text. 24. The method according to Claim 18 further comprising the step of displaying one of said plurality of data groups by selecting its representative said pictorial icon. 25. The method according to Claim 18 further comprising the step of automatically displaying one of said plurality of icons on one of said screens when one of said plurality of data groups represented by said one of said plurality of icons contains information related to emergent conditions with respect to said soldier.

Inventors: AUSMAN ANDREW, BROGAN MICHAEL W, KAVNER DOUGLAS M, MCCANN THOMAS E, ROSSER LARRI, TIFFANY JOHN R, AUSMAN, ANDREW, BROGAN, MICHAEL, W., KAVNER, DOUGLAS, M., MCCANN, THOMAS, E., ROSSER, LARRI, TIFFANY, JOHN, R.
Applicants: HUGHES AIRCRAFT CO, HE HOLDINGS, INC., DOING BUSINESS AS HUGHES ELECTR
Application year: 1997
Main classification: G06F1
IPC codes: G06F1