Abstract:
A device or computer program that takes the data and/or video from separate sensory devices and then combines this information for display in a standardized, but user configurable Graphical User Interface. Since each input device is paired with a device driver to properly parse the data for the input device, new devices may be added by writing new device drivers. The GUI has a standard set of elements that the individual user can modify to suit their personal preferences and needs. Since the Graphical User Interface is standardized, the user knows where to look for the data that they need no matter which sensory device is attached.

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
       [0001]    The present application claims priority to U.S. Provisional Patent Application Ser. No. 61/637,499, filed Apr. 24, 2012, entitled “INTUITIVE INTERFACE INITIATIVE,” the disclosure of which is expressly incorporated by reference herein. 
     
    
     STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT 
       [0002]    The invention described herein includes contributions by one or more employees of the Department of the Navy made in performance of official duties and may be manufactured, used, and licensed by or for the United States Government for any governmental purpose without payment of any royalties thereon. 
     
    
     FIELD OF THE INVENTION 
       [0003]    The field of the invention is related to interface systems. In particular, embodiments of the invention are related to standard graphical user interface (GUI) that is compatible with a variety of sensor and information systems such as, for example, Electro Optical and Sensory Equipment designed around the needs of the ends user. 
       BACKGROUND OF THE INVENTION 
       [0004]    There is a need for a GUI video overlay to provide users the ability to use original equipment manufacturers (OEM) generated GUIs. OEM purchasers utilize a vast and varying inventory of electro-optic (EO) devices for every service and mission. It has proven most cost effective to utilize contractor equipment to fulfill the requirement of the sponsors and operators. Each OEM develops unique methods and implementations of displaying necessary system information to the user. Sponsors, users, or government representatives rarely dictate the design, format or characteristics of required display information. Usually only specific data required by the operator to effectively complete their mission is captured in requirements, but not the placement and characteristics of this information. 
         [0005]    Operational effectiveness is affected in many ways including: too much or too little on-screen information; varying needs of the operators based on roles or missions (ex: gunner versus officer in charge); over-engineered overlays causing option overload; design without operator input; text readability and un-intuitive symbology; and minimal or no capability to configure the display for varying mission requirements. 
         [0006]    Existing EO device video interfaces cause multiple training issues. When new EO systems are introduced, there is a learning curve in order to be proficient in the operation of the system due to the GUI. When operators are required to operate multiple, unique EO devices, each has distinct placement and characteristics that, at times, cause confusion. Training curriculum is required to be developed for each unique system. Changes to the display interface require additional funds in order for the OEM to implement required changes. 
         [0007]    During operation within a system of systems environment, multiple hardware displays can be required to be co-located in order to provide the operator with necessary information to complete their objectives. This increases the cost and decreases the effectiveness of information processing required of the operator. 
       SUMMARY OF THE INVENTION 
       [0008]    The present invention is a system for standardizing data received for display purposes. The device comprises device drivers that are configured to receive data and video from a plurality of sensory devices that the device drivers are configured for. The data is then used to create standardized GUI overlays. The GUI overlays show the data using both symbology and text. The GUI overlays are then combined with the video and displayed in such a way that the data is shown in the same way, independent of the source of the sensory. 
         [0009]    Embodiments of the invention were developed to provide features such as a standardized GUI that is compatible with existing EO and sensory equipment designed around the needs of the end user. 
         [0010]    In certain aspects, the present invention provides a novel device that can effectively display data from a variety of devices in a manner that is standardized and intuitive to use. Accordingly, in one embodiment, the present invention provides a device that includes one or more electro-optical devices, a processor, a plurality of storage media, a display, a software program, a library of device drivers and a video data grabber. The software program is executed by the processor and stored on the storage media. The software program is configured to detect each electro-optical device connected, select a device driver from the library of device drivers which are stored on the storage media, and use that device driver to translate data from the attached electro-optical device. The software program then uses the data that it translated to create a standardized overlay. The overlay is designed to communicate critical information quickly and be intuitive to use. The video data grabber receives video from each electro-optical device and then the software combines the video with the overlay to be shown on the display. 
         [0011]    Additional features and advantages of the present invention will become apparent to those skilled in the art upon consideration of the following detailed description of the illustrative embodiment exemplifying the best mode of carrying out the invention as presently perceived. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0012]    The detailed description of the drawings particularly refers to the accompanying figures in which: 
           [0013]      FIG. 1  shows an overview of hardware and software architecture in accordance with of one embodiment of the invention that allows that data and video from a variety of devices to be collated and sent display units; 
           [0014]      FIG. 2  shows another embodiment of the system including an operator system, a client system, a sensor system, and a video encoder system; 
           [0015]      FIG. 3  shows a GUI display in accordance with another embodiment of the invention; 
           [0016]      FIG. 4  shows a GUI display in accordance with another embodiment of the invention where the camera view shows the camera in an Infrared (IR) Mode; 
           [0017]      FIG. 5  shows a model of a software architecture framework that translates data into a common and configurable feed in accordance with one embodiment of the invention; 
           [0018]      FIG. 6  shows a high-level overview of the software architecture in accordance with one embodiment of the invention; 
           [0019]      FIG. 7  shows a high-level view of the directories of the code of one embodiment of the invention; 
           [0020]      FIG. 8  shows the directory tree of the contents of the away3d directory; 
           [0021]      FIG. 9  shows the contents of the audio directory in the away3d directory; 
           [0022]      FIG. 10  shows the contents of the cameras directory in the away3d directory; 
           [0023]      FIG. 11  shows the contents of the containers directory in the away3d directory; 
           [0024]      FIG. 12  shows the contents of the core directory in the away3d directory; 
           [0025]      FIG. 13  shows the contents of the debug directory in the away3d directory; 
           [0026]      FIG. 14  shows the contents of the events directory in the away3d directory; 
           [0027]      FIG. 15  shows the contents of the exporters directory in the away3d directory; 
           [0028]      FIG. 16  shows the contents of the extrusions directory in the away3d directory; 
           [0029]      FIG. 17  shows the contents of the graphs directory in the away3d directory; 
           [0030]      FIG. 18  shows the contents of the lights directory in the away3d directory; 
           [0031]      FIG. 19  shows the contents of the loaders directory in the away3d directory; 
           [0032]      FIG. 20  shows the contents of the materials directory in the away3d directory; 
           [0033]      FIG. 21  shows the contents of the modifiers directory in the away3d directory; 
           [0034]      FIG. 22  shows the contents of the overlays directory in the away3d directory; 
           [0035]      FIG. 23  shows the contents of the physics directory in the away3d directory; 
           [0036]      FIG. 24  shows the contents of the primitives directory in the away3d directory; 
           [0037]      FIG. 25  shows the contents of the sprites directory in the away3d directory; 
           [0038]      FIG. 26  shows the contents of the test directory in the away3d directory; 
           [0039]      FIG. 27  shows the contents of the tools directory in the away3d directory; 
           [0040]      FIG. 28  shows the contents of the controller directory; 
           [0041]      FIG. 29  shows a representation of the contents of FLIRServiceStartupCommand; 
           [0042]      FIG. 30  shows a representation of the contents of NemaParseCommand; 
           [0043]      FIG. 31  shows the contents of the events directory; 
           [0044]      FIG. 32  shows a representation of the contents of AzUpdateEvent; 
           [0045]      FIG. 33  shows a representation of the contents of CameraUpdateEvent; 
           [0046]      FIG. 34  shows a representation of the contents of ElUpdateEvent; 
           [0047]      FIG. 35  shows a representation of the contents of FeedbackMessage; 
           [0048]      FIG. 36  shows a representation of the contents of FOVEvent; 
           [0049]      FIG. 37  shows a representation of the contents of NemaUpdateEvent; 
           [0050]      FIG. 38  shows a representation of the contents of TargetUpdateEvent; 
           [0051]      FIG. 39  shows the contents of the fonts directory; 
           [0052]      FIG. 40  shows a representation of the contents of MyClearviewHwy; 
           [0053]      FIG. 41  shows the contents of the Interface3 directory; 
           [0054]      FIG. 42  shows a representation of the contents of InterfaceContext; 
           [0055]      FIG. 43  shows the contents of the model directory; 
           [0056]      FIG. 44  shows a representation of the contents of Interface3Model; 
           [0057]      FIG. 45  shows the contents of the services directory; 
           [0058]      FIG. 46  shows a representation of the contents of CameraService; 
           [0059]      FIG. 47  shows a representation of the contents of FLIRService; 
           [0060]      FIG. 48  shows the contents of the views directory; 
           [0061]      FIG. 49  shows a representation of the contents of ArrowIndicator; 
           [0062]      FIG. 50  shows a representation of the contents of Boat; 
           [0063]      FIG. 51  shows a representation of the contents of Compass; 
           [0064]      FIG. 52  shows a representation of the contents of FOVBar; 
           [0065]      FIG. 53  shows a representation of the contents of FOVBarMediator; 
           [0066]      FIG. 54  shows a representation of the contents of Gimble3D; 
           [0067]      FIG. 55  shows a representation of the contents of Gimble3DMediator; 
           [0068]      FIG. 56  shows a representation of the contents of GimbleUI; 
           [0069]      FIG. 57  shows a representation of the contents of Interface3Mediator; 
           [0070]      FIG. 58  shows a representation of the contents of Reticle; 
           [0071]      FIG. 59  shows a representation of the contents of SystemStateMediator; 
           [0072]      FIG. 60  shows a representation of the contents of SystemStateUI; 
           [0073]      FIG. 61  shows a representation of the contents of TargetDistanceMediator; 
           [0074]      FIG. 62  shows a representation of the contents of TargetDistanceView; 
           [0075]      FIG. 63  shows a representation of the contents of TargetUI; 
           [0076]      FIG. 64  shows a representation of the contents of TargetUIMediator; and 
           [0077]      FIG. 65  shows a representation of the contents of Interface3Rebuild. 
       
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
       [0078]    The embodiments of the invention described herein are not intended to be exhaustive or to limit the invention to precise forms disclosed. Rather, the embodiments selected for description have been chosen to enable one skilled in the art to practice the invention. 
         [0079]    One exemplary system accepts video and data from a wide array of systems utilizing multiple formats. This video and data is then displayed via the GUI. Data transmission formats and characteristics can vary among multiple OEM vendors. The invention may operate with commercially standard formats that can comprise, for example, 90% of the EO devices. Embodiments of the invention can include a system where power required to operate an exemplary Intuitive Interface Initiative (3I) system would need to be developed around military standards and also would need to operate independently of craft/vehicle power sources. Exemplary GUI embodiments in accordance with an embodiment of the invention are intuitive and easy to interpret. Exemplary embodiments display information unobtrusively to allow effective operation of the EO sensor. 
         [0080]    Hardware architecture required to accomplish 3I objectives is flexible and adaptive to cover the broad spectrum of existing systems. Embodiments of the 3I system also may be upgraded to evolve as technology continues to increase performance of the system. 
         [0081]    One embodiment of the invention uses an object oriented architecture written in Actionscript 3.0 language (AS3). The exemplary framework was written to be modular which allows the code to be updated to add additional devices by writing a “translator” class. This class converts the device&#39;s raw data into a format that the software&#39;s “parser” class could use and distribute to the device specific objects displayed in the user&#39;s GUI. The device would then be added to the framework&#39;s library of compatible devices that the software recognizes along with any device specific GUI assets (target reticles, icons, etc.). 
         [0082]    Many advantages are realized from implementation of an example of a 3I system including an increase in effectiveness. Data displays can be designed and tailored to meet different user&#39;s needs. Consistency of overlays can be realized for a wide variety of systems and sensors. Response times to interpret and act on data can be reduced. Training costs and curriculum can be standardized and on the job training reduced resulting in minimal cost for changes to an exemplary overlay as opposed to utilizing OEM contractors to implement. As new systems are fielded to replace obsolete or ineffective systems, minimal impacts will be realized by the operators related to an exemplary video overlay. As video is ported off the EO system to be displayed by other users, multiple devices can be used. Screen over clutter can be tailored and reduced to meet the needs of the operator. Multiple sensors can feed data to a common display reducing operator workload and duplicative display costs. Customization can be realized by operators to efficiently get the information needed to complete objectives and thus reducing errors or misinterpretation of the data. 
         [0083]    An embodiment of a  31  system provides capabilities including an ability to make both the video and serial data from a common off-the-shelf (COTS) camera available to a single application. In this example, combinations of hardware for conversion and type of code chosen (e.g., AS3) allows for the capabilities for 3I&#39;s utilization in many types of operating systems and devices. 
         [0084]    An embodiment of 3I can be modified and used for any system that receives data and displays to an operator. The system could adapt to provide individual overlays for sensors on radars, lethal effectors, less than lethal effectors, global positioning systems, other visual augmentation systems, and any other military hardware that provides a video source or interface with information displayed on the screen. 
         [0085]    Referring to  FIG. 1 , an exemplary overview of the hardware and some aspects of software architecture of the Translator Device  10  is shown. An assembly  31  is shown with a variety of inputs, outputs, and internal components. Translator Device  10  has Data Ports  1 , which allow Translator Device  10  to receive data feeds from Sensor Devices  5 . Data received through the Data Ports  1  is then sent to the Software  13 . Within the Software, interface software, such as for example, Modular Mission Payload Architecture (MMPA) Software  11 , provides an interface between a sensor, e.g., Sensor Device  5 , and other software systems such as Operator Monitor  26  or Client Systems  28 . In this example, MMPA Software  11  detects which Sensor Device  5  attached to Data Port  1 . MMPA Software  11  then selects the appropriate Device Driver  14   a - 14   c  to use with the Sensor Device  5 . Then the selected Device Driver  14   a - 14   c  is used to convert the data from the Sensor Device  5  into a standardized format. The data is then sent to Flash Software  12 . Flash Software  12  uses that data to create text and symbology overlays. 
         [0086]    The Translator Device  10  also has Video-in Ports  3  which allows the Translator Device  10  to read video data from Sensor Devices  5 . This video information is then sent to a Software Development Kit (SDK) Frame Grabber  9  and a Hardware Encoder  7 . SDK Frame Grabber  9  combines the video with the text and symbology overlay from the Flash Software  12  and sends the combined data and video to Video Out Port  24 . Video Out Port  24  is connected to one or more Operator Monitor  26  to display the combined information. 
         [0087]    In this example, Hardware Encoder  7  converts the red, green, blue (RGB) video to H.264. The H.264 data is then combined with the text and symbology overlay from the flash software  12  and sent to Ethernet Switch  16 . This allows the information to be sent out through a RJ45 Port  20  to Client Systems  28 . Alternatively, information may be sent out via a Wireless Transmitter  18  which is connected to an Antenna Jack  22  to transmit the information to be picked up by Mobile Client Systems  30 . 
         [0088]      FIG. 2  details an exemplary  31  architecture with one sensor type, e.g., forward looking infrared (FLIR) laser camera software classes. Referring to  FIG. 2 , an exemplary EO Device  40  is connected to an Operator/Gunner Configuration system  44 . EO Device  40  sends data and video to a Ruggedized Mobile Computing Device, e.g., a Personal Computer (PC)  52 , which transmits data to Monitor  54  for display. Rugged PC  52  also receives information from a Controller  56  which allows a user to control where EO Device  40  is aimed. Video from the EO device  40  is also sent to a Video Controller  42  which sends video data to a Client Configuration System  50 . This exemplary Client Configuration System  50  allows data to be sent through Wireless Network  58  to be displayed on Mobile Devices  60 . Client Configuration System  50  also allows data to be sent via Wired Network  62  for display on Computers  64 . 
         [0089]      FIGS. 3 and 4  show different potential embodiments of a standardized GUI. In these example embodiments, the combined video information and symbology overlays are displayed in the User View  200 . In User View  200 , a Targeting Reticule  210  is provided which is located over a video image of a Target  205 . Targeting Reticle  210  also displays the range to Target  205 . An exemplary GUI also contains a Target Text Box  220  which displays Target&#39;s current position in latitude and longitude, and bearing and distance from the Sensory Device  5 . 
         [0090]    Following Craft Text Box  240  displays information about a device associated with the GUI, e.g., EO Device  40 , including location in latitude and longitude and date and time the video is captured. An Orientation Element  260  displays a graphical representation of a gyroscopic orientation of a device associated with an exemplary GUI, e.g., direction of a field of view of a sensor associated with an EO Device  40 . Compass  230  further displays the direction of orientation of the EO Device  40 . Camera Status Icons  250  allows a user to quickly view the state of the EO Device  40 . Camera Status Icons  250  update with regards to the EO Device  40  attached, only displaying those Icons that are available for use with an attached EO Device  40 . The Camera Status Icons  250  can include Contrast  251 , Brightness  252 , Camera Mode  253 , and Focus  254 . 
         [0091]      FIG. 4  shows User View  201  for when a FLIR camera attached and is engaged in Infrared Mode. In this mode, the video feed from the camera is changed. This change in mode is shown by Camera Mode Icon  253  which in  FIG. 4  is shown as IR. Also a Magnification Icon  255  may also be used. Unlike the other Camera Status Icons  251 - 254 , some Icons like Magnification  255  and GPS (not shown) are only visible when those modes are activated or the devices attached support them. 
         [0092]      FIG. 5  shows an exemplary programming interface in accordance with one embodiment of the invention. In particular,  FIG. 5  shows exemplary software code organized by classes along with high level communication flow for segments of software code within the Context  53 . In this embodiment the application utilizes the Robotlegs coding architecture and 3rd party plugins/engines. Robotlegs is an Action Script 3 architecture/framework that utilizes automated metadata based dependency injection. The event driven Robotlegs framework made the 3I code modular enough to enable the addition of future functionality with new sensors. 
         [0093]    Context  53  initializes the dependency injection between Mediators (e.g.  83 - 97 ) and Views (e.g.  101 - 119 ). StartupCommand  51  activates FLIRService  55  and NemaParseCommand  63 . FLIRService  55  defines the data socket settings including an internet protocol (IP) address and an assigned computer interface port address, connects to a socket service provided by an operating system on a computer e.g., the PC  52  or client system  28 , and configures the FLIR communication to be transmitted. When a message comes in from FLIR  57 , FLIRService  55  stores the data from a socket in buffer, and parses the socket data. Then NemaUpdateEvent  59  gets socket data as a NMEA string which is based on the NMEA 0183 standard. NemaUpdateEvent  59  sends that string to NemaParseCommand  63  which parses the NMEA string and dispatches events with the parsed data to update the values associated with the parsed values. NemaParseCommand  63  also activates Interface3Model  85 , which gets and sets azimuth, elevation, system mode, bit status, tracker mode, video sensor, sensor type, field of view, current zoom, and range. 
         [0094]    Events associated with NemaParseCommand  63  include AzUpdateEvent  69  (updates azimuth value), ElUpdateEvent  71  (updates elevation value), FOVEvent  73  (updates field of view value), CameraUpdateEvent  75  (updates camera type), TargetUpdateEvent (updates range value)  77 , and FeedbackMessage (returns a message string)  81 . 
         [0095]    FOVBarMediator  87  waits for an update from FOVEvent  73  to update the scale of field-of-view mask in FOVBar  107 . FOVBar  107  then defines, generates and animates the field-of-view components to be displayed. 
         [0096]    Gimble3DMediator  85  waits for an update from AzUpdateEvent  69  to update the azimuth in Gimble3D  105 . Gimble3DMediator  85  also waits for an update from ElUpdateEvent  71  to update the elevation in Gimble3D  105 . Gimble3D  105  defines, generates, and animates the camera graphics to be displayed. 
         [0097]    Interface3Mediator  83  waits for an update from TargetUpdateEvent  77  to update the target range objects. Interface3Mediator  83  also interacts with Interface3  101  to create GUI objects shown in  FIGS. 3 &amp; 4 . 
         [0098]    SystemStateMediator  93  waits for an update from CameraUpdateEvent  75  to update a camera type in SystemStateUI  113 . SystemStateUI  113  defines and generates Camera Status Icons  250 . 
         [0099]    TargetDistanceMediator  97  waits for an update from TargetUpdateEvent  77  to update target reticle range on screen values in TargetDistanceView  117 . TargetDistanceView  117  then defines and generates target reticle range data as seen in close proximity to Targeting Reticle  210 . 
         [0100]    TargetUIMediator  95  waits for an update from TargetUpdateEvent  77  to update the range to the target history data in TargetUI  115 . TargetUI  115  then defines and generates target history data on the lower right side of the screen in Target Text Box  220 . 
         [0101]    GimbleUIMediator  89  interacts with the GimbleUI  109  to define and generate the following craft UI of time, latitude, and longitude as seen in the Following Craft Text Box  240 . 
         [0102]    ReticleMediator  91  interacts with Reticle  111  to display Targeting Reticle  210  on the screen. 
         [0103]    Compass  119  defines, generates, and animates Compass  230 . 
         [0104]      FIG. 6  shows a functional diagram of computer software modules for a system in accordance with one embodiment of the invention. Context  53  initializes dependency injection and other utilities. It is the beginning of the application. Context  53  initializes the Commands  152  and matches Mediators  157  with Views  158 . Commands  152  represent individual actions that the application can perform. Commands  152  initialize Events  154  and Model  157 . In this embodiment, Commands  152  consists of StartupCommand  51  and NemaParseCommand  63 . 
         [0105]    Events  154  are used to pass parameters to event listeners when an event occurs. Events  154  in this embodiment include AzUpdateEvent  69 , ElUpdate Event  71 , FOVEvent  73 , CameraUpdateEvent  75 , TargetUpdateEvent  77 , NemaUpdateEvent  59 , and FeedbackMessage  81 . 
         [0106]    Event, e.g.,  154 , are passing parameters, e.g.,  81 , to Mediators  157  which in turn are stored in Model  164  and passed to Views  158  by Mediators  157 . Model  164  stores data and represents the current state of the application. Interface3Model  85  stores parameters including azimuth, elevation, system mode, bit status, tracker mode, video sensor, sensor type, field of view, current zoom, and range. 
         [0107]    Mediators  157  manage data between the application view components, Views  158 , and other objects within the application. As shown in  FIG. 5 , there are several Mediators which interact with the Views, each keeping track of a different Event  154  and parameter, which is then used to update the appropriate View  158 . Mediators  157  include Interface3Mediator  83 , Gimble3DMediator  85 , FOVBarMediator  87 , GimbleUIMediator  89 , ReticleMediator  91 , SystemStateMediator  93 , TargetUIMediator  95  and TargetDistanceMediator  97 . 
         [0108]    Views  158  are viewable components that contain methods that can be called by the Mediators  157 . Views  158  interact with Images  160  and Away3d  162  to draw the images and text seen on the screen. Views  158  also retrieve the data from the Model  164  to display on the screen. In this embodiment, Views  158  include ArrowIndicator (defines characteristics of compass arrow generation), Boat (displays boat graphic), Compass  119 , FOVBar  107 , Gimble3D  105 , GimbleUI  109 , Interface3  101 , Reticle  111 , SystemStateUI  113 , TargetDistanceView  117 , and TargetUI  115 . 
         [0109]    Services  156  are used to communicate with programs and/or devices outside the application to get data for the models. In this embodiment CameraService defines the video stream settings and FLIRService  55  sets up and communicates with the FLIR Camera  57 . 
         [0110]    Images  160  contain basic images such as buttons. Away3d  162  contains the classes that draw on the display and show the video. 
         [0111]    The Robotlegs architecture of this embodiment also includes several plug-ins, including Away3D, TweenLite, and TweenMax. Away3D is an open source 3D engine for Adobe Flash used to render 3D models and perform additional 3D related tasking. Away3D was used in the  31  software to implement 3D models such as the camera and Taurus graphic. TweenLite is a fast, lightweight, tweening engine that handles the movement of graphics and animations. TweenMax is a fast, lightweight, tweening engine with additional capabilities and handles the movement of graphics and animations. 
         [0112]      FIGS. 7-28 ,  31 ,  39 ,  41 ,  43 ,  45 , and  48  show various views of an exemplary file structure of example code of one embodiment of the invention. 
         [0113]      FIGS. 29 and 30  show representations of the code in the controller directory. 
         [0114]      FIGS. 32-38  show representations of the code in the events directory. 
         [0115]      FIGS. 40 ,  42  and  44  show representations of the code in the fonts, interface3 and model directories. 
         [0116]      FIGS. 46-47  show representations of the code in the services directory. 
         [0117]      FIGS. 49-64  show representations of the code in the views directory. 
         [0118]      FIG. 65  shows a representation of the Interface3Rebuild. 
         [0119]    Although the invention has been described in detail with reference to certain preferred embodiments, variations and modifications exist within the spirit and scope of the invention as described and defined in the following claims.