Patent Description:
Head wearable displays are used in various circumstances to provide symbology that enhances the information content of viewed scenes. In particular, pilot head wearable displays provide crucial flight information which is presented conformally to the viewed scene and to objects in the scene.

<CIT> and <CIT>, by BAE Systems PLC, disclose a method and system for aligning a helmet mountable display with respect to a helmet. The method comprises the steps of: aligning a first helmet mounted display guide symbol with a reference direction; selecting the alignment of the first helmet mounted display guide symbol with the reference direction; aligning a second helmet mounted display guide symbol with the reference direction; and, selecting the alignment of the second helmet mounted display guide symbol with the reference direction.

The following is a simplified summary providing an initial understanding of the invention. The summary does not necessarily identify key elements nor limit the scope of the invention, but merely serves as an introduction to the following description.

One aspect of the present invention provides a method of automatic reliability verification of a head wearable display "HWD" operated by a user, the method comprising: tracking a position and orientation "P&O" of the HWD and of a camera affixed thereto, relative to at least one hardware element having a known position, which is selected from a surroundings of the user and is visible to the user through the HWD, capturing images of the at least one hardware element by the camera, calculating at least one position of the at least one selected hardware element with respect to the tracked P&O of the HWD and of the camera, and calculating therefrom at least one expected imaged location of the at least one hardware element, processing the captured images to derive respective at least one imaged location of the at least one hardware element, and indicating a discrepancy between the at least one derived imaged location and the at least one expected imaged location, wherein at least one of the tracking, the calculating, the processing and the indicating is carried out by at least one computer processor.

A further aspect of the present invention provides an automated HWD reliability verification kit comprising: a camera affixed to a HWD and configured to capture at least one hardware element having at least one known position, wherein the at least one hardware element is selected from a surroundings of a user operating the HWD and is visible to the user through the HWD, an image processing module configured to process the captured images to derive respective at least one imaged location of the at least one hardware element, a position and orientation P&O tracker configured to track the P&O of the HWD and of the camera, and a display reliability verification module configured to calculate at least one position of the at least one hardware element with respect to the tracked P&O of the HWD and of the camera, calculate therefrom at least one expected imaged location of the at least one hardware element, and to indicate a discrepancy between the at least one derived imaged location and the at least one expected imaged location.

These, additional, and/or other aspects and/or advantages of the present invention are set forth in the detailed description which follows; possibly inferable from the detailed description; and/or learnable by practice of the present invention.

Prior to the detailed description being set forth, it may be helpful to set forth definitions of certain terms that will be used hereinafter.

The term "hardware element" as used in this application refers to any element that has a known position (possibly fixed elements) in surroundings of a user such as an aircraft cockpit or any other surrounding in which the user operated the HWD. Hardware elements, as used herein, must be visible to the user during the operation of the HWD, e.g., visible to a pilot during flight. The term "hardware element" as used in this application may also refer to geometric elements which relate to fixed elements in the surroundings, such as the geometric shapes of hardware elements in the surroundings, e.g., a shape of a display or a window.

The term "coordinate system" as used in this application refers to a frame of reference which maybe associated and fixed with respect to the hardware element or any other frame of reference.

The term "head wearable display system" as used in this application refers to an operative combination of a head wearable display (HWD) and associated tracker unit(s) and computer processor(s) that provide a conformal symbology display on the HWD. The term "tracker" as used in this application refers to one or more units that measure a position and orientation (P&O) of objects such as the HWD and associated elements in the coordinate system of the cockpit or the aircraft. The term "user" as used in this application refers to any person operating the HWD. While examples are given of pilots operating the HWD in a cockpit, user may be any operators of any vehicle or system, as well as persons using the HWD irrespective of any platform.

In the following description, various aspects of the present invention are described. For purposes of explanation, specific configurations and details are set forth in order to provide a thorough understanding of the present invention. However, it will also be apparent to one skilled in the art that the present invention may be practiced without the specific details presented herein. Furthermore, well known features may have been omitted or simplified in order not to obscure the present invention. With specific reference to the drawings, it is stressed that the particulars shown are by way of example and for purposes of illustrative discussion of the present invention only, and are presented in the cause of providing what is believed to be the most useful and readily understood description of the principles and conceptual aspects of the invention. In this regard, no attempt is made to show structural details of the invention in more detail than is necessary for a fundamental understanding of the invention, the description taken with the drawings making apparent to those skilled in the art how the several forms of the invention may be embodied in practice.

Before at least one embodiment of the invention is explained in detail, it is to be understood that the invention is not limited in its application to the details of construction and the arrangement of the components set forth in the following description or illustrated in the drawings. The invention is applicable to other embodiments that may be practiced or carried out in various ways as well as to combinations of the disclosed embodiments.

Unless specifically stated otherwise, as apparent from the following discussions, it is appreciated that throughout the specification discussions utilizing terms such as "processing", "computing", "calculating", "determining", "enhancing" or the like, refer to the action and/or processes of a computer or computing system, or similar electronic computing device, that manipulates and/or transforms data represented as physical, such as electronic, quantities within the computing system's registers and/or memories into other data similarly represented as physical quantities within the computing system's memories, registers or other such information storage, transmission or display devices. Any of the disclosed modules or units may be at least partially implemented by a computer processor.

Displaying conformal symbology over a head wearable display (HWD such as HMD - head mounted display, NTE - near to eye, etc.) involves a HWD tracking system which calculates the HWD orientation and enables presenting the symbology over the HWD in a conformal manner with respect to the real world. The conformity of the symbology to the real world may be critical, with lack of conformity rendering the HWD unreliable. For example, in civil aviation, displaying a non-conformal Flight Path Symbol (FPV) over a head up display may result, in certain phases of flight, in a catastrophic event. Other examples include symbology of a horizon line, of landing place parameters or of other aircrafts - which must represent correctly the respective real world items in order for the symbology and HWD to be reliable. In the following it is suggested to use a special symbol - 'sanity symbol' - to ensure that the HWD tracking system is functioning correctly and its output does not lead to non-conformal and unreliable symbology. Manual verification that the location of the symbol over a pre-defined fixed target is correct provides a verification of the correctness of the HWD tracking system. The 'sanity symbol' may be utilized to detect other failure conditions that may lead to a misleading, non-conformal symbology, such as optics/visor movements and display/image shift.

The 'sanity symbol' is a conformal symbol over a fixed target. The fixed target is chosen such that it doesn't change its location relative to the aircraft. The 'sanity symbol' is drawn on the HWD in a position that corresponds, conformally, with the fixed target (when the pilot looks at the fixed target, the symbol is visually aligned with the fixed target).

Head wearable displays (HWDs) and methods of display reliability verification with respect to the HWD are provided, which comprise selecting hardware element(s) (e.g., cockpit hardware elements or aircraft parts outside the cockpit, which are visible to the pilot during the flight, such as the aircraft's nose) with known position(s) in a coordinate system that is fixed relative to the aircraft, tracking a position and orientation (P&O) of the HWD in the coordinate system, calculating a position of the selected hardware element(s) with respect to HWD, and displaying verification symbol(s) in the HWD that correspond to the hardware element(s), such that as long as the HWD P&O is accurately tracked, the verification symbol(s) appear to the pilot to be in visual alignment with the corresponding selected hardware element(s). Identification, by the pilot, of deviating verification symbols, may be used to initiate a correction process or merely to indicate that the presented symbology should not be relied upon when considering future actions. Display reliability verification may be carried out automatically using a camera and image processing, optionally with an additional illuminator, to derive the element(s)'s position independently of the HWD and indicate possible discrepancies.

<FIG> presents high level schematic illustrations of a cockpit <NUM>, according to the prior art. Cockpit <NUM> has various static hardware elements <NUM> and provides a cockpit spatial reference frame with a coordinate system <NUM> that is fixed relative to the aircraft. Hardware elements <NUM> which are fixed in the aircraft and visible to the pilot during flight are also shown schematically.

<FIG> is a high level schematic illustration of a cockpit as seen through the HWD, with a verification symbol <NUM> displayed conformally to a hardware element <NUM>, according to some embodiments of the invention. Hardware element <NUM> may be a beam between cockpit windows, a ball element on the beam, or any other element in the cockpit, or any element that is fixed in an aircraft, which is visible to the pilot during flight (see e.g., <FIG>). Hardware element <NUM> may comprise multiple elements <NUM>, associated with coordinate system <NUM>, being a frame of reference which is associated and fixed with respect to hardware element <NUM>. <FIG> schematically illustrates view <NUM> of cockpit <NUM> with overlaid HWD symbology <NUM>, from which, for simplicity, only verification symbol <NUM> is shown (HWD <NUM> itself is shown in <FIG> and <FIG>). Hardware element <NUM> may be selected to be a well observed and identified point in the cockpit or on visible aircraft parts, e.g., a distinguishable point or an element between the front windows.

When the pilot looks toward the hardware element (as a fixed target), verification ("sanity") symbol <NUM> is displayed overlaid on the fixed target (hardware element <NUM>) and/or the fixed target (hardware element <NUM>) is located inside a perimeter of verification ("sanity") symbol <NUM>. If the fixed target is not inside the perimeter of the sanity symbol, the symbology may be non-conformal and should not be relied upon. The pilot may responsively ignore the symbology, switch off the display and/or initiate a correction process.

It is noted that hardware element(s) <NUM> may comprise geometric elements which relate to fixed elements in the surroundings, such as the geometric shapes or contours of hardware elements in the surroundings, e.g., a shape of a display or a window, as indicated e.g., in <FIG> or of visible features outside the windows, such protrusions from the aircraft nose. Selected hardware element(s) <NUM> are visible to a user operating the HWD during operation.

Moreover, verification symbol(s) <NUM> may comprise graphical elements that indicate different levels of tolerance, or thresholds, to the degree of overlap of verification symbol(s) <NUM> and hardware element(s) <NUM>. For example, as illustrated in <FIG>, verification symbol(s) <NUM> may comprise an inner circle <NUM>-<NUM> indicating close tolerance (high threshold) and an outer circle <NUM>-<NUM> indicating relaxed tolerance (low threshold) to the degree of required conformity.

<FIG> schematically illustrate wrong and correct positioning, respectively, of verification symbol <NUM> with respect to hardware element <NUM>, according to some embodiments of the invention. In <FIG>, verification symbol <NUM> is positioned off-center with respect to hardware element <NUM> and indicates thereby that symbology <NUM> provided by HWD <NUM> is wrong. In <FIG>, verification symbol <NUM> is centered with respect to hardware element <NUM> and indicates thereby that symbology <NUM> provided by HWD <NUM> is correct.

<FIG> and <FIG> are high level schematic illustrations of a HWD system <NUM> used by a pilot <NUM>, according to some embodiments of the invention. HWD system <NUM> comprises a HWD <NUM> and one or more position and orientation (P&O) tracker <NUM> configured to track the P&O of HWD <NUM> relative to coordinate system <NUM> that is fixed relative to the aircraft. It is noted that tracking the HWD may be carried out in any coordinate system. P&O tracker <NUM> may comprise a tracker element fixed to the cockpit, a tracker element fixed to HWD <NUM> and a tracker processor configured to provide the tracking data. Any of these elements is considered part of HWD system <NUM> and/or HWD <NUM>. HWD system <NUM> is further configured to calculate the position of selected at least one hardware element <NUM> with respect to HWD <NUM> (e.g., by processor <NUM>). HWD system <NUM> and/or HWD <NUM> further comprises a display reliability verification module <NUM> configured to display, as an addition to symbology <NUM> displayed on HWD <NUM>, at least one verification symbol <NUM> that corresponds to selected at least one hardware element <NUM>. Verification symbol(s) <NUM> are presented at locations that allow immediate identification of deviations thereof from the positions of corresponding hardware element(s) <NUM> so that as long as the HWD P&O is accurately tracked, verification symbol(s) <NUM> appear to the pilot to be in visual alignment with corresponding hardware element(s) <NUM>. A misalignment of verification symbol(s) <NUM> with respect to the corresponding hardware element(s) <NUM> may be used to indicate unreliability of HWD <NUM>. It is noted that the misalignment is with respect to the expected position of symbol <NUM> and the position of corresponding elements <NUM>, and is referred to herein also by the term misplacement. Similar configurations may be applied to any HWD system <NUM>, with HWD <NUM> operated by any user, with respect to any type of user surroundings.

<FIG> shows schematically two hardware elements 91A, 91B with associated verification symbols 111A, 111B, respectively, displayed along the lines of sight (LOS's) 85A, 85B of pilot <NUM> to elements 91A, 91B, respectively, as calculated from the HWD P&O and the positions of the hardware elements. It is noted that while regular symbology of head wearable displays usually requires only pilot head orientation information as the targets are considered to be at optical infinity, verification symbols <NUM> relate to physical elements at optical proximity to the head wearable display and hence the presentation thereof requires relating to the HWD position as well as to the HWD orientation. HWD <NUM> may be at least partially associated with a computer processor <NUM> and/or computer processor <NUM> may be part of HWD <NUM>. Computer processor <NUM> may carry out at least some of the functionality of HWD <NUM> such as managing the display of verification symbol(s) <NUM>, tracking the P&O of the pilot HWD and interfacing with pilot <NUM>. HWD system <NUM> may comprise any number of processors <NUM>, which may be dedicated to different tasks such as tracking the HWD, providing the symbology, providing the verification symbol(s) etc..

<FIG> schematically illustrates an additional camera <NUM> associated with or affixed to HWD <NUM> at a constant spatial relation (P&O relative to the HWD coordinate system). Camera <NUM> may be configured to continuously capture images <NUM> of hardware elements <NUM> in the cockpit and/or of visible aircraft parts. It is noted that aircraft parts outside the cockpit, such as hardware elements on the aircraft's nose which are visible from the pilot's position, may be used as hardware elements <NUM> in certain embodiments. Camera <NUM> is further associated with an image-processing module <NUM> that receives captured images <NUM> from camera <NUM> and is configured to identify hardware element(s) <NUM> in images <NUM> and compare the location of hardware element(s) <NUM> in captured images <NUM> to their expected locations, which are derived from the camera's known P&O relative to the HWD coordinate system, the calculated P&O of HWD <NUM> relative to coordinate system <NUM>, and the known position of hardware element(s) <NUM> in coordinate system <NUM>.

Head wearable display <NUM> may be further configured to receive user input concerning a misalignment of verification symbol(s) <NUM> with respect to the position(s) of hardware element(s) <NUM> and to identify an error source related to the misalignment. Head wearable display <NUM> may be further configured to correct software identified as being associated with the error source and/or suggest hardware corrections identified as being associated with the error source. For instance, when tracker <NUM> is an optical tracker, if the user indicates a misalignment, HWD <NUM> may be configured to suggest to the user (e.g., via a message on the display) to clean the lens of tracker unit <NUM> (the error source may be dirt on tracker unit <NUM>, e.g. dust on the lens of a tracker camera may cause an erroneous P&O).

Certain embodiments comprise an automated HWD reliability verification kit <NUM> comprising one or more camera <NUM> affixed to HWD <NUM> and configured to capture at least one selected hardware element <NUM>, and image processing module <NUM> configured to process images <NUM> of at least one hardware element <NUM> which is/are captured by camera <NUM> and to derive an imaged location of at least one hardware element <NUM>. Image processing module <NUM> and/or HWD <NUM> may be further configured to indicate a discrepancy between the derived imaged location of hardware element(s) <NUM> and the expected imaged location thereof. Automated HWD reliability verification kit <NUM> may further comprise at least one retroreflector <NUM>, associated with corresponding at least one hardware element <NUM> (91B in <FIG>), and an illuminator <NUM> associated with camera <NUM> and configured to illuminate at least one cockpit hardware element <NUM>. Images <NUM> captured by camera <NUM> may comprise hardware element images 151A and/or illumination reflections 151B from retroreflector(s) <NUM>, and image processing module <NUM> may be configured to derive the imaged location using retroreflected illumination 151B as well as hardware element images 151A. Discrepancies may be used to indicate non-conformity and unreliability of the HWD symbology <NUM> and/or an error source in HWD <NUM>, and may be corrected by software or hardware.

<FIG> is a high level flowchart illustrating a method <NUM> of display reliability verification with respect to a head wearable display, according to some embodiments of the invention. The method stages may be carried out with respect to head wearable display <NUM> described above, which may optionally be configured to implement method <NUM>. Method <NUM> may be at least partially implemented by at least one computer processor <NUM>. Certain embodiments comprise computer program products comprising a computer readable storage medium having computer readable program embodied therewith and configured to carry out of the relevant stages of method <NUM>.

Method <NUM> may comprise verifying display reliability by displaying conformal symbol(s) of cockpit element(s) (stage <NUM>). At least one hardware element with known position(s) in an associated coordinate system (which is fixed relative to the aircraft) may be selected (stage <NUM>) to be used as reference element(s). Selected hardware element(s) are visible to the user operating the HWD during operation. Method <NUM> may comprise tracking a position and orientation (P&O) of a HWD (stage <NUM>) relative to the coordinate system, calculating position(s) of the selected hardware element(s) with respect to the HWD (stage <NUM>) and displaying at least one verification symbol that corresponds to at least one hardware element in the HWD (stage <NUM>). Method <NUM> may comprise displaying the verification symbol(s) to appear to the pilot to be in visual alignment with the hardware element(s) as long as the HWD P&O is tracked accurately (stage <NUM>). A misplacement (or misalignment) of the at least one verification symbol with respect to the corresponding at least one hardware element may be used to indicate unreliability of the HWD. Method <NUM> may be applied to any HWD system <NUM>, with HWD <NUM> operated by any user, with respect to any type of user surroundings.

Method <NUM> may further comprise receiving a user input indicative of a misplacement of the at least one verification symbol with respect to at least one position of the at least one hardware element (stage <NUM>), and identifying an error source related to the misplacement (stage <NUM>). Method <NUM> may further comprise correcting software identified as being associated with the error source (stage <NUM>) and/or suggesting hardware corrections identified as being associated with the error source (stage <NUM>). Method <NUM> may further comprise verifying automatically the display reliability with respect to a pilot's HWD (stage <NUM>) by affixing a camera to the HWD to capture the at least one selected hardware element (stages <NUM>, <NUM>), tracking the P&O of the HWD and of the camera affixed thereto, relative to the coordinate system (stage <NUM>), calculating at least one position of the at least one hardware element with respect to the camera (stage <NUM>), calculating therefrom at least one expected imaged location of the at least one hardware element (stage <NUM>) based on the known position(s) of the hardware element(s), capturing images of the at least one hardware element by the camera (stage <NUM>), processing the captured images to derive respective at least one imaged location of the at least one hardware element (stage <NUM>), and indicating a discrepancy between the at least one derived imaged location and the at least one expected imaged location (stage <NUM>). Indications <NUM> may be used to point at possible error sources which may then be identified and then corrected by hardware and/or software corrections according to the indicated discrepancy (stages <NUM>, <NUM>, <NUM>, see above).

Method <NUM> may further comprise using at least one corresponding retroreflector as the at least one hardware element (stage <NUM>) or associating retroreflectors to selected hardware elements, illuminating the retroreflector(s) (stage <NUM>) and deriving the imaged location(s) using the retroreflected illumination (stage <NUM>).

Any of the stages of method <NUM> may be at least partially implemented using at least one computer processor (stage <NUM>).

Aspects of the present invention are described above with reference to flowchart illustrations and/or portion diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each portion of the flowchart illustrations and/or portion diagrams, and combinations of portions in the flowchart illustrations and/or portion diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or portion diagram portion or portions.

These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or portion diagram portion or portions.

The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or portion diagram portion or portions.

The aforementioned flowchart and diagrams illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each portion in the flowchart or portion diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the portion may occur out of the order noted in the figures. For example, two portions shown in succession may, in fact, be executed substantially concurrently, or the portions may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each portion of the portion diagrams and/or flowchart illustration, and combinations of portions in the portion diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

Claim 1:
A method of automatic reliability verification of a head wearable display "HWD" (<NUM>) operated by a user (<NUM>) within a cockpit of an aircraft, the method comprising:
tracking a position and orientation "P&O" of the HWD and of a camera (<NUM>) affixed thereto, during flight of said aircraft, relative to at least one hardware element having a known position, which is selected from a surroundings of the user and is visible to the user through the HWD (<NUM>) wherein the hardware element comprises one of: cockpit hardware elements or aircraft parts outside the cockpit, which are visible to the pilot during the flight,
capturing images (<NUM>) of the at least one hardware element (<NUM>) by the camera (<NUM>),
calculating at least one position of the at least one hardware element (<NUM>) with respect to the tracked P&O of the HWD (<NUM>) and of the camera (<NUM>), and calculating therefrom at least one expected imaged location of the at least one hardware element (<NUM>),
processing the captured images (<NUM>) to derive respective at least one imaged location of the at least one hardware element (<NUM>),
indicating a discrepancy between the at least one derived imaged location and the at least one expected imaged location, and
indicating possible sources of the discrepancy for identification and correction,
wherein the tracking, the calculating, the processing and the indicating are carried out by at least one computer processor.