Patent Description:
<CIT> discloses an optical see-through augmented reality modified-scale display method and system.

<CIT> discloses a helmet mounted display adaptive predictive tracking system.

It is known to include a helmet or head tracker system in a helmet or head-mounted display system in which the positioning of symbols or other artefacts in the display is to be determined according to orientation of the user's head relative to a movable reference frame, for example relative to a cockpit or other interior region of an aircraft or to a driver's or other position in a land vehicle or other type of vehicle.

Various different types of tracker system are known, using a variety of different sensor technologies. For example, <NPL>) describes a hybrid optical/inertial helmet tracker system for use with head and helmet-mounted displays. An example of a helmet tracker using electromagnetic techniques is available from Polhemus® Inc, for example as described in a White Paper entitled 'Accuracy and Resolution in Electromagnetic <NUM> Degree-of-freedom (6DOF) Measurement Systems', APB <NUM>-001A, available from http:///www. However, known tracker systems are not generally able to anticipate some of the particular needs of head or helmet-mounted digital display systems relying upon an associated tracking system to supply orientation data that can be used to generate and position space-stabilised symbols or other artefacts in the display, i.e. symbols that appear to a user of the display to be fixed in a far-field view of space, aligned with an externally visible point or object, despite head movement. In particular, known tracker systems are not resilient to certain operating conditions known to occur in aircraft, such as buffeting.

<CIT> relates to a method and system for providing an optical see-through Augmented Reality modified-scale display. This aspect includes a sensor suite which includes a compass, an inertial measuring unit, and a video camera for precise measurement of a user's current orientation and angular rotation rate. A sensor fusion module may be included to produce a unified estimate of the user's angular rotation rate and current orientation to be provided to an orientation and rate estimate module. The orientation and rate estimate module operates in a static or dynamic (prediction) mode. A render module receives an orientation; and the render module uses the orientation, a position from a position measuring system, and data from a database to render graphic images of an object in their correct orientations and positions in an optical display.

<CIT> relates to a system for predicting head movements by an aircrew member, such as a pilot flying a tactical aircraft, from input data which includes an ongoing record of head positions. The input is without head acceleration or velocity input data and is applied to a Least Mean Square prediction algorithm. The disclosure also includes an improved data dependent convergence algorithm for the prediction system, an algorithm shown to provide rapid convergence to the final prediction output. Use of the prediction system to make the delay between actually viewed and HMD presented images and other flight video images acceptable from a human engineering and disoriented pilot avoidance viewpoint is also disclosed.

The present invention is defined, in its various aspects, in the attached claims. In particular, according to a first aspect of the present invention, there is provided a tracker system for a head or helmet mounted digital display device arranged to display space-stabilised symbols or other space-stabilised display artefacts, the tracker system for determining orientation of an object, comprising the features of claim <NUM>, which among other features include:.

Advantageously, the present invention is able not only to help to overcome latency in responding to changes in orientation of an object by user systems, but also to detect certain operating conditions of buffeting or other induced rapid changes in orientation of the tracked object that would have potentially adverse effects upon the performance of known tracker systems, and to modify tracker operation to reduce their effects. Moreover, the present invention is able to detect the presence and severity of buffeting using the same source orientation rate of change data as used by the predictor component to determine orientation of the object.

In an example embodiment, the buffeting measurement component is arranged to calculate a measure of buffeting severity using a two-stage filtering process whereby, at a first stage, upon receipt of each new instance of rate data from the sensor arrangement, a difference is determined between a weighted average of prior-received rate data and a mid-history instance of received rate data and, at a second stage, the measure of buffeting severity in respect of the new instance of rate data is determined as the difference between a weighted average of the prior-determined differences and a mid-history determined difference.

In an example embodiment, the weighted averages are determined using a symmetric weighting profile applied across a history of the received rate data. For example, the symmetric weighting profile may comprise a half-sine-wave profile or a profile of uniform weighting.

Advantageously, the buffeting measurement component further comprises a third filter stage arranged to apply a low-pass filter with uniform weighting to the determined measures of buffeting severity.

In an example embodiment, the predictor component is arranged, upon receipt of a new instance of rate data from the sensor arrangement, to estimate an average rate of change in orientation of the object over an estimation time period ahead of a time point of validity of the new instance of rate data and to integrate the estimated average rate of change over an integration time period to give an estimate of the change in orientation of the object, adding the estimated change in orientation to an orientation determined by integrating the received new instance of rate data to give a predicted orientation of the object.

The buffeting compensation component may be arranged to adjust at least one of the estimate time period and the integration time period of the predictor component in dependence upon the determined measure of buffeting severity.

In another example embodiment, the tracker system may further comprise an output filter component arranged to receive output by the predictor component and to apply thereto a finite impulse response filter with adjustable latency and wherein the buffeting compensation component is arranged further to adjust the latency of the applied finite impulse response filter according to the determined measure of buffeting severity. The output filter may for example implement a Hanning finite impulse response filter.

In example embodiment, the object is a head or helmet or other body-worn item to which the inertial sensor arrangement is attached and the sensor arrangement is arranged to determine and to output rate data indicative of a rate of change in orientation of the head or helmet or other body-worn item.

According to a second aspect of the present invention there is provided a computer-implemented method for use with a tracker system for a head or helmet mounted digital display device arranged to display space-stabilised symbols or other space-stabilised display artefacts, the computer-implemented method determining orientation of an object, comprising the steps of claim <NUM>, which among other steps include:.

In an example embodiment of the method, at step (ii), the measure of buffeting severity is determined using a two-stage filtering process whereby, at a first stage, upon receipt of each new instance of rate data from the sensor arrangement at step (i), a difference is determined between a weighted average of prior-received rate data and a mid-history instance of received rate data and, at a second stage, the measure of buffeting severity in respect of the new instance of rate data is determined as the difference between a weighted average of the prior-determined differences and a mid-history determined difference.

The weighted averages may be determined using a symmetric weighting profile applied across a history of the received rate data. The symmetric weighting profile may for example comprise a half-sine-wave profile or a profile of uniform weighting.

In another example embodiment, step (ii) further comprises applying a low-pass filter with uniform weighting to the determined measure of buffeting severity.

In an example embodiment, the configurable prediction process comprises, upon receipt of a new instance of rate data at step (i), estimating an average rate of change in orientation of the object over an estimation time period ahead of a time point of validity of the received new instance of rate data and to integrate the estimated average rate of change over an integration time period to give an estimate of the change in orientation of the object, adding the estimated change in orientation to an orientation determined by integrating the received new instance of rate data to give a predicted orientation of the object.

Advantageously, at step (iii), at least one of the estimate time period and the integration time period of the configurable prediction process is adjusted in dependence upon the determined measure of buffeting severity.

In an example embodiment, the method further comprises the step of applying a finite impulse response filter with adjustable latency to an output by the predictor component wherein the latency of the finite impulse response filter is adjusted according to the determined measure of buffeting severity. The applied filter is a Hanning finite impulse response filter, for example.

According to an example embodiment, there is provided a head or helmet-mounted digital display device arranged to display space-stabilised symbols or other space-stabilised display artefacts, including a tracker system for determining orientation of the head or helmet according to the first aspect of the present invention in which the object is a head or a helmet on which is mounted the head or helmet-mounted display and wherein the predetermined prediction time period corresponds to an expected time of displaying a space-stabilised symbol or other space-stabilised display artefact positioned by the display device to take account of an orientation of the head or helmet indicated by an output of predicted orientation by the tracker system.

According to a further aspect of the present invention, there is provided a computer program product according to claim <NUM> and comprising, among other elements, a data carrier having stored thereon software code means, or means for access thereto, which when installed and executed upon a digital processor are arranged to implement the method steps according to the second aspect of the present invention.

According to a further aspect of the present invention, there is provided a digital processor programmed to implement the method steps according to the second aspect of the present invention.

Example embodiments of the present invention will now be described in more detail with reference to the accompanying drawings, of which:.

A system according to example embodiments of the present invention is arranged to determine and to monitor the orientation of an object taking account of the effects of buffeting or other effects involving high and varying induced rates of change in orientation of the object over short time periods. The orientation of the object may be determined in inertial space or relative to a moveable platform on which the object may be travelling or is otherwise associated. The object may be considered associated with the moveable platform if the magnitude of relative velocity of the object and the moveable platform is substantially zero at a time of determining the object's orientation relative to the platform. Embodiments of the present invention will be described below in the example context of a helmet tracker system intended for use with the helmet of a pilot of an aircraft. More particularly, embodiments of the present invention will be described in the context of such a helmet tracker system when used in association with a helmet-mounted display system. However, the present invention may be applied to the tracking of orientation of other types of object associated with a moveable platform.

The description that follows is of a logical system architecture designed to explain the principles underlying operation of a helmet tracker system incorporating embodiments of the present invention. It will be clear to a notional skilled person that the helmet tracker system may be implemented using an object-oriented data processing architecture in which functional components of the system may be combined and distributed differently to that in a logical sequential architecture to be described below. However, the principles of operation are likely to remain substantially unchanged and the description that follows may at least serve as a detailed requirements specification for the functions to be implemented in whatever data processing architecture is chosen.

For the purposes of the description that follows, the following conventions and terminology will be assumed for the labelling of axes and rotations. Axes will be labelled 'x', 'y' and 'z' and angles of rotation will be labelled as ψ, θ and φ. Where the angles of rotation as defined as Euler angles: ψ is an Azimuth angle; θ is an Elevation angle; and φ is a Roll angle. Where axes and rotations are defined with respect to Avionic Axes, the following conventions will apply:.

For convenience, the orientation of a tracked object or helmet and the orientation of a moveable platform or aircraft, relative to a given frame of reference, will be defined using quaternions.

In a typical head or helmet-mounted display system, a waveguide display or a visor projection display arrangement is provided; generating images visible to one or both eyes of a user. The images are typically in the form of symbols or other display artefacts, displayed so as to appear overlain on an external scene viewable through the otherwise transparent waveguide or visor, the waveguide or visor acting as a combiner. The symbols or other display artefacts may include anything from monochrome symbols in a predetermined symbology set, data or other text information through to full colour video imagery or a view provided of the external scene by a night vision camera during night-time use.

There are two particular situations in which helmet orientation needs to be taken into account when positioning symbols within a helmet-mounted display: display of symbols of a virtual head-up display, geometrically aligned to a centre line of the aircraft; and display of symbols such that they appear in fixed alignment with a ground-based object or other object in a fixed position relative to the Earth, viewable outside the aircraft. In each case the orientation of one or both of the helmet relative to the aircraft and the aircraft relative to the Earth may be changing, the former potentially rapidly. In particular, the helmet and hence the tracker system sensors may be subject to periods of buffeting which may seriously impair the functionality of known tracker systems during such periods.

Various different head or helmet tracking systems are known based upon inertial sensor technologies, non-inertial sensor technologies or a mixture of inertial and non-inertial sensor technologies. For example, in a helmet tracker system for use in an aircraft or other type of enclosed space, an optical tracker system may be used comprising a fixed arrangement of infra-red light-emitting diodes (LEDs) mounted on the helmet and an array of cameras mounted at known positions in a surrounding cockpit or cabin. Helmet orientation may be tracked by energising the LEDs in controlled patterns in a regular pulse cycle and processing the camera images of the helmet LEDs. Each pulse cycle provides an independent opportunity to determine helmet orientation.

It is also known to use an electro-magnetic tracker arrangement or an acoustic tracker arrangement in such confined spaces comprising a combination of sensors and transmitters, with the transmitters mounted on the helmet and the sensors at fixed positions within the space, or vice versa.

One disadvantage of optical or equivalent non-inertial tracker systems when used with display systems is an inherent latency in determining updates to helmet orientation. This latency may be compounded with any inherent latency in adjusting symbol positions in an associated display system such that the overall system response to determined changes to helmet orientation can cause pilot nausea. This occurs when movements of the head do not result in substantially simultaneous and corresponding movements in displayed symbols intended to appear fixed to some external reference point. Attempts to minimise latency in such applications have led to the use of miniature inertial sensors, be they accelerometers, tilt sensors or miniature gyroscopic sensors, small enough to be mounted on the helmet with two or three such sensors being aligned with respective orthogonal axes ('gyro axes'). Inertial sensors are capable of producing data indicative of a rate of change of orientation more quickly than optical, magnetic or acoustic sensors, for example, are capable of indicating a change in orientation. However, miniature inertial sensors of a type suitable for helmet-mounting may output orientation rate data that is prone to varying bias errors, noise or other types of 'installation' or 'misalignment' error, for example arising from imprecise mounting on the helmet and non-orthogonality of the gyro axes from manufacture.

It is known to combine inertial sensors on the helmet with a non-inertial helmet orientation sensor system and to combine data from the two sources in a process to calculate and to apply corrections to the inertial sensor rate data. This helps to prevent the accumulation of errors when calculating helmet orientation from the inertial sensor measurements.

In the present invention a hybrid helmet tracker system has been devised that is particularly suited for use with helmet-mounted display systems, addressing the problems in data quality from the helmet movement sensor systems, the effects of helmet buffeting and of latency in the associated display system.

The hybrid tracker system of the present invention combines inertial sensors mounted on the helmet with an optical helmet tracker system. A number of processing components are provided to process the data output from those sources in a novel and inventive way to determine helmet orientation relative to the inertial reference frame, or 'World axes', and helmet orientation relative to a moveable reference frame, such as an aircraft or other type of land or sea vehicle that the user may be piloting or in which the user may otherwise be riding. Moreover, helmet orientation is determined in such a way as substantially to eliminate delay in adjusting the position of displayed imagery in an associated helmet-mounted display system when responding to changes to helmet orientation, while minimising the effects of helmet buffeting.

In the description that follows, data variables will be named according to the naming structure
<Treatment><Tracked Frame><Data Type><Data Source><Reference Frame>
in which:.

Core components and functionality of an example embodiment of a hybrid helmet tracker system of the present invention, or in which embodiments of the present invention may be incorporated, will now be described in outline with reference to <FIG>. For convenience, orientation of a tracked entity with respect to defined axes will be expressed as a quaternion, unless otherwise specified.

Referring to <FIG>, a logical architecture depicts, from a functional perspective, a set of core components of the hybrid helmet tracker system <NUM> for determining helmet orientation relative to a host aircraft. In such an application, the hybrid helmet tracker system <NUM> makes use of sensor data from three sources.

A typical Aircraft Navigation System <NUM> has an interface to and outputs data over a standard data bus, one example being MIL-STD-1553B, provided within the aircraft. The Aircraft Navigation System <NUM> outputs an updated measure of aircraft orientation AW over the data bus so that it may be received by whichever of the installed aircraft systems require it. The hybrid helmet tracker <NUM>, in particular, may be linked to the standard data bus so that it may also receive updated aircraft orientation data from the Aircraft Navigation System <NUM>. However, as an alternative source of data defining aircraft orientation relative to the inertial reference frame, an inertial or other type of sensor arrangement able to determine aircraft orientation in inertial space may be mounted within the aircraft for use by the hybrid tracker system <NUM>.

In example embodiments of the present invention, helmet orientation is determined on a cyclic basis, corresponding to a need to reposition certain types of symbol, according to changing helmet orientation, being displayed in an associated digital helmet-mounted display. During each cycle - to be referred to as a 'tracker cycle' - helmet orientation is determined from data supplied by the Helmet-mounted Gyros <NUM>, any errors in those inertial sensor data being firstly corrected with reference to data supplied by the Optical Helmet Tracker <NUM> in combination with aircraft orientation data received over the data bus. Conveniently, the duration of a tracker cycle may be arranged to match the period between successive image refreshes in the associated digital display, although this is not essential.

In embodiments of the present invention, when calculating corrections to inertial sensor data, for example, it has been found convenient to work with data defining orientation of a helmet as sensed by the different associated sensor systems (<NUM>, <NUM>) referenced to a common frame of reference - to the inertial frame of reference, also referred to herein as 'inertial space' or 'World axes'. Data output by the Helmet-mounted Gyros <NUM> relate to motion of the gyros in inertial space whereas data output by the Optical Helmet Tracker <NUM> define motion of the helmet relative to a frame of reference fixed to the aircraft. It is therefore desirable to relate aircraft-referenced data from the Optical Helmet Tracker <NUM> to inertial space. With this objective in mind, a Transform component <NUM> is provided to receive a measure of helmet orientation HoA relative to the aircraft from the Optical Helmet Tracker <NUM> expressed as a quaternion and to transform it by quaternion multiplication into a derived helmet orientation <MAT> relative to the inertial reference frame using the aircraft orientation AW received over the standard bus from the Aircraft Navigation System <NUM>. If the Optical Helmet Tracker <NUM> uses a different convention for defining orientation to that used by the Aircraft Navigation System <NUM> or by the system of Helmet-mounted Gyros <NUM>, then the Transform component <NUM> is also arranged to convert the output of the Optical Helmet Tracker <NUM> so as to express helmet orientation using the same convention as used by the other two sensor systems <NUM>, <NUM>.

As a further convenience, for the purposes of subsequent data processing combining rate-of-change data from the Helmet-mounted Gyros <NUM> with data from the Optical Helmet Tracker <NUM>, a Calculate Rate component <NUM> is provided to 'differentiate' the optically-sensed orientation measurements HW to provide a rate of change of orientation vector HRateo, over a predetermined time interval, representing a point in a 'derived object orientation rate space' or, in the case of a helmet tracker, a 'derived helmet orientation rate space'. That is, the components of a rate vector HRateo define the derived optically-sensed rates of change in helmet orientation in inertial space resolved along the object-referenced or helmet-referenced axes. In an ideal arrangement, the axes of the Helmet-mounted Gyros <NUM> are substantially aligned with the helmet-referenced axes. However, in practice, there are likely to be differences in alignment or other installation-related or manufacturing errors.

Besides errors arising during installation or manufacture, rate data output by miniature gyros of a type suitable for helmet mounting may be subject to 'noise', characterised by the distribution in rate measurements output over a series of rate vectors GRateg for a given helmet movement, or even when the gyros are stationary. Additional data filtering components may also be included (not shown explicitly in <FIG>) to apply various filtering techniques to rate data received from the Helmet Mounted Gyros <NUM> and to rate data derived (<NUM>) from measurements by the Optical Helmet Tracker <NUM> to reduce the effects of noise.

More significantly, rate data output from individual gyros are also likely to include bias errors, typically of the order of <NUM>°/min or more, which may also change, for example with varying temperature of the gyros, causing accumulating errors in determined orientation of the helmet if not corrected for.

For these reasons it is also known to include a non-inertial sensor arrangement - in example embodiments of the present invention the Optical Helmet Tracker <NUM> provides such a non-inertial sensor arrangement - to provide data from which corrections to gyro rate data may be calculated.

A processing component - Calculate Gyro Corrections <NUM> - is provided to receive filtered HRateo rate vectors and the corresponding filtered GRate gyro rate vectors from the Helmet-mounted Gyros <NUM> and to determine corrections for any installation or manufacturing-related errors in the gyro rate data. Specifically, such errors may include: errors in alignment of the gyro axes with the helmet-referenced axes during installation of the Helmet-mounted Gyro system <NUM> upon a helmet; errors due to non-orthogonality of the gyro axes; and errors due to relative scaling of rate measurements along the gyro axes. The Calculate Gyro Corrections component <NUM> is also arranged to determine corrections for bias.

The determined corrections are then applied for that tracker cycle by an Apply Gyro Corrections component <NUM> to a rate vector GRate derived from a respective sample of rate vectors GRateg output from the Helmet-mounted Gyros <NUM> thereby to relate the movement sensed by the gyro system <NUM> more accurately to the actual movement of the helmet. The result is a fully corrected measure of the rate of change in orientation of the helmet as sensed by the Helmet-mounted Gyros <NUM>, expressed as a vector HRatec.

Having applied (<NUM>) corrections to the latest inertial sensor data to result in the fully corrected rate vector HRatec, a measure of helmet orientation HW relative to the inertial frame of reference, or World axes, is determined by integration in an Integrator component <NUM>. If required, the derived helmet orientation HoW with respect to the inertial frame of reference, output by the Transform component <NUM>, may be supplied to the Integrator component <NUM> for use in correcting an integration error.

A Predictor component <NUM> is provided to determine how the helmet orientation may have changed relative to the inertial frame of reference over a predetermined time period into the future - a Prediction Time period Tp - from the time of validity of the orientation HW determined by the Integrator <NUM>. That time of validity is a time point of validity of the GRate vector for the tracker cycle. The Predictor component <NUM> receives the corrected rate data HRatec and applies a prediction technique, to be described in more detail below, to determine the change in helmet orientation over the Prediction Time Period Tp taking account of a configurable length of recent history of helmet movement. The change determined by the Predictor component <NUM> is added in a summing component <NUM> to the orientation HW to give a predicted orientation HpW for the helmet relative to World axes.

A particular benefit of including the Predictor component <NUM> in the helmet tracker <NUM> of the present invention is an ability to anticipate the total latency in a process that begins with the capture of data from the Helmet-mounted Gyros <NUM>, indicative of a movement of the helmet, and ends with the display of a repositioned symbol by an associated helmet-mounted display system. The Predictor Component <NUM> enables the symbol to be repositioned in the display to take account of a helmet orientation expected by the time the repositioned symbol becomes viewable in the display by the pilot - the end of the Prediction Time period Tp.

The Integrator component <NUM> may be implemented separately from the Predictor component <NUM>, or the Integrator component <NUM> may be incorporated as part of the functionality of the Predictor component <NUM>, as convenient.

The display system may use the predicted helmet orientation HpW relative to World axes to position symbols in the display that are intended to be aligned with particular ground-referenced features or positions in an external scene visible to the pilot at that time through the display. However, to be able to reposition symbols in a virtual head-up display (HUD) fixed in space relative to the pilot's view (subject to parallax) of the interior of the aircraft (i.e. relative to aircraft axes), a further Transform component <NUM> is provided to transform the predicted orientation HpW relative to World axes into a predicted helmet orientation HpA relative to the aircraft-referenced axes using the latest measure of aircraft orientation AW as supplied over the standard aircraft data bus by the Aircraft Navigation System <NUM>. Any expected latency in the output of updates to aircraft orientation AW by the Aircraft Navigation System <NUM> has been found unlikely to have a significant effect on the accuracy of the predicted helmet orientation HpA relative to the aircraft-referenced axes.

The tracker cycle period of the hybrid helmet tracker system <NUM> is selected, conveniently, to support the needs of the associated display system or other user of the determined helmet orientation data. In the particular example of a digital helmet-mounted display system, it is desirable to be able to update the position of symbols in the display within a time period of no more than <NUM>, but in practice a shorter period is desirable, enabling not only a more rapid update to displayed information but also to a corresponding reduction in the Prediction Time period Tp.

As a further convenience, the tracker cycle period may be selected to correspond to the period between successive measurements of helmet orientation by the Optical Helmet Tracker <NUM>, <NUM> in the example mentioned above, being generally less responsive to helmet movement than the Helmet-mounted Gyros <NUM>. However, as mentioned above, the tracker cycle period need not necessarily be linked to the performance of the Optical Helmet Tracker <NUM>. In an alternative arrangement, the rate at which data output by the Helmet-mounted Gyros <NUM> are sampled may be set to be a multiple of the rate at which data are available from the Optical Helmet Tacker <NUM>, enabling a corresponding reduction in the tracker cycle period.

In a particular embodiment, data output from the Helmet-mounted Gyros <NUM> may be sampled at <NUM>, for example, and the Optical Helmet Tracker <NUM> may be operated at a slower rate, for example at <NUM> - a period of <NUM> between successive measurements. For convenience, the tracker cycle rate (the rate at which gyro data are sampled) should be a predetermined multiple of the rate at which the Optical Helmet Tracker <NUM> is operated.

There are at least two potential advantages of operating the Optical Helmet Tracker <NUM> at a slower rate than the tracker cycle rate. Firstly, the power of light signals emitted by the Optical Helmet Tracker <NUM> may be reduced, with consequent savings in power consumption. Secondly, such light signals may be rendered less easily detectable by external agencies, particularly during night-time operation.

It is generally preferred that the tracker cycle of the hybrid helmet tracker <NUM> is synchronised to the display cycle of the associated display system. If, for any reason, the display cycle is not always synchronised to the tracker cycle, a hardware or software-implemented component may be included at an output of the hybrid helmet tracker system <NUM> to interpolate one or both of the outputs HpW and HpA over successive tracker cycles to determine helmet orientation at some other time point appropriate to the needs of the display system. This ensures that the Predictor <NUM> is able to operate to a stable Prediction Time Tp.

For any given tracker cycle, a sample of rate data from the Helmet-mounted Gyros <NUM> provides a measure GRate of rate data that is then fully corrected (<NUM>) using the latest calculated corrections for errors. The Integrator <NUM> and Predictor <NUM> use the corrected rate data HRatec to determine predicted measures HpW and HpA of helmet orientation for that tracker cycle. However, the processes for updating the corrections for errors, calculated by the Calculate Gyro Corrections component <NUM>, may operate over a different cycle - a 'corrections update cycle' - for convenience, corresponding to the rate at which the Optical Helmet Tracker <NUM> is operated and so the rate at which new data become available for such calculations.

In describing an example embodiment of the present invention, it will be assumed that the Optical Helmet Tracker <NUM> is operated at the tracker cycle rate and that data from the helmet orientation sensors <NUM>, <NUM> are available to the correction calculation component <NUM> at the tracker cycle rate.

It is possible that, from time to time, output from the Optical Helmet Tracker <NUM> will be temporarily unavailable, for example when light signals passing between the helmet-mounted LEDs and the cockpit-mounted sensors are temporarily blocked by pilot hand movement or otherwise. In such a situation, operation of the helmet tracker process may enter a so-called 'coasting' period during which operation of the Calculate Gyro Corrections component <NUM> is suspended - there being no new data from the Optical Helmet Tracker <NUM> on which to base correction calculations. In this situation, the correction being applied to GRate data by the Apply Gyro Corrections component <NUM> remains unchanged until data from the Optical Helmet Tracker <NUM> next becomes available and the corrections update cycle may restart. A coasting period may last for one or more tracker cycles, limited by a configurable maximum coasting period to ensure that errors, in particular those due to gyro bias, don't accumulate to unusable levels.

As mentioned above, it is convenient, though not essential, to operate a corrections update cycle at the same rate as the tracker cycle rate. It is also convenient, though not essential, to operate the Optical Helmet Tracker <NUM> at the tracker cycle rate, the tracker cycle rate being thereby limited to a frequency of <NUM>, in this example. The rate at which the Optical Helmet Tracker <NUM> is able to measure changes in helmet orientation would be considered sufficiently rapid to support the needs of an associated helmet-mounted display in repositioning symbols in the display and so, for the purposes of explaining the principles, the following section of description will assume these convenient choices for equalising the correction cycle, tracker cycle and Optical Helmet Tracker rates. These choices are illustrated in <FIG>, showing the tracker cycle periods for the helmet tracker <NUM> and hence the periods over which the Predictor <NUM> operates to determine predicted helmet orientation.

Referring to <FIG> and additionally to <FIG>, a time line is shown spanning two complete tracker cycles, n-<NUM> and n, of duration <NUM>, beginning at time points Sync(n-<NUM>) and Sync(n), respectively. A sample of GRateg vectors may be captured from the Helmet-mounted Gyros <NUM> over sampling periods <NUM> extending over substantially the full duration of each tracker cycle, between time points Sync(n-<NUM>) and tGRate(n-<NUM>), Sync(n) and tGRate(n), etc. In a first type of filtering (not shown in the figures) that may be applied to each sample of gyro rate data GRateg, an average rate vector GRate may be calculated to provide a single rate vector representative of the captured sample for a given tracker cycle.

For the purposes of deriving a predicted helmet orientation HpA and HpW in respect of tracker cycle n for example, the most recently available fully corrected gyro rate vector HRatec is that calculated based upon a sample of GRateg vectors captured during the capture period <NUM> beginning at the time point Sync(n-<NUM>) and ending at the time point tGRate(n-<NUM>). For the next complete tracker cycle, HRatec is calculated based upon a sample of GRateg vectors captured during the capture period <NUM> beginning at the time point Sync(n) and ending at the time point tGRate(n). For the purposes of predicting helmet orientation by the Predictor <NUM> during tracker cycle n, the time point of validity of the most recently available sample of rate data from the Helmet-mounted Gyros <NUM> is the time point tGRate(n-<NUM>) at the end of the previous tracker cycle.

Updates to helmet orientation are determined in the Optical Helmet Tracker <NUM> using regular sets of pulses of infrared light emitted by light-emitting diodes (LEDs), each set of pulses occurring within a short time interval <NUM> near to the start of each tracker cycle (these pulses would only occur at the beginning of one tracker cycle in every T tracker cycles if the tracker cycle rate was T-times as fast as the Optical Helmet Tracker rate). Each set of LED pulses provides an opportunity to determine helmet orientation and therefore represents a point in time at which a respective determination of helmet orientation HoA would be considered valid. In Figure 2a, the labels tmHA(n-<NUM>) and tmHA(n) represent the nominal points in time at which an optically-tracked helmet orientation is valid in respect of LED pulses generated during the cycles n-<NUM> and n respectively.

For the purposes of nulling an error of integration, the helmet orientation HoW should be synchronised to the end of a tracker cycle to match the assumed time points of validity tGRate(n-<NUM>), tGRate(n), etc. of the gyro rate data. Given that the Optical Helmet Tracker <NUM> is only able to determine helmet orientation as it would apply for a tracker cycle n-<NUM> after the set of LED pulses at the time point tmHA(n) during the tracker cycle n, the Transform component <NUM> may be arranged to derive helmet orientation relative to World axes HoW for the time point tGRate(n-<NUM>) by interpolation between the two measures of helmet orientation HoW determined in respect of the time points tmHA(n-<NUM>) and tmHA(n) to give an interpolated estimate iHoW of the helmet orientation valid at the end of the tracker cycle n-<NUM>. The interpolation step has the effect of delaying the determination of HoW applicable in respect of tracker cycle n-<NUM> by the time period <MAT>.

That is, for the purpose of correcting an integration error during a tracker cycle n, the time point of validity of the most recently available measure HoW is an interpolated measure, iHoW, valid at the time point tGRate(n-<NUM>), or at the equivalent time point tGRate() during an earlier tracker cycle, rather than the measure HoW generated in respect of the most recent LED pulses at the time point tmHA(n). Availability of a suitably timed measure AW of aircraft orientation in inertial space is required to generate each measure HoW such that the most recently available combination of synchronised outputs of helmet orientation HoA from the Optical Helmet Tracker <NUM> and AW from the Aircraft Navigation System <NUM> may relate to an earlier tracker cycle than the cycle n-<NUM>.

The chosen convention of working with data timed to the end of a tracker cycle is taken into account when determining the Prediction time period Tp to be applied by the Predictor component <NUM> so that the Prediction Time period Tp is defined relative to the end of the tracker cycle of the latest processed gyro data - the time point of validity of the most recently available integrated helmet orientation HW.

The calculations of gyro error corrections by the component <NUM> requires, as input, filtered orientation rate data HRateo, derived (<NUM>) using successive measures HoW of orientation data output by the Optical Helmet Tracker <NUM>, substantially synchronised with filtered GRate data captured from the Helmet-mounted Gyros <NUM>, so ensuring that the two input measures relate to the same period of helmet movement. Those data forming an input to the correction calculation component <NUM> do not necessarily relate to the 'current' tracker cycle during which a fully corrected rate vector HRatec is to be generated, but they do need to be synchronised with each other. Furthermore, aircraft orientation data AW output by the Aircraft Navigation System <NUM> and used to derive the helmet orientation HoW should also be valid at the time of validity of the helmet orientation data HoA output by the Optical Helmet Tracker <NUM>, i.e. at respective time points tmHA(n-<NUM>), tmHA(n), etc. and is assumed valid at a time point midway between the respective pair. Various techniques may be applied to ensure that the derived helmet orientation HoW relative to World axes is calculated using measures of helmet orientation HoA relative to the aircraft and aircraft orientation AW relative to World axes that are correctly synchronised.

The Integrator component <NUM> receives the fully corrected rate of change in helmet orientation vector HRatec in respect of the latest complete tracker cycle and inserts it into an integration process to determine a helmet orientation relative to World axes expressed as a quaternion HW. In parallel with this, the Predictor <NUM> operates to predict how the helmet orientation will change over the Prediction time period Tp from the time point of validity of the helmet orientation HW at the end of the latest complete tracker cycle.

The Predictor <NUM> operates by firstly estimating the average rate of change of orientation of the helmet over the Prediction time period Tp or over some other 'Estimation Time' period that may vary in certain circumstances, as will be discussed below. The average rate of change may be calculated by means of a Recursive Least-Squares (RLS) filter, acting as an adaptive FIR filter in which the weights are continuously adjusted to match the output to a desired signal. In this case, the desired signal is the average rotation rate vector over the Estimation Time period immediately preceding the latest input. The adaptive part of the filter finds the weights that, when applied to the n inputs immediately preceding the time of the latest input minus the Estimation Time period, best match the rotation rate vector averaged over the most recent period of Estimation Time, where n is the tap count of the filter. These weights are then applied to the n most recent inputs to estimate the mean rotation rate vector over the forthcoming period of Estimation Time.

The estimated rotation rate vector is then integrated over the Prediction Time period Tp to give the predicted change in helmet orientation HpH relative to the latest determined helmet orientation HW. The predicted helmet orientation HpW relative to World axes is then obtained by the summing component <NUM> by quaternion multiplication of the latest helmet orientation HW output by the Integrator <NUM> and the predicted change in orientation according to the equation: <MAT>.

The integration process implemented by the Integrator component <NUM> to integrate received corrected rate vectors HRatec and to calculate HW may include a process for nulling an integration error, for example with reference to an interpolated estimate iHoW of helmet orientation derived by the Transform component <NUM>.

As mentioned above, a final stage of filtering may be applied by a Filter component <NUM> to the predicted helmet orientation HpA with respect to aircraft axes in a further level of refinement. The final stage of filtering, together with the functionality of the Predictor component <NUM>, may be arranged to take account of the effects of buffeting that may be affecting the helmet in a process that will now be described with reference to <FIG>.

Referring to <FIG>, a logical architecture for the final stages of the hybrid tracker system <NUM> is shown in more detail as a collection of interworking functional blocks designed to implement a process for detecting helmet buffeting, transferred from the aircraft, or other instances of rapid helmet movement and for helping to mitigate their effects upon the quality of data output by the tracker system and hence the user's ability to read an associated display. The purpose of this process is to produce a buffeting measurement that indicates the severity of buffeting to which the helmet is being subjected. Buffeting is classed as involuntary changes in helmet rotation, usually at frequencies in the region of <NUM> or above, caused by the transmission of aircraft buffeting to the user's head. The buffeting measurement may be used to modify the behaviour of the Integrator <NUM>, Predictor <NUM> and Filter <NUM> to help counteract its effect, to the benefit of the associated helmet-mounted display system.

In <FIG>, those functional blocks relating to the data sources (<NUM>, <NUM> and <NUM>) and correction of inertial sensor errors (<NUM>, <NUM>, <NUM> and <NUM>), resulting in a fully corrected helmet orientation rate vector HRatec, are represented by a single Sensor Arrangement component <NUM>.

A Measure Buffeting component <NUM> is provided, arranged to detect helmet movements indicative of buffeting or rapid head movement from the received data, and a Buffeting Compensation component <NUM> arranged to alter the operating parameters of the Integrator <NUM>, Predictor <NUM> and Filter <NUM> to mitigate the effects of the detected buffeting or rapid head movement. The measurement of buffeting is derived from the received corrected helmet rate vectors HRatec and is performed by a two-stage filter to produce an instantaneous measurement followed by a filtering of the measurement itself.

The first stage filter operates directly on the received helmet rate vectors HRatec and comprises an FIR filter with symmetric weighting over a predefined time period, including a number of tracker cycles, representing a history of corrected rate vectors. The weighting is at a maximum at the midpoint of the time period and follows a half sine-wave function. Alternatively, with the benefit of reduced processing requirements, the weightings may be uniform across the whole history. For each received rate vector, the filter produces a weighted average rate vector over the history of received rate vectors. This is compared with the mid-history rate vector to produce a delta vector.

The second stage filter takes the same form as the first stage filter but acts on the delta vectors. Again, the filter output is compared with the mid-history input to provide a second order delta vector. The magnitude of this vector is then taken as the instantaneous buffet measurement and is limited to a preconfigured maximum value.

The instantaneous buffet measurement is put through a final filter stage, consisting of a uniformly weighted average, to produce the buffeting severity measurement that is output from the Measure Buffeting component <NUM> to the Buffeting Compensation component <NUM>.

The Buffeting Compensation component <NUM> receives the latest buffet level and the latest helmet rotation rate vector HRatec and calculates parameters that may modify the behaviour of the Predictor component <NUM> and the output Filter component <NUM>. The calculated parameters are:.

Under normal conditions, Estimation Time and Integration Time are both equal to the demanded Prediction Time Tp and the Filter Latency is set to a configured base value. The Estimation Time can be reduced and the filter latency increased in the presence of high levels of buffeting. During rapid helmet rotation, both the Estimation Time and the Integration Time can be extended by the base latency of the output filter.

The first step in this process is to determine the state of a buffet flag. The Buffet Flag indicates whether the level of buffeting warrants any change from the "normal conditions" behaviour of the predictor. The flag is set if the Buffet Level BL calculated by the Measure Buffeting component <NUM> exceeds a predefined upper threshold and reset if the Buffet Level BL falls below a lower threshold. If the Buffet Level BL is between the two thresholds, the flag is unchanged. This provides hysteresis in the control of the flag.

The Buffeting Compensation component <NUM> calculates a scaled buffet level as the input buffet level divided by the preconfigured maximum buffet level value used to set the buffet flag. If the buffet flag is set, the scaled buffet level is used to calculate the latency for the predictor output Filter component <NUM>, where greater latency means heavier filtering: <MAT> where L is set to the predefined base latency of the output filter (which is less than BL) and is limited to a maximum latency for the Filter <NUM>, the log term is replaced by zero if it goes negative, SL is a configurable sensitivity factor, BL is a baseline factor for filtering in buffet conditions and b is the scaled buffet level. The maximum permitted latency for the Filter <NUM> is a predefined constant.

The Estimation Time and Integration Time of the predictor are nominally both equal to the demanded Prediction Time Tp but may be modified by conditions of high buffeting or high rotation rate. If the buffet flag is set and the scaled buffet level exceeds a predefined high level, the Estimation Time is reduced to one cycle. If the buffet flag is not set but the helmet rotation rate exceeds a predefined threshold, the base filter latency is added to the demanded prediction time for both estimation and integration. Under all other conditions both Estimation Time and integration Time are set to the demanded prediction time. An example logic for calculating Estimation Time and Integration Time is shown in the following table.

The Filter component <NUM> is arranged to apply a final low-pass filter to the predicted helmet orientation HpA, normally with a low latency. The latency may be increased according to equation (<NUM>) to mitigate the effects of buffeting and is always defined as an exact number of tracker cycles.

The basic filter type is a Hanning FIR filter. A separate set of weights is pre-configured for each allowed latency from <NUM> to the maximum allowed latency. In the case of zero latency, the weight set is just the single value <NUM>. In general the weight set comprises (<NUM>+ <NUM>) values where L is the required latency. The selected set of n weights is applied to each component of the most recent inputs of the HpA quaternion. It is assumed that rotation over the maximum latency period is small enough for a simple weighted average of quaternions to be valid, provided the result is normalized. To smooth out discontinuities in the output caused by changes in latency, the following method is used.

The previous cycle's input is re-filtered with the current weights, i.e. the weights selected by the latest filter Latency input L from equation (<NUM>). The difference between the last output and the re-filtered output is used to update a delta which is then added to the current filtered output: <MAT> <MAT> where.

Finally, the filtered output is normalized to a unit quaternion by the Filter <NUM>.

Having calculated and filtered the predicted helmet orientation HpA with respect to aircraft axes as a quaternion, it may be convenient to convert the quaternion representation of orientation into an Euler angle representation, providing the equivalent azimuth, elevation and roll angles, by a well-known method, according to the requirements of a user of the calculated orientation.

A typical implementation of a helmet tracker system according to example embodiments of the present invention described above will now be described with reference to <FIG>.

Referring to <FIG>, a helmet tracker system is shown integrated with a helmet <NUM>. The helmet tracker system comprises a Helmet-mounted Gyros <NUM> component, typically a single inertial sensor module containing three gyro elements, fitted to the external shell of the helmet <NUM>, and an Optical Helmet Tracker system. The Optical Helmet Tracker system comprises an arrangement of LEDs <NUM> integrated within or mounted upon the external shell of the helmet <NUM> and an arrangement of one or more Cameras <NUM>, each mounted at a fixed position, e.g. within an aircraft cockpit, such they have a line of sight view to at least some of the LEDs <NUM> at any likely orientation of the helmet <NUM>.

An Optical Helmet Tracker Controller <NUM> is linked to each of the Cameras <NUM> to receive image data resulting from the detection of light emitted by the LEDs <NUM> within the camera's line of sight. The Controller <NUM> is also linked to the LEDs <NUM> in the helmet <NUM> by means of an appropriate Cable <NUM> such that it may control the illumination of the LEDs <NUM>. The Controller <NUM> is arranged, for example, to trigger a cyclic illumination of the LEDs <NUM> in predetermined patterns, to receive the resultant image data from the Cameras <NUM>. The Controller <NUM> includes a digital processor programmed to implement an appropriate Optical Helmet Tracker algorithm, or a combination of hardware and software-implemented functionality, arranged to illuminate the LEDs <NUM> and to interpret the resultant image data from the cameras <NUM> to determine helmet orientation relative to a frame of reference associated with the aircraft, in this example.

A further Processing Module <NUM> is provided to implement the functionality of the overall helmet tracker system as described above with reference to <FIG>. The Processing Module <NUM> is linked to the helmet <NUM> by means of an appropriate Cable <NUM>, principally to receive rate data GRateg output from the Helmet-mounted Gyros <NUM>, but also to convey image data received from an aircraft-installed image generation system (not shown in <FIG>), via an interface <NUM>, to a helmet-mounted display integrated within the helmet <NUM> (not shown explicitly in <FIG>). The helmet-mounted display system may be a visor-projection display system, or a waveguide-based display, arranged in either case to present images to the pilot such that they appear overlain upon the pilot's view through a visor <NUM> of an external scene. The Processing Module <NUM> is arranged to output predicted measures of helmet orientation HpA and HpW to the aircraft-installed image generation system via the interface <NUM> for use in calculating the position of certain types of symbol and data intended for display in the helmet-mounted display, as appropriate.

In a typical aircraft installation of a helmet-mounted display system, incorporating the hybrid helmet tracker system of the present invention, one or more further modules (not shown in <FIG>) are likely to be installed within the aircraft for the purposes of generating images for display by any display systems installed or in use on the aircraft. Such display systems may include a head-up display (HUD) and/or a head-down display (HDD) in addition to the helmet-mounted display of relevance to applications of the present invention. Such further modules are likely to be linked to a standard aircraft data bus and so have access to aircraft orientation data AW output by an Aircraft Navigation System and to data from other systems with a need to display information to the pilot. Image data including symbols and data for display on the helmet-mounted display, together with updates to aircraft orientation AW received over the aircraft data bus by such further modules, may be passed to the Processing Module <NUM> by means of the interface <NUM>.

Claim 1:
A tracker system for a head or helmet mounted digital display device arranged to display space-stabilised symbols or other space-stabilised display artefacts, the tracker system for determining orientation of an object, comprising:
a sensor arrangement (<NUM>) associated with the object, arranged to output rate data providing a cyclic indication of a rate of change in orientation of the object;
a predictor (<NUM>) component arranged to receive rate data output by the sensor arrangement and to determine from the received rate data an orientation of the object at a predetermined prediction time period ahead of a time point of validity of the received rate data;
a buffeting measurement component arranged to receive rate data output by the sensor arrangement and to determine therefrom a measure of buffeting severity upon the object;
a buffeting compensation component (<NUM>) arranged to adjust the operation of the predictor component to compensate for a determined measure of buffeting severity by the buffeting measurement component; and wherein
the object is a head or a helmet on which is mounted the head or helmet-mounted display and wherein the predetermined prediction time period corresponds to an expected time of displaying a space-stabilised symbol or other space-stabilised display artefact positioned by the display device to take account of an orientation of the head or helmet indicated by an output of predicted orientation by the tracker system.