Patent Description:
A wearable head-up display typically includes a near-eye display or projection member, which is often mounted on a temple piece of an eyewear and displays an image that is visible to the user at an oblique angle whilst the user sees real objects straight through the eye-pieces. These heads-up display systems are useful for mission-critical or time-critical situations in which personnel are provided with or interact with intuitive augmented information about the surrounding so as to allow the personnel to make quick and accurate decisions. Such augmented information is transmitted via intuitive user interfaces (UIs), which minimise cognitive loads when making decisions, sending/receiving information and communicating amongst the personnel in the field, between the personnel and their equipment and/or between the personnel and a remote command centre.

These heads-up display systems are useful for mission-critical or time-critical tasks because the personnel are constantly on the look out and stay vigilant whilst augmented information is made available at an angle to the line of sight, and their hands are always ready on their equipment for action (in so-called heads-up, eyes-out and hands-on trigger concepts). These systems are useful if the systems can be adapted for use with prescription glasses, non-prescription glasses, protective eyewear used by enforcement or rescue personnel, helmet mounted eyewear or safety goggle (ie. host eyewear). Often, the near-eye display or projection member is fixed onto the eyewear and this may obstruct the vision of the user. Also, this eyewear is often not foldable.

It can thus be seen that there exists a need to provide a modular add-on head-up augmented reality functionalities to an eyewear of a host. Preferably, this modular add-on is foldable and the near-eye display or projection member is movable to an unobstructed view position during mission-critical, time-critical or safety-critical situations; these heads-up AR display systems are designed for ease of use and to augment productivity and efficiency during training or work.

The following presents a simplified summary to provide a basic understanding of the present invention. This summary is not an extensive overview of the invention, and is not intended to identify key features of the invention. Rather, it is to present some of the inventive concepts of this invention in a generalised form as a prelude to the detailed description that is to follow.

The present invention seeks to provide a modular head-up augmented reality (AR) display system to add-on to a host spectacle or an AR spectacle for mounting onto a helmet. Modules of the AR display system are for removeable attachment onto the host spectacle/helmet, and for extending functionalities of the host spectacle, which include multi-modal controls of forward equipment deployed ahead of the user.

In one embodiment, the present invention provides a modular augmented reality head-up display system as defined in claims <NUM>-<NUM>.

In another embodiment, the present invention provides a modular augmented reality head-up display kit as defined in claims <NUM> and <NUM>.

In yet another embodiment, the present invention provides a realtime control system for an unmanned vehicle as defined in claim <NUM>, and an alternative autonomous control system as defined in claims <NUM>-<NUM>.

This invention will be described by way of non-limiting embodiments of the present invention, with reference to the accompanying drawings, in which:.

One or more specific and alternative embodiments of the present invention will now be described with reference to the attached drawings. It shall be apparent to one skilled in the art, however, that this invention may be practised without such specific details. Some of the details may not be described at length so as not to obscure the present invention.

<FIG> shows a perspective view of a modular head-up display system <NUM> for removeable attachment onto an eyewear or spectacle <NUM> of a host. Preferably, the host spectacle <NUM> is a type of ballistic protection spectacle, safety goggle, non-prescription spectacle and/or a prescription spectacle. The modular head-up display system <NUM> includes a processor module <NUM>, a battery module <NUM> and a trunking module <NUM>. The processor and battery modules are attachable onto separate temple pieces 11a, 11b of the host spectacle <NUM> whilst the trunking module <NUM> is attachable onto a front frame member <NUM> of the spectacle. When assembled onto the spectacle, the trunking member is located between the processor and battery modules, thereby protecting wires <NUM> between the battery and processor modules. The battery and processor modules are configured for a right-hand or left-hand user depending on the user and an inside face of each module has a hook <NUM>, <NUM> for removeable snapping-on onto the respective temple piece 11a, 11b; FIG. IE shows a plan view of the left-handed configuration 100a of the modular AR head-up display system. Weights of the battery and processor modules are distributed and balanced to maintain comfortable wearing of the host spectacle.

The processor module <NUM> includes an electronic processing unit <NUM>, a wireless communication unit <NUM>, a barometer <NUM>, a pedometer <NUM>, a nine-axis inertia measuring unit (IMU) <NUM>, a GPS unit <NUM>, a touch pad <NUM> and some select buttons <NUM>. At a forward distal end of the processor module <NUM> is a drop-in slot connector <NUM>, which allows pivotable connection with a display or projection module <NUM>. The projection module <NUM> includes a body member <NUM> and a transparent prism member <NUM>. The body member <NUM> houses a micro-display unit <NUM>, a camera/video unit <NUM>, a light sensor <NUM> and a thermal camera <NUM>. In low light conditions, the camera unit <NUM> and the thermal camera <NUM> are both activated, and an image fusion algorithm <NUM> corrects the outputs of both the camera unit and thermal camera, and provides thermal images with defined outlines to assist the user with night vision; with these thermal images, the user <NUM> can better identify targets or objects of interest (OoI) <NUM>; this night vision function can be activated by toggling on a night vision button <NUM>, as seen in <FIG>. In addition, some buttons <NUM>, 240a (such as on/off or select buttons) are provided on an upper face of the body member <NUM>. The transparent prism member <NUM> has an internal reflection surface on which an image projected from the micro-display unit <NUM> is formed and is visible to the user <NUM> by side glancing. <FIG> shows a plan view of the head-up display system shown in <FIG>, whilst <FIG> shows the projection module <NUM> is moved to an unobstructed position, and <FIG> shows the head-up display system with the temple pieces 11a, 11b of the spectacle being folded-up. In use, the modular head-up display system <NUM> may wireless connect to a third party support devices, such as, a laser finder <NUM>, an acoustic targeting system (ATS) <NUM> and forward equipment <NUM> (such as, an unmanned ground vehicle (UGV) 103a, an unmanned aerial vehicle (UAV) or drone 103b, and an unmanned underwater vehicle (UUV) 103c. In another embodiment, such as that for helmet mounting, the barometer, pedometer, IMU, and so on, may be housed in a separate unit, which is wearable on the user's clothing. In other embodiments, the touch pad <NUM> can be configured as a tactile track point.

The modular heads-up display system <NUM> also includes user interfaces that allow augmented reality information to be provided to the user intuitively (ie. a head-up concept). In other words, during mission-critical, time-critical, or safety-critical moments, augmented information about the surrounding is provided to the user <NUM> and/or users/peers <NUM> to allow them to make critical decisions quickly and accurately whilst the user(s) is/are in the frontline (ie. with eyes-out and hands-on trigger concepts). The electronic components in the processor module <NUM> respond equally quickly and automatically to support these mission-critical, time-critical or safety-critical moments. When the user is in a less pressing moment that the user is able to use the touch pad/track point <NUM> and select buttons <NUM> to input data, such as, using <NUM> the laser finder <NUM>, using the forward equipment <NUM>, taking <NUM> photos/videos with the camera/video unit <NUM>, tagging <NUM> objects of interest (OoI) <NUM> on the photos, navigating the GPS map, sending <NUM> information to other users/peers <NUM> and command centre <NUM>, and so on. These intuitive user interfaces help to minimize cognitive load on the users <NUM> so that they can concentrate on mission-critical, time-critical or safety-critical decision making. Thus, with this heads-up display system <NUM>, the user <NUM> decision making and task effectiveness and efficiency is enhanced.

<FIG> shows the drop-in slot connector <NUM> which allows the projection module <NUM> to click or detent lock in the extended position (in the AR view) or in the retracted position (in the full reality view) until the projection module <NUM> is moved by the user. The click or detent lock mechanism is provided by two grooves <NUM> on the processor module <NUM> and two cooperating projections 162a, 162b on the drop-in slot connector <NUM>. The drop-in slot connector is hollow and allows electrical connection between the processor and display/projection modules. The drop-in slot connector <NUM> is held in the processor module <NUM> by two pins <NUM> as can be visualized in <FIG>.

<FIG> show various views of a modular head-up AR display system 100b that is mountable on a helmet <NUM>. As shown in <FIG>, the modular head-up display system 100b is made up of a battery module 170a mounted on a Picatinny rail <NUM>. A trunking module 180a connects the battery module 170a to a processor module 110a, which distal end supports a body 210a of a projection module 200a, such that a see-through prism 260a is proximate to the eye of the user <NUM>. The Picatinny rail <NUM> allows the AR display system 100b to be mounted or dismounted with ease. In <FIG>, the AR display system 100b is shown for left-hand mounting on the helmet <NUM>; this AR display system can also be configured for mounting on the right hand side of a helmet. The functions and components of the processor module 110a, trunking module 180a and battery module 170a are similar to those described above; for eg. , on the processor module, there are: a tactile track point 145a, an on/off button <NUM>, a select button 240a, and so on. On the projection body 210a, there are: a camera/video unit 220a, a light sensor 225a and a thermal sensor 230a.

<FIG> shows various functionalities of the above modular heads-up display system <NUM>, 100a,100b whilst <FIG> show various view modes <NUM>, <NUM>, <NUM>, <NUM>, <NUM> of the user interfaces. A default AR view mode <NUM> allows the user to see-through the spectacle <NUM> for identifying OoI <NUM> (ie. reality view 300a) and be provided with AR information. As seen in <FIG>, with the camera <NUM> set in video taking as the default mode, video is presented near the centre of the AR view where a man-identification algorithm <NUM> in the electronic processing unit <NUM> is provided to identify and to mark/highlight <NUM> a human face <NUM> in the video. Automatic marking of a human face in the frontline vision helps to locate a target or to warn the user of a potential threat. The human feature identified in the video can also be tagged, for eg, with an icon <NUM> and may be additionally distinguished as a friend/peer <NUM> or foe <NUM>. The tagging functionality may be supported by a tagging algorithm <NUM> disposed inside the processor unit <NUM>. At the same time, a bottom right side of the AR view is a minimized GPS map <NUM> view of the surrounding. The position of the human feature or any OoI <NUM> can be planted on the GPS map <NUM> by firing a laser finder <NUM> at the target and obtaining estimates of distance and location coordinates from the user position. Preferably, the minimized GPS map view occupies about <NUM>% of the entire AR view. On the left hand edge of the AR view, there are <NUM> view mode buttons/icons <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, respectively for activating the camera/video unit <NUM>, activating the thermal camera <NUM>, activating a forward equipment <NUM> (such as controlling a turret or camera of an unmanned vehicle, including an UGV (unmanned ground vehicle) 103a, UAV (unmanned aerial vehicle) or drone 103b or UUV (unmanned underwater vehicle) 103c), interfacing with third party support devices <NUM>, <NUM>, activating an emergency hotkey <NUM> to communicate with the command centre <NUM> and logging out <NUM> of the AR mode <NUM>. Across the top of the default AR video view mode is an incoming message panel <NUM> that pops up an incoming message from other user/peer <NUM> or the command centre to the user <NUM>. When a user wears a vest with a health monitor and battery power sensor, health conditions of the user (such as, body temperature <NUM>, health state <NUM> (like, tired, exhausted, heat stroke, etc.) and electric power level <NUM> may be displayed in a health and supplies panel <NUM> located on the right hand side of the AR view mode. Preferably, when a view mode button/icon is active, its colour toggles, for eg.

In the default AR view mode <NUM>, the camera <NUM> can be activated to take <NUM> video or photo <NUM> by activating the camera icon <NUM> or button <NUM> on the projection body <NUM>. With a photo taken, the user can activate the touch pad/track point <NUM>, move a highlight box to select a target or OoI <NUM>, click within the highlight box to attach <NUM> a tag or icon <NUM>. Alternatively, the entire photo <NUM> can be tagged. If the user takes no action after capturing a photo, the view mode reverts to the default AR video mode after a lapse of a predetermined time, such as about <NUM>; alternatively, when the user does not need to tag a target or OoI, the user can click on a cancel button on the touch pad <NUM> to switch immediately to the default AR video mode. If the touch pad is still active, the predetermined view mode toggling will not take place and the user can continue tagging other targets or OoIs. The predetermined view mode toggle will occur when the AR display system detects inactivity from the user. The videos, photos and tagged photos <NUM> are sent to the command centre <NUM> for recording and analyzing.

In dim light situation or night time, the user <NUM> can activate the camera <NUM> and thermal camera <NUM> for assisted night vision. As in the default video view mode, the man-identification algorithm <NUM> automatically helps to identify and to mark/highlight a human face <NUM> in the field of vision to lessen visual cognitive load on the user.

<FIG> shows a turret view <NUM> when a forward equipment <NUM> (such as a camera, remote detectors, etc.) of an UGV 103a, UAV/drone 103b or UUV 103c is deployed in the frontline ahead of the user. When the turret button/icon <NUM> is activated, the forward equipment is automatically paired with the modular AR display system <NUM>,100a,100b. When pairing is not successful, the user has the option to reconnect the pairing; if the pairing fails, the user or command centre is notified. In the turret view <NUM>, a sighting cross-hair <NUM> appears in the centre of the video streamed from the forward equipment (or camera/detector) mounted on the unmanned vehicle <NUM>. Pan and tilt movements of the forward equipment's turret directions are controlled by outputs of the <NUM>-axis IMU <NUM> by moving the head of the user in realtime (ie. requiring substantially continuous, manual inputs to actuators and sensors in the forward equipment). Alternatively, gesture control and/or brain sensory control may be used to control movements of the forward equipment <NUM> in an autonomous mode, thereby allowing heads-up, eyes-out and hands-on readiness. A multi-modal controller <NUM> in association with control of the drone 103b will be described in a later section.

Clicking on the mimimised GPS map <NUM> brings up the full GPS map view in a new view. <FIG> shows the full GPS map view <NUM>. In the GPS map view, the user <NUM> is identified by a circle with an apex of a triangle <NUM> indicating the direction the user is facing as enabled by the GPS unit <NUM> and <NUM>-axis IMU <NUM>. , peers or friends <NUM> are identified by water-drop location pin, whilst foes <NUM> are identified by quadrilateral speech location pin. On the right hand edge are <NUM> buttons <NUM>, <NUM>, <NUM>, <NUM>, respectively for returning to the default view mode, for zooming in, zooming out and refreshing the GPS map. A user can tap on the peer location pin to send <NUM> a message (text, icon, photo or video). On taping a peer location, a keypad (such as that shown in <FIG>) appears for the user to enter and send the message. Refreshing updates location of the tagged, peer/friends and foes locations from the last viewing position (instead of refreshing from the user's location).

When a user enters an indoor area, GPS communication is lost and the user is directed to an indoor navigation mode. From the last known GPS coordinates and direction, the pedometer <NUM> provides the user with the distances moved, even by the rates of movements, whilst the <NUM>-axis IMU <NUM> provides the directional changes and the barometer <NUM> provides the ascending and descending distances.

<FIG> shows a third party support device view mode <NUM>. The third party support device may be a laser finder <NUM> or an ATS <NUM>. With the use of the ATS, a clock face <NUM> visually shows positions of the targets or OoI <NUM>, with concentric circles indicating distances from the user and an apex of a triangle <NUM> at the centre shows the direction the user is facing. The user has the option to tag <NUM> a target location within the predetermined view mode toggle time. These known target locations and tags are automatically planted on the GPS map <NUM> and are made available to other users/peers <NUM> and the command centre <NUM>.

Preferably, after tagging a target or OoI <NUM> position, the modular AR display system <NUM>,100a,100b automatically switches to the photo taking view mode. The AR display system switches over to the default AR video mode <NUM> after the predetermined time or the user has the option to cancel the photo taking view mode to immediately switch to the default AR video view mode.

<FIG> shows an emergency view mode <NUM> of the user interface for communicating with the command centre <NUM>. In the emergency view mode <NUM>, text and/or icons are selected from a keypad <NUM> to compose <NUM> messages in an input view panel <NUM>. The message is sent by activating a send button <NUM>. In addition, the camera <NUM> can be activated to provide live video of the frontline surrounding. In the situation when the user <NUM> requires assistance or attention, the camera <NUM> can also be programmed to provide live video stream to the command centre <NUM>.

Communication with other users/peers, with the forward equipment, with the third party support devices and with the command centre may be via secure wireless connection. As described above, the incoming message panel <NUM> pops up at the top of the default AR video view mode <NUM>. There is a tick button <NUM> near the incoming message panel <NUM> for the user to acknowledge receipt; if no acknowledgement is made, the incoming message panel <NUM> fades away after a predetermined time, such as about <NUM>. The incoming message panel disappears immediately after acknowledging receipt. To send a message to a peer/friend, the user clicks on the GPS map <NUM> and clicks on the relevant position icon representing the peer/friend and a message panel similar to that in <FIG> appears.

Now, the modular heads-up display system <NUM>,100a,100b is described to bring out more clearly each of the various functionalities: (<NUM>): Feeding of intuitive AR information to users <NUM>, <NUM> to help improve situation awareness and to allow better decision making. The user interfaces help in minimizing the cognitive load on the users so as to lessen the burden on the users in making a decision by providing AR information in an intuitive manner and to lessen the demand on them to capture and to send information at the frontline to other users/peers and command centre. In addition, various view modes in the user interface allow the users to quickly switch to the various functionalities, such as:.

Typically, a drone 103b requires more complex controls from a user; generally, a conventional drone controller also requires both hands of a user to operate when navigating the drone. In the following description, the multimodal controller <NUM> is now described for use to control a drone 103b, which is deployed as a forward equipment <NUM> in the frontline. The multimodal controller <NUM> includes a command integration unit <NUM> and a command priority weighting unit <NUM>. In the present invention, the users are often called to execute mission-critical, time-critical or safety-critical tasks; the multimodal controller <NUM> thus allows a user natural, intuitive and autonomous control of a drone, yet allowing the user to stay vigilant with heads-up, eyes-out and hands-on trigger readiness (with the drone control not requiring both hands to operate); at the same time, with AR information being presented to the user or made available to the user, this heads-up AR system <NUM>,100a,100b enhances the user's task effectiveness and efficiency. In addition, the multimodal controller <NUM> adapts to the user (instead of the user having to adapt to the controller, as the case for the conventional drone controller).

The command integration unit <NUM> fuses three control modes, namely, voice, gesture and brain signals <NUM>,<NUM>,<NUM> to generate an autonomous command signal; in response, the drone executes the autonomous command signal and incrementally updates its state; for eg. , by executing a "turn" autonomous command signal, the drone navigates a predetermined distance in an azimuth direction from a present position. In another eg. , by executing an "up" autonomous command signal, the drone moves up a predetermined altitude from its present position. With this autonomous control, during a critical moment, a user can continue to hold a weapon with both hands, with eyes-out and staying vigilant whilst navigating the drone. This autonomous control with predetermined motion and direction is distinguished from the realtime control provided by outputs from the IMU <NUM>. Further, the user has a choice of initiating a voice control <NUM> (where a certain degree of sound is acceptable), gesture control <NUM> (using hand signals or inertia of a wearable device) or brain sensory control <NUM> (during silent watch). Each of these three controls has respective characteristics of response time, accuracy and sensitivity, and they augment each other (instead of creating redundancies). In this way, the command integration unit-<NUM> is driven by a dominant control mode in one particular state according to its characteristic response time, accuracy, sensitivity and environment compatibility, thereby dispensing with any need for command error correction. This also prevents commands from two different modes being treated as two distinct commands, which would result in unwanted movements of the drone, for eg. , when the beginning part of a command stream from a mode with short response/process time catches an ending part of a command stream with longer response/process time. In one embodiment, the command integration unit <NUM> operates on a rule-based mechanism (for eg. , by comparing a current state of the drone with the state of the drone when command signals were received).

In addition, sensitivities of the different input modes of control are different; for eg. , some input modes are error-prone to the environment, such as, noise level and lighting condition, whilst other modes are subject to mental distractions. The command priority weighting unit <NUM> determines the most appropriate command generated from the three control modes.

In the multimodal controller, priority weighting unit <NUM> is given to the control mode which gives the highest sensitivity under a particular environment. , in an environment where brightness is low, gesture control <NUM> will be given low weightage in generating the command signal. However, the user can over-ride the command mode. A reason for giving the user this authority is because the user is often able to assess the most suitable control mode in a given environment. , the user can switch to use brain sensory control <NUM> for commanding the drone in an environment where voice control <NUM> was given the highest priority where a noisy environment could cause a false signal to mix in.

Claim 1:
A modular heads-up augmented reality (AR) display system (<NUM>,100a,100b) wearable by a user in a rescue or combat mission comprising:
a processor module (<NUM>);
a battery module (<NUM>);
a trunking module (<NUM>) disposed between the processor module (<NUM>) and the battery module (<NUM>) to protect wires connecting the processor module (<NUM>) and the battery module (<NUM>); and
a host spectacle (<NUM>) having two temple pieces (11a,11b) and a front frame (<NUM>);
characterised in that:
the processor and the battery modules (<NUM>,<NUM>) are removeably attachable with a hook (<NUM>,<NUM>) onto the respective separate temple pieces (11a,11b), whilst the trunking module (<NUM>) is removeably attachable onto the front frame (<NUM>) of the host spectacle (<NUM>); and
a forward distal end of the processor module (<NUM>) is pivotably connected to a display or projection unit (<NUM>) by a drop-in slot connector (<NUM>), so that the display/projection unit (<NUM>) is pivotable about one axis and is adjustable between an extended position (for AR view) and a retracted position (for full reality view), thereby allowing the user to stay vigilant during a mission-critical, time-critical or safety-critical situation.