Patent Description:
In the operation of a mobile electronic device, there may be a need for an accurate determination of the position and orientation of the device. In an example, an augmented reality image may combine images of an environment containing actual objects with images of virtual objects with the intent of providing a combined image that seamlessly merges such image components.

However, conventional systems often have difficulty in merging virtual and real images because of inaccuracy in the computed position for such elements in relation to a display being used to depict the augmented image. The inaccuracy in said computed position arises from insufficient accuracy in the measurement of position and orientation of the display being used to depict the augmented image. As a result, virtual images are often not well anchored to the environment, causing such images to often "swim" in the environment rather than naturally existing or moving in the environment.

Further, the merging of real and virtual objects requires an accurate depiction of occlusions in which a virtual object is partially hidden behind a real object. If positions and orientations are not accurately presented, the border between real objects and virtual objects will be inaccurate, which may create gaps in the augmented reality images rather than natural looking delineations between virtual and actual objects.

<CIT> describes a navigation system for a marine vessel that employs a composite image produced by superimposing a virtual image comprising navigational data over a real image of a portion of a surrounding environment, comprising: an ocular device images light reflected from the portion of the surrounding environment so as to directly form a field of view of the real image of the portion of the surrounding environment on a user's retina; a virtual image generator that produces an optical signal directed toward the user's retina so as to form a virtual image on the retina; an azimuth sensor that produces an azimuth signal being indicative of an orientation of the field of view formed by the ocular device; an absolute position sensing instrument that produces an absolute position signal indicative of a position of the marine vessel; and a navigational computer coupled in communication with the ocular device and the plurality of instruments, said navigational computer processes the azimuth signal and the absolute position signal and determines the field of view produced by the ocular device; determines a portion of the navigational data that should be included in the virtual image; and includes only said portion of the navigational data thus determined in the composite image.

<CIT> describes a method for navigating concurrently and from point-to-point through multiple reality models, which includes: generating, at a processor, a first navigatable virtual view of a first location of interest, wherein the first location of interest is one of a first virtual location and a first non-virtual location; and concurrently with the generating the first navigatable virtual view of the first location of interest, generating, at the processor, a second navigatable virtual view corresponding to a current physical position of an object, such that real-time sight at the current physical position is enabled within the second navigatable virtual view.

<CIT> describes a system for combining virtual images with a real-world scene within a field of interest for an observer, said system comprising: a range scanner scanning the field of interest and generating range data indicating the distance of real-world objects within the field of interest; a computer model simulating a virtual entity and producing a virtual image of said virtual entity at a location within the field of interest; means for generating masked virtual objects from said range data and said virtual image indicating those portions of said virtual image that are visible in the field of interest; means for combining said masked virtual objects and a realworld image of the field of interest to create a combined image in which said virtual image appears in the real-world image; and display means for displaying said combined image to the observer.

<CIT> describes a device, comprising: a personal electronic device including a logic circuit and a user interface; and at least one micro-impulse radar operatively coupled to the logic circuit and configured to probe one or more regions near the personal electronic device; wherein the logic circuit is configured to receive or generate micro-impulse radar data including information about the probed one or more regions and, in cooperation with the user interface, at least conditionally provide information about the one or more regions to a user.

Embodiments described here are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings in which like reference numerals refer to similar elements.

Embodiments described herein are generally directed to determination of mobile display position and orientation using micropower impulse radar.

For the purposes of this description:
"Mobile device" means a tablet computer, <NUM>-in-<NUM> or detachable computer, smartphone, handheld computer, mobile Internet device, or other mobile electronic device that includes processing and image presentation capability.

"Micropower Impulse Radar" or "MIR" means a lower power radar system producing short electromagnetic pulses. An MIR component or system generally produces a ultra-wideband (UWB) signal using a large radio spectrum (greater than <NUM>) in short pulses, and such component or system may be referred to as an ultra-wideband micropower impulse radar, or "UMIR". As used herein, radar systems are generally referred to as MIR systems or components, which include UMIR.

"Cover" or "case" means an outer portion of an electronic device that may enclose internal components of the device.

Electronic devices such as mobile devices may be used in augmented reality (AR) operations. Augmented reality systems render virtual objects visually superimposed upon the real world. In order for an augmented reality system that attempts to simulate actual, physical objects to appear natural to a viewer, the system is required to render the virtual objects in such a manner that the objects appear to the user to behave as real, physical objects. Among other factors, the virtual objects need to appear to be firmly anchored to the environment, and such objects must be occluded in the same manner as real objects if there are occluding portions of the real environment.

Augmented reality operation has been attempted with varying degrees of success by previous conventional systems. However, a common problem with conventional systems is that rendered virtual objects tend to "swim", or appear poorly anchored to the real environment, due to errors in sensing the position of the display relative to the environment as well as errors regarding the position of the user's dominant eye (or vantage point) in relation to the display. Further, an additional problem is that occluding features of the environment are difficult to identify and characterize, and thus conventional systems often poorly represent the occluding of virtual objects.

Conventional augmented reality systems have employed digital processing of video to address object location. However, an issue with such systems is that the systems only provide a 2D projection of the viewing frustum, where a viewing frustum is a three-dimensional space that is viewed from a particular viewpoint, such as the region viewed by a viewer through a display. Information about the actual depths in the scene is lost in the 2D projection, and it is computationally expensive to undo the projection. In addition, it is not usually possible to directly determine absolute distances or depths in the video scene without resorting to reference objects in the scene, and the use of such reference objects defeats the purpose of, for example, a self-contained mobile device used for augmented reality.

The claimed teachings provide augmented reality operation that addresses the "swimming" and occlusion problems by actively interrogating the real environment through means of radar elements (such as micropower impulse radar elements) to accurately determine the position and orientation of the display relative to the surrounding environment, and accurately determine the position and orientation of the display relative to the user's viewpoint.

<FIG> illustrates an embodiment of an electronic mobile device providing determination of position and orientation using micropower impulse radar. The device operates by utilizing ultra-wideband micropower impulse radar to construct a depth map of the environment within the viewing frustum of the mobile display, and to generate and update a viewing matrix to accurately place the user's viewing frustum in relation to the device. The generation of the position data includes determination of:.

As illustrated in the top view provided in <FIG>, a mobile device <NUM> is utilized for an augmented reality application. The device <NUM> includes an imaging element <NUM>, including but not limited to a rear facing camera, to capture an image of a real environment, where the real environment includes an actual object <NUM> within the camera
viewing frustum <NUM> of the device, where <FIG> also illustrates the user viewing frustum <NUM>. The mobile device <NUM> generates and provides an augmented reality image to a user <NUM> on a display screen <NUM>, wherein the augmented reality includes the addition of a virtual image <NUM>.

However, the augmented reality image will not appear natural in its environment unless virtual objects are properly rooted to the environment and appear properly occluded by actual objects, such as object <NUM>, that are closer to the user in terms of the virtual image. As illustrated in <FIG>, the actual object silhouette ray from the camera viewpoint <NUM> may not indicate an occlusion of virtual object <NUM> by actual object <NUM>, but the actual object silhouette ray from the user's viewpoint <NUM> demonstrates that from the user's viewpoint there is an occlusion of a portion of the virtual object <NUM>.

The mobile device includes multiple radar antennae, the antennae including multiple rear facing MIR antennae <NUM>-<NUM> for use in determining locations and distances of actual objects such as object <NUM>, and may further include multiple front facing MIR antennae <NUM>-<NUM> for use in determining a position of the vantage point of the user <NUM>. The antennae are used to transmit and receive ultra micropower impulse radar signal pulses. MIR provides exceptional accuracy in determining positions of objects in very small time steps, and thus is a useful and powerful technology to perform these functions.

Commonly, 3D rendering programs use 3D transformation matrices to orient virtual models within the virtual world, and to position a virtual camera within the virtual world. In augmented reality applications, the viewing frustum <NUM> must correspond accurately to the user's viewpoint <NUM> and to the position and orientation of the mobile display in the real world. While depicted in two dimensions in <FIG>, the viewing frustum has a shape that is approximately that of a four-sided pyramid with the user's viewpoint <NUM> at the apex of the pyramid shape.

<FIG> further illustrates the different points of origin for the camera viewing frustum and the user frustum. The difference between the viewpoints may be corrected by pre-processing the camera video to make it appear to have been captured from the user's viewpoint. To accommodate a user <NUM> looking at the display <NUM> from off axis (i.e., not straight in front of the display), the camera viewing frustum <NUM> may be wider than the user viewing frustum such that the pre-processing of the camera video is to be capable of selecting the appropriate subset of the camera viewing frustum to match the user's viewing frustum. A viewing matrix to describe such a viewing frustum is continuously calculated based upon the sensing of positions of key features in the environment. The viewing matrix is then provided as a service of the mobile platform operating system to applications that wish to render virtual objects in a manner that appears well anchored to the real environment.

To handle occlusions in which a virtual object is hidden at least in part by a real object in augmented reality, 3D (three dimensional) imaging programs typically include the use of a Z- or depth-buffer. A device's MIR components are used to update a depth map to provide current and accurate depth information. A depth map gathered by the MIR can be used to pre-initialize a 3D program's Z-buffer, thus providing plausible information for occluding virtual objects by real objects. Objects in the real world that occupy a depth closer to the observer than the virtual object's depth will prevent occluded portions of the virtual object from being rendered, further enhancing the illusion that the virtual object inhabits the real world, as viewed through the mobile display. Specifically, the pre-initialized depths are required to be the distances from the user's eye-point, through each pixel of the display, to the nearest real object in the environment, as opposed to the orthogonal distance from the display plane to the objects.

MIR technology provides a direct means of interrogating the environment and determining absolute distances and angles, which provides a superior means of determining orientation and position of a display in an environment. In general, MIR components run at a speed that is at many times the 3D rendering frame rate, and thus the viewing matrix and depth map the MIR system provides are always fresh and locked to the positions of the user and the display.

Further, a MIR system is not generally subject to variations in ambient lighting conditions, as is video that passively gathers light from the scene. The radar energy used by MIR is not visible to human observers, so it can be used without interfering with the user's experience of the AR environment. This is generally not possible with video, which typically uses light spectra that is visible to human observers.

<FIG> illustrates a block diagram of elements of an apparatus according to an embodiment including determination of position and orientation using micropower impulse radar. The apparatus such as the mobile device <NUM> includes a micropower impulse radar system including one or more MIR modules or sub-systems <NUM>, wherein the mobile device further includes multiple MIR antennae, which are illustrated in <FIG> as an antenna at each corner of a cover <NUM> of the mobile device <NUM>, the antennae being antenna <NUM> at a first corner, antenna <NUM> at a second corner, antenna <NUM> at a third corner, and antenna <NUM> at a fourth corner of the cover <NUM> of the device. Transmit and receive antennae may be one and the same. While connections for only a first set of radar antennae are shown in <FIG> for ease of illustration, the mobile device includes a first set of radar antennae to track object positions in first direction, such as the illustrated antennae <NUM>, <NUM>, <NUM>, and <NUM>, and a second set of radar antennae to track a position of the vantage point of user the mobile device (the dominant eye of the user of the mobile device), such as an additional set of antennae <NUM>, <NUM>, <NUM>, and <NUM>. The mobile device may request and receive information regarding which eye of the user is the user's dominant eye, such as receiving such information from the user in an initial phase of the personalization of the mobile device by the user or requesting such information when a user first utilizes an augmented reality application.

The MIR system <NUM> generates object position data <NUM> and user position data <NUM> that is provided to a processing unit <NUM>. The processing unit <NUM> further renders virtual images, which may include the use of certain virtual object data <NUM>, wherein the data may be stored in a memory <NUM>. The processing unit <NUM> generates merged image data <NUM> including utilization of the updated viewing matrix and depth map, wherein the merged data is provided to a display <NUM> to provide the augmented reality image for the user. See, for example, the processes illustrated in <FIG> and described below.

<FIG> illustrates a micropower impulse radar component or system according to an unclaimed aspect. A MIR system <NUM> for augmented reality operation includes a noise source <NUM> to provide noise data to a pulse repetition generator <NUM> to vary the amount of time between signal pulses. The noise source is provided for the purpose of allowing multiple MIR radars to operate in the same space at the same time. The variable time between pulses, together with the extremely low duty-cycle (pulse width compared to pulse repetition rate) makes it extremely unlikely that a pulse from one MIR radar would be received and mistaken for a pulse sent from a second MIR radar operating in the same space. Therefore, MIR radar units have the desirable feature that they do not interfere with one another even when multiple units are simultaneously operating in the same space, such as, for example, multiple MIR units operating simultaneously in a mobile device. The pulse repetition generator provides a pulse signal to an impulse generator and to a delay <NUM>, the delay including a range control <NUM>.

The impulse generator <NUM> generates impulses for transmission via send/receiver antenna <NUM>. A send antenna <NUM> produces a transmitted signal <NUM>, which is reflected as a return signal <NUM> upon impacting an object. The return signal is received at a receive antenna <NUM> (which may be the same unit as the send antenna <NUM>). The received signal is provided to an impulse receiver <NUM>, which also receives a delayed pulse from the delay <NUM> as controlled by a range gate <NUM>. The delay time is swept to take samples with an impulse receiver <NUM> at different times, and thus distances from the antennae, resulting in a record of pulses correlated to object distances. The impulse receiver <NUM> provides a resulting signal to a processor <NUM> to process the received signals to generate one or both of object position data and user position data. The processor utilizes the object position data and virtual object data based upon a viewpoint of a user in the generation of images for an augmented reality display. During rendering of a virtual object, the z-buffer causes appropriate parts of the virtual object to be occluded and thus are not generated, which occurs prior to the alpha compositing operations, which merge camera and virtual image data. The alpha-compositing operations do not need to be aware of the z-buffer. However, the images that are generated match up correctly with the actual environment due to their accurate registration during the 3D rendering process.

As structured, a MIR system <NUM> can be very small in size and consume little power, and thus may be utilized in a mobile device having limited space and power capacity. The generated MIR impulses are very short compared to the repetition rate, even if said repetition rate is several Megahertz. For this reason, many MIR systems can be operating simultaneously in the same space, wherein the impulses from one MIR are not time-correlated with impulses from another because of the noise generation in the MIR system. Therefore, a MIR system <NUM> with the architecture illustrated in <FIG> has great immunity to extraneous signals, including impulses from multiple MIR components operating in the same space. This property allows several MIR components to operate in concert, operating simultaneously, in a mobile device without significant interference. It is further noted that MIR technology uses extremely low power levels, and thus the human exposure to the signal output from a device is generally a fraction of the RF power employed by a typical cellular phone.

A mobile device such as a tablet computer includes multiple mounted MIR antennae in a spread pattern, such as an antenna mounted at each corner of the cover of the mobile device. In this example, four antennae may be facing the user (from the front of the device) and four antennae may be facing away from the user (from the rear of the device). From each such antenna it is possible to record time-of-flight for an impulse signal that is reflected. Each discrete time of flight of the radar impulse represents a surface of a sphere with center at a particular MIR that transmitted it. In this example, by combining measurements from each corner antenna, the intersection of four spheres of radius determined by the time of flight at each antenna is obtained. An object at a particular distance and heading from any three antennas facing in the same direction (toward or away from the user) will have returns in the three spheres that intersect at the point in space occupied by the object. Thus, by examining returns in appropriate radius spheres and by the use of well-known trigonometric algorithms applied to these measurements, radial distances from the user's eyepoint to objects in the environment can be computed and used to initialize a z-buffer.

By making an assumption that objects appear stationary relative to a tablet computer at the rate at which measurements are made, and by discarding objects or features in the environment that do not remain in static spatial relationships to one another (or in other words, by discarding objects that move), it is thus possible to accurately determine the position and orientation of the tablet computer relative to the environment sampled by the array of MIRs.

<FIG> illustrates generation of an augmented reality image by a mobile device according to an embodiment. In <FIG>, a mobile device <NUM>, such as a tablet computer, is utilized for augmented reality in a natural setting. As illustrated, a virtual image <NUM> (the dinosaur) is superimposed upon the real environment in the viewing screen <NUM>.

In order to portray a virtual object such as the dinosaur image <NUM> in <FIG> realistically for a viewer, it is necessary that the position of the virtual object does not appear to move unnaturally relative to the environment, such as, in this example, moving unnaturally relative to the ground. Although the dinosaur virtual image itself may appear to move in the scene, the feet of the dinosaur should not appear to move relative to the ground for as long as the feet are in contact with the ground. In addition, a portion of the virtual object, the tail of the dinosaur image <NUM>, is partially occluded by a real object, a tree. In order to provide a realistic and natural augmented reality image, it is necessary that the intersection of the tree and the tail be aligned accurately relative to the user's viewpoint such that the portion of the virtual object that is occluded appears to disappear at the precise location (sometimes called the silhouette edge) where the tree begins.

The position of a virtual object depends upon the numerical values in the viewing matrix, which in turn depends upon the shape and orientation of the viewing frustum. A viewing matrix is produced using MIR data with values that precisely position the virtual object in a virtual world that is superimposed upon the real world everywhere within the viewing frustum defined by the user's viewpoint (more specifically, the user's dominant eye), and the edges of the display area of the tablet.

<FIG> illustrate the generation and superimposition of images to create the augmented reality image provided in <FIG> according to an embodiment:
<FIG> illustrates an adjusted camera view of the environment, showing real objects <NUM> such as the illustrated trees of the environment. The adjusted camera view is generated by capturing the actual environment and adjusting the view of the environment such the view matches the view of the dominant eye of the user. This may be seen as the adjustment of the camera viewing frustum <NUM> to match the current user viewing frustum <NUM>, as illustrated in <FIG>.

<FIG> illustrates the rendering of a virtual object view for use in an augmented reality image. A virtual image <NUM> is rendered for augmented reality imaging based on current position information, wherein the rendering of the image excludes image portions that would be occluded by real objects. In this example, based on the virtual distance of the image and the distances of the actual objects as determined by MIR, a portion of the virtual image is not rendered. The portion that is not rendered is located at the determined actual object silhouette from the user's viewpoint, such as defined by the actual object silhouette ray from the user's viewpoint <NUM> illustrated in <FIG>.

<FIG> illustrates a virtual information view for an augmented reality image. The rendering of a 3D image further includes the rendering of an information view <NUM> that provides information regarding one or more virtual objects that are contained in the virtual view of <FIG>. The virtual view may appear only when the virtual object <NUM> illustrated in <FIG> is wholly or mostly within the user's viewing frustum. It may not be necessary for the information view <NUM> to be anchored to the environment in the same manner as virtual object <NUM>, and thus the information view <NUM> may swim in relation to the actual objects in a view, and may, for example, be rendered to be in front of any actual objects in an augmented image such that the image is not occluded by the actual objects.

<FIG> illustrates a compositing stack for the generation of augmented reality image. As illustrated in <FIG>, a compositing stack for the generation of the augmented reality image for the view of the dominant eye of the user <NUM> includes a first layer providing adjusted camera video containing real objects <NUM> (as shown in <FIG>),
which is overlaid with a second layer containing the virtual 3D dinosaur virtual object <NUM> (as shown in <FIG>), which is overlaid with a third layer containing the information view virtual object <NUM>.

<FIG> illustrates virtual objects for the generation of an augmented reality image. As shown, a combined image includes the virtual image <NUM>
overlaid with the information view <NUM>.

<FIG> illustrates a resulting composited augmented reality view. As shown, the real objects <NUM> of the adjusted video camera view are overlaid with the virtual image <NUM> and the information view <NUM>. With the adjusted camera view of the real environment the virtual image <NUM> is rendered to precisely match the silhouette boundary of
the real objects <NUM>, and the virtual image <NUM> is rendered to tie the virtual image <NUM> to the environment such that the virtual image does not appear to swim in the real environment.

<FIG> illustrates placement of a virtual object in an augmented reality image. A mobile device utilizes multiple MIR elements <NUM> (which may include individual MIR sub-systems) to transmit radar signals <NUM> and to receive return signals <NUM>. As illustrated in <FIG>, a viewing frustum <NUM> is from a perspective behind and to the right of the virtual object <NUM> (a dinosaur image), from the mobile device user's vantage point <NUM> (the dominant eye of the mobile device user). From the vantage point of a camera capturing <FIG>, if the virtual object <NUM> actually existed in the real world, the occluded portion of the tail would be seen, but such portion of virtual object <NUM> is occluded by the real object <NUM> (tree) from the mobile device user's view.

By means of the multiple antennae (four antennae in the <FIG> illustration) that face the user (wavefronts moving towards the mobile device user in the figure), the position of the user's dominant eye <NUM> can be accurately determined. The position <NUM> is the apex of the viewing frustum <NUM> illustrated in <FIG>. Knowing the apex point, and knowing the precise position and orientation of the mobile device relative to the environment, (obtained by means of the four antennae and the wavefronts moving away from the user) a viewing matrix is generated that defines the viewing frustum <NUM> in the virtual world in alignment with the physical world as seen at an instant in time by the mobile device user, with the viewing matrix being made available to a 3D rendering program that is drawing the dinosaur virtual object <NUM>.

In order to occlude the appropriate portion (the tail, in this example,) of virtual objects, the virtual objects in the environment are rendered using a depth map that has been pre-initialized with a depth map. The initialization of the depth map may be captured by the MIR components, wherein the depth of each point in the video raster is measured by the MIRs, or using a 3D camera component.

<FIG> is a flow chart to illustrate an embodiment of a process for determination of device position and orientation using micropower impulse radar. Upon operation of an electronic device being enabled <NUM> and the device entering an augmented reality application <NUM>, the MIR system of the electronic device may be enabled <NUM> (if the MIR components or sub-systems are not currently enabled) and an image depth map and viewing matrix may be initialized (where such initialization may be based on MIR operation, video camera operation, or other similar operation). The augmented reality application may include a request for identification of the user's dominant eye if this information has not previously been received.

The processing may include both a radar data process <NUM> and a 3D image generation process <NUM>. The radar data process <NUM> may include transmitting radar signals from multiple radar antennae of the electronic device <NUM>, where such transmission includes transmission of signals both towards the real environment for determination of the display's position and orientation and towards the user for determination of the user's vantage point. Return signals are received at the MIR antennae <NUM>, the return signals providing reflection from actual objects in the environment and from the user. The process further includes determining display position data and user position data <NUM> and the updating of the depth map and viewing matrix <NUM>, which define the depth of objects in the environment and provide an accurate location and orientation for the mobile device in relation to the actual objects in the environment and the position of the user in relation to the mobile device. In some embodiments, the generated data may be stored, as indicated by position data <NUM>, for use in generation of augmented reality images. In some embodiments, the radar data process <NUM> operates quickly, and thus current position data <NUM> is available when needed for generation of images.

In some embodiments, the 3D image generation process <NUM> may include rendering of virtual image objects <NUM>, which may include obtaining the position data <NUM> and certain virtual image data from memory <NUM>. In some embodiments, the process further includes obtaining digital camera video data <NUM> and processing the video data to superimpose the camera data and the user's dominant eye point <NUM> to generate an adjusted camera video for the user's viewpoint.

In some embodiments, the process includes generation of merged image data <NUM> by layering of the virtual image with the adjusted camera view, wherein the merged image data utilizes the image depth map and viewing matrix to accurately position virtual images and provide accurate occlusion of the virtual images, resulting in a natural placement of the virtual image in the merged image data according to the current viewing frustum of the user. In some embodiments, the process includes displaying an augmented reality image based upon the merged image data <NUM>.

In some embodiments, if there is no exit from the augmented reality application, the radar data process <NUM> and the 3D image generation process <NUM> continue. If there is an exit from the augmented reality application <NUM>, then the MIR system or systems may be disabled <NUM> (if appropriate) and process ended <NUM>.

<FIG> is an illustration of an embodiment of an electronic apparatus or system to provide for determination of display position and orientation using micropower impulse radar. In this illustration, certain standard and well-known components that are not germane to the present description are not shown. Elements shown as separate elements may be combined, including, for example, an SoC (System on Chip) combining multiple elements on a single chip. The apparatus or system may include, but is not limited to, a mobile device such as a tablet computer.

The apparatus or system <NUM> (referred to generally herein as an apparatus) includes a cover <NUM>. The apparatus further includes multiple radar antennae, including a first radar antenna <NUM>, a second radar antenna <NUM>, a third radar antenna <NUM>, and a fourth radar antenna <NUM> for the transmission and reception of video images. While connections for only a first set of antennae are shown for ease of illustration, the mobile device includes a first set of antennae to track object positions in first direction, such as the illustrated antennae <NUM>, <NUM>, <NUM>, and <NUM>, and a second set of antennae to track a position of the vantage point of a user of the mobile device (the dominant eye of the user of the mobile device) in a second direction, such as an additional set of antennae <NUM>, <NUM>, <NUM>, and <NUM>. The radar antennae may be spread spatially to allow for separation of the timing of each impulse signal, such as, for example, the location of an antenna in each corner of the apparatus <NUM>. Each of the antennae is in a different location in the apparatus. The antennae <NUM>, <NUM>, <NUM>, <NUM> transmit and receive radar data a plurality of MIR components or sub-systems <NUM>. The antennae are directed towards an environment for determination of a position and orientation of the apparatus <NUM> with respect to the objects in the environment. In some embodiments, there is an additional set of antennae <NUM>, <NUM>, <NUM>, <NUM> directed in an opposite direction (towards a user of the apparatus) to determine the position of the vantage point of the user in relation to the apparatus <NUM>.

The apparatus <NUM> includes an interconnect <NUM> or other communication means for transmission of data, the MIR components <NUM> being coupled with the interconnect <NUM>. The interconnect <NUM> is illustrated as a single interconnect for simplicity, but may represent multiple different interconnects or buses and the component connections to such interconnects may vary. The interconnect <NUM> shown in <FIG> is an abstraction that represents any one or more separate physical buses, point-to-point connections, or both connected by appropriate bridges, adapters, or controllers.

The apparatus <NUM> may include a processing means such as one or more processors <NUM> coupled to the interconnect <NUM> for processing information. The processors <NUM> may comprise one or more physical processors and one or more logical processors. In some embodiments, the processors may include one or more general-purpose processors or special- processor processors. In some embodiments, the processors <NUM> operate to process radar signal data to generate data describing the position and orientation of the apparatus in relation to real objects in an environment, the vantage point of a user of the apparatus <NUM>, or both.

The apparatus includes one or more image capture elements <NUM>, such as a camera, which may include the capture of images of an environment for generation of augmented reality images. The image capture elements include a 3D camera, wherein the 3D camera may be utilized in the initialization of position data for the apparatus <NUM>.

In some embodiments, the apparatus <NUM> further comprises a random access memory (RAM) or other dynamic storage device or element as a main memory <NUM> for storing information and instructions to be executed by the processors <NUM>. The apparatus <NUM> may include one or more non-volatile memory elements <NUM>, including, for example, flash memory, for the storage of certain elements. The apparatus <NUM> also may comprise a read only memory (ROM) <NUM> or other static storage device for storing static information and instructions for the processors <NUM>, and data storage <NUM>, such as a solid state drive, for the storage of data. Memory of the apparatus <NUM> may include storage of data related to the determination of position data, such as which of the user's eyes is dominant, and the generation of image data for the presentation of augmented reality images.

The apparatus <NUM> includes one or more transmitters or receivers <NUM> coupled to the interconnect <NUM>. The apparatus <NUM> may include one or more antennae <NUM> for the transmission and reception of data via wireless communication, where the antennae may include bipole and monopole antennae, and one or more ports <NUM> for the transmission and reception of data via wired communications.

The apparatus <NUM> includes one or more input devices <NUM> for the input of data, including hard and soft buttons, ajoy stick, a mouse or other pointing device, voice command system, or gesture recognition system.

The apparatus <NUM> includes an output display <NUM>, where the display <NUM> may include a liquid crystal display (LCD) or any other display technology, for displaying information or content to a user. In some environments, the display <NUM> may include a touch-screen that is also utilized as at least a part of an input device <NUM>. The output display <NUM> may be utilized in the display of augmented reality images.

The apparatus <NUM> may also comprise a battery or other power source <NUM>, which may include a solar cell, a fuel cell, a charged capacitor, near field inductive coupling, or other system or device for providing or generating power in the apparatus <NUM>. The power provided by the power source <NUM> may be distributed as required to elements of the apparatus <NUM>.

In the description above, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the claimed invention. It will be apparent, however, to one skilled in the art that the inventions may be practiced without some of these specific details. In other instances, well-known structures and devices are shown in block diagram form. There may be intermediate structure between illustrated components. The components described or illustrated herein may have additional inputs or outputs that are not illustrated or described.

The claimed invention includes various processes. These processes may be performed by hardware components or may be embodied in computer program or machine- executable instructions, which may be used to cause a general-purpose or special-purpose processor or logic circuits programmed with the instructions to perform the processes. Alternatively, the processes may be performed by a combination of hardware and software.

Claim 1:
A mobile device (<NUM>, <NUM>) comprising
a camera;
a display (<NUM>) to present images captured with the camera;
radar components (<NUM>) to generate a plurality of radar signal pulses and to generate distance data based on received return signals;
a first plurality of radar antennae (<NUM>, <NUM>, <NUM>, <NUM>) to transmit the plurality of radar signal pulses in a rear facing direction of the apparatus and a second plurality of radar antennae (<NUM>, <NUM>, <NUM>, <NUM>) to transmit the plurality of radar signal pulses in a front facing direction of the apparatus and to receive the return signals; and
a processor (<NUM>) to process signals and data, wherein the processor is to:
process the return signals received by the first and second plurality of radar antennae to determine a position and orientation of the display with respect to real objects in an environment and to determine a position of a vantage point of a user of the apparatus, and
generate an augmented image including rendering a virtual object and superimposing the virtual object on an image including one or more real objects, the rendering of the virtual image being based at least in part on the determined position and orientation of the display and the determined vantage point of the user of the apparatus;
wherein processing the return signals includes updating a depth map providing distances of real objects and updating a viewing matrix, the viewing matrix defining a viewing frustum of a user of the apparatus.