Patent Description:
At the same time, augmented reality (AR) applications programs (apps) executing on the mobile computing devices have become ubiquitous. These apps executing on the mobile computing devices overlay virtual content with images captured, typically, by rear-facing cameras of the devices. <CIT> discloses a mobile platform that displays a corrected view of an image and/ or augmented reality (AR) data based on the position of the user with respect to the mobile platform. The corrected view is produced by determining a position of the user with respect to the mobile platform using an image of the user from a backward facing camera. The display information is provided in the form of an image or video frame of the environment captured with a forward facing camera or AR data. The position of the user with respect to the mobile platform is used to determine the portion of the display information to be displayed that is aligned with the line of sight between the user and the mobile platform so that the displayed information is aligned with the real world environment
<CIT> discloses a handheld device that has a display and a front-facing sensor and a back-facing sensor and is able to render 3D content in a realistic and spatially correct manner using position-dependent rendering and view-dependent rendering. In one scenario, the 3D con-tent is only computer-generated content and the display on the device is a typical, non-transparent (opaque) display. The position-dependent rendering is performed using either the back-facing sensor or a front-facing sensor having a wide-angle lens. In another scenario, the 3D content is composed of computer-generated 3D content and images of physical objects and the display is either a transparent or semi-transparent display where physical objects behind the device show through the display. In this case, position-dependent rendering is performed using a back-facing sensor that is actuated (capable of physical panning and tilting) or is wide-angle, thereby enabling virtual panning. <CIT> discloses an image sensor, in which some pixels in an array contain a sampling circuit to sample the light intensity and a capacitor to store an analog value representing the intensity at that pixel. Alternatively, a group of pixel circuits will be equipped with such sampling and capacitor circuits. This allows simple redundancy-reducing computations with a relatively simple pixel architecture.

Often, the rendering of the captured images and virtual content is performed from the perspective of the rear-facing camera. This has the effect of creating discontinuities between the image displayed around the periphery of the device's display and the surrounding environment, from the perspective of the user, i.e., the AR scene does not blend with the surrounding real world scene from the viewpoint of the user. This impairs the AR illusion since the captured images and the AR content are not aligned with respect to the surrounding environment and the same object may be seen twice, once on the device's display and once within the user's visual perspective.

Some have proposed to track the user's viewpoint and achieve better blending between the AR scene and the real world scene. If the device could display images that were calibrated based on the user's perspective, both in terms of magnification and position, then the device's display could blend into the surrounding environment creating an illusion of the device's transparency.

Successful blending requires high speed tracking of the user's viewpoint and real world scene by the mobile computing device. A critical figure of merit of machine vision systems is the latency, which is the time that passes between the moment the light is received and the moment the rendered AR scene is updated.

Event-based vision sensors offer many advantages, mainly by intrinsically compressing the data stream and thus reducing the amount of data that a processing unit needs to analyze. Furthermore, the event-based vision sensors' pixels continuously sense the visual changes in the scene and report them with a very low latency. This makes the event-based vision sensor an ideal sensor for always-on tasks such as visual tracking and smart sensor control or data enhancement of secondary sensing modalities.

In general, according to one aspect, the invention features a method for rendering an augmented reality scene on a mobile computing device and a mobile computing device as defined in the independent claims. It should be noted that the present invention is solely defined by the claims. Embodiments not covered by the claims do not form part of the present invention.

The claimed subject-matter will now be more particularly described with reference to the accompanying drawings.

It will be understood that the particular method and device embodying the invention are shown by way of illustration and not as a limitation of the invention.

This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention as set out in the appended claims to those skilled in the art.

<FIG> depicts a typical scenario. A user <NUM> views the surrounding environment characterized by the user's field of view <NUM>. In the illustrated example, within that user's field of view <NUM> is a user device <NUM>.

In the typical example, the user device <NUM> is a mobile computing device such as a smartphone or tablet computing device. Other devices include televisions, transparent displays (see-through displays), light field displays, holographic displays, projector based systems, 3d displays, stereoscopic displays, and automobile heads-up display systems.

As is common, the mobile computing device <NUM> often has a touch screen display <NUM>. And, as also common to such devices, the user device <NUM> includes a front sensor assembly <NUM> and a rear sensor assembly <NUM>.

The front sensor assembly <NUM> and the rear sensor assembly <NUM> have corresponding fields of view <NUM>, <NUM>. These fields of view <NUM>, <NUM> are characterized by the fields of view of the various sensors that are included in each of these sensor assemblies <NUM>, <NUM>. In many cases, the individual sensors of these assemblies may have respective wider or narrower fields of view. However, for the purposes of the following description, the fields of view are treated collectively and generally coextensive with each other.

In the illustrated example, the user device <NUM> intersects a portion of the field of view <NUM> of the user <NUM>. As a result, certain real world objects, such as the real world cube <NUM>, will be fully or partially obscured by the body of the user device <NUM> as it obscures part of the user's field of view <NUM>. Nevertheless, a portion <NUM>-<NUM> of this real world object <NUM> may still be directly observable by the user around an edge of the device <NUM>.

The user device <NUM> in the typical example executes an augmented reality application. As a result, additional virtual objects <NUM> will be generated by that AR application executing on the user device <NUM> and displayed on the display <NUM>.

In most AR applications, the AR scene <NUM> rendered on the display <NUM> of the computing device <NUM> will be a combination of the real world objects <NUM> that are captured by a camera of the rear sensor assembly <NUM> along with virtual objects <NUM> that are generated by the AR app and rendered in the AR scene <NUM>.

As opposed to device-perspective rendering, where the display <NUM> shows the AR scene <NUM> from the perspective of the rear sensor assembly <NUM>, the transparent smartphone AR app employs user-perspective rendering, which makes the AR scene <NUM> on the display <NUM> blend with the real world scene <NUM> when observed from the viewpoint <NUM> of the user <NUM>. Hence, the AR scene <NUM> content is defined by the user-perspective-rendering field of view <NUM> defined as the portion of the scene obscured by the display <NUM> when viewed from the user viewpoint <NUM>.

Often, the user viewpoint is deliberately not specifically the eye, since it can be the right, the left or even in between the user's eyes. The viewpoint can also change during the runtime, for instance, as a function of the user distance to the display <NUM>. If close, the app executing on the device <NUM> might select a specific eye (dominant eye). Furthermore some displays, for example 3D displays, might require two viewpoints (left and right eyes).

<FIG> shows the user device <NUM> from the perspective of the user <NUM>. The AR scene <NUM> rendered on the display <NUM> includes a background AR scene that was captured by the rear sensor assembly <NUM>. At the same time, the AR scene <NUM> contains virtual objects <NUM> that have been rendered along with the real objects <NUM>.

In this user perspective display, the real world objects <NUM> might be partially observable <NUM>-<NUM> within the user's field of view. These real-world objects are displayed in a blended fashion so that from the standpoint of the user <NUM>, the real world objects will be displayed within the AR scene <NUM> such that they are blended with the other parts of those objects that are directly observable. In addition, other features of the scene <NUM> will form part of a continuous scene with those same parts of the scene that are displayed in the AR scene <NUM> to yield the illusion of the transparency of the device <NUM>.

<FIG> shows the basic block diagram of the user device <NUM>.

It includes a processor assembly <NUM>. Often this includes a central processing unit and possibly a graphic processing unit. In addition, the processor assembly will include random access memory along with program memory. Typically, the AR app <NUM> will be loaded into the program memory of the processor assembly <NUM> for execution by the central processing unit.

In addition, it is also common for the device <NUM> to include an inertial measurement unit (IMU) <NUM>. Often, these IMUs include magnetometers, accelerometers, and gyroscopes. These are used to determine the pose of the user device <NUM> by determining its orientation within the earth's gravitational field along with rotational and translational movement.

Also shown are exemplary front sensor assembly <NUM> and rear sensor assembly <NUM>. In the typical example, the front sensor assembly includes a front event-based vision sensor <NUM>, a front image sensor such as a CMOS image sensor <NUM>, and a front depth sensor <NUM>. In a similar vein, the rear sensor assembly <NUM> includes a rear event-based vision sensor <NUM>, and a rear image sensor <NUM>, and a rear depth sensor <NUM>.

More generally, a list of possible depth sensor technologies for the front sensor assembly <NUM> includes but is not limited to structured light source and image sensor, time of flight (ToF) depth sensor, Lidar, Radar, stereo camera systems, stereo event-based vision sensor systems and all possible combinations of the latter.

List of possible sensors in the rear sensor assembly <NUM> possibly include (not limited to) structured light source and image sensor system, ToF sensor, Lidar, Radar, stereo camera systems, and stereo event-based vision sensor systems.

The processor assembly <NUM> typically via an integrated graphics processing unit drives the display <NUM> specifically to render the AR scene <NUM> as generated by the AR app <NUM>.

<FIG> is a flow diagram showing the operation of the AR app <NUM> in order to generate an AR scene <NUM> that is blended with the real world scene <NUM>. Generally, the AR app <NUM> must resolve the user's viewpoint as captured by the front sensor assembly <NUM>, the pose of the device <NUM>, and map the real world scene <NUM> as captured by the rear sensor assembly <NUM>.

The sub methods (dashed boxes) are typically running in separate threads and run at different update frequencies, but they can also be run sequentially. The rendering and display method for the user-perspective rendering of the scene can be performed using state of the art computer graphics theory and methods.

The rendering and display method could employ machine learning methods to more seamlessly blend the display with the scene. The method could adjust the display pixel brightness, contrast or further properties using for instance a deep neural network like a CNN. The rendering of the scene map images may also be improved or completely performed using machine learning (e.g. CNN). A possible training method may be mounting a camera between the user's eye (e.g. with using glasses) and then optimize for the smoothness in the transition from screen image to the background image.

In more detail, in step <NUM>, the AR app <NUM> tracks the user's viewpoint position relative to the display <NUM> using the front sensor assembly <NUM>.

Typically at the same time, in step <NUM>, the AR app <NUM> updates the resolved pose of the device <NUM> relative to the real world scene <NUM> based on data from the inertial measurement unit (IMU) <NUM> and/or from the rear sensor assembly <NUM>.

Further, often in parallel, the AR app <NUM> also receives the information such as images from the rear sensor assembly <NUM> and updates the scene map in steps <NUM>.

In step <NUM>, the AR scene <NUM> is resolved from the user perspective in step <NUM> and this AR scene <NUM> is then provided to the display <NUM> in step <NUM>.

<FIG> breaks-down the viewpoint tracking step <NUM> of <FIG>.

In more detail, event-based vision sensor data is received from the front event-based vision sensor <NUM> of the front sensor assembly <NUM>. Then, in step <NUM>, this data is analyzed to determine whether the user viewpoint has displaced. For example, the data is analyzed by the processor assembly <NUM> to determine whether the user's gaze has shifted by determining eye movement. If analysis of the information from the front event-based vision sensor <NUM> indicates that there has been no viewpoint change in step <NUM>, then no update to the user's viewpoint needs to be made.

These steps are based on, for example, observation of event counts or event rate in specific regions of interest to decide if a tracking update is necessary. This helps to save power since the more expensive tracking algorithm and potentially readings from further sensors are only executed when necessary.

The whole method can be executed at a fixed frequency or asynchronously for every event or every N-th event (where N can vary for different executions of the method) or every event packet or with any other suitable policy.

On the other hand, if it is determined that the viewpoint has displaced in step <NUM>, then, in step <NUM>, additional data is acquired from the front sensor assembly <NUM> in step <NUM>. Specifically, the depth sensor <NUM> is often activated to determine whether or not the user has moved their head to change their direction of gaze and thus viewpoint. In addition, the images of the user are often captured by the front image sensor <NUM>. These images are analyzed to determine whether or not the viewpoint of the user <NUM> has changed and determine what is the user's new viewpoint. Then, in step <NUM>, the user's viewpoint position is updated and tracked relative to the display using the data from the front sensor assembly <NUM>. In step <NUM>, a new viewpoint position is provided to the AR app <NUM> so that the AR scene <NUM> can update so that it blends with the real world scene <NUM>.

In general, the tracking can be based on events only or based on event data and other sensors data. In one embodiment at the beginning of the tracking and/or periodically one would acquire additional front image sensor and front depth sensor data for the detection of the viewpoint and or enhance the tracking procedure. In between, the viewpoint tracking happens solely based on event-based vision sensor data employing state of the art tracking algorithms, which allows for a very high update rate and low update latency due to the event-based vision sensor's low latency and high frequency data readout.

<FIG> shows the details of the scene mapping step <NUM> of <FIG>.

In more detail, in step <NUM>, AR app executing on the processor assembly <NUM> receives rear sensor assembly data from the rear event based vision sensor <NUM>.

Preferably, instead of mapping the scene at a constant readout rate of the rear sensor assembly <NUM>, this method only triggers expensive sensor readings and software updates when the real world scene <NUM> as observed by the rear sensor assembly <NUM> has changed due to motion of objects in the scene or of apparatus itself. The event-based vision sensor data is employed to sense object motion at high frequency and low latency.

The processor assembly processes apparatus motion and pose estimation information in step <NUM>.

In step <NUM>, the AR app <NUM> identifies the various objects within the real world scene <NUM> and estimates their motion using the data from the rear event based vision sensor <NUM> and the estimated apparatus motion aquired in step <NUM> and the current AR scene map.

In step <NUM>, the AR app assesses whether an object <NUM> in the real world scene <NUM> has moved or not. At the same time, it is determined whether a map update is required based on apparatus motion in step <NUM>.

In either of these two events, it is determined whether additional data is required and that data is acquired from the rear sensor assembly in steps <NUM> and <NUM>. From this information, in step <NUM>, the AR scene map is updated and, in step <NUM>, it is provided to the AR app <NUM> so that the AR scene <NUM> can update so that it blends with the real world scene <NUM>.

<FIG> shows the details of the device pose estimation method of step <NUM> of <FIG>.

In general, a state of the art SLAM (Simultaneous localization and mapping) method is used to estimate the position of the apparatus in space. The SLAM method estimates the apparatus motion constantly based on IMU readings. In order to reduce the pose estimation drift caused by integrating IMU measurement readings, which are subject to noise, the SLAM method periodically requires additional measurements from sensors allowing the measurement of absolute position. Event-based vision sensor data can be used for this purpose by, for instance, creating image frames using state of the art algorithms (intensity images and/or gradient images and/or edge images and/or event accumulation images) that allow the position estimation of the camera in space. The high update rate nature of the event-based vision sensor allows very high pose update rates, beneficial during fast motions of the apparatus. Also, the events might be used to enhance sensor data from other sensors, for instance the CMOS image sensor, by employing them for removing motion blur on the CMOS sensor image. Furthermore, events could be used to give the SLAM methods queues about the ego motion of the device, in the case where no IMU data is available. Also the event data reveal the SLAM pipeline information about moving objects in the scene, which can render the SLAM method more robust.

The whole method can be executed at a fixed frequency or every N-th IMU reading or asynchronously for every event or every N-th event (where N can vary for different executions of the method) or every event packet or with any other suitable policy.

In more detail, in step <NUM>, the AR app <NUM> receives information from the IMU <NUM> and/or from the rear event-based vision sensor <NUM>. Then in step <NUM>, it is determined whether or not an update of the pose of the device <NUM> is required.

If an update is required then additional data from the rear sensor assembly including the rear event-based vision sensor <NUM> is acquired in steps <NUM> and <NUM>. This information and/or the IMU data is used to update the estimation of the pose in step <NUM>. Finally, new apparatus pose update is provided to the scene mapping method and the rendering method in step <NUM>.

For background, <FIG> shows components of one possible pixel circuit <NUM>, for the event-based vision sensors <NUM>, <NUM>.

The major components of a pixel circuit <NUM> are enumerated below.

The peripheral circuits contain the controller <NUM> which applies threshold signals to the comparator A1, sends control signals to memory <NUM> and selects times when the conditional reset circuit R1 becomes active.

The peripheral circuits may also contain a readout circuit RO which reads the content of the memory <NUM>, determines if the light intensity for a given pixel has increased, decreased, or unchanged, and sends the output (computed from the current memory value) to a processor.

Generally, an OFF-event is a discrete decrease in light intensity for a given pixel. On the other hand, an ON-event is a discrete increase in light intensity for a given pixel.

In more detail, the comparator tells if light has increased/decreased. For OFF event: if Vdiff is lower than the threshold Voff (on Vb), the comparator output is high, and this level is stored in the memory. This means a decrease is detected. If Vdiff is not lower than the threshold, the comparator output is low: no decrease detected.

The only difficulty is that for ON-event, a low comparator output means an increase, while high means no change; but for OFF event high comparator output means decrease while low means no change.

So the readout must know the memory content and which threshold was applied. Or, as in the preferred embodiment, described later, there is an inverter for On so that the memory polarity is the same for both on and off.

In one preferred embodiment of the pixel circuit <NUM> of the present invention, each pixel circuit <NUM> contains one comparator only, which sequentially acts first as comparator for ON-events, then as comparator for OFF-events (or vice-versa).

The pixel circuit <NUM> and controller <NUM> operate as follows.

A change in light intensity received by the photosensor PD will translate to a change in photoreceptor signal Vpr. When the reset circuit R1 is not conducting, the changes in Vpr will be reflected also in the voltage Vdiff at a comparator node at the inverting input (-) to the comparator A1. This occurs because the voltage across the memory capacitor C1 stays constant.

At times selected by the controller <NUM>, the comparator A1 compares the voltage at the comparator node at the second terminal of the memory capacitor C1 (Vdiff) to a threshold voltage Vb (from controller) applied to the non-inverting input (+) of the comparator A1.

The controller <NUM> operates the memory <NUM> to store the comparator output Vcomp. The memory <NUM> is typically implemented as part of the pixel circuit <NUM> as shown. In other embodiments, however, the memory <NUM> is implemented as part of column logic circuit (peripheral circuit, one per each column of the pixel array).

If the state of the stored comparator output held in the memory <NUM> indicates a change in light intensity AND the global reset signal GlobalReset signal from the controller <NUM> is active, the conditional reset circuit R1 is conducting. Here "AND" indicates the logical AND operator. With the conditional reset circuit R1 in a conductive state, the voltage at the comparator node at the inverting input of the comparator A1 (Vdiff) is reset to a known level. Thus, it stores the current photoreceptor signal Vpr on the memory capacitor C1.

<FIG> shows an exemplary event-based vision sensor <NUM>, <NUM> comprising a two-dimensional array of pixels <NUM>-<NUM> through <NUM>-<NUM>. The illustrated sensor shows only two rows and only three columns to avoid cluttering the figure. In practice the sensor <NUM>, <NUM> would comprise of m rows (typically much greater than <NUM>) and n columns (typically much greater than <NUM>) of pixels. A pixel in a two dimensional array can be identified by its address which is the pixel's row number and column number. , pixel <NUM>-<NUM> has row <NUM> (counting from top) and column <NUM> (counting from left) as its address.

The controller <NUM> controls pixels <NUM> and the other components such as the row selection circuit <NUM>, the readout circuit <NUM>, and transmission of data from the array to the processor assembly <NUM> and possibly a low power co-processor that handles the event-base vision sensor.

In the illustrated example, the row selection circuit <NUM> is shown as part of the controller <NUM>. This row selection circuit <NUM> selects one or multiple subsets of rows. When a row of pixels <NUM> is selected, the comparator outputs of the pixels in the selected row are conveyed to the readout circuit <NUM>.

The readout circuit <NUM> reads the data (the memorized comparator outputs) from the pixel array. Often the readout circuit <NUM> will further encode this data into a more efficient representation before transmitting to the processor assembly <NUM>.

The readout circuit <NUM>, divided into several column logic circuits <NUM>-<NUM> through <NUM>-n, where there are n columns, determines from reading comparator outputs if the light intensity for the corresponding pixel has increased, decreased or remained unchanged.

Claim 1:
A method for rendering an augmented reality scene on a mobile computing device (<NUM>), comprising:
tracking a real-world scene and/or a viewpoint (<NUM>) of a user (<NUM>) with one or more event- based vision sensors (<NUM>, <NUM>); and
blending an augmented reality scene displayed on the mobile computing device (<NUM>) based on the viewpoint (<NUM>) of the user (<NUM>) and a scene map of the real-world scene and on the tracking of the one or more event-based vision sensors (<NUM>, <NUM>), characterised in that
the event-based vision sensors (<NUM>, <NUM>) detect OFF-events as discrete decreases in light intensity and/or ON-events as discrete increases in light intensity for pixels of the event-based vision sensors (<NUM>, <NUM>); and
the event-based vision sensors (<NUM>, <NUM>) include comparators that compare a difference between photoreceptor signal and past photoreceptor signal to a threshold to determine the OFF-events and/or ON-events.