Patent Description:
For a conventional camera system to perform light-in-flight imaging, the system would require sensitive time-tagging hardware that is relatively expensive and further require ultra-sensitive cameras operating at a trillion frames per second. Accordingly, the hardware required for light-in-flight imaging is not readily able to be commercialized due to these costs and equipment complexities.

According to a first aspect, the present invention provides a method comprising:.

According to further aspects, the method comprises one or more (e.g., all) of the following features (or any combination thereof).

Generating the three-dimensional data structure comprises:
for a given pixel of the depth map, mapping the given pixel to a respective location of the three-dimensional data structure based on two-dimensional coordinates of the given pixel in the depth map and based on a depth value associated with the given pixel.

The convolutional operator defines a Gaussian sphere.

The method further comprises:
identifying a slice comprising a subset of voxels of the filtered three-dimensional data structure centered at a wavefront depth corresponding to the location of the simulated wavefront at the time corresponding to the defined timestamp.

Adjusting the brightnesses of the subset of pixels of the optical image based on the optical simulation output comprises:
performing an affine-transform of the optical image with the identified slice.

The method further comprises:
determining the wavefront depth based on the speed of light and based on the time corresponding to the defined timestamp, wherein the wavefront depth is indicative of the location of simulated wavefront in the optical image at the time corresponding to the defined timestamp.

According to a second aspect, the present invention provides a mobile device comprising:.

According to further aspects, the mobile device comprises one or more (e.g., all) of the following features (or any combination thereof). The mobile device and/or the components comprised within the mobile device are adapted to perform the following features.

To generate the optical simulation output, the processor is further configured to:.

To generate the three-dimensional data structure, the processor is configured to:
for a given pixel of the depth map, map the given pixel to a respective location of the three-dimensional data structure based on two-dimensional coordinates of the given pixel in the depth map and based on a depth value associated with the given pixel.

The processor is configured to:
identify a slice comprising a subset of voxels of the filtered three-dimensional data structure centered at a wavefront depth corresponding to the location of the simulated wavefront at the time corresponding to the defined timestamp.

To adjust the brightnesses of the subset of pixels of the optical image based on the optical simulation output, the processor is configured to:
affine-transforming the optical image with the identified slice to produce the light-in-flight image.

The processor is further configured to:
determine the wavefront depth based on the speed of light and based on the time corresponding to the defined timestamp, wherein the wavefront depth is indicative of the location of the simulated wavefront in the optical image at the time corresponding to the defined timestamp.

According to a third aspect, the present invention provides a method comprising:.

According to a fourth aspect, the present invention provides a computer program product having instructions stored thereon, the instructions, when executed by a processor, cause the processor to perform the steps of a method as described herein.

Embodiments of the present disclosure provide systems and methods by which light-in-flight imaging can be simulated using the camera system of a smartphone or other device. For example, light-in-flight effects can be simulated using a light-in-flight image that employs a red-green-blue (RGB) image (sometimes referred to herein as an "optical image") and depth image (sometimes referred to herein as a "depth map") pair as inputs, which may respectively be captured using an RGB camera (sometimes referred to herein as an "optical camera") and a depth camera of a device such as a smart phone, tablet, pair of smart glasses, or another applicable device that includes such cameras.

In some embodiments, the device may simulate light-in-flight imaging of a wavefront corresponding to the simulated emission of a light pulse from the device or from an object (e.g., a light source) in an augmented reality (AR) scene. For example, when generating a light-in-flight image frame based on a captured RGB image, a light-in-flight engine calculates where the wavefront will be located in the scene depicted in the RGB image at a particular time based on depth information derived from a depth map corresponding to that RGB image, and based on timing parameters that define the timing of the movement of the wavefront over the scene, such as the speed at which the wavefront is to travel across the scene and/or the time period over which the wavefront is to travel across the scene. In some embodiments, to improve the perceptibility of the wavefront and to better simulate how the light packets of the wavefront would bounce off of the objects of the scene, the light-in-flight engine may transform the RGB image by generating an optical simulation of the wavefront, such as by applying a convolutional operator to wavefront pixels of the corresponding depth map, then affine-transforming the original RGB image with the output of the optical simulation to produce a light-in-flight image frame. Affine-transforming the original RGB image with the output of the optical simulation modifies the brightness of the wavefront pixels as well as pixels within a region of influence of the wavefront pixels in three-dimensional space, thus improving the perceptibility of the wavefront pixels.

In some embodiments, the optical simulation may be generated and applied to the original RGB image over multiple sequential timestamps in order to generate a light-in-flight video effect. For example, since the wavefront moves across a given scene over time, the light-in-flight engine may generate a sequence of light-in-flight image frames for multiple timestamps, such that the sequence of light-in-flight image frames depicts motion of the wavefront across the scene of the original RGB image. The light-in-flight engine may then assemble the sequence of light-in-flight image frames to produce a light-in-flight video or animated graphics interchange format (GIF) file, for example, which may be stored on a storage device (e.g., solid state drive, hard disk drive, flash drive, memory, etc.) included in or coupled to the device that executes that executes the light-in-flight engine.

Simulating light-in-flight imaging in this way enables various applications. For example, the light-in-flight engine may be used to apply a unique photo filter to captured images, allowing photographers to introduce novel simulated lighting to captured scenes. As another example, the light-in-flight engine may be applied in AR scenes to simulate the motion of light packets from a selected light source in such AR scenes, which may be desirable for its aesthetic visual effect or which may be used to teach students about the mechanics of light paths.

<FIG> is a block diagram illustrating a mobile device <NUM> that is configured to simulate light-in-flight imaging. As shown the mobile device <NUM> includes one or more RGB cameras <NUM> (sometimes referred to herein as "optical cameras <NUM>"), one or more depth cameras <NUM>, one or more processors <NUM>, a display <NUM>, and one or more storage devices <NUM>. Each of components <NUM>, <NUM>, <NUM>, <NUM>, and <NUM> are coupled (physically, communicatively, and/or operatively) via communication channels <NUM>. In some examples, the communication channels <NUM> may include a system bus, a network connection, an inter-process communication data structure, or any other method for communicating data.

The processor(s) <NUM> may be configured to implement functionality and/or process instructions for execution. For example, the processor(s) <NUM> may be capable of processing instructions stored on the storage device(s) <NUM>. The processor(s) <NUM> may include any or each of a microprocessor, a controller, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or equivalent logic circuitry.

The storage device(s) <NUM> may be configured to store information within the mobile device <NUM> during operation. For example, in some embodiments, the storage device(s) <NUM> include one or more computer-readable storage media, computer-readable storage devices, temporary memory devices, and/or volatile memories such as random access memories (RAM), static random access memories (SRAM), dynamic random access memories (DRAM), or other applicable volatile memories. In some examples, the storage device(s) are used to store program instructions for execution by the processor(s) <NUM>. In some examples, the storage device(s) may further include one or more non-volatile storage elements such as magnetic hard discs, optical discs, floppy discs, flash memories, or forms of electrically programmable memories (EPROM) or electrically erasable and programmable (EEPROM) memories. In some embodiments, the storage device(s) <NUM> are used by software or applications (e.g., the light-in-flight engine <NUM>) running on the mobile device <NUM> to temporarily store information during program execution.

The RGB camera(s) <NUM> may include one or more cameras that include a standard image sensor through which visual (e.g., colored) images of scenes of persons and objects are acquired. Optical images captured by the RGB camera(s) <NUM> may be stored at the storage device(s) <NUM>. Herein, optical images captured by an RGB camera such as the RGB camera(s) <NUM> may sometimes be referred to as "RGB images".

The depth camera(s) <NUM> may capture depth maps of a scene that define the distance between the mobile device <NUM> and some or all of the objects in the scene. In some embodiments, for each RGB image captured by the RGB camera(s) <NUM>, the depth camera(s) <NUM> may capture a corresponding depth map. In some embodiments, the depth camera(s) <NUM> (sometimes referred to as "range cameras") may include one or more Time-of-Flight (ToF) depth cameras, which use pulsed light or continuous wave amplitude modulation to illuminate a scene with infrared light and measure the ToF of the infrared light. ToF may be characterized as the time period spanning the time at which infrared light is emitted from an infrared light source of the depth camera to the time at which the infrared light returns to an infrared light sensor of the depth camera after being reflected off of objects in the scene. It should be understood that other types of depth cameras, such as stereo triangulation cameras and coded aperture cameras, could additionally or alternatively be included in the depth camera(s) <NUM>, according to various embodiments.

The mobile device <NUM> can include additional components that are not shown in <FIG> to reduce clutter. For example, the mobile device <NUM> can include a battery to provide power to the components of the mobile device <NUM>.

In the example of <FIG>, the storage device(s) <NUM> may store RGB images <NUM> captured by the RGB camera(s) <NUM> and depth maps <NUM> captured by the depth camera(s) <NUM>. The storage device(s) <NUM> may include instructions for executing a light-in-flight engine <NUM> with the one or more processors <NUM>. The light-in-flight engine <NUM> receives the RGB images <NUM> and corresponding depth maps <NUM> and generates light-in-flight images <NUM> based on the RGB images <NUM> and the depth maps <NUM>. For example, for each of the RGB images <NUM>, the light-in-flight engine <NUM> generates a sequence of light-in-flight images <NUM> depicting a wavefront moving across the scene of that RGB image. Herein, the terms "wavefront" or "simulated wavefront" generally correspond to the simulated emission of a short light pulse. In some embodiments, the light-in-flight engine <NUM> may simulate the emission of such a short light pulse from a source located at the depth camera(s) <NUM> or RGB cameras <NUM>. In some embodiments, the light-in-flight engine <NUM> may simulate the emission of such a short light pulse from a source disposed at a location within the scene depicted in the RGB image or at a location outside of the scene. In some embodiments, the light-in-flight engine <NUM> may simulate the emission of such a short light pulse from a source at a location in an environment, which may correspond to a physical object in that environment or which may correspond to a virtual object that is associated with the location as part of an augmented reality scene or virtual reality scene.

When simulating light-in-flight imaging of a wavefront of a light pulse moving across the scene of one of the RGB images <NUM> over a given time period, the light-in-flight engine <NUM> is able to determine which pixels of the RGB image <NUM> and which pixels of the corresponding depth map <NUM> correspond to the depth of the wavefront at a given time following emission of a corresponding light pulse based on pixel depth information defined in the depth map <NUM>. For example, like an RGB image, a depth map includes an array of pixels, each having a respective pixel value. However, rather than representing color values, each pixel of a depth map represents a respective depth value corresponding to the distance between the depth camera that captured the depth map and the portion of the scene (i.e., "object patch") to which that pixel corresponds.

Each depth value of the depth map is a function of the time that it takes for light to reach the depth camera multiplied by the speed of light (e.g., d =c*t, where d is distance, c is the speed of light, and t is time). This means that the light-in-flight engine can determine the amount of time it would take for a wavefront to travel to a location corresponding to a specific depth within the scene based on the speed of light. Accordingly, when simulating a wavefront traveling across the scene of an RGB image, the light-in-flight engine <NUM> uses the corresponding depth map of the scene to determine the location of the wavefront in the scene at a given time following the simulated emission of the wavefront. That is, the pixels of the RGB image that correspond to the location of the wavefront at a given time during the wavefront's simulated traversal of the scene may be identified by the light-in-flight engine <NUM> based on the depth values of corresponding pixels of the depth map <NUM> associated with the RGB image <NUM>.

In some embodiments, the light-in-flight engine <NUM> may determine, based on defined timing parameters stored in the storage devices <NUM>, a time period over which the traversal of the wavefront across the scene in a given RGB image is to be simulated. In some embodiments, the light-in-flight engine <NUM> may generate timestamps corresponding to discrete times occurring within the determined time period, and may generate a respective light-in-flight image for each timestamp or for a subset of the timestamps. Herein, a "timestamp" refers to a record of a discrete time occurring during the time period over which propagation of a wavefront across a scene is simulated or is to be simulated. In some embodiments, the light-in-flight engine <NUM> may generate a light-in-flight video by assembling a sequence of the generated light-in-flight images (e.g., temporally sequential light-in-flight images).

Alternatively, in some embodiments, the light-in-flight engine <NUM> may generate a single light-in-flight image corresponding to a single timestamp within the determined time period, which may be defined in the timing parameters or via user input provided by the user of the mobile device <NUM>. For example, it may be desirable to generate only one light-in-flight image from an RGB image at a defined timestamp in some photography applications. It should be understood that, in some embodiments, a computing device other than the mobile device that captured the RGB image and corresponding depth map may execute the light-in-flight engine <NUM> after receiving the RGB image and the depth map. Accordingly, embodiments of the light-in-flight engine described herein are not limited to execution by mobile devices or, specifically, the mobile device that captured the images being processed by the light-in-flight engine.

To visualize the location of the wavefront at a given time following wavefront emission, light-in-flight engine <NUM> may increase the brightness of each of the identified pixels (sometimes referred to herein as the "wavefront pixels") relative to the remaining pixels of the RGB image (sometimes referred to herein as the (non-wavefront pixels"), or may decrease the brightness of each of the non-wavefront pixels of the RGB image relative to the brightness of each of the wavefront pixels, depending on the desired effect.

<FIG> depicts a sequence <NUM> of three different depth images corresponding to three different timestamps of a wavefront's simulated traversal across a scene, where each depth image indicates pixels within the scene corresponding to the simulated wavefront's location (e.g., depth) at a corresponding time. In the present example, the light-in-flight engine <NUM> simulates the wavefront as though it is a short pulse emitted from the mobile device that has captured the images and corresponding depth maps. The sequence <NUM> includes a first depth image <NUM> that corresponds to a first time and that includes a first set of wavefront pixels <NUM>, a second depth image <NUM> that corresponds to a second time and that includes a second set of wavefront pixels <NUM>, and a third depth image <NUM> that corresponds to a third time and that includes a third set of wavefront pixels <NUM>, where the first time occurs before the second time, the second time occurs before the third time, and the first, second, and third times each occur during a time period in which the wavefront is simulated to traverse the scene depicted in the depth images <NUM>, <NUM>, and <NUM> and corresponding RGB images. In some embodiments, pixels of the depth images <NUM>, <NUM>, and <NUM> having lower pixel values correspond to objects that are nearer to the mobile device, while pixels having higher pixel values correspond to objects that are further away from the mobile device. That is, the value of each pixel of a given depth map <NUM>, <NUM>, <NUM> corresponds to that pixel's depth within the scene depicted in the depth map and that of the corresponding pixel in the corresponding RGB image. Accordingly, the pixel value of a pixel of a depth map may be referred to as a "depth value" herein. Pixels of the depth maps having the same pixel value are approximately the same distance away from the mobile device. It should be understood that the wavefront pixels <NUM>, <NUM>, and <NUM> have been darkened to improve their visibility and the depths of the wavefront pixels <NUM>, <NUM>, and <NUM> are, therefore, not indicated by their color or brightness shown in the illustration of <FIG>.

To identify which pixels of the depth map are wavefront pixels at a given time, the light-in-flight engine <NUM> identifies a depth value corresponding to that time, and identifies all pixels in the depth map with the identified depth value as being wavefront pixels. In the present example, the wavefront pixels <NUM> are disposed closer to the mobile device than the wavefront pixels <NUM>, which are disposed closer to the mobile device than the wavefront pixels <NUM>, such that the wavefront is shown to move further away from the mobile device over time.

Returning to the example of <FIG>, while in some embodiments the light-in-flight engine <NUM> may brighten only the identified wavefront pixels in each light-in-flight image, this brightening may be difficult to perceive, depending on the scene depicted in the original RGB image. Additionally, brightening only the identified wavefront pixels does not account for light scattering that may occur when the wavefront hits various objects within the scene, beam widening that occurs as the wavefront propagates across the scene, and light dispersion that occurs as the wavefront propagates across the scene. Accordingly, in some embodiments, the perceptibility of the simulated wavefront and the accuracy of the simulation may be improved, by simulating how the light packets of the wavefront are reflected or scattered off of the objects of the scene (i.e., light scattering) and simulating the effects of beam widening and light dispersion by applying a convolutional operator to the wavefront pixels. For example, the light-in-flight engine <NUM> may generate a light-in-flight image for an RGB image, simulating a pulse wavefront at a given timestamp, by generating an optical simulation of the wavefront and affine-transforming the original RGB image with the output of the optical simulation. When generating the optical simulation, the light-in-flight engine <NUM> may apply a convolutional operator to a three-dimensional transformation of the pixels of the depth map to produce a simulation output that approximates non-idealities of the wavefront's propagation across the scene. A slice of the simulation output centered at the wavefront depth for a given timestamp may then be affine-transformed with the original RGB image to produce a light-in-flight image frame. For example, the convolutional operator may approximate the effects of short light pulse scattering, beam widening, and light dispersion in space.

For example, <FIG> illustrates how a short pulse of light <NUM>, output by a laser <NUM>, propagates over time. As shown, the cross-sectional area <NUM> of the light <NUM> may expand outwardly along the x and y axes over time. As the light <NUM> travels away from the laser <NUM>, it may experience various non-idealities such as scattering, beam widening, and dispersion. The effects of such non-idealities on the amplitude of the light <NUM> in the x, y, and t directions may be defined as respective Gaussian functions. Here, the x and y axes define a plane that is transverse to the vector of propagation of the emitted light <NUM>. The t axis is orthogonal to the transverse plane, is parallel to the vector of propagation of the light <NUM>, and represents both the time and distance traveled by the light <NUM>.

For example, the pulse function, denoted here as a vector vT, that defines the amplitude of the light <NUM> over time may be approximated by a Gaussian function, since it is not exactly localized in time. Additionally, at a given point along its transverse plane, the light <NUM> may define a Gaussian spot <NUM>. <FIG> depicts a chart <NUM> that illustrates the beam amplitude of the Gaussian spot <NUM> in the transverse plane. As shown, the amplitude of the light <NUM> may be approximated along the x and y direction by respective Gaussian functions. The amplitude of the light <NUM> in the x direction is denoted here as a vector vx, and the amplitude of the light <NUM> in the y direction is denoted here as vy. Accordingly, the amplitude of the light <NUM> over three-dimensional space may be approximated according to Equations <NUM>-<NUM>: <MAT> <MAT> <MAT> <MAT> where G(t), G(x), and G(y) are Gaussian functions over the t, x, and y axes, respectively, and H is the outer product of the vectors vx, vy, and vt, such that Hrepresents a three-dimensional matrix. Returning to the example of <FIG>, the light-in-flight engine <NUM> may simulate non-idealities such as scattering, beam widening, and dispersion (e.g., as demonstrated for the light <NUM> of <FIG>) of the quanta of the simulated pulsed wavefront by applying H from Equation <NUM> as a convolutional operator to the pixels of the depth map associated with the original RGB image to produce an optical simulation output.

When generating the optical simulation output for a given RGB image, the light-in-flight engine <NUM> may transform the RGB image's corresponding depth map into a three-dimensional data structure, such as a cubic data structure, referred to herein as a transform to the "lift domain", in which a pixel is transformed into a voxel having a position in space that is defined with respect to a sparsely populated cube by the original pixel's position in the depth map and the depth value of the original pixel. <FIG> shows an example of the transformation of a depth map from a two-dimensional domain to the three-dimensional lift domain. As shown, a two-dimensional depth map <NUM> initially includes an array of pixels, each defining a respective depth value, Di,j, where i and j define the coordinates of a given pixel in the array. In the present example, the light-in-flight engine <NUM> transforms the depth map <NUM> to the lift domain, in which the previously two-dimensional depth map <NUM> is represented as a three-dimensional cubic data structure <NUM>. The cubic data structure <NUM> is sparsely populated with three-dimensional pixels (i.e., voxels) <NUM>. A subset of the voxels524 corresponding to the depth of the wavefront at a given timestamp may be referred to herein as "wavefront voxels". Upon transformation to the lift domain, a given voxel's position with respect to the x and y axes is maintained, while the voxel's position along a new third axis, t axis or "time" axis, is determined based on the depth value defined by the corresponding depth map pixel. For example, the depth value of a given pixel divided by the speed of light defines the corresponding voxel's position along the t axis since, as established above, d =c*t. In some embodiments, the third axis may explicitly define depth, rather than time.

After transforming the depth map to the lift domain, the light-in-flight engine <NUM> applies the convolutional operator to the voxels of the resultant cubic data structure. <FIG> illustrates the application of a convolutional operator <NUM> to voxels <NUM> of a cubic data structure <NUM>. The cubic data structure <NUM> may correspond to a lift-domain transformation of a two-dimensional depth map that corresponding to an RGB image / depth map pair. In the cubic data structure <NUM>, the position of each voxel <NUM> may be defined by assigning wavefront voxels <NUM> with a value of <NUM> or another non-zero value and assigning all other voxels of the cubic data structure <NUM> a value of zero, for example. The convolutional operator <NUM> may define a Gaussian sphere <NUM> corresponding, for example, to the convolutional operator H of Equation <NUM> and, accordingly, may be defined by respective Gaussian functions along the x, y, and t axes. The light-in-flight engine <NUM> convolves the convolutional operator <NUM> with the cubic data structure <NUM>, to produce a filtered cubic data structure <NUM> that includes Gaussian spheres <NUM>, each centered at the location of a respective one of the voxels <NUM>. For example, the size and amplitude profile of each Gaussian sphere <NUM> of the cubic data structure <NUM> may be the same. In some embodiments, the dimensions of the Gaussian sphere <NUM> along the x, y, and t axes are customizable by a user to achieve different visual effects when simulating the wavefront. By convolving the convolutional operator <NUM> with the voxels <NUM> of the cubic data structure <NUM>, the values of voxels around each voxel <NUM> are increased according to the distributions defined by the Gaussian spheres <NUM>. For example, the amount by which the amplitude of a given non-wavefront voxel is increased depends on its proximity to one or more of the voxels <NUM>, according to distribution(s) of the Gaussian sphere(s) defined by H, with the affected voxels having greater amplitudes the closer they are to the voxels <NUM>. Herein, the filtered cubic data structure <NUM> is considered the simulation output.

When generating a light-in-flight video, the light-in-flight engine <NUM> may logically divide the filtered cubic data structure <NUM> into a number of slices, each centered at a different wavefront depth associated with a respective timestamp (e.g., with each wavefront depth corresponding to the location of the wavefront at the associated timestamp), and each having a width (i.e., along the t axis) corresponding to the diameter of H in the t dimension. It should be understood that the slices may overlap one another along the t axis in some embodiments. The light-in-flight engine <NUM> may then perform an affine-transform of each slice of the filtered cubic data structure <NUM> with the original RGB image to generate a sequence of light-in-flight image frames. The light-in-flight engine <NUM> may then assemble the light-in-flight image frames to produce a light-in-flight video.

For example, returning to <FIG>, upon generating the optical simulation output, the light-in-flight engine <NUM> may perform an affine-transform of the original RGB image with slices of the optical simulation output to generate a light-in-flight image. Equation <NUM> shows an example of the affine-transformations that may be performed by the light-in-flight engine <NUM> to generate a sequence of light-in-flight images: <MAT> where S(I) is a set of light-in-flight images output by the light-in-flight engine <NUM> for an original RGB image, I, D is the depth map corresponding to the original RGB image, Y is the optical simulation output, H is the convolutional operator, X is the cubic data structure corresponding to the depth map, F(D) is the function used to transform the depth map to the lift domain to produce the cubic data structure X, α is a constant that determines the weight assigned to the simulation output (e.g., the amount by which brightnesses of the affected pixels of the original RGB image are increased), k is the index of the sequence of light-in-flight images, and N is the number of light-in-flight images in the sequence. For example, each affine-transformation of the original RGB image performed by the light-in-flight engine <NUM> effectively adjusts the brightness of pixels in the original RGB image where light from the wavefront would be directly incident (i.e., wavefront pixels) and where scattered light from the wavefront would be incident (i.e., approximated by the convolutional operator used to generate the optical simulation output) for a given timestamp. While Equation <NUM> indicates the generation of a sequence of light-in-flight images, it should be understood that Equation <NUM> could also be used to generate a single light-in-flight image.

The light-in-flight engine <NUM> may be configured to generate a light-in-flight video by generating a sequence of light-in-flight frames (e.g., as provided in connection with Equation <NUM>) simulating motion of the wavefront across the scene depicted in an original RGB image. The speed at which the wavefront is depicted to move across the scene (i.e., the "wavefront speed") in such a light-in-flight video may be predefined, may be defined by a user of the device that executes the light-in-flight image (e.g., the mobile device <NUM>), or may be defined automatically based on a defined time period (e.g., a predefined or user-defined time period for the light-in-flight video). The light-in-flight engine <NUM> may determine a sequence of timestamps to be used when generating the light-in-flight image frames of the light-in-flight video based on one or more predefined or user-defined timing parameters, which may include the wavefront speed, a desired light-in-flight video length, and/or a predefined sampling rate. In some embodiments, the light-in-flight engine <NUM> may determine the number of light-in-flight frames to be generated based on the depth of the scene depicted in the RGB image and one or more of the timing parameters. With respect to light-in-flight video, it should be understood that the discrete movement of a wavefront across a scene at the speed of light would generally not be visible to the human eye or reproducible on a conventional display, so the light-in-flight engine <NUM> may generate the light-in-flight video to simulate movement of the wavefront across the scene at a fraction of the speed at which an actual light wavefront would move or over an expanded time period in order to make the discrete movement of the wavefront perceptible to a viewer. The amount by which the light-in-flight engine <NUM> modifies the speed of the wavefront or expands the time period in which the wavefront traverses the scene for playback of the light-in-flight video may be selected based on one or more defined timing parameters (e.g., which may be stored in the storage device(s) <NUM>), which may be preset or user defined. For example, the timing parameters may include one or more of: a wavefront speed that defines the speed at which the wavefront is to move in the light-in-flight video, a wavefront time period that defines the time period over which the wavefront is to traverse the scene in the light-in-flight video and which may be defined as the length of the light-in-flight video to be generated, or a frame rate that defines the number of light-in-flight image frames that are to be generated over a given time period of the light-in-flight video.

<FIG> shows a process flow for a method <NUM> of generating a light-in-flight video. For example, the method <NUM> may be performed by a mobile device having an RGB camera and a depth camera. In the present example, the method <NUM> is described in the context of the mobile device <NUM> of <FIG>.

At block <NUM>, an RGB camera <NUM> and a depth camera <NUM> of the mobile device <NUM> capture an RGB image and a corresponding depth map. The RGB image and the depth map may be stored in storage devices <NUM> of the mobile device <NUM> for subsequent processing.

At block <NUM>, the light-in-flight engine <NUM> generates an optical simulation output for the RGB image and the depth map by transforming the depth map pixels to a three-dimensional domain (e.g., cubic data structure), then applying a convolutional operator Hto the non-zero voxels of the cubic data structure to produce the optical simulation output (e.g., a filtered cubic data structure).

At block <NUM>, the light-in-flight engine <NUM> determines the number of light-in-flight image frames, N, to be generated. In some embodiments, the light-in-flight engine <NUM> determines the number of light-in-flight image frames to be generated based on one or more timing parameters, which may include one or more of wavefront speed, wavefront time period, frame rate, and/or another applicable timing parameter.

At block <NUM>, the light-in-flight engine <NUM> determines a sequence of timestamps including a respective timestamp for each light-in-flight image frame to be generated, where the timestamp for a given light-in-flight image frame determines the location/depth at which the wavefront is to be simulated in the given light-in-flight image frame. In some embodiments, adjacent timestamps in the sequence may be equally spaced.

At block <NUM>, the light-in-flight engine <NUM> sets a variable k equal to <NUM>, where k represents the index, with respect to a sequence of frames, of the next light-in-flight image frame to be generated.

At block <NUM>, the light-in-flight engine <NUM> generates the kth light-in-flight image frame of the sequence, corresponding to the kth timestamp of the sequence of timestamps. For example, the light-in-flight engine <NUM> may generate each image frame according to blocks <NUM> and <NUM> of the method <NUM> of <FIG>.

At block <NUM>, the light-in-flight engine <NUM> determines whether the most recently generated light-in-flight image frame is the last image frame in the sequence of light-in-flight image frames by determining whether k = N - <NUM>. If the light-in-flight engine <NUM> determines that k = N - <NUM>, the method <NUM> proceeds to block <NUM>. Otherwise, if the light-in-flight engine <NUM> determines that k < N - <NUM>, the method <NUM> proceeds to block <NUM>.

At block <NUM>, the light-in-flight engine <NUM> increments the value of k by <NUM>. The method <NUM> then returns to block 712at which the light-in-flight engine <NUM> generates the next light-in-flight image frame in the sequence.

At block <NUM>, the light-in-flight engine <NUM> assembles the sequence of light-in-flight image frames into a light-in-flight video depicting a simulated wavefront travelling across the scene of the RGB image.

At block <NUM>, the light-in-flight engine <NUM> causes the light-in-flight video to be stored on the storage device <NUM>.

<FIG> shows a process flow for a method <NUM> of generating a light-in-flight image. For example, some or all of the steps of the method <NUM> may be performed at blocks <NUM> and <NUM> of <FIG> when generating the optical simulation output and light-in-flight image frames for light-in-flight images. As another example, the method <NUM> may be performed to generate a single, stand-alone light-in-flight image. For example, the method <NUM> may be performed by a mobile device having an RGB camera and a depth camera. In the present example, the method <NUM> is described in the context of the mobile device <NUM> of <FIG>.

At step <NUM>, the light-in-flight engine <NUM> receives an RGB image, a depth map corresponding to the RGB image, a timestamp, and one or more predefined or user-defined timing parameters. The timestamp defines the time during the traversal of the wavefront across the scene depicted in the RGB image for which the light-in-flight image is to be generated, and the wavefront's location at that time may be derived from the timestamp. The timing parameters may include one or more of wavefront speed, wavefront time period, frame rate, and/or another applicable timing parameter.

At step <NUM>, the light-in-flight engine <NUM> transforms the depth map to a three dimensional domain. In some embodiments, the light-in-flight engine <NUM> transforms the depth map to the lift domain to produce a three-dimensional structure, referred to in the present example as a cubic data structure (e.g., the cubic data structure <NUM> of <FIG>). The coordinates of a given voxel in the cubic data structure may be selected based on the coordinates of the corresponding pixel of the original depth map and the depth value of the corresponding pixel.

At step <NUM>, the light-in-flight engine <NUM> applies a convolutional operator (e.g., the convolutional operator H of Equation <NUM>) to the cubic data structure to produce a filtered cubic data structure. For example, the filtered cubic data structure may include one or more Gaussian spheres, each centered at a non-zero voxel of the cubic data structure.

At step <NUM>, the light-in-flight engine <NUM> identifies a slice of the filtered cubic data structure of the optical simulation output that corresponds to the current timestamp for which the light-in-flight image frame is being generated. For example, the identified slice of the filtered cubic data structure may be a matrix that is a subset of the filtered cubic data structure. The matrix may consist of all voxels filtered cubic data structure within a depth range (i.e., a range of values along the t axis) centered at the depth that corresponds to the current timestamp (i.e., corresponding to the location of the wavefront at the time represented by the timestamp), where the size of the depth range corresponds to the diameter of convolutional operator H in the t direction. At step <NUM>, the light-in-flight engine <NUM> performs an affine-transform of the original RGB image with the identified slice of the optical simulation output to generate a light-in-flight image frame. For example, the affine-transformation may increase the brightnesses of pixels in the RGB image corresponding to surfaces on which the wavefront is simulated to be incident at the time of the timestamp and may increase the brightnesses of pixels corresponding to surfaces on which scattered light from the wavefront is simulated to be incident at the time of the timestamp.

In some embodiments, certain aspects of the techniques described above may be implemented by one or more processors of a processing system executing software.

Claim 1:
A method comprising:
generating (<NUM>) an optical simulation output that simulates light scattering effects for a simulated wavefront based on a depth map (<NUM>) that corresponds to an optical image (<NUM>), wherein generating (<NUM>) the optical simulation output comprises:
generating a three-dimensional data structure based on the depth map (<NUM>); and
convolving the three-dimensional data structure with a convolutional operator to generate a filtered three-dimensional data structure; and
producing (<NUM>) a light-in-flight image (<NUM>) by adjusting brightnesses of a subset of pixels of the optical image (<NUM>) based on the optical simulation output and based on a location of the simulated wavefront at a time corresponding to a defined timestamp.