Patent Description:
In the pursuit of intelligent machine perception, it is necessary to endow systems, like autonomous cars and robots, with an awareness of the scene content beyond their immediately visible field-of-view (FoV). This entails training and configuring those systems to predict additional FoV information from past information, for example to leverage information from past narrow FoV frames to infer the present scene at a wider FoV.

To the best of the knowledge of the inventors of the present application, FoV extrapolation from a narrow FoV to a wide FoV has never been addressed in the prior art. Several challenges can be envisioned with solving this problem. First, a large image size discrepancy may exist between the input narrow FoV frames and the output wide FoV frame. This discrepancy has to be bridged while achieving adequate temporal consistency in the video output. Second, certain areas in the wide FoV frame may change significantly, or may even not appear at all in any of the past narrow FoV frames. Thus, lots of details need to be hallucinated in the wide FoV frame. And, third, there may be ambiguity between information contained in the narrow FoV frames and the wide FoV ground truth. This ambiguity which may vary depending on frame region can mislead the prediction training process.

Related problems in the art can be found in the area of video-based image synthesis. For example, video inpainting aims to hallucinate missing pixels, conditioned on visible pixels, in a narrow FoV frame. Similarly, future video prediction focuses on hallucinating future frames conditioned on past and present frames, all within narrow FoV.

Video extrapolation generally adopts 2D or 3D geometry-based image warping and stitching techniques to blend observed pixels of adjacent narrow FoV frames in order to extend the FoV. However, video extrapolation does not address the problem of pixels not observed in the narrow FoV.

Novel view synthesis aims to generate images of a given object or scene from different viewpoints by blending the observed pixels, as well as hallucinating a few missing pixels mainly for dis-occlusion. The technique is heavily reliant on highly-accurate multi-view geometry to produce good results, especially when applied to a video scene.

The following scientific article describes a method for generating an extended view frame from a narrower view frame:<NPL>.

The following scientific article proposes an attention gate model for image analysis: <NPL>.

The present disclosure overcomes one or more deficiencies of the prior art by proposing a system for image completion, comprising:.

said system being characterized in that said frame aggregation module comprises:.

According to embodiments, the first and second FoV may be equal.

In an embodiment, the second FoV is larger than the first FoV. For example, the second FoV may have a greater width and/or length in pixels than the first FoV.

Depending of the application, the first FoV may be considered a "narrow FoV" and the second FoV may be considered a "wide FoV.

In an embodiment, the first FoV and the second FoV may be specified as parameters of the system at initialization time.

Through the coordinate maps, the coordinate generation module thus enables the propagate information contained in the past frames to the coordinate system of the first present frame. This allows for the information from past frames to be combined with information from the first present frame. Specifically, the frame aggregation module uses the coordinate maps to appropriately warp the information from past frames for their ultimate combination.

In an embodiment, the coordinate generation module comprises:.

The depth map for a given frame indicates, for every pixel in the frame, an estimate of the distance between the camera and the object represented by the pixel.

The relative camera pose pt-i corresponding to time-adjacent frames (It-i, It-i+<NUM>) represents an estimate of the relative rotation and translation of the camera position from time (t-j) to time (t-j+<NUM>).

In an embodiment, the coordinate calculation module is configured, for each first past frame, to calculate a rigid flow from the first present frame to the past frame and to calculate the coordinate map for the first past frame based on the calculated rigid. The rigid flow from the first present frame to the first past frame indicates respective pixel displacements that would be applied to pixels of the first present frame to warp the first present frame to the first past frame.

In an embodiment, the frame aggregation module is configured to propagate information contained in the received first past frames to the coordinate system of the first present frame using the set of coordinate maps generated by the coordinate generation module.

In an embodiment, the AFA module is configured in the aggregation to emphasize, for each frame of the first past frames and the first present frame, region-specific features of the frame based on a timing of the frame relative to the first present frame.

In an embodiment, the AFA module is configured to emphasize, for older frames (of the first past frames and the first present frame), frame regions farther from the center of the frame (e.g., regions more than a predetermined distance from the center); and for later frames, frame regions near the center of the frame (e.g., regions less than a predetermined distance from the center). The insight behind such aggregation scheme is that frame regions far from the center are more likely to have been observed, and with lower depth/pose errors, in the older frames than in the more recent frames. In contrast, frame regions near the center of the frame are more likely to have been observed, and with lower depth/pose errors, in the more recent frames than the older frames. As such, robustness to depth/pose errors is improved.

In an embodiment, the AFA module is configured to, for each frame of the first past frames and the first present frame:.

In an embodiment, the AFA module is further configured to sum, over all of the first past frames and the first present frame, the generated respective feature maps to generate the set of aggregated feature maps.

In an embodiment, the frame aggregation module further comprises a U-net module configured to generate the second present frame having the second FoV based on the set of aggregated feature maps.

In an embodiment, the U-net module comprises:.

In an embodiment, the GSA sub-module is configured to spatially aggregate the feature maps output by the decoder sub-module based on weights that are dynamically generated per feature vector (i.e., per pixel) based on a spatial location of the feature vector (or pixel) in the frame.

In an embodiment, the ambiguity level between an estimated frame having the second FoV and the ground truth associated with that second FoV may vary from one region to another in the second FoV frame (i.e., there is a correlation between ambiguity level and location). As such, aggregating feature maps based on weights dynamically generated based on location allows for the feature aggregation to be dynamically adapted based on the ambiguity level of the feature vectors being aggregated. As such, the impact of ambiguity is reduced, improving feature aggregation performance.

In an embodiment, the proposed system comprises a hallucination uncertainty module configured to generate an uncertainty map associated with the second present frame.

By generating the uncertainty map associated with the second present frame, the hallucination uncertainty module provides a mechanism to interpret the hallucination uncertainty at each pixel of the second present frame. This can assist systems using the image completion system to better process any additional FoV information generated by the image completion system. Such systems may be decision-making systems such as self-driving cars, autonomous robots, and VR/AR systems, to name a few examples.

During training, the uncertainty map may be used to weight a loss function spatially to reduce supervision mismatch (supervision mismatch is the mismatch between a prediction result and ground truth; spatial displacement may cause the supervision mismatch to be large even though the prediction result may be visually acceptable, causing training convergence difficulties). Specifically, the weighting of the loss function by the uncertainty map attenuates the effect of pixels with high hallucination uncertainty on the loss function value and helps temper the training objective.

In an embodiment, the hallucination uncertainty module is configured to generate the uncertainty map to minimize a loss function incorporating hallucination uncertainty.

In an embodiment, the hallucination uncertainty module is configured to generate the uncertainty map based on predicting regions of the second present frame that will have high hallucinating uncertainty and those that will have low hallucinating uncertainty.

In an embodiment, a portion of the first past frames received by the depth network and the frame aggregation module are replaced by second past frames (having the second FoV) generated by the frame aggregation module and corresponding to said portion of the first past frames. This helps improves temporal consistency (i.e. color and structure jitter) in a video of the generated wide FoV frames.

In an embodiment, the second past frames are each concatenated with a respective uncertainty map generated by the hallucination uncertainty module, before providing them to the frame aggregation module. As such, the uncertainty maps are used to introduce a confidence signal into the input of the frame aggregation module, which reflects an estimation confidence level for each of the second past frames. This allows the system to account for hallucination uncertainty per pixel per estimated second past frame.

In an embodiment, any of the above-described features may be implemented as instructions of a computer program. As such, the present disclosure provides a computer program including instructions that when executed by a processor cause the processor to implement a system for image completion as described above.

The computer program can use any programming language and may take the form of a source code, an object code, or a code intermediate between a source code and an object code, such as a partially compiled code, or any other desirable form.

The computer program may be recorded on a computer-readable medium. As such, the present disclosure is also directed to a computer-readable medium having recorded thereon a computer program as described above. The computer-readable medium can be any entity or device capable of storing the computer program.

Further features and advantages of the present disclosure will become apparent from the following description of certain embodiments thereof, given by way of illustration only, not limitation, with reference to the accompanying drawings in which:.

<FIG> illustrates an example system <NUM> for image completion according to an embodiment of the present disclosure. Example system <NUM> is provided for the purpose of illustration and not limitation of embodiments of the present disclosure.

As shown in <FIG>, example system <NUM> is based on a two-stage recurrent framework that includes a coordinate generation module <NUM> and a frame aggregation module <NUM>. A hallucination uncertainty module <NUM> may also be provided.

Coordinate generation module <NUM> is configured to receive first past frames <NUM> and a first present frame <NUM> and to generate a set of coordinate maps <NUM>, one per received first past frame. The coordinate map corresponding to a first past frame provides a spatial mapping of the first past frame to the first present frame. In an embodiment, the first past frames <NUM> and the first present frame <NUM> have a first FoV (e.g., <NUM> x <NUM> pixels).

Frame aggregation module <NUM> is configured to receive the first past frames <NUM> and the first present frame <NUM>, as well as the coordinate maps <NUM> from module <NUM>. Frame aggregation module <NUM> is configured to synthesize a second present frame <NUM> based on the received input. The second present frame <NUM> has a second FoV.

The second FoV may be equal to or greater than the first FoV. In an embodiment, the second FoV is larger than the first FoV. For instance, the second FoV may have a greater width and/or length in pixels than the first FoV. For example, where the first FoV is <NUM> x <NUM> pixels, the second FoV may be <NUM> x <NUM> pixels.

In an embodiment, to improve temporal consistency in the generated second frames (having the second FoV), a portion of the first past frames, input into the modules <NUM> and <NUM>, may be replaced with their corresponding second frames generated by frame aggregation module <NUM>.

Frame aggregation module <NUM> may be configured to propagate information contained in the past frames (which include first past frames, and optionally also second past frames) to the coordinate system of the first present frame using the coordinate maps <NUM>.

In an embodiment, frame aggregation module <NUM> may be configured to generate a plurality of feature maps based on each input frame. The feature maps may be multi-scale feature maps.

The frame aggregation module <NUM> may use the coordinate map <NUM> corresponding to a given past frame to warp the plurality of feature maps associated with the past frame. For the first present frame, as it is already in the correct coordinate system, no warping is necessary and the warped feature maps correspond to the original feature maps.

Subsequently, frame aggregation module <NUM> may be configured to aggregate the warped feature maps over all frames to generate aggregated feature maps. In an embodiment, the aggregation of the warped feature maps uses an attention-based feature aggregation scheme. The attention-based feature aggregation scheme is trained to learn to select the useful features among the frames in order to address issues caused by depth/pose errors (i.e., errors in the coordinate maps) and by frame inconsistency. This improves the fusion of the multi-frame information on the feature level.

The frame aggregation module <NUM> may be configured to generate the second present frame <NUM> based on the aggregated feature maps.

In an embodiment, the frame aggregation module <NUM> may use a context normalization based technique to out-paint (extrapolate) regions falling outside the first FoV.

In an embodiment, the frame aggregation module <NUM> may use a gated convolution technique to in-paint (complete) occluded or unobserved regions falling within the first FoV.

In an embodiment, the frame aggregation module <NUM> may implement a Gated Self-Attention (GSA) mechanism to allow the system to be adaptable to observations with different ambiguity levels. The GSA mechanism may be configured to spatially aggregate feature maps based on weights that are dynamically generated according to local information (ambiguity being different from region to region).

In an embodiment, the frame aggregation module <NUM> may implement an uncertainty mechanism. Specifically, hallucination uncertainty module <NUM> may be provided to generate an uncertainty map <NUM> associated with the second present frame <NUM>. The uncertainty map serves to interpret the hallucination uncertainty at each pixel. During training, the uncertainty map may be used to weight a loss function spatially to reduce supervision mismatch (supervision mismatch is the mismatch between a prediction result and ground truth; spatial displacement may cause the supervision mismatch to be large even though the prediction result may be visually acceptable, causing training convergence difficulties). Specifically, the weighting of the loss function by the uncertainty map attenuates the effect of pixels with high hallucination uncertainty on the loss function value and helps temper the training objective.

In an embodiment, system <NUM> may be implemented on a computer system such as computer system <NUM> shown in <FIG>. Specifically, system <NUM>, and any of its modules and mechanisms, may be implemented as a computer program including instructions that, when executed by a processor <NUM> of computer system <NUM>, cause the processor <NUM> to execute methods or functions of system <NUM> as described herein. In an embodiment, the computer program may be recorded on a computer-readable medium <NUM> of computer system <NUM>.

In the following, detailed operation of system <NUM> is presented with reference to <FIG>.

Without loss of generality, system <NUM> will be described for the particular embodiment in which the second FoV is larger (in terms of width and/or length) than the first FoV. For example, the first FoV may be <NUM> x <NUM> pixels and the second FoV may be <NUM> x <NUM> pixels. Accordingly, for simplification, the first FoV will be referred to as "narrow FoV" and the second FoV will be referred to as "wide FoV. " As would be understood by a person of skill in the art based on the teachings herein, embodiments are not limited by this particular embodiment.

For the simplification of presentation, the operation of system <NUM> is described from the processing perspective of a present narrow FoV frame It to generate a present wide FoV frame Ot. Accordingly, system <NUM> may be considered as a FoV extrapolation system. However, as described above, system <NUM> is not limited to FoV extrapolation.

<FIG> illustrates an example coordinate generation module <NUM> according to an embodiment. Example coordinate generation module <NUM> is provided for the purpose of illustration and not limitation of embodiments of the present disclosure. Example coordinate generation module <NUM> may be an embodiment of coordinate generation module <NUM>.

As shown in <FIG>, coordinate generation module <NUM> includes a depth network <NUM>, a pose network <NUM>, and a coordinate calculation module <NUM>.

Depth network <NUM> is configured to receive a plurality of past frames. The past frames may include narrow FoV frames and past wide FoV frames. For the purpose of illustration, it assumed in the following description that the depth network <NUM> receives k frames. The k frames may include (k-j) past narrow FoV frames (denoted in <FIG> as It-k,. , It-j-<NUM><NUM>) and j past wide FoV frames (denoted in <FIG> as Ot-j,. , Ot-<NUM>). In an embodiment, k may be equal to <NUM> and j may be between <NUM> and <NUM>.

The past frames may be RGB frames, depth frames, or semantic segmentation frames, for example. The frames may be derived from the same camera source or from different camera sources and translated to the same camera reference.

Depth network <NUM> generates a depth map d for each of the received k frames. In an embodiment, depth network <NUM> generates depth maps dt-k,. , dt-j-<NUM> corresponding respectively to past narrow FoV frames It-k,. , It-j-<NUM> and generates depth maps dt-j,. , dt-<NUM> corresponding respectively to past wide FoV frames Ot-j,. , Ot-<NUM>.

Pixel depth estimation is well-known to a person of skill in the art and will not be described herein. In an embodiment, depth network <NUM> may be implemented as a fully convolutional U-net as described in detail in "<NPL>". Specifically, the depth network <NUM> may include a well-known VGG16BN encoder and a decoder of several convolutional layers. The input may be a RGB image frame (<NUM> channels), and the output is a depth map (<NUM> channel) of the same resolution.

Pose network <NUM> receives as input k pairs of narrow FoV frames and generates a relative camera pose for each of the k frame pairs. In an embodiment, the k frame pairs include the frame pairs (It-k, It-k+<NUM>),. , (It-<NUM>, It), i.e., adjacent frame pairs over the time t (present) narrow FoV frame and the k-<NUM> past narrow FoV frames. The resulting relative camera poses are denoted as pt-k,. , pt-<NUM>.

The relative camera pose pt-i corresponding to adjacent narrow FoV frames (It-i, It-i+<NUM>) represents an estimate of the relative rotation and translation of the camera position from time (t-j) to time (t-j+<NUM>).

Relative camera pose estimation is well-known to a person of skill in the art and will not be described herein. In an embodiment, pose network <NUM> may be implemented as described in detail in "<NPL>". Specifically, pose network <NUM> may include a ResNet18 encoder that receives as input a pair of RGB images (<NUM>+<NUM> channels) and that produces as output a <NUM>-channel vector.

Coordinate calculation module <NUM> is configured to calculate k coordinate maps <NUM> based on the outputs of depth network <NUM> and pose network <NUM>. In an embodiment, (k-j) maps (et-k,. , et-j+<NUM>) corresponding respectively to the (k-j) past narrow FoV frames (It-k,. , It-j-<NUM> ) and j maps (et-j,. , et-<NUM>) corresponding respectively to the j past wide FoV frames (Ot-j,. , Ot-<NUM>) are calculated.

The coordinate map corresponding to a past (narrow or wide FoV) frame provides a spatial mapping of the past frame to the present narrow FoV frame It. In other words, the coordinate map indicates for each pixel of the past frame its corresponding coordinates in the present frame It.

In an embodiment, the coordinate map for a past frame Ii or Oi(i = t-k,. , t-<NUM>) is obtained by first calculating a rigid flow matrix from the present frame It to the past frame according to: <MAT> where K denotes the intrinsic matrix of the camera, Ti→t denotes the relative camera pose from the past frame Ii to the present frame, ci represents a matrix of the homogeneous (or projective) coordinates of the pixels in the present frame Ii, and Di(ci) represents the depth value of the position ci.

The rigid flow from the present frame It to the past frame indicates respective pixel displacements that would be applied to pixels of the present frame It to warp the present frame It to the past frame.

Using the calculated rigid flow, a coordinate map êt→i that spatially matches the present frame It to the past frame can then be computed. The coordinate map êt→i can be obtained by adding the rigid flow to a regular 2D grid (<NUM> channels) (e.g., a 3x3 2D grid with the values [ [[<NUM>,<NUM>,<NUM>], [<NUM>,<NUM>,<NUM>], [<NUM>,<NUM>,<NUM>]], [[<NUM>,<NUM>,<NUM>], [<NUM>,<NUM>,<NUM>], [<NUM>,<NUM>,<NUM>]]]). Finally, the coordinate map êt→i is reversed to obtain the coordinate map coordinate map ei, which spatially matches the past frame to the present frame It. In the reversal of the coordinate map êt→i to obtain the coordinate map coordinate map ei, if a pixel (x0, y0) of the present frame It is spatially matched to the pixel (u0, v0) of the past frame in the coordinate map êt→i, then the pixel (u0, v0) of the past frame will be spatially matched to the pixel (x0, y0) of the present frame It in the coordinate map ei.

<FIG> illustrates an example frame aggregation module <NUM> according to an embodiment. Example frame aggregation module <NUM> is provided for the purpose of illustration and not limitation of embodiments of the present disclosure. Example frame aggregation module <NUM> may be an embodiment of frame aggregation module <NUM>.

As shown in <FIG>, example frame aggregation module <NUM> includes an encoder <NUM>, a warping module <NUM>, an attention-based feature aggregation (AFA) module <NUM>, and a U-net module <NUM>.

Encoder <NUM> is configured to receive as input k+<NUM> frames. At initialization, the k+<NUM> frames correspond to the k past narrow FoV frames (i.e., It-<NUM>,. , It-k) and to the present narrow FoV frame It. After j iterations, to improve temporal coherency, the inputs corresponding to the past narrow frames {It-i}i=<NUM>,. , j are replaced with the previous outputs {Ot-i}i=<NUM>,. , j and their associated uncertainty maps {Ut-i}i=<NUM>,. , j (Ot-j and Ut-j may be concatenated with each other channel by channel). For the purpose of simplification, <FIG> illustrates the encoder inputs after j iterations have taken place.

Encoder <NUM> is configured to generate a plurality (N) of feature maps <NUM> based on each received input frame. In an embodiment, the plurality of feature maps <NUM> may be multi-scale feature maps (i.e., having different spatial scales or sizes). In an embodiment, N may be equal to <NUM>, though a greater number may be used.

In an embodiment, the encoder <NUM> may include a first convolutional layer configured to generate a first feature map based on the input frame. The first feature map may be a tensor of size H x W x C, where H is the frame height, W is the frame width, and C is a number of channels. For example, C may be equal to <NUM>. The first feature map may be referred to as "level <NUM>" feature map.

The encoder <NUM> may also include a second convolutional layer configured to receive the first feature map and to generate an intermediate second feature map based on the first feature map. The intermediate second feature map may be of size H1 x W1 x C, where at least one of H1 and W1 is lower respectively than H and W. The intermediate second feature map is then added to a downsized version of the first feature map of the same size (i.e., H1 x W1 x C) to generate a second feature map of size H1 x W1 x C. The second feature map may be referred to as "level <NUM>" feature map.

The process described above may repeated with respect to the second feature map to obtain a third feature map of size H2 x W2 x C, where at least one of H2 and W2 is lower respectively than H and W. The third feature map may be referred to as "level <NUM>" feature map.

The feature maps <NUM>, generated for each of the k+<NUM> input frames, are then provided to warping module <NUM>.

Additionally, warping module <NUM> receives the k coordinate maps <NUM> from coordinate calculation module <NUM>. As noted above, the k coordinate maps <NUM> include (k-j) maps (et-k,. , et-j+<NUM>) corresponding respectively to the (k-j) past narrow FoV frames (It-k,. , It-j-<NUM>) and j maps (et-j,. , et-<NUM>) corresponding respectively to the j past wide FoV frames (Ot-j,. , Ot-<NUM>).

In an embodiment, for each of the past frames (i.e., each of (k-j) past narrow FoV frames (It-k,. , It-j-<NUM>) and the j past wide FoV frames (Ot-j,. , Ot-<NUM>)), warping module <NUM> may be configured to use the respective coordinate map corresponding to the frame to propagate the feature maps <NUM> associated with the frame to the present narrow FoV frame It. The propagation of the feature maps <NUM> warps the feature maps <NUM>, according to the coordinate map, to generate a plurality of warped feature maps <NUM> for the frame.

In an embodiment, the warping module <NUM> uses bilinear sampling as described in detail in Jaderberg, Max, Karen Simonyan, and Andrew Zisserman, "Spatial transformer networks," In NIPS. to propagate the multi-scale feature maps <NUM> based on the coordinate maps <NUM>.

It is noted that the feature maps <NUM> corresponding to the present narrow FoV frame It are not warped by warping module <NUM> because they are already in the coordinate system of the present frame. As such, the feature maps <NUM> are identical to the feature maps <NUM> for the present narrow FoV frame It.

The warped feature maps <NUM> (which may be different levels, e.g., level <NUM>, <NUM>, and <NUM>) are then provided to AFA module <NUM>, which is configured to aggregate the warped feature maps <NUM>, over all of the k+<NUM> frames, to generate aggregated feature maps <NUM>. Warped feature maps resulting from narrow FoV frames may be padded with zeros to have the same size as warped feature maps resulting from wide FoV frames.

In an embodiment, AFA module <NUM> may be implemented as shown in <FIG>, which illustrates an example AFA module <NUM> according to an embodiment of the present disclosure. Example AFA module <NUM> is provided for the purpose of illustration only and is not limiting of embodiments.

As shown in <FIG>, example AFA module <NUM> includes a plurality of channels each configured to receive the warped feature maps <NUM> (e.g., level <NUM>, <NUM>, and <NUM>) corresponding to a given frame of the k+<NUM> frames.

Within each channel, the warped feature maps <NUM> of a respective frame are each fed into a convolutional layer <NUM>, followed by a softmax normalization module <NUM>, to generate a respective frame-wise spatial attention map <NUM>. In an embodiment, a level <NUM> spatial attention map, a level <NUM> spatial attention map, and a level <NUM> spatial attention map are generated. For the purpose of illustration, <FIG> shows example level <NUM> attention maps corresponding to a sequence of example narrow FoV frames shown in <FIG>.

Each of the warped feature maps <NUM> is then multiplied by its respective spatial attention map <NUM> to generate a respective feature map <NUM>. Thus, for each frame, a plurality of feature maps <NUM> (e.g., level <NUM>, level <NUM>, and level <NUM>) are obtained.

The use of spatial attention maps as described above allows to focus on or select specific features of each frame for subsequent aggregation with other frames. In an embodiment, the spatial attention maps <NUM> are configured to emphasize, for older frames of the k+<NUM> frames (e.g., frames t-k to t-j-<NUM>), frame regions farther from the center of the frame (e.g., regions more than a predetermined distance from the center); and for later frames of the k+<NUM> frames (e.g., frames t-j to t), frame regions near the center of the frame (e.g., regions less than a predetermined distance from the center).

The resulting feature maps <NUM> are then summed, across all frames, to generate the aggregated feature maps <NUM>. In an embodiment, this includes summing, across all frames, all level <NUM> feature maps <NUM> together, all level <NUM> feature maps <NUM> together, and all level <NUM> feature maps <NUM> together.

Returning to <FIG>, the aggregated feature maps <NUM> are then provided to U-net module <NUM>. U-net module <NUM> is configured to synthesize the present wide FoV frame Ot based on the aggregated feature maps <NUM>. Additionally, in an embodiment, U-net module <NUM> also outputs an uncertainty map Ut associated with the wide FoV frame Ot. The uncertainty map serves to interpret (explain) the hallucination uncertainty at each pixel and to guide the learning by reducing supervision ambiguity.

In an embodiment, U-net module <NUM> implements mechanisms for hallucinating missing regions, for example by in-painting (completing) occluded or unobserved regions falling within the narrow FoV and/or out-painting (extrapolating) regions falling outside the narrow FoV. Image inpainting and out-painting are known techniques in the art. Specifically, image in-painting aims to hallucinate the missing pixels through warping, or to generate the missing pixels conditioned on the neighboring (spatial or temporal dimensions) visible pixels. Image out-painting typically adopts 2D or 3D geometry-based image warping and stitching techniques to blend the observed pixels of adjacent narrow FoV frames in order to extend the FoV.

Optionally, U-net module <NUM> may include a Gated Self-Attention (GSA) mechanism. The motivation for the GSA mechanism is that, typically, the ambiguity level between an estimated wide FoV frame and the wide FoV ground truth may vary from one region to another in the wide FoV frame. For example, as illustrated in <FIG>, the pixels in the wide FoV frame can be roughly divided into four categories: (a) the observed narrow FoV pixels in the present frame (e.g., region <NUM> of the frame), for which there is no ambiguity; (b) the propagated pixels from past frames with accurate propagation (e.g., regions 806a and 806b), for which ambiguity is low; (c) the propagated pixels from past frames with noisy propagation (e.g., regions 808a and 808b), characterized by a medium ambiguity level; and (d) the pixels corresponding to unobserved regions (e.g., region <NUM>), for which the ambiguity level is high. As further described below, the GSA mechanism is configured to ensure that the model is adaptable to observations with different ambiguity levels.

In an embodiment, U-net module <NUM> may be implemented as shown in <FIG>, which illustrates an example U-net module <NUM> according to an embodiment of the present disclosure. Example U-net module <NUM> is provided for the purpose of illustration only and is not limiting of embodiments.

As shown in <FIG>, U-net module <NUM> includes a bottleneck module <NUM> and a decoder module <NUM>.

In an embodiment, bottleneck module <NUM> includes a plurality of successive layers <NUM>-<NUM>,. , <NUM>-<NUM>. Layers <NUM>-<NUM>,. , <NUM>-<NUM> may each be implemented as a residual dilated convolutional layer. Such a layer can be described by the equation y = x + conv(x), where y is the layer output, x is the layer input, and conv(x) denotes a dilated convolution of the input x.

In an embodiment, decoder module <NUM> includes a context normalization sub-module <NUM>, a decoder sub-module <NUM>, a gated self-attention (GSA) sub-module <NUM>, and up-sampling modules <NUM>.

Context normalization sub-module <NUM> may be configured to outpaint (extrapolate) regions falling outside the narrow FoV.

In an embodiment, sub-module <NUM> comprises a plurality of context normalization layers <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM>. In an embodiment, normalization layers <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM> may each be implemented as described in <NPL>. As such, layers <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM> may be configured to transfer the mean and the variance from the observed region features to the unobserved region features.

However, unlike in Wang et al. , where the mask is given in the input, an aggregated mask that indicates the unobserved regions after propagating the past frames may be used. As such, it can be recognized that a large amount of wide view information has been observed in the past frames and this information can simply be propagated into the present wide FoV frame, rather than hallucinated.

Decoder sub-module <NUM> may be configured to in-paint (complete) occluded or unobserved regions falling within the narrow FoV.

In an embodiment, sub-module <NUM> includes a plurality of decoder layers <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM>. Decoder layers <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM> may each be implemented as a gated convolution layer as described in<NPL>.

GSA sub-module <NUM> may be configured to perform feature aggregation with self-attention. Specifically, unlike a traditional convolution operator which performs feature aggregation using a convolution kernel of fixed pre-trained weights (to combine features from nearby locations), sub-module <NUM> may be configured to dynamically generate the kernel weights per feature vector (i.e., per pixel) based on the location of the feature vector. In an embodiment, as the ambiguity is directly correlated with location (as described above with respect to <FIG>), the kernel weights may be dynamically adapted per feature vector based on the ambiguity level of the feature vectors being aggregated. As such, the impact of ambiguity is reduced, improving feature aggregation performance.

In an embodiment, GSA sub-module <NUM> includes a plurality of GSA layers <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM>. GSA layers <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM> may each be implemented as a patch-wise self-attention block as described in <NPL>. Specifically, the patch-wise self-attention block can be described by the following equation: <MAT> where α(xR(i))j = γ(δ(xR(i))), yi denotes the new aggregated feature, R(i) denotes the aggregation footprint (e.g., <NUM> x <NUM> or <NUM> x <NUM> pixels), xR(i) denotes a tensor corresponding to the patch of feature vectors in the aggregation footprint, α(xR(i))j represents the attention vector (i.e., weight vector) at location j in the tensor α(xR(i)) and which corresponds spatially to the feature vector xj in the tensor xR(i), β is a function that generates the feature vectors β(xj), and ⊙ is the Hadamard product.

The functions β and γ are mappings implemented via one convolution layer, respectively. The function δ combines the feature vectors xj from the patch xR(i) and may be implemented using a concatenation operation.

In an embodiment, to reduce the impact of vanishing gradients during training, the self-attention block may be wrapped by a residual structure: z = Conv, (y) + x, where Conv, denotes a residual convolutional layer, y is the output of the self-attention block, and x is the input of the self-attention block.

In another embodiment, the self-attention may further include a gating mechanism to deal with regions with high ambiguity, formulated as: <MAT> where Convg and Conva denote a gating convolutional layer and an attention convolutional layer. The gating mechanism controls the path(s) through which information flows in the network. Particularly, in an embodiment, the gating mechanism may be configured to allow only feature vectors with an ambiguity above a certain level to flow through the network and/or to limit the flow of feature vectors with ambiguity above a certain level. Image quality can thus be improved.

In an embodiment, bottleneck module <NUM> may be configured to receive as input, via the first layer <NUM>-<NUM>, an aggregated feature map <NUM>-<NUM> and to generate a modified aggregated feature map <NUM>. Feature map <NUM>-<NUM> may be a level <NUM> aggregated feature map of size (H/<NUM> x W/<NUM> x C), where H is the frame height, W is the frame width, and C is the number of channels. Feature map <NUM> may be of the same size as feature map <NUM>-<NUM>.

Context normalization layers <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM> are configured to receive respective aggregated feature maps <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM>. Aggregated feature maps <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM> may correspond respectively to level <NUM>, level <NUM>, and level <NUM> aggregated feature maps. As an example, feature map <NUM>-<NUM> may be of size (H/<NUM> x W/<NUM> x C), feature map <NUM>-<NUM> may be of size (H/<NUM> x W/<NUM> x C), and feature map <NUM>-<NUM> may be of (H x W x C), where H is the frame height, W is the frame width, and C is the number of channels.

In an embodiment, context normalization layers <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM> are configured to feed respectively decoder layers <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM>, which in turn are configured to feed respectively GSA layers <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM>.

Concurrently with receiving the respective outputs of context normalization layers <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM>, decoder layers <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM> further receive as input respectively the output <NUM> of bottleneck module <NUM>, the output of GSA layer <NUM>-<NUM> (up-sampled by up-sampling module <NUM>-<NUM>), and the output of GSA layer <NUM>-<NUM> (up-sampled by up-sampling module <NUM>-<NUM>). For example, the combined inputs of decoder layers <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM> may be of size (H' x W' x 2C), and the outputs of the layers may be of size (H' x W' x 2C), where H' = H/<NUM>, H/<NUM>, and H respectively and W' = W/<NUM>, W/<NUM>, and W respectively for layers <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM>.

Decoder layer <NUM>-<NUM> receives an input the output of GSA layer <NUM>-<NUM> and generates an output <NUM> comprising the present wide FoV frame Ot.

Returning to <FIG>, as mentioned above, in an embodiment, system <NUM> may include a hallucination uncertainty module <NUM>. Specifically, hallucination uncertainty module <NUM> may be provided to generate an uncertainty map <NUM> associated with the generated wide FoV frame <NUM>. The uncertainty map serves to interpret the hallucination uncertainty at each pixel of the wide FoV frame.

Additionally, the uncertainty map may be used to temper the training objective by attenuating the effect of pixels with high hallucinating uncertainty on the loss function value, thereby reducing supervision mismatch and speeding up training convergence.

In an embodiment, the hallucination uncertainty module <NUM> may be trained to generate an uncertainty map based on predicting the regions of the wide FoV frame that will have high hallucinating uncertainty and those that will have low hallucinating uncertainty. The generated uncertainty map reflects this prediction by assigning an estimated hallucination uncertainty per pixel of the wide FoV frame.

In an embodiment, hallucination uncertainty module <NUM> may be trained to generate the uncertainty map <NUM> to minimize a loss function incorporating hallucination uncertainty.

In an embodiment, the loss function incorporating hallucination uncertainty is a pixel level reconstruction L1 loss function.

In an embodiment, the loss function incorporating hallucination uncertainty is given by the equation: <MAT> where Ot is the present wide FoV frame, Wt is the ground truth wide FoV frame, Ut is the predicted uncertainty map associated with Ot, Mview is a mask for out-of-narrow view regions, and the ⊙ operator denotes element-wise multiplication. The loss function is computed per pixel and then averaged over all pixels.

According to the above equation, it is noted that the narrow FoV region, given by (<NUM>-Mview), is not weighted by the uncertainty map Ut. This is because this region corresponds to pixels that are observed in the narrow FoV frame. The right-most Ut term is a regularization term that helps stabilize loss gradients.

In an embodiment, to make the uncertainty Ut more interpretable and further stabilize the training process, Ut is constrained in the range (<NUM>, <NUM>) using a sigmoid function.

Additionally, as shown in <FIG>, previously generated uncertainty maps {Ut-i}i = <NUM>. j may be used in the present input to act as a confidence signal. In an embodiment, this is done by concatenating the past uncertainty maps with respective past estimated wide FoV frames.

Claim 1:
A system (<NUM>) for image completion, comprising:
a coordinate generation module (<NUM>) configured to receive first past frames (<NUM>) and a first present frame (<NUM>), the first past frames (<NUM>) and the first present frame (<NUM>) having a first field-of-view, FoV, and to generate a set of coordinate maps (<NUM>), one for each of the received first past frames (<NUM>), wherein the coordinate map corresponding to a first past frame provides a spatial mapping of the first past frame to a coordinate system of the first present frame (<NUM>); and
a frame aggregation module (<NUM>) configured to receive as input the first past frames (<NUM>), the first present frame (<NUM>), and the coordinate maps (<NUM>) and to synthesize, based on said input, a second present frame (<NUM>) having a second FoV equal or greater than the first FoV;
said system being characterized in that said frame aggregation module (<NUM>) comprises:
an encoder (<NUM>) configured to generate a plurality of feature maps (<NUM>) based on each of the first past frames (<NUM>) and the first present frame (<NUM>);
a warping module (<NUM>) configured, for each of the first past frames (<NUM>) and the first present frame (<NUM>), to warp the plurality of feature maps (<NUM>) associated with said each frame, using the respective coordinate map (<NUM>) associated with said each frame, to generate a plurality of warped feature maps (<NUM>) for said each frame; and
an attention-based feature aggregation, AFA, module (<NUM>) configured to aggregate, over all of the first past frames (<NUM>) and the first present frame (<NUM>), the generated warped feature maps (<NUM>) to generate a set of aggregated feature maps (<NUM>).