ADAPTIVE HUMAN INSTANCE SEGMENTATION WITH STEREO VIEW CONSISTENCY

A system stores first and second images generated by first and second cameras; applies a segmentation model to the first image to generate a first segmentation mask identifying object instances; applies the segmentation model to the second image to generate a second segmentation mask identifying the object instances; projects the first segmentation mask to a viewpoint of the second camera to generate a first projected segmentation mask; converts the first projected segmentation mask and the second segmentation mask to first and second semantic masks, respectively; and computes a first similarity value based on the first and second semantic masks. This may be repeated exchanging the first and second images to compute a second similarity value. The system determines a loss value based on the first similarity value and the second similarity value and trains the segmentation model based on the loss value.

TECHNICAL FIELD

This disclosure relates to systems for image segmentation.

BACKGROUND

Image segmentation is a technique that divides an image into multiple parts or regions. For example, image segmentation may be used to identify which parts of an image represent specific objects, such as individual people or individual vehicles. In general, an image segmentation process generates labels individual pixels of an image so that pixels having the same characteristics have the same labels. For example, each of the pixels in an image corresponding to an individual person may have the same label. A variety of segmentation models have been developed to perform image segmentation. These models include Mask R—CNN and others.

Image segmentation is useful in a wide variety of scenarios. For example, image segmentation may be used in autonomous driving applications. In another example, image segmentation may be used for extended reality (XR), such as mixed reality (MR) or augmented reality (AR). In another example, image segmentation may be used in robotics applications.

SUMMARY

In general, this disclosure describes techniques for adaptively training image segmentation models. Training a segmentation model may be difficult for several reasons. For example, there may be only a limited number of segmentation datasets with accurate ground truth annotations that are available for training, especially with respect to segmenting instances of humans in images. In another example, it may be difficult to obtain training dataset that cover a large variety of environments that may be encountered during deployment of the segmentation model. As a result of these challenges, there may be a need to use unlabeled data (i.e., data that was not manually labeled by a human) for training the segmentation model to account for distributions of data that are not covered by existing training datasets. However, the use of pseudo-labels (i.e., labels not generated by humans) on unlabeled data may contribute to noise and there would need to be a robust way to deal with that noise.

The techniques of this disclosure may address these challenges. As described herein, the techniques of this disclosure use consistency between left and right images of a stereoscopic camera to determine a loss value. The system may obtain the consistency between a predicted segmentation mask of a first view and a projected mask of a second view onto the first view. A system may use the loss value to further train a segmentation model.

In one example, a method of processing image data includes a system comprising: a storage system comprising one or more computer-readable media, the storage system configured to store a pair of stereoscopic images, the stereoscopic images including a first image generated by a first camera and a second image generated by a second camera; and one or more processors implemented in circuitry, the one or more processors configured to: apply a segmentation model to the first image to generate a first segmentation mask identifying one or more object instances; apply the segmentation model to the second image to generate a second segmentation mask identifying the one or more object instances; project the first segmentation mask to a viewpoint of the second camera to generate a first projected segmentation mask; convert the first projected segmentation mask to a first segmentation mask; convert the second segmentation mask to a second semantic mask and a second semantic mask, respectively; compute a first similarity value based on the first semantic mask and the second semantic mask; project the second segmentation mask to a viewpoint of the first camera to generate a second projected segmentation mask; convert the second projected segmentation mask to a third semantic mask; convert the first segmentation mask to a fourth semantic mask; compute a second similarity value based on the third semantic mask and the fourth semantic mask; determine a loss value based on the first similarity value and the second similarity value; and train the segmentation model based on the loss value.

In another example, this disclosure describes a method comprising: storing a pair of stereoscopic images, the stereoscopic images including a first image generated by a first camera and a second image generated by a second camera; and applying a segmentation model to the first image to generate a first segmentation mask identifying one or more object instances; applying the segmentation model to the second image to generate a second segmentation mask identifying the one or more object instances; projecting the first segmentation mask to a viewpoint of the second camera to generate a first projected segmentation mask; converting the first projected segmentation mask to a first semantic mask; converting the second segmentation mask to a second semantic mask; computing a first similarity value based on the first semantic mask and the second semantic mask; projecting the second segmentation mask to a viewpoint of the first camera to generate a second projected segmentation mask; converting the second projected segmentation mask to a third semantic mask; converting the first segmentation mask to a fourth semantic mask; computing a second similarity value based on the third semantic mask and the fourth semantic mask; determining a loss value based on the first similarity value and the second similarity value; and training the segmentation model based on the loss value.

In another example, this disclosure describes non-transitory computer-readable storage media having stored thereon instructions that, when executed, cause one or more processors to: store a pair of stereoscopic images, the stereoscopic images including a first image generated by a first camera and a second image generated by a second camera; and apply a segmentation model to the first image to generate a first segmentation mask identifying one or more object instances; apply the segmentation model to the second image to generate a second segmentation mask identifying the one or more object instances; project the first segmentation mask to a viewpoint of the second camera to generate a first projected segmentation mask; convert the first projected segmentation mask to a first semantic mask; convert the second segmentation mask to a second semantic mask; compute a first similarity value based on the first semantic mask and the second semantic mask; project the second segmentation mask to a viewpoint of the first camera to generate a second projected segmentation mask; convert the second projected segmentation mask to a third semantic mask; convert the first segmentation mask to a fourth semantic mask; compute a second similarity value based on the third semantic mask and the fourth semantic mask; determine a loss value based on the first similarity value and the second similarity value; and train the segmentation model based on the loss value.

DETAILED DESCRIPTION

Training a segmentation model may be challenging for several reasons, including a lack of human-labeled training data sufficient to cover enough situations during deployment of the segmentation model. If the segmentation model is not sufficiently trained, the segmentation model may produce poor segmentation results, which can lead to further problems, especially in the context of autonomous navigation, robotics, and XR.

This disclosure describes techniques that may help to address these challenges. A computing system may obtain pairs of stereoscopic images. For instance, the system may obtain pairs of images from a first camera (e.g., a left image camera) and a second camera (e.g., a right image camera). Thus, the stereoscopic images may include a first image generated by a first camera and a second image generated by a second camera. The computing system may apply a segmentation model to the first image to generate a first segmentation mask identifying one or more object instances. Additionally, the computing system apply the segmentation model to the second image to generate a second segmentation mask identifying the one or more object instances. The computing system may project the first segmentation mask to a viewpoint of the second camera to generate a first projected segmentation mask. The computing system may then convert the first projected segmentation mask and the second segmentation mask to a first semantic mask and a second semantic mask, respectively.

The computing system may compute a first similarity value based on the first semantic mask and the second semantic mask. The computing system may also perform this operation for the second segmentation mask. Thus, the computing system may project the second segmentation mask to a viewpoint of the first camera to generate a second projected segmentation mask. The computing system may convert the second projected segmentation mask and the first segmentation mask to a third semantic mask and a fourth semantic mask, respectively. Additionally, the computing system may compute a second similarity value based on the third semantic mask and the fourth semantic mask. The computing system may determine a loss value based on the first similarity value and the second similarity value. The computing system may train the segmentation model based on the loss value.

By determining a loss value in this way and training the segmentation model based on this loss value in this way, the computing system may, in effect, reduce differences in segmentation between the stereoscopic images. This may effectively increase the number of training images. Increasing the number of training images in this way may increase the performance of the segmentation model without requiring more human-labeled images.

FIG.1is a block diagram illustrating an example system100according to techniques of this disclosure. In various examples, system100may be part of a vehicle, smartphone, mobile device, computing device, robot, or other type of device. In the example ofFIG.1, system100includes a plurality of image cameras102, a plurality of depth cameras104, and a computing system106. Computing system106may include one or more computing devices, such as personal computers, chipsets, mobile devices, or other types of devices.

Image cameras102are configured to generate image data, such as Red-Green-Blue (RGB) images or images in other color spaces. Image cameras102may be positioned at various locations around system100. For instance, in an example where system100is a vehicle, image cameras102may include two or more forward-facing image cameras, two or more rear-facing image cameras, and so on.

Depth cameras104are configured to generate depth images. Depth images represent the depths of objects. In some examples, there is a depth camera for each of image cameras102. For instance, in an example where image cameras102include a left image camera and a right image camera, depth cameras104may include a left depth camera corresponding to the left image camera and a right depth camera corresponding to the right image camera. Depth images generated by a depth camera may represent the depths of objects shown in images generated by an image camera corresponding to the depth camera.

In the example ofFIG.1, computing system106includes one or more processors108, one or more output devices110, and a storage system112. Processors108may be implemented in circuitry. Example types of processors108may include microprocessors, digital signal processors, application-specific integrated circuits (ASICs), and so on. Output devices110may include display screens, XR display devices, and other devices for displaying output. In some examples, such as examples involving robotics or autonomous driving, output device110may include actuators to perform various physical actions. Storage system112may include one or more non-transitory computer-readable storage media. Example types of non-transitory computer-readable storage media may include random access memory (RAM) units, disk drives, and so on. Processors108, output device110, and the computer-readable storage media of storage system112may be distributed among two or devices of computing system106, or may be consolidated within a single device of computing system106.

Storage system112may be configured to store various types of data and computer-readable instructions. In the example ofFIG.1, storage system112stores data and instructions associated with a segmentation system113, such as stereoscopic images114generated by image cameras102. Storage system112may also store depth images116generated by depth cameras104. Additionally, segmentation system113may also store a segmentation model118. Segmentation model118may include data defining a model for segmenting images. In examples where segmentation model118is implemented using one or more artificial neural networks, the data defining segmentation model118may include input weights for parameters.

Storage system112may store computer-readable instructions of segmentation system113associated with a segmentation unit120and training unit122. Processors108may execute instructions of segmentation unit120and training unit122. Execution of instructions associated with segmentation unit120and training unit122may configure processors to perform the functionality ascribed in this disclosure to segmentation unit120and training unit122. Thus, when this disclosure indicates that segmentation unit120or training unit122(or sub-units thereof) perform specific actions, this may be the result of processors108executing instructions associated with segmentation unit120or training unit122. In other examples, specific actions described in this disclosure as being performed by segmentation unit120or training unit122(or sub-units thereof) may be performed by special purpose circuitry.

In general, segmentation unit120may apply segmentation model118to stereoscopic images114to generate segmentation masks124. Storage system112may store segmentation masks124. Each of segmentation masks124may identify one or more object instances within stereoscopic images114. An object instance may be an instance of an object, such as an individual human, animal, vehicle, plant, barrier, building, or other type of object.

Training unit122may perform a process to adaptively train segmentation model118. Training segmentation model118may improve the ability of segmentation model118to accurately segment images to identify object instances. In the example ofFIG.1, training unit122includes a projection unit126, a mask conversion unit128, a loss determination unit130, and a model update unit132.

As described in greater detail below, stereoscopic images114may include a pair of stereoscopic images that includes first image generated by a first image camera of image cameras102and a second image generated by a second image camera of image cameras102. Segmentation unit120may apply segmentation model118to the first image to generate a first segmentation mask identifying one or more object instances. Additionally, segmentation unit120may apply segmentation model118to the second image to generate a second segmentation mask identifying the one or more object instances. Projection unit126of training unit122may project the first segmentation mask to a viewpoint of the second camera to generate a first projected segmentation mask.

Mask conversion unit128may convert the first projected segmentation mask and the second segmentation mask to a first semantic mask and a second semantic mask, respectively. In general, a semantic mask indicates object instances associated with locations in an image. A semantic mask differs from a segmentation mask in that there is not a dimension corresponding to different object instances. Loss determination unit130may compute a first similarity value based on the first semantic mask and the second semantic mask. Additionally, projection unit126may project the second segmentation mask to a viewpoint of the first camera to generate a second projected segmentation mask. Mask conversion unit128may convert the second projected segmentation mask and the first segmentation mask to a third semantic mask and a fourth semantic mask, respectively. Loss determination unit130may compute a second similarity value based on the third semantic mask and the fourth semantic mask. Additionally, loss determination unit130may determine a loss value based on the first similarity value and the second similarity value. Model update unit132may train segmentation model118based on the loss value.

The process performed by training unit122may allow segmentation model118to be trained with fewer human-labeled images. In essence, a projected segmentation mask projected to a given viewpoint and an original segmentation mask from an image generated from the given viewpoint should be substantially the same. Differences between the projected segmentation mask and the original segmentation mask may therefore be considered errors that can be used to further train segmentation model118. The use of such errors to further train segmentation model118may avoid the need for segmentation model118to be trained using additional human-labeled images. Avoiding the need for segmentation model118to be trained using additional human-labeled images may reduce costs and may accelerate training of segmentation model118. The techniques of this disclosure for adaptively training segmentation model118may be applied within a device that uses segmentation model118for image segmentation, e.g., with or without sending or receiving data from other devices. In other examples, training segmentation model118may be performed on a device separate from a device that uses segmentation model118for image segmentation.

FIG.2is a block diagram illustrating an example process for generating a similarity value according to techniques of this disclosure. In the example ofFIG.2, a left image camera102A generates a left image200A and a right image camera102B generates a right image200B. Left image200A and right image200B may be stereoscopic images in the sense that both left image200A and right image200B may be images of the same scene at the same time from different viewpoints. In the example ofFIG.2, it is assumed that left image camera102A has a viewpoint that is to the left of the viewpoint of right image camera102B. In other examples, left and right may be reversed.

Segmentation unit120may apply segmentation model118to left image200A to generate a left segmentation mask202A (i.e., a first segmentation mask). InFIG.2and elsewhere in this disclosure, “seg.” is used as an abbreviation for “segmentation.” Segmentation unit120may also apply segmentation model118to right image200B to generate a right segmentation mask202B. Segmentation model118may be implemented in one of a variety of ways. For instance, as described in greater detail elsewhere in this disclosure, segmentation model118may be implemented as a SparseInst segmentation model.

Projection unit126may then project left segmentation mask202A to a viewpoint of right image camera102B to generate a right projected segmentation mask204. Conceptually speaking, right projected segmentation mask204is an estimate of how left segmentation mask202A would appear if left segmentation mask202A were generated from the viewpoint of right image camera102B. Projection unit126may project left segmentation mask202A to the viewpoint of right image camera102B to generate right projected segmentation mask204based on a left camera intrinsic matrix208(i.e., a camera intrinsic matrix of left image camera102A), a relative pose210of left image camera102A and right image camera102B, a left depth image206, and left segmentation mask202A. Left depth image206may be a depth image generated at the same time and substantially the same viewpoint as left image200A. In general, a camera intrinsic matrix is a matrix for converting points from a camera coordinate system to a pixel coordinate system. Left camera intrinsic matrix208is a matrix for converting points from a coordinate system of left image camera102A to a pixel coordinate system. The relative pose of left image camera102A and right image camera102B defines the positions of left image camera102A relative to right image camera102B. In some examples, projection unit126performs a matrix multiplication of left camera intrinsic matrix208, a matrix representing the relative pose of left image camera102A and right image camera102B, a 2-dimensional (2D) array comprising left depth image206, and left segmentation mask202A. For instance, projection unit126may generate right projected segmentation mask204according to the following equation.

In the equation above, Ml→rrepresents right projected segmentation mask204, K represents left camera intrinsic matrix208of left image camera102A, Tl→rrepresents the relative pose210of left image camera102A and right image camera102B, Dlrepresents left depth image206, K−1represents an inverse of left camera intrinsic matrix208, and Mlrepresents left segmentation mask202A.

Furthermore, mask conversion unit128may convert right projected segmentation mask204and right segmentation mask202B to a right projected semantic mask212A and a right semantic mask212B, respectively. In general, a semantic map is a 2-dimensional map that indicates object instances associated with locations in an image. The semantic map may be a 2-dimensional map or a 3-dimensional map. For example, right projected semantic mask212A is a 2-dimensional map that indicates object instances associated with locations in the projected right image (i.e., an image estimated based on left image200projected to the viewpoint of right image cameras102B). Similarly, right semantic mask212B is a 2-dimensional map that indicates object instances associated with locations in right image200B.

In some examples, right projected segmentation mask204includes a first array of values and right segmentation mask202B includes a second array of values. Each of the first and second arrays of values have a width dimension, a height dimension, and an object instance dimension. For each 3D coordinate combination of width coordinates in the width dimension, height coordinates in the height dimension, and instance indexes in the object instance dimension, a value in the first array having the 3D coordinate combination indicates a level of confidence that a pixel at a width coordinate of the 3D coordinate combination and a height coordinate of the 3D coordinate combination in a first projected image belongs to an object instance having an instance index of the 3D coordinate combination. For example, a value in the first array at width coordinate w, height coordinate h, and object instance coordinate d may indicate a level of confidence that a pixel in the first projected image at width coordinate w and height coordinate h belongs to an object corresponding to object instance coordinate d. For instance, if object instance coordinate d corresponds to a specific person, the value in the first array at width coordinate w, height coordinate h, and object instance coordinate d may indicate a level of confidence that a pixel in left image200A corresponds to the specific person. Furthermore, a value in the second array having the 3D coordinate combination indicates a level of confidence that a pixel at the width coordinate of the 3D coordinate combination and the height coordinate of the 3D coordinate combination in right image200B belongs to the object instance having the instance index of the 3D coordinate combination.

For each 2D coordinate combination of the height coordinates in the height dimension and the width coordinates in the width dimension, mask conversion unit128may, as part of converting right projected segmentation mask204to right projected semantic mask212A, identify a first array maximum value among values in the first array that have a height coordinate of the 2D coordinate combination and a width coordinate of the 2D coordinate combination. For example, there may be D values in the object instance dimension of right projected segmentation mask204at width coordinate w and height coordinate h. In this example, mask conversion unit128may determine which of the D values at width coordinate w and height coordinate h is greatest.

Additionally, mask conversion unit128may generate a ground-truth value by applying a threshold to the first array maximum value. For example, mask conversion unit128may set the ground-truth value equal to 1 if the first array maximum value is greater than (or greater than or equal to) to the threshold (e.g., 0.5 or another value) and may set the ground-truth value equal to 0 if the first array maximum value is less than (or less than or equal to) the threshold. In other examples, mask conversion unit128may set the ground-truth value to a value other than 0 or 1.

Mask conversion unit128may set a value in right projected semantic mask212A at the height and width coordinates of the 2D coordinate combination to the ground-truth value. For example, when the 2D coordinate combination specifies a width coordinate w and a height coordinate h, mask conversion unit128may set a value in right projected semantic mask212A at width coordinate w and height coordinate h to the ground-truth value.

As part of converting right segmentation mask202B to right semantic mask212B, mask conversion unit128may identify a second array maximum value among values in the second array that have the height and width coordinates of the 2D coordinate combination. For example, there may be D values in the object instance dimension of right segmentation mask202B at width coordinate w and height coordinate h. In this example, mask conversion unit128may determine which of the D values in right segmentation mask202B at width coordinate w and height coordinate h is greatest.

Mask conversion unit128may set a value in right semantic mask212B at the height and width coordinates of the 2D coordinate combination to the second array maximum value. For example, when the 2D coordinate combination specifies a width coordinate w and a height coordinate h, mask conversion unit128may set a value in right semantic mask212B at width coordinate w and height coordinate h to whichever of the D value in right segmentation mask220B at width coordinate w and height coordinate h is greatest.

After mask conversion unit128has generated right projected semantic mask212A and right semantic mask212B, loss determination unit130may compute a first similarity value based on right projected semantic mask212A and right semantic mask212B. For example, mask conversion unit128may compute the first similarity value as a first Dice similarity value (DSC) with the first semantic mask as a first ground truth and the second semantic mask as a first prediction. A DSC is a statistic to gauge the similarity of two samples, such as the first semantic mask and the second semantic mask. A DSC may be used to compare the pixel-wise agreement between a predicted segmentation and a corresponding ground truth. Mask conversion unit128may use the following formula to compute a DSC.

In the formula above, c can be computed as:

In the equations above, DSC is a continuous DSC, A is a predicted set (e.g., right projected semantic mask212A), B is a ground truth set (e.g., right semantic mask212B), ∥ indicates the cardinality of a set, c is a mean value of B over the values where both A and B are positive, a represents an individual value in A, and b represents an individual value in B. The sign( ) function may be defined as:

In another example, mask conversion unit128may use a binary cross-entropy value as a similarity value. For instance, mask conversion unit128may calculate a binary cross-entropy value from right semantic mask212B and right projected semantic mask212A.

Additionally, in the example ofFIG.2, training unit122may repeat the process described above while replacing left and right. Thus, projection unit126may generate a left projected segmentation mask (e.g., a second projected segmentation mask) based on right segmentation mask202B, mask conversion unit128may generate a left projected semantic mask (e.g., a third semantic mask) based on the left projected segmentation mask, and a left semantic mask (e.g., a fourth semantic mask) based on left segmentation mask202A.

Loss determination unit130may compute a second similarity value based on the left projected semantic mask and the left semantic mask. Loss determination unit130may determine a loss value based on the first similarity value and the second similarity value. For example, loss determination unit130may determine an intermediate loss value by adding the first similarity value and the second similarity value. In this example, loss determination unit130may determine the loss value by adding the intermediate loss value to a loss regularization factor. Inclusion of the loss regularization factor in the loss value may prevent trivial solutions. Thus, in some examples, loss determination unit130may calculate the loss value as shown in the following equations:

whererepresents the intermediate loss value, SegmentationLoss( ) represents a function (e.g., a Dice loss function, binary cross-entropy loss, etc.) for calculating a similarity value, Ml→rsemrepresents the semantic mask generated based on the projected segmentation mask generated by projecting left segmentation mask202A to the viewpoint of right camera102B, Mrgtsemrepresents the semantic mask generated based on right segmentation mask202B, Mr→lsemrepresents the semantic mask generated based on the projected segmentation mask generated by projecting right segmentation mask202B to the viewpoint of left image camera102A, Mlgtsemrepresents the semantic mask generated based on left segmentation mask202A,represents the loss value, A represents a Lagrangian multiplier, andrepresents a loss regularization factor. In other examples, the loss value may be calculated in other ways.

FIG.3is a block diagram illustrating an example segmentation model300in segmentation system113according to techniques of this disclosure. Segmentation model300may be an instance of segmentation model118(FIG.1,FIG.2). Although not shown inFIG.3, segmentation system113may include the elements of segmentation system113shown inFIG.2. In the example ofFIG.3, segmentation model300is implemented using a SparseInst architecture as described in Cheng et al., “Sparse Instance Activation for Real-Time Instance Segmentation” arXiv:2203.12827v1 [cs.CV]. In other examples, segmentation model118may be implemented using other architectures.

In the example ofFIG.3, segmentation model300includes a backbone component302, an encoder component304, and a decoder component306. Backbone component302receives an input image308for segmentation. Backbone component302generates multiple scaled versions of input image308at different scales, labeled C3, C4, and C5inFIG.3. In the example ofFIG.3, scaled image C3is scaled down by a factor of 8 in both the height and width dimensions, scaled image C4is scaled down by a factor of 16 in both the height and width dimensions, and scaled image C5is scaled down by a factor of 32 in both the height and width dimensions.

Encoder component304applies a pyramid pooling module (PPM)310to scaled image C5. PPM310may enlarge a receptive field and fuse multi-scale features. In other words, PPM310may generate ensemble high-level feature maps that represent global context information of multiple scales. PPM310may apply different max pooling operations (e.g., 1×1, 2×2, 3×3, 6×6 max pooling operations) to the input, apply separate 2-dimensional convolution operations to the results of the max pooling operations, and then concatenate the results of the 2-dimensional convolution operations with the original input to PPM310.

Additionally, a summation unit312of encoder component304may combine C4with 2× up-sampled output of PPM310. A summation unit314of encoder component304may combine C3with 2× up-sampled output of summation unit312. Encoder component304may apply a first convolutional network316to the output of PPM310, a second convolutional network318to the output of summation unit312, and a third convolutional network320to the output of summation unit314. Convolutional networks316,318, and320may apply 3×3 convolution. A concatenation unit322of encoder component304may concatenate a 4× up-sampled output of convolutional network316, a 2× up-sampled output of convolutional network318, and the output of convolutional network320, thereby generating a 3-dimensional array324. Decoder component306has two branches: an instance branch and a mask branch.

In the instance branch, 3-dimensional array324may be provided as input to an instance activation map (IAM) module326. IAM326predicts activation maps to acquire instance features for recognition and mask kernels. The mask branch includes a convolutional network328that generates mask features M. Convolutional network328may be a 3×3 stack of convolutions with 256 channels. Decoder component306includes a multiplier unit329that multiplies the mask features with predicted kernels338generated by the instance branch to generate a segmentation mask330.

Instance activation maps are instance-aware maps that highlight informative regions for objects. An array325is input to IAM module326. Array325may be a copy of array324. IAM module326may apply Fiam332to array325. Fiam332is a neural network (e.g., a 3×3 convolution) with sigmoid non-linearity. The output of Fiam332is denoted inFIG.3as A and may be an array dimensions N×H×W, where N is a number of instance maps, H and W are a height and width, respectively. IAM module326may then apply three linear layers334to the output of Fiam332. Each of the linear layers applies a linear transform to incoming data, such as the output of Fiam332. A multiplication unit336of IAM module326may multiply the output of the linear layers334with array325to obtain an array of classification values, an array of objectness score values, and a mask kernel. Fiam332generates instance activation maps.

FIG.4is a flowchart illustrating an example method according to techniques of this disclosure. In the example ofFIG.4, storage system112may store a pair of stereographic images, including a first image (e.g., left image200A) generated by a first image (e.g., left image camera102A) and a second image (e.g., right image200B) generated by a second image camera (e.g., right image camera102B) (400). Furthermore, segmentation unit120may apply segmentation model118to the first image to generate a first segmentation mask (e.g., left segmentation mask202A) identifying one or more object instances (402). Segmentation unit120may apply segmentation model118to the second image to generate a second segmentation mask (e.g., right segmentation mask202B) identifying the one or more object instances (404).

Projection unit126may project the first segmentation mask to a viewpoint of the second image camera to generate a first projected segmentation mask (e.g., right projected segmentation mask204) (406). Projection unit126may generate the first projected segmentation mask according to the examples provided elsewhere in this disclosure. For instance, projection unit126may obtain a first depth image (e.g., left depth image206) representing estimates of depths in the first image. Projection unit126may project the first segmentation mask to the viewpoint of the second camera to generate the first projected segmentation mask based on a camera intrinsic matrix of the first camera, a relative pose of the first camera and the second camera, the first depth image, and the first segmentation mask. Computing system106may include a second depth camera configured to generate the second depth image. In some examples, projection unit126may perform a matrix multiplication of the camera intrinsic matrix of the first camera, a matrix representing the relative pose of the first camera and the second camera, a 2-dimensional (2D) array comprising the first depth image, an inverse of the camera intrinsic matrix of the first camera, and the first segmentation mask.

Mask conversion unit128may convert the first projected segmentation mask to a first semantic mask (e.g., right projected semantic mask212A) (408). Additionally, mask conversion unit128may convert the second segmentation mask to a second semantic mask (e.g., right semantic mask212) (409). Mask conversion unit128may convert the segmentation masks to semantic masks in accordance with the examples provided elsewhere in this disclosure.

Loss determination unit130may compute a first similarity value based on the first semantic mask and the second semantic mask (410). In some examples, loss determination unit130may compute the first similarity value as a first Dice similarity value with the first semantic mask as a first ground truth and the second semantic mask as a first prediction.

Furthermore, projection unit126may project the second segmentation mask (e.g., right segmentation mask202B) to a viewpoint of the first image camera (e.g., image camera102A) to generate a second projected segmentation mask (412). For instance, projection unit126may obtain a second depth image representing estimates of depths in the second image and may project the second segmentation mask to the viewpoint of the first camera to generate the second projected segmentation mask based on a camera intrinsic matrix of the second camera, the relative pose of the first camera and the second camera, the second depth image, and the second segmentation mask. Computing system106may include a second depth camera configured to generate the second depth image. In some examples, projection unit126may perform a matrix multiplication of the camera intrinsic matrix of the second camera, the matrix representing the relative pose of the first camera and the second camera, a 2-dimensional (2D) array comprising the second depth image, an inverse of the camera intrinsic matrix of the second camera, and the second segmentation mask.

Mask conversion unit128may convert the second projected segmentation mask to a third semantic mask (414). Additionally, mask conversion unit128may convert the first segmentation mask (e.g., left segmentation mask202A) to a fourth semantic mask (415). Thus, in some examples, the first projected segmentation mask includes a first array of values, the second segmentation mask includes a second array of values, the second projected segmentation mask includes a third array of values, the first segmentation mask includes a fourth array of values, and each of the first, second, third, and fourth arrays of values has a width dimension, a height dimension, and an object instance dimension. For each 3D coordinate combination of width coordinates in the width dimension, height coordinates in the height dimension, and instance indexes in the object instance dimension:a value in the first array having the 3D coordinate combination indicates a level of confidence that a pixel at a width coordinate of the 3D coordinate combination and a height coordinate of the 3D coordinate combination in a first projected image belongs to an object instance having an instance index of the 3D coordinate combination,a value in the second array having the 3D coordinate combination indicates a level of confidence that a pixel at the width coordinate of the 3D coordinate combination and the height coordinate of the 3D coordinate combination in the second image belongs to the object instance having the instance index of the 3D coordinate combination,a value in the third array having the 3D coordinate combination indicates a level of confidence that a pixel at the width coordinate of the 3D coordinate combination and the height coordinate of the 3D coordinate combination in a second projected image belongs to the object instance having the instance index of the 3D coordinate combination, anda value in the fourth array having the 3D coordinate combination indicates a level of confidence that a pixel at the width coordinate of the 3D coordinate combination and the height coordinate of the 3D coordinate combination in the first image belongs to the object instance having the instance index of the 3D coordinate combination,
For each 2-dimensional (2D) coordinate combination of the height coordinates in the height dimension and the width coordinates in the width dimension, mask conversion unit128may, as part of converting the first projected segmentation mask to the first semantic mask, identify a first array maximum value among values in the first array that have a height coordinate of the 2D coordinate combination and a width coordinate of the 2D coordinate combination, generate a ground-truth value by applying a threshold to the first array maximum value, and set a value in the first semantic mask at the height and width coordinates of the 2D coordinate combination to the ground-truth value. Furthermore, as part of converting the second segmentation mask to the second semantic mask, mask conversion unit128may identify a second array maximum value among values in the second array that have the height and width coordinates of the 2D coordinate combination, set a value in the second semantic mask at the height and width coordinates of the 2D coordinate combination to the second array maximum value. As part of converting the second projected segmentation mask to the third semantic mask, mask conversion unit128may identify a third array maximum value among values in the third array that have the height and width coordinates of the 2D coordinate combination, generate a ground-truth value by applying the threshold to the third array maximum value; and set a value in the third semantic mask at the height and width coordinates of the 2D coordinate combination to the ground-truth value. As part of converting the first segmentation mask to the fourth semantic mask, mask conversion unit128may identify a fourth array maximum value among values in the fourth array that have the height and width coordinates of the 2D coordinate combination, set a value in the fourth semantic mask at the height and width coordinates of the 2D coordinate combinations to the fourth array maximum value.

Loss determination unit130may compute a second similarity value based on the third semantic mask and the fourth semantic mask (416). In some examples, loss determination unit130may compute the second similarity value as a second Dice similarity value with the third semantic mask as a second ground truth and the fourth semantic mask as a second prediction. Additionally, loss determination unit130may determine a loss value based on the first similarity value and the second similarity value (418). Loss determination unit130may determine the loss value as described elsewhere in this disclosure. For instance, loss determination unit130may determine the loss value based on the first similarity value, the second similarity value, and a loss regularization factor.

Model update unit132may train segmentation model118based on the loss value (420). For example, model update unit132may use the loss value in a backpropagation process that updates weights of inputs to neurons in segmentation model118.

Various examples of the techniques of this disclosure are summarized in the following clauses.

Clause 1. A system comprising: a storage system comprising one or more computer-readable media, the storage system configured to store a pair of stereoscopic images, the stereoscopic images including a first image generated by a first camera and a second image generated by a second camera; and one or more processors implemented in circuitry, the one or more processors configured to: apply a segmentation model to the first image to generate a first segmentation mask identifying one or more object instances; apply the segmentation model to the second image to generate a second segmentation mask identifying the one or more object instances; project the first segmentation mask to a viewpoint of the second camera to generate a first projected segmentation mask; convert the first projected segmentation mask to a first segmentation mask; convert the second segmentation mask to a second semantic mask and a second semantic mask, respectively; compute a first similarity value based on the first semantic mask and the second semantic mask; project the second segmentation mask to a viewpoint of the first camera to generate a second projected segmentation mask; convert the second projected segmentation mask to a third semantic mask; convert the first segmentation mask to a fourth semantic mask; compute a second similarity value based on the third semantic mask and the fourth semantic mask; determine a loss value based on the first similarity value and the second similarity value; and train the segmentation model based on the loss value.

Clause 2. The system of claim1, wherein the one or more processors are configured to: obtain a first depth image representing estimates of depths in the first image; obtain a second depth image representing estimates of depths in the second image; project the first segmentation mask to the viewpoint of the second camera to generate the first projected segmentation mask based on a camera intrinsic matrix of the first camera, a relative pose of the first camera and the second camera, the first depth image, and the first segmentation mask; and project the second segmentation mask to the viewpoint of the first camera to generate the second projected segmentation mask based on a camera intrinsic matrix of the second camera, the relative pose of the first camera and the second camera, the second depth image, and the second segmentation mask.

Clause 3. The system of claim2, further comprising: a first depth camera configured to generate the first depth image; and a second depth camera configured to generate the second depth image.

Clause 4. The system of any one of claims2-3, wherein the one or more processors are configured to: as part of projecting the first segmentation mask, perform a matrix multiplication of the camera intrinsic matrix of the first camera, a matrix representing the relative pose of the first camera and the second camera, a 2-dimensional (2D) array comprising the first depth image, an inverse of the camera intrinsic matrix of the first camera, and the first segmentation mask, and as part of projecting the second segmentation mask, perform a matrix multiplication of the camera intrinsic matrix of the second camera, the matrix representing the relative pose of the first camera and the second camera, a 2-dimensional (2D) array comprising the second depth image, an inverse of the camera intrinsic matrix of the second camera, and the second segmentation mask.

Clause 5. The system of any one of claims1-4, wherein the one or more processors are configured to determine the loss value based on the first similarity value, the second similarity value, and a loss regularization factor.

Clause 6. The system of any one of claims1-5, wherein: the first projected segmentation mask includes a first array of values, the second segmentation mask includes a second array of values, the second projected segmentation mask includes a third array of values, the first segmentation mask includes a fourth array of values, each of the first, second, third, and fourth arrays of values has a width dimension, a height dimension, and an object instance dimension, for each 3-dimensional (3D) coordinate combination of width coordinates in the width dimension, height coordinates in the height dimension, and instance indexes in the object instance dimension: a value in the first array having the 3D coordinate combination indicates a level of confidence that a pixel at a width coordinate of the 3D coordinate combination and a height coordinate of the 3D coordinate combination in a first projected image belongs to an object instance having an instance index of the 3D coordinate combination, a value in the second array having the 3D coordinate combination indicates a level of confidence that a pixel at the width coordinate of the 3D coordinate combination and the height coordinate of the 3D coordinate combination in the second image belongs to the object instance having the instance index of the 3D coordinate combination, a value in the third array having the 3D coordinate combination indicates a level of confidence that a pixel at the width coordinate of the 3D coordinate combination and the height coordinate of the 3D coordinate combination in a second projected image belongs to the object instance having the instance index of the 3D coordinate combination, and a value in the fourth array having the 3D coordinate combination indicates a level of confidence that a pixel at the width coordinate of the 3D coordinate combination and the height coordinate of the 3D coordinate combination in the first image belongs to the object instance having the instance index of the 3D coordinate combination, for each 2-dimensional (2D) coordinate combination of the height coordinates in the height dimension and the width coordinates in the width dimension, the one or more processors are configured to: as part of converting the first projected segmentation mask to the first semantic mask: identify a first array maximum value among values in the first array that have a height coordinate of the 2D coordinate combination and a width coordinate of the 2D coordinate combination; generate a ground-truth value by applying a threshold to the first array maximum value; and set a value in the first semantic mask at the height and width coordinates of the 2D coordinate combination to the ground-truth value; as part of converting the second segmentation mask to the second semantic mask: identify a second array maximum value among values in the second array that have the height and width coordinates of the 2D coordinate combination; set a value in the second semantic mask at the height and width coordinates of the 2D coordinate combination to the second array maximum value; as part of converting the second projected segmentation mask to the third semantic mask: identify a third array maximum value among values in the third array that have the height and width coordinates of the 2D coordinate combination; generate a ground-truth value by applying the threshold to the third array maximum value; and set a value in the third semantic mask at the height and width coordinates of the 2D coordinate combination to the ground-truth value; and as part of converting the first segmentation mask to the fourth semantic mask: identify a fourth array maximum value among values in the fourth array that have the height and width coordinates of the 2D coordinate combination; and set a value in the fourth semantic mask at the height and width coordinates of the 2D coordinate combinations to the fourth array maximum value.

Clause 7. The system of any one of claims1-6, wherein the one or more processors are configured to: compute the first similarity value as a first Dice similarity value with the first semantic mask as a first ground truth and the second semantic mask as a first prediction; and compute the second similarity value as a second Dice similarity value with the third semantic mask as a second ground truth and the fourth semantic mask as a second prediction.

Clause 8. The system of any one of claims1-7, wherein the segmentation model is implemented using a SparseInst architecture.

Clause 9. A method comprising: storing a pair of stereoscopic images, the stereoscopic images including a first image generated by a first camera and a second image generated by a second camera; and applying a segmentation model to the first image to generate a first segmentation mask identifying one or more object instances; applying the segmentation model to the second image to generate a second segmentation mask identifying the one or more object instances; projecting the first segmentation mask to a viewpoint of the second camera to generate a first projected segmentation mask; converting the first projected segmentation mask to a first semantic mask; converting the second segmentation mask to a second semantic mask; computing a first similarity value based on the first semantic mask and the second semantic mask; projecting the second segmentation mask to a viewpoint of the first camera to generate a second projected segmentation mask; converting the second projected segmentation mask to a third semantic mask; converting the first segmentation mask to a fourth semantic mask; computing a second similarity value based on the third semantic mask and the fourth semantic mask; determining a loss value based on the first similarity value and the second similarity value; and training the segmentation model based on the loss value.

Clause 10. The method of claim9, wherein: the method further comprises: obtaining a first depth image representing estimates of depths in the first image; and obtaining a second depth image representing estimates of depths in the second image; projecting the first segmentation mask comprises projecting the first segmentation mask to the viewpoint of the second camera to generate the first projected segmentation mask based on a camera intrinsic matrix of the first camera, a relative pose of the first camera and the second camera, the first depth image, and the first segmentation mask; and projecting the second segmentation mask comprises projecting the second segmentation mask to the viewpoint of the first camera to generate the second projected segmentation mask based on a camera intrinsic matrix of the second camera, the relative pose of the first camera and the second camera, the second depth image, and the second segmentation mask.

Clause 11. The method of claim10, wherein: projecting the first segmentation mask comprises performing a matrix multiplication of the camera intrinsic matrix of the first camera, a matrix representing the relative pose of the first camera and the second camera, a 2-dimensional (2D) array comprising the first depth image, an inverse of the camera intrinsic matrix of the first camera, and the first segmentation mask, and projecting the second segmentation mask comprises performing a matrix multiplication of the camera intrinsic matrix of the second camera, the matrix representing the relative pose of the first camera and the second camera, a 2-dimensional (2D) array comprising the second depth image, an inverse of the camera intrinsic matrix of the second camera, and the second segmentation mask.

Clause 12. The method of any one of claims9-11, wherein determining the loss value comprise determining the loss value based on the first similarity value, the second similarity value, and a loss regularization factor.

Clause 13. The method of any one of claims9-12, wherein: the first projected segmentation mask includes a first array of values, the second segmentation mask includes a second array of values, the second projected segmentation mask includes a third array of values, the first segmentation mask includes a fourth array of values, each of the first, second, third, and fourth arrays of values has a width dimension, a height dimension, and an object instance dimension, for each 3-dimensional (3D) coordinate combination of width coordinates in the width dimension, height coordinates in the height dimension, and instance indexes in the object instance dimension: a value in the first array having the 3D coordinate combination indicates a level of confidence that a pixel at a width coordinate of the 3D coordinate combination and a height coordinate of the 3D coordinate combination in a first projected image belongs to an object instance having an instance index of the 3D coordinate combination, a value in the second array having the 3D coordinate combination indicates a level of confidence that a pixel at the width coordinate of the 3D coordinate combination and the height coordinate of the 3D coordinate combination in the second image belongs to the object instance having the instance index of the 3D coordinate combination, a value in the third array having the 3D coordinate combination indicates a level of confidence that a pixel at the width coordinate of the 3D coordinate combination and the height coordinate of the 3D coordinate combination in a second projected image belongs to the object instance having the instance index of the 3D coordinate combination, and a value in the fourth array having the 3D coordinate combination indicates a level of confidence that a pixel at the width coordinate of the 3D coordinate combination and the height coordinate of the 3D coordinate combination in the first image belongs to the object instance having the instance index of the 3D coordinate combination, the method comprises, for each 2-dimensional (2D) coordinate combination of the height coordinates in the height dimension and the width coordinates in the width dimension: as part of converting the first projected segmentation mask to the first semantic mask: identifying a first array maximum value among values in the first array that have a height coordinate of the 2D coordinate combination and a width coordinate of the 2D coordinate combination; generating a ground-truth value by applying a threshold to the first array maximum value; and setting a value in the first semantic mask at the height and width coordinates of the 2D coordinate combination to the ground-truth value; as part of converting the second segmentation mask to the second semantic mask: identifying a second array maximum value among values in the second array that have the height and width coordinates of the 2D coordinate combination; setting a value in the second semantic mask at the height and width coordinates of the 2D coordinate combination to the second array maximum value; as part of converting the second projected segmentation mask to the third semantic mask: identifying a third array maximum value among values in the third array that have the height and width coordinates of the 2D coordinate combination; generating a ground-truth value by applying the threshold to the third array maximum value; and setting a value in the third semantic mask at the height and width coordinates of the 2D coordinate combination to the ground-truth value; and as part of converting the first segmentation mask to the fourth semantic mask: identifying a fourth array maximum value among values in the fourth array that have the height and width coordinates of the 2D coordinate combination; and setting a value in the fourth semantic mask at the height and width coordinates of the 2D coordinate combinations to the fourth array maximum value.

Clause 14. The method of any one of claims9-13, wherein: computing the first similarity value comprises computing the first similarity value as a first Dice similarity value with the first semantic mask as a first ground truth and the second semantic mask as a first prediction; and computing the second similarity value comprise computing the second similarity value as a second Dice similarity value with the third semantic mask as a second ground truth and the fourth semantic mask as a second prediction.

Clause 15. The method of any one of claims9-14, wherein the segmentation model is implemented using a SparseInst architecture.

Clause 16. Non-transitory computer-readable storage media having stored thereon instructions that, when executed, cause one or more processors to: store a pair of stereoscopic images, the stereoscopic images including a first image generated by a first camera and a second image generated by a second camera; and apply a segmentation model to the first image to generate a first segmentation mask identifying one or more object instances; apply the segmentation model to the second image to generate a second segmentation mask identifying the one or more object instances; project the first segmentation mask to a viewpoint of the second camera to generate a first projected segmentation mask; convert the first projected segmentation mask to a first semantic mask; convert the second segmentation mask to a second semantic mask; compute a first similarity value based on the first semantic mask and the second semantic mask; project the second segmentation mask to a viewpoint of the first camera to generate a second projected segmentation mask; convert the second projected segmentation mask to a third semantic mask; convert the first segmentation mask to a fourth semantic mask; compute a second similarity value based on the third semantic mask and the fourth semantic mask; determine a loss value based on the first similarity value and the second similarity value; and train the segmentation model based on the loss value.

Clause 17. The non-transitory computer-readable storage media of claim16, wherein the instructions cause the one or more processors to: obtain a first depth image representing estimates of depths in the first image; obtain a second depth image representing estimates of depths in the second image; project the first segmentation mask to the viewpoint of the second camera to generate the first projected segmentation mask based on a camera intrinsic matrix of the first camera, a relative pose of the first camera and the second camera, the first depth image, and the first segmentation mask; and project the second segmentation mask to the viewpoint of the first camera to generate the second projected segmentation mask based on a camera intrinsic matrix of the second camera, the relative pose of the first camera and the second camera, the second depth image, and the second segmentation mask.

Clause 18. The non-transitory computer-readable storage media of any one of claims16-17, wherein: the first projected segmentation mask includes a first array of values, the second segmentation mask includes a second array of values, the second projected segmentation mask includes a third array of values, the first segmentation mask includes a fourth array of values, each of the first, second, third, and fourth arrays of values has a width dimension, a height dimension, and an object instance dimension, for each 3-dimensional (3D) coordinate combination of width coordinates in the width dimension, height coordinates in the height dimension, and instance indexes in the object instance dimension: a value in the first array having the 3D coordinate combination indicates a level of confidence that a pixel at a width coordinate of the 3D coordinate combination and a height coordinate of the 3D coordinate combination in a first projected image belongs to an object instance having an instance index of the 3D coordinate combination, a value in the second array having the 3D coordinate combination indicates a level of confidence that a pixel at the width coordinate of the 3D coordinate combination and the height coordinate of the 3D coordinate combination in the second image belongs to the object instance having the instance index of the 3D coordinate combination, a value in the third array having the 3D coordinate combination indicates a level of confidence that a pixel at the width coordinate of the 3D coordinate combination and the height coordinate of the 3D coordinate combination in a second projected image belongs to the object instance having the instance index of the 3D coordinate combination, and a value in the fourth array having the 3D coordinate combination indicates a level of confidence that a pixel at the width coordinate of the 3D coordinate combination and the height coordinate of the 3D coordinate combination in the first image belongs to the object instance having the instance index of the 3D coordinate combination, for each 2-dimensional (2D) coordinate combination of the height coordinates in the height dimension and the width coordinates in the width dimension, the instructions cause the one or more processors to: as part of converting the first projected segmentation mask to the first semantic mask: identify a first array maximum value among values in the first array that have a height coordinate of the 2D coordinate combination and a width coordinate of the 2D coordinate combination; generate a ground-truth value by applying a threshold to the first array maximum value; and set a value in the first semantic mask at the height and width coordinates of the 2D coordinate combination to the ground-truth value; as part of converting the second segmentation mask to the second semantic mask: identify a second array maximum value among values in the second array that have the height and width coordinates of the 2D coordinate combination; set a value in the second semantic mask at the height and width coordinates of the 2D coordinate combination to the second array maximum value; as part of converting the second projected segmentation mask to the third semantic mask: identify a third array maximum value among values in the third array that have the height and width coordinates of the 2D coordinate combination; generate a ground-truth value by applying the threshold to the third array maximum value; and set a value in the third semantic mask at the height and width coordinates of the 2D coordinate combination to the ground-truth value; and as part of converting the first segmentation mask to the fourth semantic mask: identify a fourth array maximum value among values in the fourth array that have the height and width coordinates of the 2D coordinate combination; and set a value in the fourth semantic mask at the height and width coordinates of the 2D coordinate combinations to the fourth array maximum value.

Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the terms “processor” and “processing circuitry,” as used herein may refer to any of the foregoing structures or any other structure suitable for implementation of the techniques described herein. In addition, in some aspects, the functionality described herein may be provided within dedicated hardware and/or software modules. Also, the techniques could be fully implemented in one or more circuits or logic elements.