Patent Description:
Convolutional neural networks (CNNs) have been successfully applied to various computer vision problems by posing these problems as image segmentation problems. Examples include road scene understanding for autonomous driving, and interpreting medical imaging. For such applications, networks are typically trained with multi-class per-pixel labels that together form an image-sized segmentation map or mask. The output of such a network is then again an image-sized map, representing per-pixel class probabilities.

In likelihood-based semantic segmentation, a neural network can be trained to perform semantic segmentation by training it on training data consisting of images (e.g. photographs) and corresponding human-labelled "ground-truth" segmentation maps. A final segmentation map for a given input image can subsequently be obtained from the trained network by applying the argmax function along the class dimension of the predicted probabilities (i.e. for each pixel, selecting the most probable classification). This segmentation map may be useful in, for example, enabling a self-driving car to determine whether an object in view is likely to be another vehicle or a pedestrian.

However, producing such pixel-level ground-truth labels is time-consuming and expensive. Recent advancements in weakly-supervised training show that reasonable performance can be obtained by using only image-level labels-i.e. training data consisting of images and corresponding sets of labels indicating which of a predetermined set of objects (classes) is present in each image. For example, an image of a street scene may be labelled as containing {"bicycle", "road", "vegetation"}.

A weakly-supervised image-level classifier neural network can be used as a proxy task for performing pixel-level image segmentation by extracting an attention map, also referred to herein as a class activation map, from the network, which identifies regions of the input image that the network considers to be discriminative for a particular target object. These regions represent where the network has focused in order to identify the object. By combining attention maps for each of the different possible classes the image might contain, it is possible to segment the image into the different objects. However, classification needs only the minimum evidence to make predictions, so such networks tend to focus on the most discriminative object regions, and so any resulting segmentation is likely to miss significant parts of the object. The network that has been trained to classifier dogs and cats may, for instance, discriminate principally on the heads of the animals, so any derived segmentation map may fail to classify the animals' bodies correctly. This "discriminative localization" problem typically leads to poor segmentation performance. It is especially prevalent in non-rigid object classes such as birds, cats, horses and sheep where the texture of the fur or skin is much less discriminative than other body parts such as heads or feet.

It has been proposed to address this problem by using a classifier network to identify the most discriminative object regions in an input image, and then training one or more further classifier networks on the same image with these regions removed, so that the further networks learn to find less discriminative regions that belong to the target object. Combining the attention maps from the multiple classifier networks can result a larger map having better coverage of the target object. See, for example, [<NPL>] and [<NPL>].

It has also been proposed to train a single network to classify an image using an attention map that covers as much of the target object as possible. The network consists of two serialised streams having shared weights. The first "classifier" stream generates an attention map that is passed as input to the second "attention mining" stream. See [<NPL>].

However, the applicant has recognised that the performance of these known approaches can be improved upon. Thus, the present invention seeks to present an improved model for use in semantic image segmentation.

From a first aspect, the invention provides a method of training a model for use in semantic image segmentation, wherein the model comprises:.

wherein the method comprises training the first network by:.

wherein the method comprises training the second network by:.

From a second aspect, the invention provides a computer processing system for training a model for use in semantic image segmentation,.

From a third aspect, the invention provides computer software comprising instructions which, when executed on a computer processing system, cause the computer processing system to train a model for use in semantic image segmentation, wherein the model comprises:.

wherein the instructions cause the computer processing system to train the first network by:.

wherein the instructions cause the computer processing system to train the second network by:.

From a fourth aspect, the invention provides computer software comprising instructions which, when executed on a computer processing system, cause the computer processing system to perform semantic image segmentation using the first classifier neural network of a model trained by a method as disclosed herein.

From a fifth aspect, the invention provides a computer processing system configured to implement the first classifier neural network of a model trained by a method as disclosed herein, for performing semantic image segmentation.

Thus it will be seen that, in accordance with the invention, two classifier neural networks are trained independently, in an adversarial manner, using two different loss functions. The first classifier neural network is trained towards identifying discriminative regions that are sufficiently large that, once they are removed from the first training image, the second classifier neural network is unable to identify the target object. Meanwhile, the second network is separately trained to improve its classification performance on modified images generated by the first network, thereby forcing the first network to improve its performance. The model can thus be trained in an adversarial minimax fashion so that the first network learns to identify regions that conform accurately to the target object.

Once trained, the first classifier neural network may be used in an inference mode to perform accurate semantic image segmentation, e.g. by identifying respective sets of pixels in an input image that are discriminative for one or more target objects, and combining these sets of pixels (e.g. using an argmax operation) to produce a segmentation map for the input image.

For convenience, the first classifier neural network will be referred to herein as a "localizer" network, while the second classifier neural network will be referred to as the "adversarial" network.

It will be appreciated that this approach of having two independently-trained networks differs from models that use a single network, trained to minimise one loss function that contains both classification and attention-mining loss terms, e.g. as described in [<NPL>]. As explained below, the present approach has been found to provide better classification and localization performance, as it can better accommodate the different distributions of data that its two independently-trained networks operate on.

The localizer and adversarial networks may be any convolutional neural networks. They may be the same or different types of network. However, the localizer and adversarial networks are preferably two independent networks-i.e. having no shared weights.

Training the second network may comprise using the first network, with updated weights determined through training of the first network, to identify pixels of a training image that are discriminative for the target object. Training the first network may comprise using the second network, with updated weights determined through training of the second network, to determine a likelihood of a modified training image containing the target object.

The adversarial training may be performed iteratively-i.e. using an iterative, alternating training process. The localizer network may be trained on one of more training images, with the classifier network operating in an inference mode, e.g. during a first time period. The classifier network may be trained on one of more training images, with the localizer network operating in an inference mode, e.g. during a second time period, distinct from the first time period. The weights of the first network may be updated independently of the weights of the second network-i.e. without changing any weights of the adversarial network when updating weights of the localizer network, and without changing any weights of the localizer network when updating weights of the adversarial network. Training the model may comprise alternately training the first network and the second network. This alternating training may be repeated a plurality of times-e.g. for tens, hundreds or thousands of cycles.

The second training image may be different from the first training image, or it may be the same. More generally, the networks may be trained on a common plurality of training images, or they may be trained on distinct (e.g. non-overlapping) sets of training images.

The localizer and adversarial networks may be the only two classifier neural networks in the model. This may make the model faster to train compared with models that require three of more classifier models to be trained.

The pixels identified by the localizer network may correspond to one or more regions that are discriminative for the target object. They may be pixels that are relatively more discriminative, for the localizer network, than any other pixels of the image. They may be determined from an attention map (i.e. class activation map) for the localizer network. They may be determined from a global average pooling layer and a fully connected layer of the localizer network. They may be identified by applying a hard or soft threshold to attention-map data obtained from the localizer network. The attention-map data may comprise values representative of how relatively discriminative respective pixels of the training image are for the target object.

Attenuating or erasing the identified pixels may comprise reducing a pixel value of each identified pixels. It may comprise reducing each pixel value to zero (i.e. erasing the pixels). However, in some embodiments, it comprises attenuating the pixels using a differentiable operation-e.g. by applying a soft thresholding function to attention-map data. This can enable gradients still to be back-propagated.

The localizer network is preferably trained to classify the target object. Training the localizer network may therefore comprise inputting a set of training images to the localizer network, the set comprising one or more training images that contain the target object and one or more training images that do not contain the target object. It may comprise, for each training image of the set, using the localizer network to determine a likelihood of the respective training image containing the target object, and updating the localizer network using a loss function that is a monotonically-decreasing function of the determined likelihood when the training image contains the target object, and that is a monotonically-increasing function of the determined likelihood when the training image does not contain the target object. This loss function may be the same loss function as disclosed above. It may comprise a first loss term relating to the likelihood determined by the adversarial network and a second loss term for correctly classifying the target object. The first network may be trained to classify the target object using the first training image.

The localizer and adversarial networks are preferably image-level classifier networks. The model is preferably trained using training data that comprises training images and associated image-level labels. The model is preferably an end-to-end model. The adversarial network is preferably trained using backpropagation, independently of the localizer network. The localizer network is preferably trained using backpropagation, independently of the adversarial network.

The localizer network is preferably trained to favour identifying smaller sets of discriminative pixels. This can help prevent the localizer from favouring a trivial solution in which it identifies the entire image as discriminative. Training the localizer network may therefore comprise updating weights of the localizer network using a loss function that is a function of the identified pixels for a training image-e.g. that is a function of the number of identified pixels. This loss function may increase monotonically with the number of identified pixels. When the pixels are identified by thresholding attention-map data, this loss function may be a function of the attention-map data output by the localizer network. The loss function may penalise attention maps that have high average or total values across some or all of the image. It may be a monotonically-increasing function of the sum of the pixel values over the whole (or over part) of an attention map for a training image.

Training the localizer network to classify the target object and/or to favour identifying smaller sets of discriminative pixels and/or to reduce the success of the adversarial network may be performed during a single weight updating operation using a single localizer loss function. The same first training image, containing the target object, may be used in the updating operation.

Thus, the localizer network may be trained using a localizer loss function that is, as well as being a monotonically-increasing function of the likelihood, determined by the adversarial network, of the first modified training image containing the target object:.

The first and/or second training image may contain more than one target object, which may be in different classes. The first and second classifier networks may be trained to classify a plurality of target object types (i.e. classes). Each target object may be a solid object (e.g. "bicycle") or it could be any other entity (e.g. "sky" or "purple"). An image may contain a target object even when it contains only part of the real-world object (e.g. half of a person)
The training may be implemented by training logic in the software or computer processing system. The training of any of the neural networks may comprise a gradient descent process. The model may be trained on a plurality of training images, which may comprise one hundred, one thousand, ten thousand, or more, images.

In some embodiments, each training image may be a photographic image from a camera-e.g. one frame from a video stream. It may be an image of a street scene.

The model may have a training mode and a trained (inference) mode. The training may take place during a training phase. After the training phase, the model may be configured to receive an input image (or an uncalibrated predicted label distribution for an input image) and to segment the input image. In some embodiments, weights may be extracted from the trained first localizer neural network and used to create a model for semantic image segmentation. This may be a standalone inference model that comprises the first classifier neural network but that does not include the second classifier neural network or any training logic. The trained model may be configured to output a segmentation map for an input image. It may generate the segmentation map by operating the first network to identify respective pixels of the input image (e.g. from activation maps) that are discriminative for each of a set of target object. It may combine a plurality of activation maps or thresholded pixel masks to generate the segmentation map.

The computer processing system may comprise an input for receiving image data from a camera. It may be an in-vehicle computer processing system. It may be configured to output segmentation data-e.g., to an autonomous driving system.

The training images, modified training images, inference input images, and activation maps, may be represented and encoded in any appropriate ways. They may be rescaled between operations. Data, including training data, may be stored in and accessed from a database or other data retrieval system. Weights for the neural networks may be stored as values in digital memory.

The methods disclosed herein may be fully or wholly computer-implemented methods. The computer software disclosed herein may be on a transitory or a non-transitory computer-readable medium. The model could be implemented on one or more further computer processing systems that are distinct from the computer processing system that is configured to train the model-e.g. with the systems being arranged to communicate over a physical interface. However, in other embodiments, the computer processing system is also configured to implement the first network and/or the second network.

The first and second classifier neural networks may each comprise any number of convolution layers, dense blocks, and other processing layers. The model logic and/or training logic may comprise software instructions for one or more processors, or may comprise dedicated hardware logic, or may be implemented using a combination of software and application-specific hardware. The software may comprise instructions that are stored in a memory of the computer processing system. The computer processing system may comprise one or more of: CPUs, DSPs, GPUs, FPGAs, ASICs, volatile memory, non-volatile memory, inputs, outputs, displays, network connections, power supplies, radios, clocks, and any other appropriate components. It may comprise one or more servers or supercomputers. It may comprise a microcontroller or system-on-chip (e.g. when implementing a trained model for inference operation). It may be configured to store or display or output a segmentation map or other segmentation data.

Semantic segmentation is the task of predicting a pixelwise class label y ∈ {<NUM>,. , K}H"x" W for an input image <MAT> of height H, width W, and colour-space C. In some use cases, such as in self-driving vehicles, the image x may be a red-green-blue (RGB) image or video frame of a street scene, and the K classes could include "road", "sidewalk", "building", "wall", "person", "car", etc..

Described here is an exemplary end-to-end adversarial erasing model, embodying the invention. It will be given the abbreviation "EADER" for convenience.

In weakly-supervised semantic segmentation (WSSS), models are trained not on images annotated by a human with pixel-level class labels, but using training data that is labelled using bounding boxes, scribbles, points, or image-level labels. The methods described below train a model using image-level labels, which reduces the labelling effort the most. This is typically the hardest form of WSSS training. The methods could disclosed herein could be adapted to support other, easier types of WSSS training in variant embodiments.

First, we present a novel adversarial model for weakly-supervised semantic segmentation, then we illustrate the effectiveness of the method by showing how end-to-end adversarial erasing can be integrated into a conventional weakly supervised semantic segmentation framework, and providing experimental test results.

<FIG> presents a high-level overview of the exemplary end-to-end adversarial model <NUM>.

The model <NUM> uses two image classifiers: a first image classifier network <NUM>, and a second, adversarial image classifier network <NUM>. Each can be instantiated by any appropriate convolutional neural network (CNN).

The first image classifier network <NUM> receives input images <NUM> and is trained to localize a target object (in this example, a bird) using attention maps <NUM>. It will therefore be referred to herein as a "localizer" network. The attention maps <NUM> are used to create new, modified images <NUM> in which the most discriminative object regions of the input images <NUM> are erased through a pixel-attenuating operation <NUM>. These modified images <NUM> are then forwarded through the second network <NUM>, which will be referred to as the "adversarial" classifier network. Its goal is to classify the images <NUM> correctly, even when the target classes are erased.

The image classifiers of the localizer <NUM> and adversarial <NUM> networks are optimized using binary cross-entropy loss, but in an alternating fashion using distinct optimizers. To force the localizer network <NUM> to not only classify the image <NUM> correctly, but also to spread its attention to less discriminative object regions, the localizer <NUM> is trained with an objective function that includes an adversarial loss term. This term captures the ability of the adversarial network <NUM> to still classify the erased object. The localizer <NUM> is regularized using an additional regularization loss term to prevent it finding a trivial solution in which it identifies the entire image <NUM> as discriminative, so that the entire image is then hidden from the adversarial network <NUM>. This regularization limits the attention of the localizer <NUM> and forces it to erase only regions that belong to the target class.

<FIG> shows an exemplary computer processing system <NUM> on which the model <NUM> may be implemented. It comprises a processor <NUM> (e.g. an Intel™ processor) arranged to executed software stored in a memory <NUM>. The processor <NUM> also uses the memory <NUM> for reading and writing data, such as input data, intermediate calculation results, and output data. Software may control the processor <NUM> to implement any of the methods or steps disclosed herein. In particular, the memory <NUM> may store training software for training the model <NUM>, which may include instructions for providing input to, and receiving output from, the localizer and adversarial networks <NUM>, <NUM>, and for implementing the pixel- attenuating operation <NUM>. The computer <NUM> has input/output peripherals <NUM>-e.g. for receiving training data and/or for exporting data encoding the trained network.

Once a training of the model <NUM> has been completed, the localizer network <NUM> can be used in an inference mode for classifying images. It may be used without the adversarial network <NUM>. The localizer network <NUM> may be replicated and installed on further computer systems, similar to the system <NUM>. It may be installed in devices such as within a control or warning system for an autonomous vehicle. It may be trained to classify objects in a street scene, and may be used in inference mode to classify object in images captures by one or more cameras mounted in a vehicle. However, it is not limited to such applications.

<FIG> shows the end-to-end adversarial erasing framework <NUM> in more detail.

The example training input image x <NUM> in <FIG> is a photo containing both a dog and a cat, and it has therefore been labelled with the image-level class labels y = {cat, dog}. Other training images may contain just a dog or just a cat (or any other target objects) and be labelled accordingly. During training, an input image x is forwarded through the localizer network Gϕ <NUM> to extract a respective attention map Ac <NUM> for each class, c, of target object. Using a soft-thresholding operation, each attention map <NUM> is converted to a respective binary masks Mc <NUM> in which the pixels of the most discriminative object parts for the respective target object (i.e. class) are "<NUM>" and all other pixels are "<NUM>". An image subtraction module <NUM> creates a respective modified image x̃ <NUM> from each mask <NUM>, by erasing or attenuating the most discriminative pixels from the input image x. The modified images <NUM> are forwarded through the adversarial network Fe <NUM>. The localizer <NUM> is optimized using both a classification loss <IMG> and an adversarial loss term <IMG>. These two terms force the localizer <NUM> to spread its attention to less discriminative object parts, while a further regularizing <IMG> loss term encourages the localizer model <NUM> to bound the activation to the minimum necessary area.

The adversarial network Fe <NUM> is optimized using a classification loss <IMG>. The adversarial network Fθ <NUM> and the localizer network Gϕ <NUM> may be trained iteratively, in alternating steps. A common data set is here used to train both networks <NUM>, <NUM>. <MAT> <MAT>.

Localizer network. The localizer <NUM> can be instantiated by any convolutional neural network from which class activation mappings (CAMs) or gradient-weighted class activation mappings (Grad-CAMs) can be extracted. Class activation mappings highlight relevant regions of an image and can be obtained by using a global average pooling layer and a fully connected layer for classification. For simplicity, the usage of CAMs is assumed in the following explanation, but the same approach can be extended to more advanced attention extraction methods such as Grad-CAM.

The localizer G <NUM> has trainable parameters ϕ and is trained as a multi-label classifier using binary cross-entropy loss on each label class, c, for C different classes, to minimize the loss term: <MAT> <MAT> <MAT>.

Ac is then normalized so that the maximum activation equals <NUM>.

Soft masks. Only the attention maps for ground-truth classes are kept (i.e. for classes that are actually present in the input images <NUM>). These attention maps <NUM> are then resized to have the same pixel dimensions as the input image <NUM>, and a soft thresholding operation is applied to generate class-specific masks <NUM> for input images Xi: <MAT> where σ is the sigmoid nonlinearity, ψ is the threshold value, and ω is a scaling parameter that ensures that values above the threshold are close to <NUM> and values below are close to <NUM>. In contrast to a regular thresholding operation, this soft threshold is differentiable which allows the gradients from any further computations to back-propagate to the localizer <NUM>. Adjustable parameters such as ψ may be set by performing a hyperparameter optimization process on a validation set.

Input images x̃i <NUM> for the adversarial network <NUM> are created by the pixel-attenuating module <NUM> as follows:
<MAT>.

Note that only the attention map <NUM> of one particular class is erased in any particular modified image <NUM>, so multiple images <NUM> are created in cases where there is more than one target class, c.

Adversarial network. The adversarial network F <NUM> has trainable parameters θ and is trained as a multi-label classifier using a binary cross-entropy loss function: <MAT>.

This is the same as the classifier loss term <IMG> in Equation (<NUM>) that is used to train the localizer network <NUM>, although applied to the modified images x̃.

Hence, the goal of this adversarial classifier network <NUM> is to classify the same targets as before, despite the erased evidence.

Attention mining loss. To encourage the localizer network <NUM> to erase the target object evidence thoroughly, we engage the localizer network <NUM> in an adversarial game with the adversarial model <NUM>.

The loss function for the localizer <NUM> therefore also includes an adversarial loss term, which encourages it to learn to force the adversarial network <NUM> into misclassifying the modified images. As an adversarial loss term, this embodiment utilizes attention-mining loss, which is the mean of the logits of the classes that have been erased: <MAT>.

Regularization loss. To regularize the localizer <NUM>, to prevent it from learning the trivial solution where it erases the entire image to globally minimize the attention mining loss, we impose an additional regularization loss term: <MAT> where W and H represent the width and height of the activations.

Incorporating this regularization loss term in the optimization process encourages the localizer <NUM> to find a minimum attention map that covers the target class.

Total loss. The total loss function used when training the localizer <NUM> is therefore: <MAT> where α and β are hyper-parameters that can be adjusted to tune the importance of the adversarial and regularization losses respectively-e.g. based on results of empirical testing.

The localizer <NUM> is trained to minimize the adversarial loss term, <IMG>, while the adversarial model <NUM> tries to maximize it, by being trained to minimize the loss function <IMG> in Equation (<NUM>).

Segmentation maps. After the computer system <NUM> trains the model <NUM> with the described loss terms, the localizer network <NUM> can be used in an inference mode by converting attention maps <NUM> to segmentation maps. This may be done by the same computer system <NUM>, or by copying the localizer <NUM> and implementing it on one or more inference systems. The attention maps are up-sampled to the image resolution and stacked together, using C + <NUM> channels. Since the classification model <NUM> is not trained for a background class, the first channel is set to a threshold value ρ. Segmentation masks can then be obtained by taking the argmax over the class dimension. These segmentation masks may be further processed as appropriate-e.g., when used in a vehicle, to detect a potential collision with a pedestrian and to sound an alarm and/or take evasive driving action in response.

The methods described here can be integrated into other systems. This was demonstrated by an integration of the above-described end-to-end adversarial erasing (EADER) scheme into an existing WSSS method known as Pixel-level Semantic Affinity (PSA), e.g. as described in Ahn et al.

PSA consists of three stages. The first stage trains a classification network to generate CAMs. However, this stage suffers from a "discriminative localization" problem in which it learns to focus on only the most discriminative object regions. In this stage there are no specific methods utilized to improve the segmentation masks, but extensive training and test-time data augmentations are utilized. The segmentation masks generated from CAMs are used to generate two sets of labels for the second stage. The first set are labels with high confidence for the background regions, while the second set is with high confidence for the object regions. In the second stage the semantic affinities are learned using Affinity Net, which propagates local responses in the maps to nearby regions which belong to the same semantic entity. The two distinct sets allow reliable training of AffinityNet, as regions of low confidence are discarded. The AffinityNet output predictions are used as pseudo-labels in the last stage, where a fully-supervised semantic segmentation network is trained with it.

The CAM-generation stage of PSA is suitable for adversarial training using the methods disclosed herein as it suffers from discriminative localization, and because the classification network is suitable for use as a localizer network. In particular, it is suitable because the CAMs are generated from the final convolutional layer without the need of any post-processing or other gradient-breaking computations.

In experimental testing, a PSA implementation was used as a localizer network <NUM>, and a soft threshold, as described above, was applied to attention maps <NUM> generated using the PSA to create masks <NUM>, which were used to erase the most discriminative object regions from the input images <NUM>. The resulting images <NUM> were forwarded through an adversarial network <NUM>, as described above, with an attention mining loss <IMG> being applied as an adversarial loss on the PSA localizer network <NUM>.

We evaluated the performance of the methods described herein on the Pascal VOC <NUM> segmentation dataset, which is a widely used benchmark for weakly supervised semantic segmentation. The dataset consists of twenty object classes and one background class and contains <NUM>, <NUM> and <NUM> images in the train, validation and test sets respectively. We augmented the dataset with annotations from Hariharan et al. [<NPL>)], resulting in a total of <NUM> training images. We determined the mean intersection-overunion (mloU) for the validation and test sets. The test set results were obtained using the official Pascal VOC evaluation server.

No post-processing was employed and the method did not employ tricks such as test-time augmentations, post-processing and saliency cues to the method we integrated with.

Network architecture details. We tested the adversarial training approach with a ResNet-<NUM> localizer network, while the adversarial network was a ResNet-<NUM> network. We utilized ImageNet pre-trained weights for the both networks. When integrating with PSA, to ensure fair comparisons, we did not change any of the existing networks, which means the localizer was a WideResNet with <NUM> convolutional layers, while the adversarial model was kept as a ResNet-<NUM>. In the final stage we trained a fully-supervised semantic segmentation network on proxy labels. We utilized the segmentation model DeepLabV3+, with ResNet-<NUM> and Xception-<NUM> backbones and the default training strategy.

Training specifications. We trained the localizer with a batch size of <NUM>, while the batch size was dynamic for the adversarial model as it depends on the number of objects in each image. For example, when each image in the batch of <NUM> has two object categories, both objects are erased from each image separately and the batch size for the adversarial network will be <NUM>. We randomly resized and cropped the input images into <NUM> × <NUM> for both the localizer and the adversarial model. Both networks were optimized for ten epochs with stochastic gradient descent with a learning rate of <NUM>. We alternatively trained the localizer and adversarial per <NUM> training steps.

Throughout the experiments, unless specified otherwise, the values of α and β in Equation (<NUM>) were set to α = <NUM> and β = <NUM>-<NUM>. Further hyperparameter values were ω = <NUM>, ψ = <NUM> in Equation (<NUM>), and ρ = <NUM> as a first-channel threshold. We followed the training settings of Ahn et al. when training PSA and use an initial learning rate of <NUM> for the adversarial network. To generate attention maps we utilized Grad-CAM [<NPL>] in our experiments, while we used CAM [<NPL>] when integrating into PSA, to keep the comparison fair.

Our first experiment was an ablation study to verify that the adversarial network forces the localizer network to spread its attention to less discriminative object regions.

Recall from Equation (<NUM>) that α controls the strength of the adversarial loss term. The following table show the results of varying the α parameter on the Pascal VOC <NUM> validation set, and reporting the mloU, precision and recall of the segmentation masks. We report both the mean and standard deviation over six runs.

<FIG> shows examples of attention maps generated for input images, obtained using Grad-CAMs from a novel end-to-end adversarial erasing (EADER) method described herein, using different values for the adversarial loss term α. The top row shows attention maps for an image of a person riding a horse, for the class "horse". The bottom row shows attention maps for an image of a person stroking a cat, for the class "person", As the α value increases, it can be seen that attention spreads to less discriminative object regions.

We now compare the original PSA results to the results where we have integrated end-to-end adversarial erasing ("EADER") as disclosed herein.

The following table shows the improvements in terms of mloU as a comparison to our baseline, the Pixel-level Semantic Affinity (PSA) method on the Pascal VOC <NUM> validation set. To enable a fair comparison we reproduced the PSA numbers and trained the proxy labels from Affinity Net on DeepLabV3+. The numbers with a † denote our reproduced results.

Additionally, we reported precision and recall after the CAM generation stage.

End-to-end adversarial erasing (EADER) improves performance in this stage for all metrics. In other words, the combination of the adversarial and regularization loss terms forces the attention map to spread to less discriminative object regions without spreading to background areas. Besides, the mloU scores in this stage are higher than those reported in the first table above, which is caused by the extensive test-time augmentations used by PSA.

In the next stage, training Affinity Net with the improved outputs of the first stage again resulted in better mloU scores. Finally, we report results when training a fully-supervised semantic segmentation model on the proxy labels generated by AffinityNet. As we utilized DeepLabV3+ instead of ResNet-<NUM> as a fully-supervised semantic segmentation model, we also report the results of DeepLabV3+ trained on proxy labels without end-to-end adversarial erasing. Again, this results in improvements in terms of mloU, proving the integrability of end-to-end adversarial erasing into existing WSSS methods.

In the following two tables we make a per-class comparison on the validation and test sets.

This table shows a per-class comparison with Pixel-level Semantic Affinity (PSA) on the Pascal VOC <NUM> validation set with only image-level supervision:.

The next table shows a per-class comparison with Pixel-level Semantic Affinity (PSA) on the Pascal VOC <NUM> test set with only image-level supervision:.

Note that in this last table, the more powerful WideResNet-<NUM> was used by the PSA method as the final fully-supervised segmentation model instead of DeepLabV3+.

The discriminative localization problem is especially prevalent in non-rigid object classes. We found that end-to-end adversarial erasing (EADER) significantly improves the results in many non-rigid object classes such as bird, cat, cow and horse. Typically in these object classes the most discriminative object region is the head or the feet, which causes the attention map to cover only a small portion of these object classes. With end-to-end adversarial erasing, the localizer is forced to capture the entire object region, as the fur or skin of these object classes are less discriminative, but still recognizable. For outdoor object classes the results are often similar to PSA, while for indoor object classes the performance is often degraded. Overall, end-to-end adversarial erasing increases the performance, even though it makes use of a less powerful fully-supervised segmentation model.

<FIG> shows some qualitative results on the Pascal VOC <NUM> validation set. The white edges in the ground-truth mask denote pixels that are ignored during evaluation. These images demonstrate the increase in precision, recall and mloU. The first four rows, for input images containing a pair of cats, a bird, a horse and a train, respectively, show that end-to-end adversarial erasing (EADER) better segments objects by capturing less discriminative object regions, including for non-rigid object classes (i.e. correctly identifying more pixels of the target object class than conventional PSA does). The last two rows, for images containing a person and a train, respectively, show increased specificity (i.e. fewer pixels outside the target object that a falsely classified as belonging to the target class), which can be attributed to the regularization term that forces the attention to spread only to areas where the localizer is confident that it is an object region.

the only other adversarial erasing methodology without saliency masks, we significantly outperform their number on the validation set.

The following table compares EADER to known WSSS methods, on the Pascal VOC <NUM> dataset. It shows the feature extractor that is used to generate object locations, the fully supervised model that is trained on proxy labels (where applicable), and the form of supervision. The methods denoted with supervision signal F denote the upper bound of the segmentation performance. Note that our method outperforms a fully supervised FCN network and we achieved ≈ <NUM>% of the upper bound, set by a fully.

Thus it can be seen that the novel end-to-end adversarial erasing (EADER) approach helps to address the "discriminative localization" problem which is an inherent issue in weakly-supervised semantic segmentation methods. This approach is integrable to existing methods, not requiring iterative classifiers, post-processing, weight sharing or saliency masks, unlike many previous adversarial erasing methods. It has been shown to improve performance on the Pascal VOC <NUM> dataset, especially on non-rigid classes, which suffer the most from the discriminative localization problem.

Claim 1:
A method of training a model (<NUM>) for use in semantic image segmentation,
wherein the model comprises:
a first classifier neural network (<NUM>); and
a second classifier neural network (<NUM>),
wherein the method comprises training the first network by:
inputting a first training image (<NUM>) containing a target object to the first network (<NUM>);
using the first network (<NUM>) to identify pixels of the first training image (<NUM>) that are discriminative for the target object;
generating a first modified training image (<NUM>) by attenuating or erasing the identified pixels of the first training image (<NUM>);
inputting the first modified training image (<NUM>) to the second network;
using the second network (<NUM>) to determine a likelihood of the first modified training image (<NUM>) containing the target object; and
updating weights of the first network (<NUM>) using a first loss function that is a monotonically-increasing function of the determined likelihood, and
wherein the method comprises training the second network (<NUM>) by:
inputting a second training image (<NUM>), containing the target object, to the first network (<NUM>);
using the first network (<NUM>) to identify pixels of the second training image (<NUM>) that are discriminative for the target object;
generating a second modified training image (<NUM>) by attenuating or erasing the identified pixels of the second training image;
inputting a set of training images to the second network (<NUM>), the set comprising the second modified training image (<NUM>) and one or more training images (<NUM>) that do not contain the target object; and
for each training image (<NUM>) of the set, using the second network (<NUM>) to determine a likelihood of the respective training image (<NUM>) containing the target object, and updating weights of the second network (<NUM>) using a second loss function, different from the first loss function, wherein the second loss function is a monotonically-decreasing function of the determined likelihood when the training image (<NUM>) contains the target object, and is a monotonically-increasing function of the determined likelihood when the training image does not contain the target object.