Systems and methods for semantic segmentation

Fully-supervised semantic segmentation machine learning models are augmented by ancillary machine learning models which generate high-detail predictions from low-detail, weakly-supervised data. The combined model can be trained over both fully- and weakly-supervised data. Only the primary model is required for inference, post-training. The combined model can be made self-correcting during training by adjusting the ancillary model's output based on parameters learned over both the fully- and weakly-supervised data. The self-correction module may combine the output of the primary and ancillary models in various ways, including through linear combinations and via neural networks. The self-correction module and ancillary model may benefit from disclosed pre-training techniques.

FIELD

This disclosure generally relates to machine learning techniques, and in particular to semantic segmentation of datasets using machine learning models.

BACKGROUND

Semantic segmentation is a family of techniques in computing involving classifying elements of items in a dataset. It is most commonly encountered as semantic image segmentation, a computer vision problem, where the task may be described as predicting object classes for each pixel in an image. Semantic segmentation has applications in medical imaging, self-driving cars, and other domains. Specialized forms of semantic segmentation include object instance segmentation, where elements are associated not only with a class but with a particular instance of a class; for example, for an image containing two cats, the pixels comprising each cat may be labelled with the class “cat” and (in object instance segmentation) may also be labelled “instance 1” or “instance 2” (or some other suitable label), depending on which cat the pixels belong to. The set of vectors for a given set of pixels (e.g. a whole image) is called a “mask” and is generally the output of a semantic image segmentation model.

A problem that arises with semantic segmentation is that it usually requires large quantities of carefully-labelled training data. For example, some proposed semantic image segmentation techniques involve training a convolutional neural network to receive images and output vectors of category probabilities for each pixel (e.g. cat=20%, tree=85%, . . . ). Training is usually fully-supervised, requiring each image in the training dataset to be labelled, usually as a polygon. Some have estimated that this requires 78 seconds per image on average for a human to label one instance of one class. This is quite laborious, particularly when one considers that even a small training dataset is likely to contain tens of thousands of images and encompass multiple classes.

One the training dataset has been obtained, perhaps at great cost, the convolutional neural network is usually trained based on a cross-entropy loss term, such as:

maxθ⁢∑(x,y)⁢log⁢⁢p⁡(y|x,θ)
where x is the input data, y is the ground truth label (i.e. the labels provided with the training dataset), θ represents the parameters of the neural network, and p(y|x,θ) is the probability that the model currently (under the then-current values of parameters θ) will yield the ground-truth label y for a given item of input data x.

Some have experimented with weakly-supervised training of semantic segmentation models. For example, Khoreva et al.,Simple does it: Weakly supervised instance and semantic segmentation,In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, propose estimating segmentation masks based on various techniques (e.g. finding the overlap between labels generated by MCG and GrabCut+) which use bounding boxes instead of fully-supervised (e.g. polygonal) labels as input. Bounding boxes are less laborious to prepare (some estimates put the cost at around 10 seconds per instance per class per image). Others suggest alternative sets of hand-crafted rules for generating segmentation masks from “weakly supervised” data such as bounding boxes; results vary between proposals and between circumstances depending on how well the authors' intuitions match the ground truth.

There is thus a general desire for systems and methods for semantic segmentation which improve the quality of results, the time taken to train, and/or the cost of obtaining suitable training data in at least some circumstances.

BRIEF SUMMARY

Aspects of the present disclosure provide systems and methods for instantiating a machine learning system for generating semantic segmentation masks over an input dataset based on a fully-supervised dataset having high-detail labels and a weakly-supervised dataset having low-detail labels. The method is executed by at least one processor in communication with at least one memory and comprises: instantiating a primary model in the at least one memory, the primary model operable to generate a high-detail prediction based on one or more parameters of the primary model and a first item from any of the input, fully-supervised, and weakly-supervised datasets; instantiating an ancillary model in the at least one memory, the ancillary model operable to generate a high-detail prediction based on one or more parameters of the ancillary model, a second item, and a low-detail label corresponding to the second item; training the one or more parameters of the ancillary model based on the fully-supervised dataset independently of the primary model; training the one or more parameters of the primary model based on the fully-supervised and weakly-supervised datasets based on one or more predictions of the ancillary model over the weakly-supervised dataset.

In some implementations, training the one or more parameters of the primary model comprises holding the one or more parameters of the ancillary model fixed while training the primary model.

In some implementations, instantiating the ancillary model comprises instantiating an encoder-decoder segmentation model comprising an encoder and a decoder and instantiating the ancillary model further comprises instantiating a mask encoder operable to transform the low-detail label into one or more representations corresponding to output of the encoder.

In some implementations, training the one or more parameters of the ancillary model comprises combining the one or more representations of the low-detail label with one or more encoded values output by the encoder to generate a combined representation and passing the combined representation to the decoder in place of the one or more encoded values.

In some implementations, training the one or more parameters of the primary model comprises determining a value of an objective function based on a first probability of a first label being generated by the primary model and further based on a second probability of the first label being generated by the ancillary model.

In some implementations, determining the value of the objective function based on the second probability comprises scaling a first term based on the first probability by a second term based on the second probability.

In some implementations, instantiating a self-correction module in the at least one memory, the self-correction module operable to generate a high-detail prediction based on a first prediction of the primary model and a second prediction of the ancillary model. In some implementations, the self-correction module is operable to generate the high-detail prediction based on a linear combination of a first distribution induced by the primary model and a second distribution induced by the ancillary model and training the one or more parameters of the primary model comprises optimizing an objective function based on the linear combination of the first and second distributions.

In some implementations, the linear combination comprises a geometric mean of the first and second distributions and training the one or more parameters of the primary model comprises optimizing an objective function based on the geometric mean of the first and second distributions.

In some implementations, training the one or more parameters of the primary model comprises determining a value of an objective function based on a first probability of a first label being generated by the primary model and further based on a second probability of the first label being generated by the ancillary model. In some implementations, determining the value of the objective function based on the second probability comprises scaling a first term based on the first probability by a second term based on the second probability. In some implementations, determining a value of an objective function based on the second probability comprises determining the second term based on the second probability of the first label being generated by the self-correction module.

In some implementations, the self-correction module is operable to generate the high-detail prediction based on a neural network having one or more parameters, the neural network operable to receive a first prediction of the primary model and a second prediction of the ancillary model as input, the method further comprising training the one or more parameters of the self-correction module based on the fully-supervised dataset. In some implementations, training the one or more parameters of the ancillary model comprises training the one or more parameters of the ancillary model over a first subset of the fully-supervised dataset; and training the one or more parameters of the self-correction module comprises pre-training the one or more parameters of the self-correction module over a second subset of the fully-supervised dataset containing one or more items not in the first subset, said pre-training done independently of the weakly-supervised dataset. In some implementations, training the one or more parameters of each of the self-correction module and primary model comprises training the self-correction module and primary model together over at least a portion of the fully-supervised dataset and at least a portion of the weakly-supervised dataset after pre-training the self-correction module.

DETAILED DESCRIPTION

Introductory Generalities

Computing Systems

FIG. 1illustrates a computing system100comprising a digital computer102. The example digital computer102includes one or more digital processors106that may be used to perform classical digital processing tasks. Digital computer102may further include at least one system memory108, and at least one system bus110that couples various system components, including system memory108to digital processor(s)106. System memory108may store a machine learning instructions module112.

The digital processor(s)106may be any logic processing unit or circuitry (e.g., integrated circuits), such as one or more central processing units (“CPUs”), graphics processing units (“GPUs”), digital signal processors (“DSPs”), application-specific integrated circuits (“ASICs”), programmable gate arrays (“FPGAs”), programmable logic controllers (“PLCs”), etc., and/or combinations of the same.

In some implementations, computing system100comprises an analog computer104, which may include one or more quantum processors114. Digital computer102may communicate with analog computer104via, for instance, a controller126. Certain computations may be performed by analog computer104at the instruction of digital computer102, as described in greater detail herein.

Digital computer102may include a user input/output subsystem116. In some implementations, the user input/output subsystem includes one or more user input/output components such as a display118, mouse120, and/or keyboard122.

System bus110can employ any known bus structures or architectures, including a memory bus with a memory controller, a peripheral bus, and a local bus. System memory108may include non-volatile memory, such as read-only memory (“ROM”), static random access memory (“SRAM”), Flash NAND; and volatile memory such as random access memory (“RAM”) (not shown).

Digital computer102may also include other non-transitory computer- or processor-readable storage media or non-volatile memory124. Non-volatile memory124may take a variety of forms, including: a hard disk drive for reading from and writing to a hard disk (e.g., magnetic disk), an optical disk drive for reading from and writing to removable optical disks, and/or a solid state drive (SSD) for reading from and writing to solid state media (e.g., NAND-based Flash memory). The optical disk can be a CD-ROM or DVD, while the magnetic disk can be a rigid spinning magnetic disk or a magnetic floppy disk or diskette. Non-volatile memory124may communicate with digital processor(s) via system bus110and may include appropriate interfaces or controllers126coupled to system bus110. Non-volatile memory124may serve as long-term storage for processor- or computer-readable instructions, data structures, or other data (sometimes called program modules) for digital computer102.

Although digital computer102has been described as employing hard disks, optical disks and/or solid state storage media, those skilled in the relevant art will appreciate that other types of nontransitory and non-volatile computer-readable media may be employed, such magnetic cassettes, flash memory cards, Flash, ROMs, smart cards, etc. Those skilled in the relevant art will appreciate that some computer architectures employ nontransitory volatile memory and nontransitory non-volatile memory. For example, data in volatile memory can be cached to non-volatile memory. Or a solid-state disk that employs integrated circuits to provide non-volatile memory.

Various processor- or computer-readable instructions, data structures, or other data can be stored in system memory108. For example, system memory108may store instruction for communicating with remote clients and scheduling use of resources including resources on the digital computer102and analog computer104. Also for example, system memory108may store at least one of processor executable instructions or data that, when executed by at least one processor, causes the at least one processor to execute the various algorithms described elsewhere herein, including machine learning related algorithms. For instance, system memory108may store a machine learning instructions module112that includes processor- or computer-readable instructions to provide a machine learning model. Such provision may comprise training and/or performing inference with a convolutional neural network and/or other machine learning model, e.g., as described in greater detail herein.

Aspects of the present disclosure provide a semantic segmentation model having a primary segmentation model and an ancillary segmentation model. (For convenience, these are generally referred to herein as the primary model and the ancillary model.) The primary model may be structured and used for inference according to existing knowledge and/or as described herein. The ancillary model assists with training the primary model in a weakly-supervised capacity. Much of the present disclosure relates to the structure of the ancillary model and methods for training the primary and ancillary models.

FIG. 2Ashows schematically an example semantic segmentation model200a.Either or both of two datasets may be available: a fully-supervised dataset202having items206and corresponding high-detail segmentation masks216and a weakly-supervised dataset204having items208and corresponding low-detail segmentation masks218. Where items206,208are images, high-detail segmentation masks216may comprise polygonal masks of class instances (e.g. people and planes, as shown in the exemplaryFIG. 2) and low-detail segmentation masks218may comprise bounding boxes which relatively coarsely correspond to class instances. (The particular example shown inFIG. 2has a first bounding box containing the area in which a person is found and a second bounding box containing the area in which a horse is found—although the bounding boxes also contain substantial areas which do not contain people or horses due to their low detail.)

Primary model232receives items206and/or208as input and outputs predictions240. It is possible to train primary model232in a fully-supervised regime by ingesting items206, generating predictions240(e.g. by processing items206via a convolutional neural network or other suitable machine learning models), and updating parameters via training module238based on ground-truth segmentation masks216(e.g. by optimizing a cross-entropy loss term over the parameters of primary model232), as described above. Primary model may comprise, for example, an encoder-decoder-based deep network (such as, e.g., a DeepLabv3+ model), although other segmentation models may be used instead or in addition. Primary model232can be represented as a distribution p defined over the space of all possible predictions240conditioned on its inputs and its parameters ϕ). This can be written in terms of pointwise probabilities as p(y|x,ϕ), where y is a prediction for a given input x.

Ancillary model234receives items206,208as input, along with corresponding low-detail segmentation masks218(which, in the case of items206, may be generated from high-detail segmentation masks216and/or otherwise obtained), and outputs a high-detail segmentation mask226. High-detail segmentation masks226can be used to assist in training of primary model232. In at least some implementations, ancillary model234and its generated high-detail segmentation masks226are not required during inference, allowing primary model232to be used for inference according to conventional (or any other suitable) techniques. Ancillary model234can be represented as a distribution pancdefined over the space of all possible predictions conditioned on its inputs (items208and low-detail segmentation masks204) and its parameters θ, which may be disjoint from or fully or partially shared with primary model232.

In some implementations, ancillary model234comprises an encoder-decoder segmentation model (which may, e.g., correspond structurally to primary model232) with an additional mask encoder to assist in processing low-detail segmentation masks208.FIG. 3shows an example mask encoder302. The example ancillary model ofFIG. 3comprises a segmentation encoder304and segmentation decoder306. Encoder304and decoder306may provide layers which produce representations of varying size, such as high-detail layers312and318and low-detail layers314and316. Information may flow some or all of the layers of encoder304before being passed to decoder306. For example, information may be passed from high-detail layers312to low-detail layers314in encoder304(thus downsampling input item204) before being passed to decoder306. Decoder306may then pass that information from low-detail layers316to high-detail layers318(thus upsampling the output of encoder304) to produce output segmentation mask226. Alternatively, or in addition, information may be passed between layers of similar size even if not fully processed by the encoder—for example, high-detail layers312of encoder304may pass information to high-detail layers316of decoder306.

Mask encoder302processes input low-detail segmentation mask208to one or more intermediate forms which may be combined with the output of encoder304. The result(s) of that combination are decoded by decoder306. Where encoder304produces multiple forms of output (e.g. high-detail output at layers312and low-detail output at layers314), mask encoder302may produce corresponding representations for one or more of those forms of output, e.g. by resizing its output to correspond to the size of encoder304's output. The output of mask encoder302and304may be combined in any suitable way, e.g. via elementwise multiplication.

In some implementations, mask encoder302is parametrized via a subset of the ancillary model's parameters θ. Encoder302may process an input low-detail segmentation mask208based on those parameters, e.g. by passing it through a convolution layer with sigmoid activation or via other suitable techniques. Mask encoder302may generate, for example, an attention map based on input low-detail segmentation mask208.

Returning toFIG. 2A, ancillary model234may be trained independently of primary model232based on a corresponding objective function (which may differ from an objective function used to train primary model232). For example, ancillary model234may be trained by training module238based on a cross-entropy loss using fully-supervised dataset202, e.g. based on:

maxθ⁢∑(x,y)∈ℱ⁢log⁢⁢panc⁡(y|x,b,θ)
wheredenotes fully-supervised dataset202, each (x, y) pair denotes an item (x) and its corresponding high-detail segmentation mask (y), and b denotes the low-detail segmentation masks received by ancillary model234. Low-detail segmentation masks may be obtained by, for example, generating them from the high-detail segmentation masks216already in. This may involve, for example, determining the appropriately-shaped (e.g. rectangular) closure of each mask layer to generate a corresponding bounding box.

Ancillary model234is used by training module238to assist in training primary model232. In some implementations, the parameters of ancillary model234are fixed while training primary model232. Primary model232may be trained over either or both datasets202and204. In some implementations, primary model232is trained over both datasets202and204in minibatches, with items from each dataset202and204present in each minibatch. Primary model238may be trained by optimizing an objective function based on a fully-supervised objective term defined over high-detail dataset202() and a weakly-supervised objective term defined over low-detail dataset204(W). In some implementations, the weakly-supervised objective term has a form corresponding to that of the fully-supervised objective term with a further scaling term based on the predictions226of ancillary model234.

For example, the objective function may be determined based on:

maxϕ⁢∑(x,y)∈ℱ⁢log⁢⁢p⁡(y|x,ϕ)+∑(x,y,b)∈𝒲⁢∑y⁢panc⁡(y|x,b,θ)⁢log⁢⁢p⁡(y|x,ϕ)
where the first term is a conventional cross-entropy term (and serves as the fully-supervised term defined over) and the second term is a cross-entropy term scaled by a probabilistic label generated by ancillary model234(i.e. panc(y|x,b,θ)). Note that in this example formulation θ (the parameters of ancillary model234) are fixed. Scaling the contribution of the output of primary model232over weakly-supervised dataset204based on predictions226of ancillary model234over the same items tends to draw primary model232's behavior toward the output of ancillary model234, thereby allowing primary model238to be trained over W despite the lack of ground-truth, high-detail segmentation masks216in weakly-supervised dataset204.

In some implementations, the contribution of the weakly-supervised term is scaled to adjust the degree to which primary model232relies on weakly-supervised dataset204in training. For example, the second term may be multiplied by a scaling factor α; setting α<1 will tend to reduce the effect of weakly-supervised dataset204(and this prioritize fully-supervised dataset202).

FIG. 5Ashows schematically a flowchart of example method500afor training model200a.At502ancillary model234is trained over all or part of fully-supervised dataset202. In at least some implementations, this is done independently of training primary model232. At506primary model232is trained over both the fully- and weakly-supervised datasets202,204(or portions thereof) based on ancillary model234's predictions, as described above. At508primary model506may be used for inference (this is technically a post-training step but is shown to assist the reader). As noted above, in at least some implementations ancillary model234is not required for inference.

Semantic Segmentation with Self-Correction

The foregoing example model200ahas been found experimentally to yield promising results in at least some circumstances where ancillary model234is trained exclusively on fully-supervised dataand its output is used directly to influence the training of primary model232over weakly-supervised data W. In some implementations, the output of primary model232(trained overand W) is mixed with the output of ancillary model234(trained over) by a self-correction module236to generate a prediction226which is based not only on learning frombut also from W.

FIG. 2Bshows an example model200bhaving an example self-correction module236. It receives output from both primary model232and ancillary model234and mixes them to generate prediction226. For example, self-correction module may induce a distribution q(y|x,b) over labels that tends to be close to both distributions p(y|x,ϕ) and panc(y|x,b,θ) of the primary and ancillary models232and234, respectively. Distribution q may have its own parameters λ or be parameter-free (other than, optionally, an implicit parametrization by ϕ and/or θ due to dependence on models232,234). Self-correction module236may come in any of several forms, including linear, convolutional, and/or otherwise.

In some implementations, self-correction module236comprises a linear combination of the output of primary and ancillary models232and234. Such a linear combination may be parameter-free. For example, training module238may recast training primary model232as training the distribution q of the self-correction module236. (It is equivalent to think of this as training primary model232based on an objective function which includes a transformation of its output induced by q—in either event, training of primary model232by training module238is based on q.) For instance, training module238may train primary model232based on a KL-divergence between q and p and also on a KL-divergence between q and panc, e.g. as follows:

In some implementations, distribution q is determined based on a mean of the primary and ancillary models'232,234distributions p, panc. For example, q may comprise an arithmetic and/or geometric mean. For instance, q may be determined based on:

In some implementations, such as those where panc(y|x,b) and p(y|x) are both factorial (e.g. where they decompose to the product of probabilities over the components of y) and distributions over components are categorical, q is factorial and may be determined by (for example) applying a softmax activation to the linear combination of logits coming from primary and ancillary models232,234. For example, q may be determined based on:

q⁡(ym=1|x,b)=σ⁡(lm+α⁢⁢lmancα+1)
where σ is the softmax function, lm:=log p(ym=1|x) and lmanc:=log panc(ym=1|x,b) are logits generated by the primary and ancillary models232,234respectively, and ymdenotes the mthelement of item y (e.g. the mthpixel). Distribution q may be determined in other suitable ways depending on the structure of the underlying distributions p and panc, the structure chosen for q (e.g. the form of mean selected), and/or other factors.

Training module238may train primary model232based on distribution q of self-correction module236by, for example, using q in place of panc. Referring back to an earlier example, a resulting example objective function may be based on:

In some implementations, self-correction module236comprises a neural network, such as a convolutional neural network, to learn q. This alleviates the need to select a suitable range of α values (in implementations where α is required), which may require a hyperparameter search. The network may be large, but this is not required; in some implementations, the network of self-correction module236is relatively small, such as the example network400ofFIG. 4.

Network400receives logits402from primary model232and logits404from ancillary model234. It combines these inputs at combiner410, e.g. via concatenation, to generate combined logits406. Combined logits406are then processed by convolutional layers420. There may be any number of these; in at least one embodiment, it comprises two convolution layers422,424each comprising a 3×3 kernel and a ReLU activation function. (Activation functions are not required and kernels may vary in size—e.g. a 1×1 kernel may be used.) In some embodiments, layer424has a number of output feature maps corresponding to the number of classes in the dataset and layer422has a fixed number (e.g. 128) which may differ from that of layer424. Prediction408is obtained based on the output of layers420; e.g. that output may be used directly as prediction408, and/or it may optionally be received at a sigmoid module and processed to generate prediction408.

Self-correction module236is parametrized in such embodiments by parameters λ (which may comprise, e.g., the parameters of network400). This network may be trained independently of primary model232and/or alongside primary model232. In some implementations, self-correction module236is trained alongside primary model232by using q in place of ancillary model234's pancin the objective function and by adding a term to train parameters λ over fully-supervised dataset202. For example, training module238may train model200bby optimizing an objective function based on:

In some implementations the parameters of self-correction module236are randomly initialized. As a result, it may be inaccurate early in training when predicting labels for items in W. In some such implementations, ancillary model234is trained over a subset of itemsin fully-supervised dataset202, with the remaining items being retained for later training. Self-correction module236is then pre-trained over fully-supervised dataset202, including items not in(e.g. module236may be trained over all of). This pre-training may be done via the first and last terms of the above objective function, i.e. omitting the terms over W. In some implementations, both ϕ (the parameters of primary model232) and λ are pre-trained at this stage. The final (or main) stage of training may then proceed over all training data (i.e. all ofand W) using all terms of the objective function; this “fine-tunes” the whole model.

FIG. 5Bshows schematically a flowchart of example method500bfor training model200bfor implementations where self-correction module234is provided and pre-training is used. At502ancillary model234is trained over a portion () of fully-supervised dataset202. In at least some implementations, this is done independently of training primary model232. At504self-correction module236is pre-trained as described above. This pre-training may be independent of any weakly-supervised data so at to limit interaction between parameters. At506primary model232is trained over both the fully- and weakly-supervised datasets202,204(or portions thereof) based on ancillary model234's predictions, as corrected by self-correction module234, as described above. At508primary model506may be used for inference. As noted above, in at least some implementations ancillary model234and self-correction module236are not required for inference.

Implementations of example model200bhave been tested on certain widely-available datasets and have achieved results which exceed that of the state of the art, indicating that in at least some circumstances the presently-disclosed systems and methods provide a machine learning model which is more powerful and/or relies less heavily on costly fully-supervised data than at least some existing techniques.

Concluding Generalities

The above described method(s), process(es), or technique(s) could be implemented by a series of processor readable instructions stored on one or more nontransitory processor-readable media. Some examples of the above described method(s), process(es), or technique(s) method are performed in part by a specialized device such as an adiabatic quantum computer or a quantum annealer or a system to program or otherwise control operation of an adiabatic quantum computer or a quantum annealer, for instance a computer that includes at least one digital processor. The above described method(s), process(es), or technique(s) may include various acts, though those of skill in the art will appreciate that in alternative examples certain acts may be omitted and/or additional acts may be added. Those of skill in the art will appreciate that the illustrated order of the acts is shown for exemplary purposes only and may change in alternative examples. Some of the exemplary acts or operations of the above described method(s), process(es), or technique(s) are performed iteratively. Some acts of the above described method(s), process(es), or technique(s) can be performed during each iteration, after a plurality of iterations, or at the end of all the iterations.

The above description of illustrated implementations, including what is described in the Abstract, is not intended to be exhaustive or to limit the implementations to the precise forms disclosed. Although specific implementations of and examples are described herein for illustrative purposes, various equivalent modifications can be made without departing from the spirit and scope of the disclosure, as will be recognized by those skilled in the relevant art. The teachings provided herein of the various implementations can be applied to other methods of quantum computation, not necessarily the exemplary methods for quantum computation generally described above.

The various implementations described above can be combined to provide further implementations. All of the commonly assigned US patent application publications, US patent applications, foreign patents, and foreign patent applications referred to in this specification and/or listed in the Application Data Sheet, including but not limited to U.S. Provisional Application No. 62/768,020, are incorporated herein by reference, in their entirety.