Weakly-supervised spatial context networks to recognize features within an image

Systems, methods and articles of manufacture for training a convolutional neural network for feature recognition within digital images. A spatial context neural network is trained using a plurality of patches cropped from a plurality of digital images, the spatial context neural network comprising a first convolutional neural network configured to predict a feature representation for a first specified portion of a first digital image, a second convolutional neural network configured to compute a feature representation for a second specified portion of a second digital image, and a spatial context module that accepts output of the first and second convolutional neural networks as input. The second convolutional neural network is refined by regressing features of the second convolutional neural network to features of the first convolutional neural network. The refined second convolutional neural network is used to recognize one or more features within a third digital image.

BACKGROUND

Field of the Invention

The present disclosure relates to digital image processing, and more specifically, to improved techniques for recognizing features within digital images using a spatial context neural network.

Description of the Related Art

Recent advances in object categorization, detection and segmentation have been fueled by high capacity deep learning models from large, labeled data sets. However, the large-scale human supervision that is required for many of these methods limits their use for some applications. This is particularly the case with fine-grained object-level tasks such as detection or segmentation, where the annotation requirements can be become so costly and unwieldy as to render the methods impractical. One solution is to use a pre-trained model for other, potentially unrelated, image tasks, and while such pre-trained models can produce effective and highly generic feature representations, fine-tuning with task-specific labeled samples is often necessary. Moreover, while unsupervised learning techniques can be used to potentially address some of these challenges, unsupervised models have not produced representations that can rival pre-trained models, much less surpass them.

SUMMARY

One embodiment provides a method and system that include training a spatial context neural network using a plurality of patches cropped from a plurality of digital images. The spatial context neural network includes a first convolutional neural network configured to predict a feature representation for a first specified portion of a first digital image, a second convolutional neural network configured to compute a feature representation for a second specified portion of a second digital image, and a spatial context module that accepts output of the first and second convolutional neural networks as input. The method and system include refining the second convolutional neural network, by regressing features of the second convolutional neural network to features of the first convolutional neural network. Additionally, the method and system include using the refined second convolutional neural network to recognize one or more features within a third digital image.

Another embodiment described herein provides a non-transitory computer-readable memory containing computer program code that, when executed by operation of the one or more computer processors, performs an operation. The operation includes retrieving a first convolutional neural network configured to predict a feature representation for a first patch of a first digital image. The operation also includes refining the first convolutional neural network by regressing features of the first convolutional neural network to features of a second convolutional neural network, wherein the second convolutional neural network is configured to predict a feature representation for a second patch of the first digital image, based on an offset between the first patch and the second patch. Additionally, the operation includes using the refined second convolutional neural network to recognize one or more features within a second digital image.

DETAILED DESCRIPTION

Recent successful advances in object categorization, detection and segmentation have been fueled by high capacity deep learning models (e.g., convolutional neural networks (CNNs)) learned from massive labeled corpora of data (e.g., ImageNet, COCO, etc.). However, the large-scale human supervision that makes these methods effective at the same time can also limit their use. This is especially true for fine-grained object-level tasks such as detection or segmentation, where annotation efforts become costly and unwieldy at scale. One solution is to use a pre-trained model (e.g., VGG 19 trained on ImageNet) for other, potentially unrelated, image tasks. Such pre-trained models can produce effective and highly generic feature representations. However, when using such pre-trained models, fine-tuning with task-specific labeled samples is often necessary. Unsupervised learning is one way to potentially address some of these challenges. Unfortunately, despite significant research efforts unsupervised models such as auto-encoders and, more recently, context encoders have not produced representations that can rival pre-trained models, much less surpass them. Among the biggest challenges is how to encourage a representation that captures semantic-level (e.g., object-level) information without having access to explicit annotations for object extent or class labels.

Generally, embodiments provided herein provide techniques for training and using a convolutional neural network for feature recognition within digital images. One embodiment includes training a spatial context neural network using a plurality of patches cropped from a plurality of digital images. The spatial context neural network can include a first convolutional neural network configured to predict a feature representation for a first specified portion of a first digital image. Additionally, the spatial context neural network can include a second convolutional neural network configured to compute a feature representation for a second specified portion of a second digital image, as well as a spatial context module that accepts output of the first and second convolutional neural networks as input. Embodiments can refine the second convolutional neural network, by regressing features of the second convolutional neural network to features of the first convolutional neural network. The refined second convolutional neural network can then be used to extract one or more features within a first digital image, and then extracted features can then be used for other tasks on a new set of images (e.g., recognizing objects depicted within each of the new set of images).

One embodiment described herein incorporates image tokenization into training the spatial context neural network. In a particular embodiment, the image tokenization is performed at the level of objects. Generally, by working with patches at object scale, the spatial context neural network can focus on more object-centric features and potentially ignore some of the texture and color details that are likely less important for semantic tasks. That is, instead of looking at immediate regions around the patch for context and encoding the relationship between the contextual and target regions implicitly, embodiments described herein can consider potentially non-overlapping patches with longer spatial contextual dependencies and explicitly condition the predicted representation on the relative spatial offset between the two regions.

In addition, when training the spatial context neural network, embodiments can make use of a pre-trained model to extract intermediate representations. Since lower levels of CNNs have been shown to be task independent, doing so can allow for a better representation to be learned. More specifically, embodiments described herein provide a spatial context network (SCN), which is built on top of existing CNN networks and is designed to predict a representation of one (object-like) image patch from another (object-like) image patch, conditioned on the relative spatial offset between the two image patches. As a result, the spatial context neural network can learn a spatially conditioned contextual representation of the image patches. In other words, given the same input patch and different spatial offsets, embodiments described herein can learn to predict different contextual representations. For example, given a patch depicting a side-view of a car and a horizontal offset, the spatial context neural network may output a patch representation of another car (e.g., as cars may typically appear horizontally adjacent to one another on images within the training data set). However, as another example, the same input patch with a vertical offset may result in a patch representation of a plane (e.g., as when a car and a plane appear together within a single image within the images in the training data set, the plane may consistently appear vertically above the car).

Embodiments described herein can use of ImageNet pre-trained model as both initialization and to define intermediate representations. Once the spatial context neural network is trained (e.g., on pairs of patches), embodiments can use one of the two streams as an image representation that can be used for a variety of tasks, including object categorization or localization (e.g., as part of Faster R-CNN). Doing so provides an improved data model for feature recognition that is trained using spatial context as an effective supervisory signal. In other words, embodiments described herein provide a spatial context network that uses offset between two patches as a form of contextual supervision. According to various embodiments described herein, a variety of tokenization schemes can be used for mining training patch pairs.

FIG. 1is a block diagram illustrating a workflow for generating a spatial context network, according to one embodiment described herein. As shown, the workflow100includes a plurality of digital images110, which are analyzed by the feature identification component120to extract the plurality of patches115. Generally, each patch represents a respective portion of one or the digital images110of a fixed size. For example, a first patch of the plurality of patches115for a digital image I could be represented as XiI. The first patch could be represented by a patch bounding box biIas an eight-tuple consisting of (x,y) positions of top-left, top-right, bottom-left and bottom-right corners of the bounding box.

Generally, embodiments described herein can be configured to operate with (category-independent) object proposals as a way to tokenize an image into more semantically meaningful parts. Embodiments can make predictions when recognizing features within digital images, where the predictions are on spatial category-independent object proposals (rather than, e.g., frames offset in time). Further, the spatial context network130can be parametrized by the real-valued offset between pairs of proposals, providing finer levels of granularity relative to conventional solutions.

Generally, the feature identification component120is configured to train the spatial context network130using the plurality of patches115. As shown, the spatial context network130includes a first convolutional neural network140, a second convolutional neural network140and a spatial context module160. An example is shown inFIG. 2, where the flow diagram200illustrates a method for training a convolutional neural network for feature recognition, according to one embodiment described herein. As shown, a patch210is extracted from a digital image and is processed as an input into the convolutional neural network140. As discussed above, the convolutional neural network140can be configured to compute a feature representation of the patch210. The feature representation is output from the convolutional neural network140into the spatial context network160, which also accepts an offset value220. For example, the offset oijIcould be determined according to oijI=biI−bjI. Generally, the offset represents the relative offset between two patches (e.g., patches240and210) and is computed by subtracting locations of their respective four corners. The spatial context network130could be trained using a plurality of training samples represented as 3-tuples XiI,XjI,oijI). The convolutional neural network140could be fine-tuned to predict feature representations of the patch240, for a given feature in patch210, based on the spatial context module160and the convolutional neural network150. For example, the convolutional neural network140could be trained such that the model generally predicts a patch with a substantially horizontal offset relative to patch210could likely contain another car, while a patch with a substantially vertical offset above the patch210could likely contain a plane. Advantageously, refining the convolutional neural network140using spatial object predictions results in more accurate feature prediction by the convolutional neural network140. The convolutional neural network140could then be extracted and used independently, e.g., for feature recognition analysis for digital images.

FIG. 3illustrates a spatial context neural network, according to one embodiment described herein. As shown, the network300includes a top data stream for the convolutional neural network150and a bottom stream for the convolutional neural network140. Generally, the goal of the top stream is to provide a feature representation for patch XiIthat will be used as soft target for contextual prediction by the learned representation of the patch XjI. In one embodiment, the top stream comprises an ImageNet pre-trained CNN such as VGG 19, GoogleNet or ResNet. However, such examples are provided for illustrative purposes only and more generally any pre-trained CNN model can be used, consistent with the functionality described herein. In a particular embodiment, the output of the top stream is the representation from the fully-connected layer (fc7) obtained by propagating patch XiIthrough the original pre-trained ImageNet model. In one embodiment, the softmax layer is removed from the pre-trained model, prior to obtaining the representation.

More formally, g(XiI; WT) can denote the non-linear function approximated by the CNN model and parameterized by weights WT. In one embodiment, the fc7 layer within the model is used for its relative simplicity and performance in many high-level visual tasks. More generally, however, according to various embodiments other layers within the model can be utilized, consistent with the functionality described herein.

The bottom data stream for the convolutional neural network140can represent an identical CNN model to the top stream. As shown, the output of the bottom stream feeds into the spatial context module160, along with an offset310between the depicted patches within the digital image. The spatial context module160can then account for spatial offset between the input pair of patches. For example, the convolutional neural network140could map the input patch to a feature representation h1=g(XjI; WB), and then the resulting h1(fc7 representation) can be used as input for the spatial context module160. In one embodiment, the bottom stream is initialized using the ImageNet pre-trained model, so initially WB=WT.

The spatial context module160is generally configured to take the feature representation of the patch XjIproduced by the bottom stream and, given the offset to patch XiI, predict the representation of patch XiIthat would be produced by the top stream. In one embodiment, the spatial context module is represented by a non-linear f([h1,oijI]; V) that is parameterized by weight matrix V={V1, Vloc, V2}. In a particular embodiment, the spatial context module160takes the feature vector h1that is computed from patch XjI, together with an offset vector oijbetween XjIand XiI, and derives an encoded representation h2=σ(V1h1+Vlocoij). Here, V1denotes a weight for h1, Vloc, represents a weight matrix for an input offset, and σ(x)=1/(1+e−x). The spatial context module160can then map h2to h3, using a linear transformation, to reconstruct a fully-connected layer feature vector computed by the first convolutional neural network140on the patch XiI.

Given the output feature representations from the aforementioned two streams, embodiments can train the network by regressing the features from the bottom stream to those from the top stream. In one embodiment, a squared loss function is used. In a particular embodiment, the squared loss function is defined asV,WBminΣI∈I;i≠j∥g(XiI;WT)−f([g(XjI;WB),oij]; V)∥2. As a result, the spatial context modeul160can essentially provide an encoder-decoder framework, with the bottom stream encoding the input image patch into a fixed representation and the spatial context module160decoding the fixed representation into a representation of another, spatially offset, patch.

Advantageously, the spatial context network130can be trained using backpropagation with stochastic gradient descent. Note that for the top stream, rather than predicting raw pixels in images, embodiments can utilize the features extracted from off-the-shelf CNN architecture as ground truth, to which the features constructed by the bottom stream regress. That is, because the pretrained CNN model contains valuable semantic information (e.g., referred to as dark knowledge) to differentiate objects and the extracted off-the-shelf features have achieved great success on various tasks.

In an alternative embodiment, rather than formulating the problem as a regression task, the training is treated as a classification problem by appending a softmax layer on top of the two streams and predicting whether a pair of features is likely given the spatial offset. However, such an embodiment can require a significant amount of negative samples (e.g., a car is not likely to be in a lake), which in some circumstances can make training more difficult. Further, the regression loss used by the spatial context module160builds on studies that soft real-valued targets can often perform better than discrete labels.

Once the spatial context network is trained, the encoded representation h1can be extracted and utilized for other tasks. An example of this is shown inFIG. 4, which illustrates a convolutional neural network trained for feature recognition, according to one embodiment described herein. As shown, the illustration400depicts the refined convolutional neural network140that has been extracted from the spatial context network. Such a refined convolutional neural network140could then be used, for example, for classification or object detection applications. Advantageously, the feature representations output by the refined CNN140are on average improved and more accurate, relative to those obtained from the original ImageNet pre-trained model for object detection and classification.

FIG. 5is a flow diagram illustrating a method of training and using a convolutional neural network to recognize features within a digital image, according to one embodiment described herein. As shown, the method500begins at block510, where the feature identification component120trains a first CNN that is configured to predict a feature representation for a first specified portion of a first digital image. The feature identification component120further trains a second CNN that is configured to compute a feature representation for a second specified portion of a second digital image (block520). The feature identification component120further trains a spatial context module that accepts the output of the first and second CNNs as input (block530). The feature identification component120refines the second CNN, by regressing features of the second CNN to features of the first CNN (block540). The refined second CNN can then be used for feature recognition analysis (block550), and the method500ends. For example, the refined second CNN can be used for feature recognition analysis in images in a new set of digital images (e.g., a set of images that does not include the first digital image or the second digital image).

FIG. 6depicts one architecture of an object recognition system600within which embodiments of the present disclosure may be implemented. This figure in no way limits or is intended to limit the scope of the present disclosure. The object recognition system600may be a desktop computer, video game console, digital assistant, rendering engine, or any other device suitable for practicing one or more embodiments of the present disclosure.

As shown, the object recognition system600includes one or more central processing units (CPUs)610, a memory620, storage640and a network interface650. CPU(s)610includes one or more processing cores, and, in operation, CPU610is the master processor of system600, controlling and coordinating operations of other system components. The system memory620stores software applications and data for use by CPU(s)610, and as shown, the memory620includes the feature identification component120and an operating system630.

Storage640represents non-volatile storage for applications and data and may include fixed or removable hard disk drives, flash memory devices, and CD-ROM, DVD-ROM, Blu-ray, HD-DVD, or other magnetic, optical, or solid state storage devices. The storage640(which can be connected to the CPU610via an I/O bridge, not shown) may be configured to store content and applications and data for use by CPU610. As shown, storage640contains digital images650, which can represent images from a training data set that are used to train the spatial context network130, as well as images analyzed using the spatial context network130.

Network interface660allows the object recognition system600to communicate with other systems via an electronic communications network, and may include wired or wireless communication over local area networks and wide area networks such as the Internet. For example, the digital images650can be received over a data communications network by the network interface660(e.g., from remote camera devices) and stored in the storage640, e.g., until analyzed by the feature identification component120.

FIG. 7is a flow diagram illustrating a method of using a convolutional neural network to recognize features within a digital image, according to one embodiment described herein. As shown, the method700begins at block710, where the feature identification component120retrieves a first convolutional neural network configured to predict a feature representation for a first patch of a first digital image. The feature identification component120refines the first convolutional neural network by regressing features of the first convolutional neural network to features of a second convolutional neural network (block715). Generally, the second convolutional neural network is configured to predict a feature representation for a second patch of the first digital image, based on an offset between the first patch and the second patch. The feature identification component120uses the refined first convolutional neural network to recognize one or more features within a second digital image (block720), and the method700ends.