Heterogeneous convolutional neural network for multi-problem solving

A heterogeneous convolutional neural network (HCNN) system includes a visual reception system generating an input image. A feature extraction layer (FEL) portion of convolutional neural networks includes multiple convolution, pooling and activation layers stacked together. The FEL includes multiple stacked layers, a first set of layers learning to represent data in a simple form including horizontal and vertical lines and blobs of colors. Following layers capture more complex shapes such as circles, rectangles, and triangles. Subsequent layers pick up complex feature combinations to form a representation including wheels, faces and grids. The FEL portion outputs data to each of: a first sub-network which performs a first task of object detection, classification, and localization for classes of objects in the input image to create a detected object table; and a second sub-network which performs a second task of defining a pixel level segmentation to create a segmentation data set.

FIELD

The present disclosure relates generally to artificial intelligence convolutional neural networks.

BACKGROUND

Convolutional neural networks (CNNs) are being used to solve problems in computer vision, including image classification, object detection, and object segmentation problems. A CNN may be comprised of one or more convolutional layers, typically including a subsampling step, followed by one or more fully connected layers similar to a standard multilayer neural network. The architecture of a CNN is designed to take advantage of the 2D structure of an input image including pixel images from a camera. This includes local connections and tied weights followed by some form of pooling which produce translation invariant features. The benefits of CNNs include they are easy to train and have fewer parameters than fully connected networks with the same number of hidden units.

Conventional CNNs do not solve two or more independent problems at the same time. For example, known CNNs when applied to automobile vehicle assist and autonomous control systems cannot perform object detection including classification and localization, and road segmentation (lane detection) problems simultaneously. This requires the vehicle computer system to perform multiple parallel or independent computational steps, thereby requiring longer computational time and increased memory.

Thus, while current automobile artificial intelligence system CNNs achieve their intended purpose, there is a need for a new and improved convolutional neural network system allowing multi-problem solving.

SUMMARY

According to several aspects, a heterogeneous convolutional neural network (CNN) system includes a visual reception system generating an input image. A feature extraction mechanism in a convolutional neural network includes feature extraction layers (FEL) of convolutional neural networks having multiple convolution, pooling and activation layers stacked together with each other, directly receiving the input image, conducting a learning operation to learn to represent a first stage of data of the input image. The FEL includes multiple different stacked layers, wherein the first set of layers (the one directly after the input image) learns to represent data in a very simple form such as horizontal and vertical lines and simple blobs of colors. The following layers capture more complex shapes such as circles, rectangles, triangles, and the like. The subsequent layers pick up complex combinations of features from the previous layer or layers to form a more meaningful representation such as wheels, faces, grids, and the like. All of the above occurs inside the FEL, therefore the HCNN saves computation, memory, and speeds up execution by performing the above actions only once for each of the sub-networks. The FEL portion outputs the first stage of data to: a first sub-network which performs a first task of object detection, classification, and localization for classes of objects in the input image to create a detected object table; and a second sub-network which performs a second task of defining a pixel level segmentation to create a segmentation data set. The first stage of data include a first feature map captured from the input image.

In another aspect of the present disclosure, the FEL portion includes a first convolution and pooling layer (CPL) portion receiving the first stage of data and in a second stage of data capturing shapes including circles, rectangles, triangles, and the like. The first CPL portion forwards the second stage of data to the first sub-network for performing the first task of object detection, classification, and localization for classes of objects in the input image to create the detected object table. The FEL is common to each of the sub-networks and is reused while separation into specialized layers occurring after the FEL provides for individual special tasks including pixel level segmentation and object detection, classification, and localization.

In another aspect of the present disclosure, the FEL portion includes a second CPL portion capturing a third stage of data defining complex geometries including combinations of the first stage of data and the second stage of data and complex feature combinations to form a representation including wheels, faces and grids, the second CPL portion forwarding the third stage of data to the first sub-network for performing the first task of object detection, classification, and localization for classes of objects in the input image to create the detected object table.

In another aspect of the present disclosure, the second sub-network is trained by minimizing a loss function of the second sub-network while freezing the first sub-network.

In another aspect of the present disclosure, the first sub-network is trained by minimizing a loss function of the first sub-network while freezing the second sub-network.

In another aspect of the present disclosure, training the second sub-network and the first sub-network in a single stage wherein a model loss total LTotalis a weighted sum of individual loss functions L1, L2of each of the second sub-network and the first sub-network.

DETAILED DESCRIPTION

Referring toFIG. 1, a heterogeneous convolutional neural network (HCNN) system10receives object attribute data in a host vehicle12as it travels on a road or highway14in a general path of travel “A” in a third lane of the highway14together with other vehicles on the highway14. The other vehicles can include for example a first vehicle16such as a truck in an adjacent second lane but in front of the host vehicle12, a second vehicle18which may be partially blocked from direct view to the host vehicle12by the first vehicle16, as well as a third vehicle20such as a car travelling in a third lane on the highway14. Object attribute data may also be received from a fourth vehicle24which is stopped or disabled, and may be on the highway14or off to a side of the highway14. The image received by the system may also include a pedestrian26.

HCNN system10receives image data via a visual reception system22such as a camera, a LIDAR, or a RADAR system which collects the object attribute data, for example as a pixel image30shown and described in reference toFIG. 2. In this manner the object attribute data may be utilized for Advanced Driver Assist (ADAS) technology by also utilizing sensors that are in an existing centralized vision processor. The visual reception system22may further receive information as object imaging data defining the pedestrian26in an immediate vicinity of the fourth vehicle24, and fixed objects such as bridges, guard rails, trees, highway signs, and the like that are all located within a host vehicle predefined sensing and transmission window28of the HCNN system10.

Referring toFIG. 2and again toFIG. 1, the heterogeneous convolutional neural network (HCNN) system10of the present disclosure receives an input image30generated by the visual reception system22. The HCNN system10may further receive additional input images over a predetermined period of time, for example once every 30 ms as the host vehicle12travels along the highway14. It is desirable to generate at least two outputs using the data in the input image30, by solving at least two independent problems. A first output defining a detected object table32provides a list of detected objects, including object types34such as a car, a truck, a pedestrian, and the like, and a confidence level36in the accuracy of defining the object type34. Production of the detected object table32requires solutions of classification and localization of the objects. A second output defining a segmentation data set38provides data to the host vehicle12related to lane detection, lane conditions, and lane positions relative to the host vehicle12within the transmission window28of the HCNN system10.

The HCNN system10includes a processor or computer40which controls the visual reception system22and processes the data in the input image30. As noted above the HCNN system10performs several parallel tasks. A first sub-network44performs a first task of object detection, classification, and localization for certain classes of objects (vehicles, pedestrians, traffic signs, traffic lights, and the like, where the output from the first sub-network44is the list of detected objects, detected object table32, which provides a confidence level and location information for the detected objects. A second sub-network46performs a second task of lane detection. A pixel level segmentation is the solution and the output for the second sub-network46providing a colored image defining the segmentation data set38that indicates the drivable roads or lanes in front of the host vehicle12. Additional third and greater sub-networks (not shown) performing further tasks may also be provided with the HCNN system10.

The HCNN system10includes convolutional neural networks (CNNs) having multiple convolution, pooling and activation layers stacked together with each other. A first set of these layers defines a feature extraction layer (FEL) portion48defining the first set of layers directly after and directly receiving the input image30. The FEL portion48may be considered part of one or more sub-networks or separate from the sub-networks. The FEL portion48conducts a learning operation and thereby learns to represent a first stage of data of the input image30. The FEL portion48includes multiple different stacked layers, wherein the first set of layers (the one directly after and receiving the input image30) learns to represent data in a very simple form such as horizontal and vertical lines and simple blobs of colors in a first stage of data. The following layers of the FEL portion48capture more complex shapes such as circles, rectangles, triangles, and the like in a second stage of data. The subsequent layers of the FEL portion48capture complex combinations of features from the previous layer or layers to form a more meaningful representation such as wheels, faces, grids, and the like in a third stage of data. The FEL portion48distributes the first stage of data to each of the first sub-network44and the second sub-network46. The first stage of data includes a first feature map captured from the input image.

In the first sub-network44, included with the layers defining the FEL portion48is a sub-set of convolution, pooling and activation layers stacked on top of each other defining a first convolution and pooling layer (CPL) portion50and a second convolution and pooling layer (CPL) portion52. The first convolution and pooling layer (CPL) portion50receives the first stage of data output from the FEL portion48and in a second stage of data captures the more complex shapes more complex shapes including circles, rectangles, triangles, and the like. Thus, the second stage of data includes a second feature map different than the first feature map. The third and final grouping of layers defining the second convolution and pooling layer (CPL) portion52of the first sub-network44then captures a third stage of data defining complex combinations of the features from the FEL portion48and the first CPL portion50to form a meaningful representation such as wheels, faces, grids, and the like needed to handle the individual special tasks such as object detection, classification, and localization. Thus, the third stage of data includes a third feature map different than the first feature map and the second feature map. Each stage of data is passed to each of the CPL portions50,52in succession. Thus, the first stage of data is passed to the CPL portion50, and the second stage of data is passed to the CPL portion52.

Data from each of the FEL portion48, including the CPL portion50and the CPL portion52are merged to generate a fully connected layer54. The fully connected layer54determines multiple confidence levels for an object identified by the first stage of data, the second stage of data, and the third stage of data. The multiple confidence levels are communicated to a non-maximum suppression module56that reduces the multiple confidence levels to a single confidence level for the object. The non-maximum suppression module56is used to generate output image data having predicted objects identified at specific x, y coordinate locations in the field of the original input image30, from which the detected object table32is generated.

The HCNN system10combines different sub-networks such as the first sub-network44and the second sub-network46to perform multiple tasks efficiently, thereby using a smaller memory footprint (memory saving) and operating faster than running the different sub-networks separately. To accomplish this, the single FEL portion48and therefore the tasks performed by the single FEL portion48are common to all the CNNs of the first sub-network44and the second sub-network46(and any additional sub-networks) for HCNN system10regardless of the final task performed by each individual CNN. Because the initial volume of data analyzed from the input image30is greatest, by performing this analysis only once and using the output of the single FEL portion48for the input of each of the sub-networks reduces computational time and memory and reduces hardware.

A third CPL portion58of the second sub-network46also directly receives the first stage of data from the FEL portion48in lieu of providing a separate feature extraction layer for the second sub-network46. A fourth CPL portion60of the second sub-network46receives the second stage of data from the CPL portion50, and a fifth CPL portion62of the second sub-network46having a 2× deconvolution member64receives the third stage of data from the CPL portion52. An output from the deconvolution member64and the output from the CPL portion62are added and passed through a 2× deconvolution member66. An output from the deconvolution member66and the output from the CPL portion58are added and passed through an 8× deconvolution member68to generate the segmentation data set38. The first stage of data, the second stage of data, the third stage of data, and any further stages of data from any further convolution layers in the first subnetwork44are communicated to the fully connected layer54.

As used herein, a convolution layer detects the presence of specific features or patterns in the original data by applying a convolution operation between a filter (the weights of the network) and the input image. The output of the convolution for a given filter is called a feature map. A pooling layer represents the process of reducing the spatial size of the input image. Max Pooling and Average pooling are considered as the two most common subsampling methods. For example, if the resolution of the input to a pooling layer is 640×540, then the resolution of the output is 320×270. Thus, a pooling layer leads to the reduction on the size, width and height, by half every time pooling layer is applied. A deconvolution layer reverses the effect of a convolution layer by transposing the corresponding convolution layer in order to recover the effect of the corresponding convolution layer.

The HCNN system10provides a method to combine different networks to perform multiple tasks efficiently in one combined heterogonous network, thus using a smaller memory footprint, saving memory resources. Due to the synergies in the FEL portion48the partially combined network performs the tasks faster than running separate networks. The HCNN system10is described in one present example for use in a perception kit for an ADAS and autonomous vehicle vision system. The HCNN system10performs two tasks simultaneously, which in the example of the ADAS and autonomous vehicle vision system includes lane detection and object detection. The first task in this example is the lane detection task, where a pixel level segmentation is the solution for the problem to determine lanes in the picture or image of the road and the output for this sub-task is a colored image that shows the drivable lanes in front of the host vehicle. The second task is object detection, classification, and localization for certain classes of objects (vehicles, pedestrians, traffic signs, traffic lights, etc.), where the output for this task is a list of detected objects with confidence level and location information.

Referring generally toFIGS. 3, 4, 5, and again toFIGS. 1 and 2, training the HCNN system10resolves heterogynous CNN tasks where a loss function for an individual sub-network differs from a loss function of any of the other sub-networks. HCNN system10training can therefore be conducted in multiple stages. In each stage a loss function of a specific task of one sub-network is minimized to train the layers that belong to that task while freezing the other sub-network or sub-networks. For example, a loss function (L1) for the road segmentation task of the second sub-network46is a binary cross entropy loss, and a loss function (L2) for the object detection task of the first sub-network44is a sum of a localization loss (e.g., smooth L1) and a confidence (e.g., a soft-max or multi-class cross entropy loss).

With specific reference toFIG. 3, a first step in a multi-step training process to train the HCNN system10provides training of the second sub-network46by minimizing the loss function L1while freezing the first sub-network44, and other sub-networks if present. With specific reference toFIG. 4, a second step in the multi-step training process to train the HCNN system10provides training of the first sub-network44by minimizing the loss function L2while freezing the second sub-network46, and other sub-networks if present. With specific reference toFIG. 5, a third step in the multi-step training process to train the HCNN system10provides training of any Nthsub-network using a loss function Lnwhile freezing the first sub-network44, the second sub-network46, and other sub-networks if present. These methods ensure that training the layers of one task or sub-network does not affect the layers of the other tasks or sub-networks.

Referring toFIG. 6and again toFIGS. 3 through 5, a single-stage training method for the HCNN system10is provided. In this method, the HCNN system10is trained in a single stage for the entire network including the first sub-network44, the second sub-network46, and other sub-networks if present. In the single-stage training method, a model loss total LTotalis a weighted sum of the individual loss functions L1, L2, Lnas described in reference toFIGS. 3 through 5.

The HCNN system10can be trained using multi-stage training or single-stage training. For fine tuning, the HCNN system10can also be trained using both the multi-stage training followed by the single-stage training.

A heterogeneous convolutional neural network (HCNN) system10of the present disclosure offers several advantages. These include the capability to merge separate machine vision challenges in a single heterogeneous network, thereby reducing overall memory usage and the time needed to run predictions. For automobile vehicle vision systems, this permits object detection including classification and localization and road segmentation including lane detection to be addressed together using output from a single feature extraction layer (FEL) portion of the convolutional neural networks. In addition, each sub-network can be trained using a loss function of that sub-network while freezing the other sub-network or sub-networks.

Additionally, in the claims and specification, certain elements are designated as “first”, “second”, “third”, “fourth”, “fifth”, “sixth”, and “seventh”. These are arbitrary designations intended to be consistent only in the section in which they appear, i.e. the specification or the claims or the summary, and are not necessarily consistent between the specification, the claims, and the summary. In that sense they are not intended to limit the elements in any way and a “second” element labeled as such in the claim may or may not refer to a “second” element labeled as such in the specification. Instead, the elements are distinguishable by their disposition, description, connections, and function.