Patent Publication Number: US-2021192752-A1

Title: Cascaded architecture for disparity and motion prediction with block matching and convolutional neural network (cnn)

Description:
BACKGROUND 
     1. Field 
     The field relates to image processing using neural networks. 
     2. Description of the Related Art 
     Electronics use in automobiles is increasing daily. In addition to the conventional engine controller, transmission controller, infotainment unit, body controller and the like, the advent of numerous safety and autonomous systems are greatly increasing the processing done inside an automobile. For example, adaptive cruise control may entail intercommunication between a radar system, an engine controller and a transmission controller. More advanced features, such as collision avoidance and autonomous operation, may require significant image processing, particularly disparity and motion prediction. 
     Traditional block matching based stereo and optical flow engines provide reasonable quality of disparity and motion streams, respectively, and they are suitable for hardware acceleration for real-time processing. The term disparity refers to the shift that occurs at each pixel in a frame between the left and right images due the different perspectives of the cameras used to capture the two images. The term motion refers to the shift that occurs at each pixel in a frame between successive frames. Hardware acceleration can be done as the basic algorithms are capable of being performed by dedicated, special function hardware logic engines. A DMPAC (Disparity and Motion Prediction Acceleration Core) hardware acceleration module implements semi-global block matching and Lucas-Kanade method for stereo and optical flow, respectively, to provide quite accurate disparity and motion streams efficiently in real time. 
     Semi-global block matching is based on the idea of pixelwise matching of mutual information and approximating a global, 2D smoothness constraint by combining many 1D constraints. The algorithm has distinct processing steps, assuming a general stereo geometry of two or more images with known epipolar geometry. First, pixelwise costs are calculated. Second, a smoothness constraint is applied. Next, the disparity is determined with sub-pixel accuracy and occlusion detection. For more details see H. Hirschmuller, “Accurate and Efficient Stereo Processing by Semi-Global Matching and Mutual Information,” IEEE Computer Science Conference on Computer Vision and Pattern Recognition, Vol. 2, Jun. 20-25, 2005, pp. 807-814 and H. Hirschmuller, “Stereo Processing by Semi-Global Matching and Mutual Information,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 30, No. 2, February 2008, pp. 328-341. 
     The Lucas-Kanade method is a widely used differential method for optical flow estimation developed by Bruce D. Lucas and Takeo Kanade. It assumes that the flow is essentially constant in a local neighborhood of the pixel under consideration, and solves the basic optical flow equations for all the pixels in that neighborhood, by the least squares criterion. For more details see B. Lucas, &amp; T. Kanade, “An Iterative Image Registration Technique with an Application to Stereo Vision,” Proceedings of the 7th International Joint Conference on Artificial Intelligence, 1981, pp. 121-130 and B. Lucas, “Generalized Image Matching by the Method of Differences,” Carnegie-Mellon University, Department of Computer Science, 1984. 
     However, the DMPAC module has limitations, most pronounced in ambiguous regions with insufficient texture, repeated pattern, occlusion, etc. In contrast, convolutional neural network (CNN)-based end-to-end learning approaches have shown superior performance over traditional block matching based approaches, especially for ambiguous regions. However, such CNNs require excessively high computation, data movement and memory and therefore are not practical for real-time processing. 
     SUMMARY 
     To improve operations in images having ambiguous regions and yet operate in real time, a CNN operates on the disparity or motion stream outputs of a block matching hardware module, such as a DMPAC module, to produce refined disparity or motion streams. As the block matching hardware module provides most of the processing, the CNN can be small and thus able to operate in real time, in contrast to CNNs which are performing all of the processing. In one example, the CNN operation is performed only if the block hardware module output confidence level is below a predetermined amount. The CNN can have a number of different configurations and still be sufficiently small to operate in real time on conventional platforms. 
    
    
     
       BRIEF DESCRIPTION OF THE FIGURES 
       The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee. 
       For a detailed description of various examples, reference will now be made to the accompanying drawings in which: 
         FIG. 1  is a drawing of a vehicle and the fields of view of various sensors. 
         FIG. 2  is a block diagram of the sensors in the vehicle of  FIG. 1 . 
         FIG. 3  is a block diagram of an SoC as used in the sensor modules of  FIG. 2 . 
         FIG. 4  is a block diagram of a cascaded CNN for stereo image processing. 
         FIG. 4A  is a block diagram of the cascaded CNN of  FIG. 4  allowing bypassing of the cascaded CNN. 
         FIG. 4B  is a flowchart of the operation of  FIG. 4A . 
         FIG. 5  is a block diagram of a cascaded CNN for optical flow image processing 
         FIG. 6  is a diagram of one example of a sequential refine network as the CNN of  FIGS. 4 and 5 . 
         FIGS. 7A and 7B  are diagrams of a first example of an encoder decoder refine network as the CNN of  FIGS. 4 and 5 . 
         FIG. 7C  is a diagram of a block in  FIG. 7A . 
         FIGS. 7D and 7E  are diagrams of a second example of an encoder decoder refine network as the CNN of  FIGS. 4 and 5 . 
         FIGS. 8A and 8B  are diagrams of one example of an hourglass refine network as the CNN of  FIGS. 4 and 5 . 
         FIG. 8C  is a diagram of a block in  FIG. 8A . 
         FIG. 9  is an illustration of the outputs of the CNNs of  FIGS. 6 and 8A and 8B  compared to the output of the DMPAC alone. 
         FIGS. 10A-10D  are illustrations of the outputs of the CNNs of  FIGS. 7D and 7E  compared to the output of the DMPAC alone. 
     
    
    
     DETAILED DESCRIPTION 
     Referring now to  FIG. 1 , a vehicle  100  is shown. The vehicle  100  includes a series of cameras or optical sensors. Left camera  102  and right camera  104  provide images from the front of the vehicle  100  for lane departure warnings, traffic sign recognition, collision alert and object detection. A left LIDAR (light detecting and ranging) sensor  106  and a right LIDAR sensor  108  provide image streams from the front of the vehicle  100  for lane and object detection. These camera and LIDAR sensors provide the input streams to various advanced driver assistance systems (ADAS). Cameras and LIDAR sensors are just examples and many other sensors, such as radar and ultrasonic and the like can be used as well. 
     Referring now to  FIG. 2 , cameras  102  and  104  are connected to a front cameras module  202 . LIDAR sensors  106  and  108  are connected to a LIDAR module  204 . The front cameras module  202  and the LIDAR module  204  are connected to a sensor fusion module  210  which integrates the various sensor outputs developed by the other modules. An autonomous processing module  212  is connected to the sensor fusion module  210  to perform autonomous processing needed for vehicle operation. More or fewer sensors can be connected to a given module and multiple sensor types can be provided to a single module. 
       FIG. 3  is a block diagram of an exemplary system on a chip (SoC)  500  as can be used in the modules  202 ,  204 ,  210  or  212 . A series of more powerful microprocessors  502 , such as ARM® A72 or A53 cores, form the primary general-purpose processing block of the SoC  500 , while a digital signal processor (DSP)  504  provides specialized computing capabilities. A simpler microprocessor  506 , such as an ARM R5F core, provides general control capability in the SoC  500 . A high-speed interconnect  508  connects the microprocessors  502 , DSP  504  and microprocessor  506  to various other components in the SoC  500 . For example, a shared memory controller  510 , which includes onboard memory or RAM  512 , is connected to the high-speed interconnect  508  to act as the onboard RAM for the SoC  500 . A DDR memory controller system  514  is connected to the high-speed interconnect  508  and acts as an external interface to external DRAM memory. A video acceleration module  516  and a radar processing accelerator (PAC) module  518  are similarly connected to the high-speed interconnect  508 . A vision processing accelerator module  520  is connected to the high-speed interconnect  508 , as is a depth and motion PAC (DMPAC) module  522 . The DMPAC module  522  is discussed in more detail in U.S. patent application Ser. No. 15/073,078, entitled “Hybrid Tiling Strategy for Semi-Global Matching Stereo Hardware Acceleration,” filed Mar. 17, 2016, now U.S. Pat. No. 10,080,007; Ser. No. 15/012,829, entitled “Semi-Global Matching (SMG) Cost Compression,” filed Feb. 1, 2016, published as Publication No. 2016/0227237; Ser. No. 15/081,118, entitled “Quasi-Parametric Optical Flow Estimation,” filed Mar. 25, 2016, now U.S. Pat. No. 10,268,901; Ser. No. 15/684,321, entitled “Handling Perspective Magnification in Optical Flow Processing,” filed Aug. 23, 2017, published as Publication No. 2018/0181816; and Ser. No. 15/695,266, entitled “Image Compression/Decompression in a Computer Vision System,” filed Sep. 5, 2017, published as Publication No. 2019/0073740, all of which are hereby incorporated by reference. 
     A graphics acceleration module  524  is connected to the high-speed interconnect  508 . A display subsystem  526  is connected to the high-speed interconnect  508  and includes conversion logic  528  and output logic  530  to allow operation with and connection to various video monitors. A system services block  532 , which includes items such as DMA controllers, memory management units, general-purpose I/O&#39;s, mailboxes and the like, is provided for normal SoC  500  operation. A serial connectivity module  534  is connected to the high-speed interconnect  508  and includes modules as normal in an SoC. A vehicle connectivity module  536  provides interconnects for external communication interfaces, such as PCIe block  538 , USB block  540  and an Ethernet switch  542 . A capture/MIPI module  544  includes a four-lane CSI-2 compliant transmit block  546  and a four-lane CSI-2 receive module and hub. 
     An MCU island  560  is provided as a secondary subsystem and handles operation of the integrated SoC  500  when the other components are powered down to save energy. An MCU ARM processor  562 , such as one or more ARM R5F cores, operates as a master and is coupled to the high-speed interconnect  508  through an isolation interface  561 . An MCU general purpose I/O (GPIO) block  564  operates as a slave. MCU RAM  566  is provided to act as local memory for the MCU ARM processor  562 . A CAN bus block  568 , an additional external communication interface, is connected to allow operation with a conventional CAN bus environment in the vehicle  100 . An Ethernet MAC (media access control) block  570  is provided for further connectivity in the vehicle  100 . External memory, generally non-volatile memory (NVM) is connected to the MCU ARM processor  562  via an external memory interface  569  to store instructions loaded into the various other memories for execution by the various appropriate processors. The MCU ARM processor  562  operates as a safety processor, monitoring operations of the SoC  500  to ensure proper operation of the SoC  500 . 
     It is understood that this is one example of an SoC provided for explanation and many other SoC examples are possible, with varying numbers of processors, DSPs, accelerators and the like. 
     The examples of  FIGS. 4 and 5  are similar, using the DMPAC module  522  followed by a cascaded CNN  402  executed on the DSP  504  to act as an image processing system to provide enhanced disparity or motion stream outputs, generically referred to as refined displacement streams. In stereo operation as in  FIG. 4 , left and right image streams  404  and  406  are provided to the DMPAC module  522  and the CNN  402 . The output of the DMPAC module  522  is considered an initial disparity stream to be refined by the CNN  402 . This initial disparity stream and the left and right image streams  404  and  406  are provided to the CNN  402 . The output of the CNN  402  is a refined disparity stream. 
     In optical flow operation as in  FIG. 5 , successive or current and previous image streams  408  and  410  are provided to the DMPAC module  522 . In this case, the output of the DMPAC module  522  is considered an initial motion stream to be refined by the CNN  402 . This initial motion stream and the current and previous image streams  408  and  410  are provided to the CNN  402 . The output of the CNN  402  is a refined motion stream. The disparity and motion streams provided from the DMPAC module  522  are generically referred to as displacement streams. 
     The use of the CNN  402  cascaded with the DMPAC module  522  provides improved disparity and motion stream outputs compared to just the DMPAC module  522  alone. The CNN  402  is much smaller than the end-to-end CNNs discussed above as it uses many fewer layers and thus requires many fewer calculations, so the combination can provide real time operation. 
     While previously the outputs of the DMPAC module  522  were used by the various functions, such as collision avoidance and autonomous operation, now the refined disparity and motion stream outputs of the CNN  402  are used in the various functions. 
     In the examples of  FIGS. 4 and 5 , the CNN  402  is developed by executing software instructions on the DSP  504 , the instructions stored in the NVM connected to the MCU ARM processor  562  and loaded into RAM  512  for use by the DSP  504 . The DMPAC module  522  outputs are provided to the RAM  512  or other memory and then retrieved by the DSP  504  to perform the CNN function. In other examples the CNN can be a hardware CNN present on the SoC or can be a combination of a hardware CNN and a DSP executing software instructions. 
     The DMPAC module  522  is one example of a block matching system and other more traditional block matching systems can be used instead of the DMPAC module  522 , the CNN  402  improving the results of those other block matching systems as the CNN  402  improves the output of the DMPAC module  522 . 
     In one example the operation of the CNN  402  is a refine network (RefineNet) that has been taught to predict a residual correction value to combine with original disparity or motion values to provide a refined disparity or motion value. For disparity or stereo operation, mathematically this is stated as: 
         d   2   =d   1   +F   r ( I   L   ,I   R   ,Ĩ   L   ,E   L   ,d   1 )         where d 1 =initial disparity
           d 2 =refined disparity   F r =correction function   I L =left image   I R =right image   Ĩ L =warped or reconstructed left image—right image and disparity   E L =error image—displacement between I L  and Ĩ L      
               
     For optical flow or motion operation, I L  becomes I t-1 , I R  becomes I t , Ĩ L  becomes Ĩ t-1 , E L  becomes E t-1 , d 1  becomes d 1x , d 1y  and d 2  becomes d 2x , d 2y . 
     In one example, illustrated in  FIG. 4A , to reduce required processing for stereo operations, a confidence stream provided by the DMPAC module  522  is used to determine if the initial disparity value will be sufficiently improved by the CNN  402  to merit the additional processing. The confidence stream is a quantitative estimate of the correctness of the disparity or motion output stream. If the confidence stream value is over a threshold value, then the initial disparity value from the DMPAC module  522  is used without being improved by the CNN  402 . Example confidence stream values vary based on the amount of improvement needed for the particular function. For example, a confidence stream value of 5 as the threshold value is used when less accuracy is required, while confidence stream values such as 6 or 7 are used when accuracy is more critical to the proper operation of the function. 
     This comparator logic shown diagrammatically in  FIG. 4A  by having the left and right image streams and the initial disparity stream as inputs to a demultiplexer  420 . The control of the demultiplexer  420  is provided by the output of a comparator  422 . The comparator  422  compares the confidence stream value from the DMPAC module  522  to the threshold value, shown as X, and provides a true output if the confidence stream value is less than the threshold value. The 1 or true outputs of the demultiplexer  420  are provided to the CNN  402 . The refined disparity output of the CNN  402  is provided to the 1 or true input of a multiplexer  424 , with control of the multiplexer  424  being the output of the comparator  422 . The initial disparity value is provided from the 0 or false output of the demultiplexer  420  to the 0 or false input of the multiplexer  424 . The output of the multiplexer  424  is then the initial disparity if the confidence stream value is greater than the threshold value and the refined disparity from the CNN  402  if the confidence stream value is less than the threshold value. 
     The CNN  402  is developed by software instructions executing on the DSP  504 ; the comparator logic is shown in flowchart format in  FIG. 4B . The comparison against the threshold value is performed by the DSP  504  at step  450  before the CNN  402  operations are performed. If the confidence stream value is greater than the threshold value, the DSP  504  simply passes the initial disparity value through in step  452  and does not perform the CNN  402  operations. If the confidence stream value is less than the threshold value, in step  454  the CNN  402  operations are performed, and the refined disparity output is provided. 
       FIG. 6  illustrates one example of a sequential refine network configuration  602  for the CNN  402  developed using seven 3×3 convolutional layers with a rectified linear unit (ReLu) for non-linearity. The first layer  604  has 13 input channels for disparity operation and 14 input channels for optical flow. The input channels for disparity operation are the red, green and blue (RGB) streams of the various images I L , I R , I L , and E L  and the d 1  stream from the DMPAC module  522 . The input channels for optical flow are the RGB streams for the various images I t-1 , I t , Ĩ t-1 , and E t-1 , and the d 1x  and d 1y  streams from the DMPAC module  522 . The first layer  604  has 16 output channels. 
     The outputs of the first layer  604  are provided as inputs to a second layer  606 , which has 16 output channels. The 16 output channels of the second layer  606  are the inputs to a third layer  608 , which also has 16 output channels. The 16 output channels of the third layer  608  are the inputs to a fourth layer  610 . The fourth layer  610  has 32 output channels, which are the inputs to a fifth layer  612 . The fifth layer  612  has 32 output channels, which are the inputs to a sixth layer  614 . The sixth layer  614  has 64 output channels, which are the inputs to a seventh layer  616 . The seventh layer  616  has one output channel for disparity and two output channels for motion. A summer  618  combines the output streams from the seventh layer  616  with the disparity or motion streams from the DMPAC module  522  to produce the refined disparity or motion streams. In one example, the sequential refine network configuration  602  has only 33,000 parameters and a receptive field size of 15×15. 
       FIGS. 7A and 7B  are a first example for an encoder-decoder structured refine network configuration  702 . A first layer  704  is 3×3 convolutional layer with ReLu with a stride of 2 and downsampling of two. The first layer  704  has 13 input channels for disparity and 14 input channels for optical flow. The input channels for disparity are the RGB streams of the various images I L , I R , Ĩ L , and E L  and the d 1  stream from the DMPAC module  522 . The input channels for optical flow are the RGB streams for the various images I t-1 , I t , Ĩ t-1 , and E t-1  and the d 1x  and d 1y  streams from the DMPAC module  522 . The first layer  704  has 32 output channels. 
     The 32 output channels from first layer  704  are provided to second layer  706 . The second layer  706  is a 3×3 depthwise convolutional layer that has 32 output channels provided to a third layer  708 . The third layer  708  is a 1×1 convolutional layer with ReLu6 and 16 output channels. 
     The output channels from the third layer  708  are provided to a first block  710  of a series of blocks. The block composition is illustrated in  FIG. 7C . The block  768  is a MobileNetV2 linear bottleneck. A first layer  770  in the block  768  is a 1×1 convolutional layer with ReLu6. A second layer  772  in the block  768  is a 3×3 depthwise convolutional layer. The third layer  774  in the block  768  is another 1×1 convolutional layer with ReLu6. The block  768  has an expansion factor or E of six. This means that the output of the first layer  770  is six channels wide for each input channel, the second layer  772  is the same six channels wide for each input channel and then the third layer  774  reduces from the six channels per input channel back down to the desired number of output channels. The first block  710  has a stride of 2 and is a normal convolution. R values are the dilated convolution rate, so R values greater than one indicate dilated convolution, while an R value of one means normal convolution. The first block  710  provides a downsampling of two and has 24 output channels. 
     A second block  712  receives the 24 output channels from the first block  710  and has an R value of 1 and a stride of 1, with 24 output channels. A third block  714  receives the 24 output channels from the second block  712  and has an R value of 1 and a stride of 2, with 32 output channels, providing a further factor of two downsampling. A fourth block  716  receives the 32 output channels from the third block  714  and has an R value of 1 and a stride of 1, with 32 output channels. A fifth block  718  receives the 32 output channels from the fourth block  716  and has an R value of 1 and a stride of 1, with 32 output channels. A sixth block  720  receives the 32 output channels from the fifth block  718  and has an R value of 1 and a stride of 1, with 64 output channels. A seventh block  722  receives the 64 output channels from the sixth block  720  and has an R value of 2 and a stride of 1, with 64 output channels. An eighth block  724  receives the 64 output channels from the seventh block  722  and has an R value of 2 and a stride of 1, with 64 output channels. A ninth block  726  receives the 64 output channels from the eighth block  724  and has an R value of 2 and a stride of 1, with 64 output channels. 
     A tenth block  728  receives the 64 output channels from the ninth block  726  and has an R value of 2 and a stride of 1, with 96 output channels. An eleventh block  730  receives the 96 output channels from the tenth block  728  and has an R value of 2 and a stride of 1, with 96 output channels. A twelfth block  732  receives the 96 output channels from the eleventh block  730  and has an R value of 2 and a stride of 1, with 96 output channels. A thirteenth block  734  receives the 96 output channels from the twelfth block  732  and has an R value of 2 and a stride of 1, with 160 output channels. A fourteenth block  736  receives the 160 output channels from the thirteenth block  734  and has an R value of 2 and a stride of 1, with 160 output channels. A fifteenth block  738  receives the 160 output channels from the fourteenth block  736  and has an R value of 2 and a stride of 1, with 160 output channels. A sixteenth block  740  receives the 160 output channels from the fifteenth block  738  and has an R value of 2 and a stride of 1, with 320 output channels. 
     The 320 output channels of the sixteenth block  740  are provided to a fourth layer  742 , which is a 3×3 depthwise convolutional layer that has 256 output channels. The 256 output channels of the fourth layer  742  are provided to an average pooling layer  744  with 256 output channels. The 256 output channels of the fourth layer  742  and the 256 output channels of the average pooling layer  744  are provided to a concatenation element  746 , which has 512 output channels. The concatenated 512 output channels are provided to a fifth layer  748 , which is a 1×1 convolutional layer and has two output channels. The two output channels are provided to an upsampling element  750 , which upsamples by a factor of eight to return to the original channel density and provides one output channel for disparity and two output channels for motion. The upsampled output channels are added by a summer  752  with the disparity or motion streams from the DMPAC module  522  to produce the refined disparity or motion streams. While the encoder-decoder structured refine network configuration  702  has many more stages than the sequential refine network configuration  602 , in one example the receptive field size is greater at 374×374 and the computational complexity, the total number of multiplications and additions, is similar because of the simplicity of the MobileNetV2 configuration and the downsampling. The larger receptive size allows further improvements in the disparity by removing more noise on flat areas and repeated patterns. 
       FIGS. 7D and 7E  are a second example of an encoder-decoder structured refine network configuration  758 . This second example encoder-decoder of  FIGS. 7D and 7E  differs from the first example encoder-decoder of  FIGS. 7A and 7B  in that only the I L  or left or previous image and d 1  or d 1x  and d 1y  disparity streams are used. Layer  704  to layer  742  are the same as the encoder-decoder structured refine network configuration  702 . The 320 output channels of the sixteenth block  740  are provided to fifth through seventh layers  760 ,  762 ,  764 , which are 3×3 depthwise convolutional layers that have 256 output channels. Fourth through seventh layers  742 ,  760 - 764  are similar except each has a different dilation or R value. The R value of the fourth layer  742  is one, the R value of the fifth layer  760  is six, the R value of the sixth layer  762  is twelve and the R value of the seventh layer  764  is 18. The 256 channel outputs of the fourth through seventh layers  742 ,  760 - 764  are provided to a first concatenation element  766 , which has 1024 output channels. The 1024 output channels are provided to an eighth layer  769 , which is a 3×3 depthwise convolutional layer that has 256 output channels. The 256 output channels are provided to a first upsampling layer  770  which provides an upsampling of two. The 256 output channels of the first upsampling layer  771  are provided to a second concatenation element  773 . 
     The 24 output channels of the first block  710  are also provided to a ninth layer  775 , which provides 48 output channels. The 48 output channels are provided to the second concatenation element  773 . The 304 output channels of the second concatenation element  773  are provided to a tenth layer  776 , a 3×3 depthwise convolutional layer that has 304 output channels. The 304 output channels are provided to an eleventh layer  778 , a convolutional layer with 256 output channels. The 256 output channels are provided to a twelfth layer  780 , a convolutional layer with 256 output channels. The 256 output channels are provided to a thirteenth layer  782 , a convolutional layer with one output channel for disparity operation and two output channels for motion operation. The output of the thirteenth layer  782  is provided to a first summer  784 . 
     The disparity or motion outputs of the DMPAC module  522  are provided to a pooling layer  786 , which downsamples the streams by a factor of four. The output of the pooling layer  786  is provided to the first summer  784 . The output of the first summer  784  is provided to a second upsampling layer  788 , which upsamples by a factor of two. The output of the second upsampling layer  788  is provided to a second summer  790 . 
     The output of the thirteenth layer  782  is also provided to a third upsampling layer  792 , which upsamples by a factor of two. The output of the third upsampling layer  792  is provided to the second summer  790 . The output of the second summer  790  is provided to a fourth upsampling layer  794 , which upsamples by a factor of two, returning to the original channel density. The output of the fourth upsampling layer  794  is provided to a third summer  798 . 
     The output of the thirteenth layer  782  is also provided to a fifth upsampling layer  796 , which upsamples by a factor of two. The output of the fifth upsampling layer  796  is provided to the third summer  798 . The output of the third summer  798  is the refined disparity or motion streams. 
       FIGS. 8A, 8B and 8C  are one example for an hourglass refine network configuration  802 . A first layer  804  is a 3×3 convolutional layer, with 13 disparity or 14 optical flow input channels as described above and 16 output channels. A second layer  806  receives the 16 output channels from first layer  804  and is a 3×3 convolutional layer, with 32 output channels. A third layer  808  receives the 32 output channels from second layer  806  and is a 3×3 convolutional layer, with a step size of two, to provide a downsampling of two, and 32 output channels. A fourth layer  810  receives the 32 output channels from third layer  808  and is a 3×3 convolutional layer, with 32 output channels. A fifth layer  812  receives the 32 output channels from fourth layer  810  and is a 3×3 convolutional layer, with a step size of two, to provide a downsampling of two, and 64 output channels. A sixth layer  814  receives the 64 output channels from fifth layer  812  and is a 3×3 convolutional layer, with 64 output channels. A seventh layer  86  receives the 64 output channels from sixth layer  814  and is a 3×3 convolutional layer, with a step size of two, to provide a downsampling of two, and 128 output channels. An eighth layer  88  receives the 128 output channels from the seventh layer  86  and is a 3×3 convolutional layer, with 128 output channels. 
     The output of the eighth layer  818  is provided to a first residuals module  822  and a tenth layer  820 . The first residuals module  822  is a residuals module as shown in  FIG. 8C  and has one output channel. A previous layer  870  has the outputs provided to a first convolutional layer  872 , whose outputs are provided to a second convolutional layer  874 . The outputs of the previous layer  870  and the second convolutional layer  874  are added elementwise by summer  876 . 
     The disparity or motion stream of the DMPAC  552  is provided to a first downsampling layer  821 , which downsamples the disparity or motion stream to match the eight times downsampled outputs of the first residuals module  822  and has one output channel. The outputs of the first downsampling layer  821  and the first residuals module  822  are summed by ninth layer  823 , which has one output channel for disparity and two output channels for motion. 
     The tenth layer  820  is an upsampling convolutional layer with 64 output channels and an upsampling of two. The output channels of the sixth layer  814 , the tenth layer  820  and the ninth layer  823  are provided to an eleventh layer  826 , which is a concatenating layer, so that the eleventh layer  826  has 129 or 130 input channels. In addition to concatenating, the eleventh layer  826  is a convolutional layer with 64 output channels. 
     The output of the eleventh layer  826  is provided to a second residuals module  830  and a twelfth layer  828 . The second residuals module  830  is a residuals module as shown in  FIG. 8C  and has one or two output channels. The disparity or motion stream of the DMPAC  552  is provided to a second downsampling layer  829 , which downsamples the disparity or motion stream to match the four times downsampled outputs of the eleventh layer  826  and has one or two output channels. The outputs of the second downsampling layer  829  and the second residuals module  830  are summed by thirteenth layer  831 , which has one or two output channels. 
     The twelfth layer  828  is an upsampling convolutional layer with 32 output channels and an upsampling of two. The output channels of the fourth layer  810 , the twelfth layer  828  and the thirteenth layer  831  are provided to a fourteenth layer  834 , which is a concatenating layer, so that the fourteenth layer  834  has 65 input channels. In addition to concatenating, the fourteenth layer  834  is a convolutional layer with 32 output channels. 
     The output of the fourteenth layer  834  is provided to a third residuals module  838  and a fifteenth layer  836 . The third residuals module  838  is a residuals module as shown in  FIG. 8C  and has one output channel. The disparity output of the DMPAC  552  is provided to a third downsampling layer  837 , which downsamples the disparity or motion output to match the two times downsampled outputs of the fourteenth layer  834  and has one or two output channels. The outputs of the third downsampling layer  837  and the third residuals module  838  are summed by sixteenth layer  839 , which has one or two output channels. 
     The fifteenth layer  836  is an upsampling convolutional layer with 16 output channels and an upsampling of two. The output of the fifteenth layer  836  is concatenated with the output of the first layer  804  and the output of the sixteenth layer  839  to a fourth residuals module  842 , so that the seventeenth layer has 33 input channels. The fourth residuals module  842  is a residuals module as shown in  FIG. 8C  and has one output channel. 
     A summer  844  combines the output of the fourth residuals module  842  and the disparity or motion stream of the DMPAC  552  to provide the refined disparity or motion stream. 
     These are four examples of CNN configurations to operate with a block matching hardware module such as a DMPAC module. These examples are small enough to operate in real time on standard SoCs. Many other CNN configurations can be developed based on the teachings provided by these examples and this description. 
     In one example, training of the stereo configuration was done using the KITTI stereo 2015 dataset, available at www.cvlibs.net/datasets/kitti/index.php and referenced generally in Andreas Geiger, Philip Lenz, and Raquel Urtasun, “Are we ready for autonomous driving? the KITTI vision benchmark suite,” Proc. Computer Vision Pattern Recognition, 2012. The dataset was randomly divided into a training set (80%) and a test set (20%). During training, for each epoch, the training set was divided into a training part (90%) and a validation part (10%). 
     In one example, the KITTI stereo 2012 dataset was used for training instead of the KITTI stereo 2015 dataset. 
     In one example, training of the optical flow configuration was done using the virtual KITTI dataset available at europe.naverlabs.com/research/computer-vision/proxy-virtual-worlds. The dataset was divided into a training set (80%) and a test set (20%). For the virtual KITTI dataset, as it contains 5 different driving scenes, the division was done according to the driving scenario. During training, for each epoch, the training set was divided into a training part (90%) and a validation part (10%). 
     In one example the training was done using the Adam optimizer, Diederik P. Kingma and Jimmy Ba, “Adam: A method for stochastic optimization,” 3rd International Conference for Learning Representations, San Diego, 2015, available at arxiv.org/abs/1412.6980. The initial learning rate R was 0.001. The validation loss was monitored to modify the learning rate. If the validation loss did not decrease for longer than N1=7 epochs, the learning rate was decreased by 50%. If the validation loss did not decrease for longer than N2=18 epochs, the training was stopped. 
     The results of using one example of the sequential configuration and one example of the hourglass configuration are shown in  FIG. 9 . The area in the blue rectangle was problematic as the image is of a shadow on a flat street, so that the image portion is highly uniform. The area in the green rectangle was problematic as the image is of a repeated pattern of a fence. As visible in  FIG. 9 , by comparing the red error portions and the results in the next to last column, the error is greatly reduced using either the sequential configuration or the hourglass configuration. 
       FIGS. 10A-10D  are examples of the improvement of the second example encode-decode configuration of  FIGS. 7D and 7E  as shown by the various images. 
     The above description is intended to be illustrative, and not restrictive. For example, the above-described examples may be used in combination with each other. Many other examples will be apparent upon reviewing the above description. The scope should, therefore, be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. In the appended claims, the terms “including” and “in which” are used as the plain-English equivalents of the respective terms “comprising” and “wherein.”