Patent Description:
In recent years, there have been proposed various devices to which neural networks are provided. For example, Patent Literature <NUM> discloses a technology capable of improving identification properties for images on the basis of likelihood that indicates probability of a detection object. <CIT> discloses a convolutional neural network (CNN) system.

Incidentally, devices represented by a digital camera and the like is provided with an image sensor including a Complementary Metal Oxide Semiconductor (CMOS) and a Digital Signal Processor (DSP). In recent years, from viewpoints of diversification and acceleration in image processing, protection of personal information, and the like; it is desired that a Deep-Neural-Network (DNN) function is provided to an image sensor so as to execute advanced processes, for example.

However, in Patent Literature <NUM>, an image sensor is to output a target image, and thus data amount to be output is large and further there presents possibility that personal information is not able to be protected.

In the present disclosure, there are proposed a solid-state image capturing system, an information processing device, an information processing method, and a program capable of reducing data amount output from an image sensor while protecting personal information.

The above problem is solved by the claimed subjectmatter, which defines the present invention.

The following describes preferable embodiments of the present disclosure in detail with reference to the attached drawings. In the present specification and the drawings, overlap of descriptions will be avoided by providing the same reference signs for constituent elements having substantially the same functional configuration.

The present disclosure will be described in the following order.

A configuration of a solid-state image capturing system according to a first embodiment of the present disclosure will be explained with reference to <FIG> is a block diagram illustrating one example of a configuration of the solid-state image capturing system according to the first embodiment of the present disclosure.

As illustrated in <FIG>, a solid-state image capturing system <NUM> includes a solid-state image capturing device <NUM> and an information processing device <NUM>.

As illustrated in <FIG>, both of the solid-state image capturing device <NUM> and the information processing device <NUM> are housed in a housing <NUM>. In other words, the solid-state image capturing device <NUM> and the information processing device <NUM> are arranged in the same housing <NUM> as separated tips. The solid-state image capturing device <NUM> and the information processing device <NUM> are implemented by using a System-on-a-chip (SoC), a Multi-Chip Module (MCM), a System In a Package (SIP), a Small Outline Package (SOP), etc. The solid-state image capturing system <NUM> may be connected to an external device by using, for example, an Internet communication network <NUM> to be able to communicate with each other. The solid-state image capturing system <NUM> illustrated in <FIG> includes the single solid-state image capturing device <NUM> and the single information processing device <NUM>. The solid-state image capturing system <NUM> may include the plurality of solid-state image capturing devices <NUM> and the plurality of information processing device <NUM>. The number of the solid-state image capturing devices <NUM> and that of the information processing devices <NUM> included in the solid-state image capturing system <NUM> may be different form each other. The solid-state image capturing system <NUM> may be applied to a Virtual Personal Assistant (VPA) and an on-vehicle camera, for example.

As illustrated in <FIG>, the solid-state image capturing device <NUM> includes an image capturing unit <NUM>, a capture processing unit <NUM>, a first Deep-Neural-Network (DNN) processing unit <NUM>, a first storage <NUM>, a first control unit <NUM>, a selector <NUM>, a communication I/F <NUM>, and a communication controlling unit <NUM>.

<FIG> is a diagram illustrating one example of a laminate structure of the solid-state image capturing device <NUM> according to the first embodiment. As illustrated in <FIG>, the solid-state image capturing device <NUM> has a laminate structure obtained by bonding a rectangular first substrate <NUM> and a rectangular second substrate <NUM> together, for example.

The first substrate <NUM> and the second substrate <NUM> may be bonded together by a Chip on Chip type (CoC type) in which each of the first substrate <NUM> and the second substrate <NUM> is separated into tips, and then the first substrate <NUM> and the second substrate <NUM> that have been separated into tips are bonded together, for example. One (for example, first substrate <NUM>) of the first substrate <NUM> and the second substrate <NUM> may be separated into tips and then the first substrate <NUM> having been separated into tips may be bonded to the second substrate <NUM> that is not separated into tips (namely, in wafer state), in other words, the first substrate <NUM> and the second substrate <NUM> may be bonded together by a Chip on Wafer type (CoW type). Moreover, the first substrate <NUM> and the second substrate <NUM> may be bonded together in a state where both of the first substrate <NUM> and the second substrate <NUM> are in a wafer state, namely, may be bonded by a Wafer on Wafer type (WoW type).

For a method for bonding the first substrate <NUM> and the second substrate <NUM> together, plasma binding and the like may be employed, for example. Not limited thereto, various bonding methods may be employed.

A size of the first substrate <NUM> and that of the second substrate <NUM> may be equal to each other, or may be different from each other. For example, the first substrate <NUM> and the second substrate <NUM> may be semiconductor substrates such as silicon substrates.

On the first substrate <NUM>, the image capturing unit <NUM> among the configuration elements of the solid-state image capturing device <NUM> illustrated in <FIG> may be arranged, for example.

On the second substrate <NUM>, the capture processing unit <NUM>, the first DNN processing unit <NUM>, the first storage <NUM>, the first control unit <NUM>, the selector <NUM>, the communication I/F <NUM>, and the communication controlling unit <NUM> among the configuration elements of the solid-state image capturing device <NUM> illustrated in <FIG> may be arranged, for example.

In other words, the image capturing unit <NUM> of the solid-state image capturing device <NUM> has a configuration in which the image capturing unit <NUM> is laminated and mounted on configuration elements other than the image capturing unit <NUM>.

Again, with reference to <FIG>. The image capturing unit <NUM> includes, for example, an optical system including a zoom lens, a focus lens, a diaphragm, and the like; and a pixel array unit having a configuration in which unit pixels, each of which includes a light receiving element such as a photodiode, are arrayed in two-dimensional matrix. Light having been made incident on from the outside goes through the optical system so as to form an image on a light receiving surface of the pixel array unit on which light receiving elements are arrayed. Each of the unit pixels of the pixel array unit photo-electrically converts light having been made incident on a light receiving element thereof so as to generate image data according to a light amount of the incident light. The image capturing unit <NUM> outputs the captured image data to the capture processing unit <NUM>.

The capture processing unit <NUM> converts image data into digital image data, for example. The capture processing unit <NUM> executes "preprocessing", "data extension", and "data normalization" on the image data having been converted into digital data, for example. The preprocessing is a process that is executed on the image capturing unit <NUM> during estimation and learning, and includes processes such as dewarping, cropping, lens shading correction, downscaling, and downscaling. The data extension is a process that is executed on the image data during learning, and includes processes for, for example, changing a length-to-width ratio of image data, moving image data in parallel, rotating image data, inverting image data, geometrically deforming image data, and the like. Moreover, the data extension includes processes for changing light and shade of colors of image data and changing colors of image data, for example. Furthermore, the data extension includes a process for adding noise to image data, for example. The data normalization is a process that is executed on image data during estimation and learning, and includes processes for setting average of pixel values of image data to zero, setting dispersion of pixel values to one, and setting a correlation between components to zero so as to whiten image data, for example. The capture processing unit <NUM> outputs, to the first DNN processing unit <NUM> and the selector <NUM>, digital image data on which the various processes have been executed.

The first DNN processing unit <NUM> executes, on image data input thereto from the capture processing unit <NUM>, a DNN on the basis of a DNN model stored in the first storage <NUM>, for example, so as to execute a recognition process of an object included in the image data. Specifically, on image data received from the capture processing unit <NUM>, the first DNN processing unit <NUM> executes a first DNN, and executes a part of an algorithm constituting a DNN model so as to generate an execution result. The above-mentioned execution result may be referred to as a feature map and a map, or may be simply referred to as an image and the like. The first DNN processing unit <NUM> executes the first DNN on image data so as to generate a map of an intermediate layer and first result. More specifically, the first DNN processing unit <NUM> outputs, as the first result, an intermediate layer map of whole of the input image data and an intermediate layer map of a part of the input image data (for example, intermediate layer map on which Region Of Interest (ROI) is executed). The first DNN processing unit <NUM> may output both of the whole intermediate map and the partial intermediate map, or may be output one of the intermediate maps. The first DNN processing unit <NUM> calculates, as first result, ROI information and existence probability of an object. When execution of the first DNN has been completed, the first DNN processing unit <NUM> outputs a completion notification to the first control unit <NUM>. The first DNN processing unit <NUM> outputs first result to the selector <NUM>.

Specifically, the first DNN processing unit <NUM> executes, on image data received from the capture processing unit <NUM>, for example, a convolution operation, bias addition, an activation operation, and a pooling process so as to perform an object detecting process.

As an activation function employed by the first DNN processing unit <NUM> in the activation operation, for example, there may be exemplified an identity function, a sigmoid function, a softmax function, a step function, a ReLU function, a Tanh function, and the like; however, not limited thereto.

For example, during learning, the first DNN processing unit <NUM> calculates an error and a value of a loss function. The first DNN processing unit <NUM> calculates an error of a target DNN model by using a method such as a method for gradient descent, a method for stochastic gradient descent, a Newton method, a quasi-Newton method, and an error backpropagation method. The first DNN processing unit <NUM> calculates a value of the loss function by using a method such as a least square error and a cross entropy error.

The first storage <NUM> stores therein at least one DNN model to be executed in the first DNN processing unit <NUM>, for example. The first storage <NUM> may be realized by using a semiconductor memory element such as a Random Access Memory (RAM), a Read Only Memory (ROM), and a Flash Memory.

The first control unit <NUM> controls the first DNN processing unit <NUM>, for example. Specifically, for example, the first control unit <NUM> controls the first DNN processing unit <NUM> so as to execute interruption or stop of first DNN. The first control unit <NUM> generates first control information, for example. The first control unit <NUM> transmits the first control information to the information processing device <NUM>, for example. The first control information includes information on a first DNN executed by the first DNN processing unit <NUM> among DNN algorithms to be executed on image data, for example. The first control information includes a synchronization signal, for example. The first control unit <NUM> transmits, to the information processing device <NUM>, an execution completion notification of the first DNN processing unit <NUM>, for example. The first control unit <NUM> receives second control information from a second control unit <NUM> of the information processing device <NUM>, for example. In this case, the first control unit <NUM> controls the first DNN processing unit <NUM> on the basis of the second control information, for example. Incidentally, the first control unit <NUM> may control the first DNN processing unit <NUM> so as to execute a DNN algorithm on image data in a pipelined manner. Specifically, the first control unit <NUM> may control the first DNN processing unit <NUM> so as to execute, after execution of first DNN on specific image data and before being informed of completion of execution of a second DNN process on the specific image data, a first DNN on the next image data. Thus, according to the present embodiment, it is possible to effectively execute DNN processes on a plurality of pieces of image data captured by the image capturing unit <NUM>, for example.

The selector <NUM> receives, from the capture processing unit <NUM>, digital image data on which various processes have been executed, for example. The selector <NUM> receives first result from the first DNN processing unit <NUM>, for example. The selector <NUM> selectively outputs, to the communication I/F <NUM>, data received from the capture processing unit <NUM> and the first DNN processing unit <NUM> in accordance with a control signal transmitted from a control unit of a selector (not illustrated), for example.

The communication I/F <NUM> includes a transmitting unit <NUM> and a reception unit <NUM>. The solid-state image capturing device <NUM> transmits data to the information processing device <NUM> via the transmitting unit <NUM>. The solid-state image capturing device <NUM> receives data from the information processing device <NUM> via the reception unit <NUM>.

The communication controlling unit <NUM> controls the communication I/F <NUM>. Thus, the communication I/F <NUM> transmits data to the information processing device <NUM>, and receives data from the information processing device <NUM>. The communication controlling unit <NUM> may encrypt data, for example. When encryption is executed, the communication controlling unit <NUM> may use a common key encryptosystem such as a Data Encryption Standard (DES) and an Advanced Encryption Standard (AES). The communication controlling unit <NUM> may use a cryptography mode such as an Electronic Codebook (ECB), a Cipher Block Chaining (CBC), a Cipher Feedback (CFB), an Output Feedback (OFB), and a Counter (CTR).

The information processing device <NUM> includes a communication I/F <NUM>, a communication controlling unit <NUM>, a second DNN processing unit <NUM>, a second storage <NUM>, and the second control unit <NUM>. In the first embodiment, the information processing device <NUM> is an application processor, for example.

The communication I/F <NUM> includes a reception unit <NUM> and a transmitting unit <NUM>. The information processing device <NUM> receives data from the solid-state image capturing device <NUM> via the reception unit <NUM>. The information processing device <NUM> transmits data to the solid-state image capturing device <NUM> via the transmitting unit <NUM>.

The communication controlling unit <NUM> controls the communication I/F <NUM>. Thus, the communication I/F <NUM> transmits data to the information processing device <NUM>, and receives data from the information processing device <NUM>. The communication controlling unit <NUM> may encrypt data so as to communicate with the solid-state image capturing device <NUM>, for example. When encryption is executed, the communication controlling unit <NUM> may use a common key encryptosystem such as DES and AES. The communication controlling unit <NUM> may use a cryptography mode such as ECB, CBC, CFB, OFB, and CTR.

The second DNN processing unit <NUM> executes, on the basis of a DNN model stored in the second storage <NUM>, for example, DNN on first result input from the solid-state image capturing device <NUM> so as to execute a recognition process of an object included in image data. Specifically, the second DNN processing unit <NUM> executes a second DNN on the first result having received from the solid-state image capturing device <NUM> so as to execute remaining part among algorithms constituting a DNN model, which is not executed in the first DNN. Thus, the second DNN processing unit <NUM> outputs second result. Specifically, the second DNN processing unit <NUM> recognizes, as second result, an object included in image data. The second DNN processing unit <NUM> merges the first result and the second result with each other so as to execute ROI and an object classification, for example. When execution of the second DNN has been completed, the second DNN processing unit <NUM> outputs a completion notification to the second control unit <NUM>. The second DNN processing unit <NUM> outputs second result to an external device, for example.

Specifically, for example, the second DNN processing unit <NUM> executes, on image data received from the capture processing unit <NUM>, a convolution operation, bias addition, an activation operation, and a pooling process so as to execute an object detecting process.

As an activation function used by the second DNN processing unit <NUM> in the activation operation, there may be exemplified an identity function, a sigmoid function, a softmax function, a step function, a ReLU function, and a Tanh function, for example.

For example, during learning, the second DNN processing unit <NUM> calculates an error and a value of a loss function. The second DNN processing unit <NUM> calculates an error of a target DNN model by using a method such as a method for gradient descent, a method for stochastic gradient descent, a Newton method, a quasi-Newton method, and an error backpropagation method. The second DNN processing unit <NUM> calculates a value of a loss function by using a method such as a least square error and a cross entropy error.

The second storage <NUM> stores therein at least one DNN model to be executed by the second DNN processing unit <NUM>, for example. A DNN model stored in the first storage <NUM> of the solid-state image capturing device <NUM> and a DNN model stored in the second storage <NUM> are combined with each other so as to constitute one DNN algorithm. Thus, one DNN algorithm is executed by using two devices of the solid-state image capturing device <NUM> and the information processing device <NUM>. In other words, in the present embodiment, one DNN algorithm is able to be divided into the two devices of the solid-state image capturing device <NUM> and the information processing device <NUM> to be executed. The second storage <NUM> is able to be realized by a semiconductor memory element such as RAM, ROM, and a flash memory.

The second control unit <NUM> controls the second DNN processing unit <NUM>, for example. Specifically, the second control unit <NUM> controls the second DNN processing unit <NUM> so as to execute interruption or stop of a second DNN, for example. The second control unit <NUM> receives first control information from the first control unit <NUM> of the solid-state image capturing device <NUM>, and controls the second DNN processing unit <NUM> on the basis of the received first control information, for example. The second control unit <NUM> generates second control information, for example. The second control unit <NUM> transmits the generated second control information to the solid-state image capturing device <NUM>, for example. The second control information includes information on a second DNN that is executed on the first result by the second DNN processing unit <NUM>, for example. The second control information includes a synchronization signal, for example. The second control unit <NUM> transmits an execution completion notification of the second DNN processing unit <NUM> to the solid-state image capturing device <NUM>, for example. The second control unit <NUM> may control the second DNN processing unit <NUM> so as to execute a DNN algorithm on image data in a pipelined manner. Specifically, the second control unit <NUM> executes a second DNN on a specific first result so as to generate second result. The second result may be transmitted to a processing device other than the second DNN processing unit <NUM>, and the second DNN processing unit <NUM> may be controlled so as to execute a second DNN on the next first result before receiving, from the other processing device, a completion notification of the process executed on the second result. Thus, according to the present embodiment, it is possible to effectively execute DNN processes on a plurality of first results.

With reference to <FIG>, processes of the first DNN processing unit <NUM> and the second DNN processing unit <NUM> will be explained. <FIG> is a diagram illustrating processes of the first DNN processing unit <NUM> and the second DNN processing unit <NUM>.

First, the first DNN processing unit <NUM> receives image data from the capture processing unit <NUM> (Step S1). The first DNN processing unit <NUM> receives image data including a dog D and a man M, for example.

Next, the first DNN processing unit <NUM> executes a first DNN on the image data received in Step S1 (Step S2). Herein, the first DNN processing unit <NUM> executes a first DNN on image data so as to generate, as first result, a feature map such as an intermediate layer. In the feature map, it is indicated that there present some kind of objects in a region R1 and a region R2 with high probability. The first DNN processing unit <NUM> output the feature map to the second DNN processing unit <NUM>. The first DNN processing unit <NUM> may output whole of the feature map to the second DNN processing unit <NUM>, or may output a feature map corresponding to the region R1 and the region R2 alone. In other words, the first DNN processing unit <NUM> encodes image data captured by the image capturing unit <NUM>, and outputs the encoded image data to the second DNN processing unit <NUM>. Thus, data output from the first DNN processing unit <NUM> does not include original image data of the dog D and the man M captured by the image capturing unit <NUM>, so that it is possible to protect personal information. Furthermore, volume of a feature map is smaller than that of image data, so that it is possible to reduce data amount transmitted from the solid-state image capturing device <NUM> to the information processing device <NUM>.

The second DNN processing unit <NUM> executes a second DNN on the feature map (first result) that is obtained in Step S2 (Step S3). Thus, the second DNN processing unit <NUM> is capable of recognizing that there presents the dog D in the region R1 of the feature map and there presents the man M in the region R2 of the feature map.

In other words, in the present embodiment, the first DNN processing unit <NUM> executes a part of a DNN algorithm, and the second DNN processing unit <NUM> executes remaining part thereof so as to execute a recognition process of an object included in image data. In other words, two DNN processing units of the first DNN processing unit <NUM> and the second DNN processing unit <NUM> execute one DNN algorithm.

With reference to <FIG>, a configuration example of a DNN algorithm will be explained. <FIG> is a diagram illustrating a configuration example of a DNN algorithm.

<FIG> is a diagram illustrating one example of a structure of an object detecting algorithm <NUM>.

First, the object detecting algorithm <NUM> executes a Convolutional Neural Network (CNN) on image data. Specifically, image data I is input to a convolution layer CL. The convolution layer CL executes a CNN on whole of the image data I so as to output a feature map FM1. The processes so far are image processing using a CNN <NUM>.

In the next step, an object-region recognizing algorithm is executed on the feature map FM1. The object-region recognizing algorithm is executed on a feature map FM so as to extract a candidate region in which there presents an object included in a feature map. Specifically, in the example illustrated in <FIG>, RoI1, RoI2, and RoI3 are extracted, from the feature map FM1, as candidate regions in which there presents an object. The processes so far are image processing using an object-region recognizing algorithm <NUM>.

In the next step, RoI1, RoI2, and RoI3 are superposed on the feature map FM1 to generate an RoI feature map FM2, and the generated RoI feature map FM2 is stored in a Pooling layer. A RoI pooling layer executes RoI pooling on the feature map FM1 on which RoI1, RoI2, and RoI3 are superposed. The RoI pooling is a process for extracting regions including RoI1, RoI2, and RoI3 as individual feature maps. Thus, regions including RoI1, RoI2, and RoI3 are respectively extracted as a feature map FM3, a feature map FM4, and a feature map FM5. The feature map FM3, the feature map FM4, and the feature map FM5 are input to a fully connected layer <NUM>.

In the next step, a classification layer <NUM> and a rectangular regression layer <NUM> share the feature map FM3, the feature map FM4, and the feature map FM5 that are input to the fully connected layer <NUM> with each other.

The classification layer <NUM> classifies types of objects included in the feature map FM3, the feature map FM4, and the feature map FM5. The classification layer <NUM> outputs a classification result <NUM>. The rectangular regression layer <NUM> generates a rectangle that surrounds an object included in the feature map FM3, the feature map FM4, and the feature map FM5.

The feature map FM3, the feature map FM4, and the feature map FM5 are input to a Fully Convolution Network (FCN) <NUM> in parallel with processing of the classification layer <NUM> and the rectangular regression layer <NUM>. Specifically, RoIAlign is executed by a RoIAlign layer on the feature map FM3, the feature map FM4, and the feature map FM5 to be input to the FCN <NUM>. The RoIAlign is a process for correcting, by using a bilinear interpolation method, a round-off error that occurs when the RoI feature map FM2 is generated from the image data I.

The FCN <NUM> is configured to classify an object included in a feature map, generate a rectangle that surrounds an object, and execute masking on an object for each of the feature map FM3, the feature map FM4, and the feature map FM5. The FCN <NUM> outputs a classification result <NUM>, a rectangular frame <NUM>, and a mask result <NUM> for each of the feature maps. The processes so far are the object detecting algorithm <NUM>.

In the present embodiment, the first DNN processing unit <NUM> and the second DNN processing unit <NUM> may execute processes that are arbitrarily divided from the processes included in the DNN algorithm illustrated in <FIG> as long as the processes are divided to be executed. For example, the first DNN processing unit <NUM> may execute processes up to the process for generating the feature map FM1, and the second DNN processing unit <NUM> may execute remaining processes. For example , the first DNN processing unit <NUM> may execute, in the object detecting algorithm <NUM>, processes up to the process for extracting the feature map FM3, the feature map FM4, and the feature map FM5; and the second DNN processing unit <NUM> may execute the process of the FCN <NUM> alone. Processes to be executed by the first DNN processing unit <NUM> may be arbitrarily decided. For example, which process is to be executed by the first DNN processing unit <NUM> may be decided in accordance with performance of the first DNN processing unit <NUM>.

With reference to <FIG>, one example of processes to be executed by the first DNN processing unit <NUM> and the second DNN processing unit <NUM> will be explained. <FIG> is a diagram illustrating one example of processes to be executed by the first DNN processing unit <NUM> and the second DNN processing unit <NUM>.

First, in the process illustrated in <FIG>, for example, a plurality of pieces of image data is input to the first DNN processing unit <NUM> from the capture processing unit <NUM> (Step S11).

Next, the first DNN processing unit <NUM> executes an image recognizing process on the image data received from the capture processing unit <NUM> so as to recognize an object included in image data (Step S12). Specifically, the first DNN processing unit <NUM> executes CNN on each piece of image data so as to recognize an object included in image data. The first DNN processing unit <NUM> generates metadata from an execution result of CNN for each piece of image data.

Next, the second DNN processing unit <NUM> recognizes, by using a Recurrent Neural Network (RNN), relationship of metadata generated by the first DNN processing unit <NUM> (Step S13). Specifically, the second DNN processing unit <NUM> recognizes relationship of metadata by using a Long short-term memory (LSTM) network.

The second DNN processing unit <NUM> recognizes relationship of metadata so as to execute captioning (Step S14). For example, the second DNN processing unit <NUM> executes captioning on image data, such as "boy", "playing", and "golf".

As described above, when object recognition and LSTM are combined with each other, relationship between image frames is able to be recognized. In this case, in the present embodiment, object recognition is executed by the first DNN processing unit <NUM> and LSTM is executed by the second DNN processing unit <NUM>, and single DNN algorithm is divided and executed. Herein, the case has been explained in which a plurality of still images is input, in the present embodiment, similarly thereto, the recognition process may be executed on videos.

With reference to <FIG>, processes of the solid-state image capturing device <NUM> and the information processing device <NUM> will be explained. <FIG> is a sequence diagram illustrating flows of processing of the solid-state image capturing device <NUM> and the information processing device <NUM>.

First, the solid-state image capturing device <NUM> controls the first DNN processing unit <NUM> (Step S101). Specifically, the solid-state image capturing device <NUM> causes the first control unit <NUM> to control the first DNN processing unit <NUM>.

Next, the solid-state image capturing device <NUM> executes a first DNN on input image data (Step S102). Specifically, the solid-state image capturing device <NUM> causes the first DNN processing unit <NUM> to execute a first DNN on input image data, and outputs first result.

Next, the solid-state image capturing device <NUM> generates first control information (Step S103). Specifically, the solid-state image capturing device <NUM> causes the first control unit <NUM> to generate first control information.

Next, the solid-state image capturing device <NUM> transmits the first result and the first control information to the information processing device <NUM> (Step S104). Specifically, the solid-state image capturing device <NUM> causes the transmitting unit <NUM> to transmit the first result and the first control information to the information processing device <NUM>. The transmitting unit <NUM> may transmit the first result to the information processing device <NUM> before execution of the second DNN processing unit <NUM>.

Next, the information processing device <NUM> controls the second DNN processing unit <NUM> (Step S105). Specifically, on the basis of the first control information, the information processing device <NUM> causes the second control unit <NUM> to control the second DNN processing unit <NUM>.

Next, the information processing device <NUM> executes a second DNN on the first result (Step S106). Specifically, the information processing device <NUM> causes the second DNN processing unit <NUM> to execute a second DNN so as to generate second result.

Next, the information processing device <NUM> generates second control information (Step S107). Specifically, the information processing device <NUM> causes the second control unit <NUM> to generate the second control information.

Next, the information processing device <NUM> transmits the second result to an external device (Step S108). Specifically, the information processing device <NUM> causes the second DNN processing unit <NUM> to transmit the second result to an external device. Note that in Step S108, the case has been explained in which the information processing device <NUM> transmits the second result to the external device; however, this is merely one example, and not intending to limit the present disclosure. For example, in Step S108, the information processing device <NUM> may hold the second result without outputting the second result to the external device.

The information processing device <NUM> transmits second control information to the solid-state image capturing device <NUM> (Step S109). Specifically, the information processing device <NUM> causes the transmitting unit <NUM> to transmit second control information to the solid-state image capturing device <NUM>.

With reference to <FIG>, the processing of the solid-state image capturing device <NUM> and the information processing device <NUM> will be more specifically explained. <FIG> is a sequence diagram illustrating one example of a processing procedure to be executed by the first DNN processing unit <NUM>, the first control unit <NUM>, the second DNN processing unit <NUM>, and the second control unit <NUM>.

First, the first control unit <NUM> outputs first DNN-processing-unit controlling information to the first DNN processing unit <NUM> (Step S201). By using the first DNN-processing-unit controlling information, the first control unit <NUM> causes the first DNN processing unit <NUM> to execute DNN, interrupt DNN, or stop DNN.

Next, in accordance with first DNN-processing-unit controlling information, the first DNN processing unit <NUM> executes a first DNN on the input image data (Step S202).

Next, when the execution of the first DNN on the image data has been completed, the first DNN processing unit <NUM> outputs a completion notification to the first control unit <NUM> (Step S203).

Next, when receiving a completion notification, the first control unit <NUM> transmits first control information to the second control unit <NUM> (Step S204). Specifically, first control information is transmitted from the transmitting unit <NUM> to the reception unit <NUM>. The second control unit <NUM> receives first control information from the reception unit <NUM>.

Next, the first DNN processing unit <NUM> transmits, to the second DNN processing unit <NUM>, first result that is execution result of a first DNN (Step S205). Specifically, first result is transmitted from the transmitting unit <NUM> to the reception unit <NUM>. The second DNN processing unit <NUM> receives first result from the reception unit <NUM>.

Next, on the basis of the first control information, the second control unit <NUM> outputs second DNN-processing-unit controlling information to the second DNN processing unit <NUM> (Step S206).

Next, in accordance with second DNN-processing-unit controlling information, the second DNN processing unit <NUM> executes a second DNN on the input first result (Step S207).

Next, when execution of the second DNN on the first result has completed, the second DNN processing unit <NUM> outputs a completion notification to the second control unit <NUM> (Step S208).

Next, the second DNN processing unit <NUM> transmits, to the outside thereof, second result that is an execution result of the second DNN (Step S209). Note that in Step S209, the case has been explained in which the second DNN processing unit <NUM> transmits second result to the outside thereof; however, this is merely one example, and not intending to limit the present disclosure. For example, in Step S209, the second DNN processing unit <NUM> may hold the second result without outputting the second result to the outside thereof.

Next, the second control unit <NUM> transmits second control information to the first control unit <NUM> (Step S210). Specifically, second control information is transmitted from the transmitting unit <NUM> to the reception unit <NUM>. The first control unit <NUM> receives first control information from the reception unit <NUM>.

With reference to <FIG>, a configuration of a solid-state image capturing system according to a second embodiment will be explained. <FIG> is a block diagram illustrating one example of a configuration of the solid-state image capturing system according to the second embodiment.

As illustrated in <FIG>, a solid-state image capturing system 1A includes the solid-state image capturing device <NUM> and an information processing device 200A. Configuration elements and operations of devices constituting the solid-state image capturing system 1A are similar to those of the solid-state image capturing system <NUM> according to the first embodiment, and thus explanation thereof is omitted.

As illustrated in <FIG>, the solid-state image capturing device <NUM> and the information processing device 200A are connected to be able to communicate with each other via the Internet communication network <NUM>, for example. In this case, it is sufficient that the communication I/F <NUM> of the solid-state image capturing device <NUM> and the communication I/F <NUM> of the information processing device 200A are connected to be able to communicate with each other via the Internet communication network <NUM>. Moreover, the communication I/F <NUM> of the solid-state image capturing device <NUM> and the communication I/F <NUM> of the information processing device 200A may be connected to be able to communicate with each other by using wireless communication. The solid-state image capturing system 1A illustrated in <FIG> includes the single solid-state image capturing device <NUM> and the single information processing device 200A; however, this is merely one example, and not intending to limit the present disclosure. The solid-state image capturing system 1A may include the plurality of solid-state image capturing devices <NUM> and the plurality of information processing devices 200A. Furthermore, the number of the solid-state image capturing devices <NUM> and that of the information processing devices 200A included in the solid-state image capturing system 1A may be different from each other. In the second embodiment, the information processing device 200A is a cloud server that is connected to the solid-state image capturing device <NUM> to be able to communicate with each other via the Internet communication network <NUM> or in a wireless manner, for example. The solid-state image capturing system 1A may be applied to Factory Automation (FA) or monitoring cameras, for example.

With reference to <FIG>, a modification of the solid-state image capturing system according to the second embodiment of the present disclosure will be explained. <FIG> is a diagram illustrating a modification of a connection relation of the solid-state image capturing system according to the second embodiment of the present disclosure.

A solid-state image capturing system 1A-<NUM> includes a solid-state image capturing device <NUM>-<NUM>, a solid-state image capturing device <NUM>-<NUM>, ···, and a solid-state image capturing device <NUM>-N (N is integer equal to or more than three) and an information processing device 200A-<NUM>, an information processing device 200A-<NUM>, ···, an information processing device <NUM>-N. In other words, in the solid-state image capturing system 1A-<NUM>, the plurality of solid-state image capturing devices and the plurality of information processing devices are connected to be able to communicate with each other via the Internet communication network <NUM>. In the solid-state image capturing system 1A-<NUM>, the number of the solid-state image capturing devices and the number of information processing devices may be equal to each other, or may be different from each other.

The solid-state image capturing system 1A-<NUM> includes the plurality of solid-state image capturing devices and the plurality of information processing devices; however, this is merely one example, and not intending to limit the present disclosure. For example, the solid-state image capturing system 1A-<NUM> may be constituted of the single solid-state image capturing device and the plurality of information processing devices. Moreover, for example, the solid-state image capturing system 1A-<NUM> may be constituted of the plurality of solid-state image capturing devices and the single information processing device.

The solid-state image capturing device <NUM> and the information processing device <NUM> according to the above-mentioned embodiments are realized by a computer <NUM> having a configuration illustrated in <FIG>, for example. Hereinafter, explanation will be provided while exemplifying the solid-state image capturing device <NUM> according to the first embodiment. <FIG> is a diagram illustrating a hardware configuration of one example of the computer <NUM> that realizes functions of the solid-state image capturing device <NUM>. The computer <NUM> includes a CPU <NUM>, a RAM <NUM>, a Read Only Memory (ROM) <NUM>, a Hard Disk Drive (HDD) <NUM>, a communication interface <NUM>, and an input/output interface <NUM>. The units of the computer <NUM> are connected to each other by a bus <NUM>.

The CPU <NUM> operates on the basis of programs stored in the ROM <NUM> or the HDD <NUM> so as to control the units. For example, the CPU <NUM> expands programs stored in the ROM <NUM> or the HDD <NUM> into the RAM <NUM> so as to execute processes corresponding to the various programs.

The ROM <NUM> stores therein a boot program of a Basic Input Output System (BIOS) which is executed by the CPU <NUM> at start-up of the computer <NUM>, and a program depending on hardware of the computer <NUM>.

The HDD <NUM> is a computer-readable recording medium that non-temporarily records therein programs to be executed by the CPU <NUM> and data to be used by the programs. Specifically, the HDD <NUM> is a recording medium that records a program according to the present disclosure that is one example of program data <NUM>.

The communication interface <NUM> is an interface for the computer <NUM> to connect to an external network <NUM> (for example, Internet). For example, via the communication interface <NUM>, the CPU <NUM> is configured to receive data from another device, or transmit data generated by the CPU <NUM> to another device.

The input/output interface <NUM> is an interface for connecting an input/output device <NUM> and the computer <NUM> to each other. For example, the CPU <NUM> receives data from an input device, such as a keyboard and a mouse, via the input/output interface <NUM>. The CPU <NUM> transmit data to an output device such as a display, a speaker, and a printer via the input/output interface <NUM>. The input/output interface <NUM> may function as a media interface that reads a program and the like recorded in predetermined recording medium (media). Herein, the media is, for example, optical storage medium such as a Digital Versatile Disc (DVD) and a Phase change rewritable Disk (PD), magneto-optic recording medium such as a Magneto-Optical disk (MO), tape medium, magnetic recording medium, a semiconductor memory, or the like.

For example, when the computer <NUM> functions as the solid-state image capturing device <NUM> according to the first embodiment, the CPU <NUM> of the computer <NUM> executes a program loaded on the RAM <NUM> so as to realize functions of units constituting the solid-state image capturing device <NUM>. The HDD <NUM> stores therein programs according to the present disclosure. The CPU <NUM> reads the program data <NUM> from the HDD <NUM> and executes the read program data <NUM>; however, in another example, the CPU <NUM> may acquire these programs from another device via the external network <NUM>.

A technology (present technology) according to the present disclosure may be applied to various products. For example, the technology according to the present disclosure may be applied to an endoscopic surgery system.

<FIG> is a diagram illustrating one example of a schematic configuration of an endoscopic surgery system to which a technology (present technology) according to the present disclosure is applied.

In <FIG>, there is illustrated a state where an operator (doctor) <NUM> performs an operation on a patient <NUM> on a patient bed <NUM> by using an endoscopic surgery system <NUM>. As illustrated in <FIG>, the endoscopic surgery system <NUM> is constituted of an endoscope <NUM>, other surgical instruments <NUM> such as a pneumoperitoneum tube <NUM> and an energy processing apparatus <NUM>, a support arm device <NUM> that supports the endoscope <NUM>, and a cart <NUM> on which various devices for endoscopic operations are mounted.

The endoscope <NUM> is constituted of a lens barrel <NUM> whose portion having a predetermined length from its leading end is to be inserted into a body cavity of the patient <NUM>, and a camera head <NUM> that is connected with a base end of the lens barrel <NUM>. In the illustrated example, there is exemplified the endoscope <NUM> configured to be a rigid endoscope including the rigid lens barrel <NUM>; however, the endoscope <NUM> may be configured to be a flexible endoscope including a flexible lens barrel.

A leading end of the lens barrel <NUM> is provided with an opening to which an objective lens is fixed. The endoscope <NUM> is connected with a light source device <NUM>, and thus light generated by the light source device <NUM> is led to a leading end of the lens barrel via a light guide extending along an inner part of the lens barrel <NUM> so as to irradiate, via the objective lens, the generated light toward an observation target in a body cavity of the patient <NUM>. Note that the endoscope <NUM> may be any of a forwardviewing endoscope, a forward-oblique viewing endoscope, and a side-viewing endoscope.

An optical system and an image capturing element are arranged in an inner part of the camera head <NUM>, reflected light (observation light) from an observation target is condensed to the image capturing element by the optical system. The image capturing element photo-electrically converts the observation light so as to generate an electrical signal corresponding to the observation light, namely, an image signal corresponding to an observation figure. The above-mentioned image signal is transmitted to a Camera Control Unit (CCU) <NUM> in a form of RAW data.

The CCU <NUM> is constituted of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and the like so as to comprehensively control operations of the endoscope <NUM> and a display <NUM>. Moreover, the CCU <NUM> receives an image signal from the camera head <NUM>, and executes, on the image signal, various kinds of image processing for displaying an image that is based on the image signal, such as a developing process (demosaicing process).

Under control by the CCU <NUM>, the display <NUM> displays an image that is based on an image signal on which image processing is executed by the CCU <NUM>.

The light source device <NUM> is constituted of a light source such as a Light Emitting Diode (LED) so as to supply irradiation light in capturing an operative part and the like to the endoscope <NUM>.

An input device <NUM> is an input interface with respect to the endoscopic surgery system <NUM>. Via the input device <NUM> and to the endoscopic surgery system <NUM>, a user is able to input various kinds of information and give input instructions. For example, a user is able to input an instruction for changing a capturing condition (type, magnification, and focus distance of irradiation light, etc.) of the endoscope <NUM>.

A treatment-tool controller <NUM> controls operations of the energy processing apparatus <NUM> for cautery and incision of a tissue, sealing of a blood vessel, and the like. In order to inflate a body cavity of the patient <NUM> for ensuring view of the endoscope <NUM> and a working space of an operator, a pneumoperitoneum device <NUM> delivers gas into the body cavity via the pneumoperitoneum tube <NUM>. A recorder <NUM> is a device capable of recording various kinds of information on operations. A printer <NUM> is a device that is capable of printing the various kinds of information on operations in a form of text, image, graph, etc..

The light source device <NUM> that supplies, to the endoscope <NUM>, irradiation light in capturing an operative part may be constituted of a white light source that includes an LED, a laser light source, or a combination thereof, for example. When the white light source is constituted of a combination of RGB laser light sources, an output intensity and an output timing of each color (each wavelength) are able to be controlled with high accuracy, so that it is possible to adjust a white balance of a captured image by using the light source device <NUM>. In this case, laser light may be irradiated to an observation target on time-sharing basis from each of the RGB laser light sources, and operations of an image capturing element of the camera head <NUM> may be controlled in synchronization with the corresponding irradiation timing to capture images corresponding to respective RGB on time-sharing basis. By employing the above-mentioned method, it is possible to obtain color images without providing a color filter to the image capturing element.

Moreover, operations of the light source device <NUM> may be controlled such that an intensity of output light is changed at predetermined time intervals. Operations of an image capturing element of the camera head <NUM> is controlled in synchronization with a change timing in a light intensity so as to acquire images on time-sharing basis, and the acquired images are synthesized to be able to generate an image having a high dynamic range without black defects and halation.

The light source device <NUM> may be configured to be capable of supplying light within a predetermined wavelength band corresponding to a special light observation. In the special light observation, by using wavelength dependence of light absorbance in a body tissue, for example, a light having a narrower band than that of irradiation light (namely, white light) used in a common observation is irradiated so as to capture a predetermined tissue such as blood vessels in a mucosal surface with a high contrast, namely, a narrow-band light observation (Narrow Band Imaging) is executed. Or in the special light observation, a fluorescence observation may be executed in which images are obtained by using fluorescence that is generated by irradiation of excited light. In the fluorescence observation, for example, excited light is irradiated to a body tissue so as to observe fluorescence from the body tissue (autofluorescence observation), or a reagent such as indocyanine green (ICG) is locally injected into a body tissue and excited light corresponding to a fluorescence wavelength of the reagent is irradiated to the body tissue so as to obtain a fluorescence image. The light source device <NUM> may be configured to supply narrow band light and/or excited light corresponding to such a special light observation.

<FIG> is a block diagram illustrating one example of functional configurations of the camera head <NUM> and the CCU <NUM> illustrated in <FIG>.

The camera head <NUM> includes a lens unit <NUM>, an image capturing unit <NUM>, a drive unit <NUM>, a communication unit <NUM>, and a camera-head controlling unit <NUM>. The CCU <NUM> includes a communication unit <NUM>, an image processing unit <NUM>, and a control unit <NUM>. The camera head <NUM> and the CCU <NUM> are connected to be able to communicate with each other by a transmission cable <NUM>.

The lens unit <NUM> is an optical system arranged at a connection portion with the lens barrel <NUM>. Observation light captured from a leading end of the lens barrel <NUM> is led to the camera head <NUM> to be made incident on the lens unit <NUM>. The lens unit <NUM> is constituted of a plurality of combined lenses including a zoom lens and a focus lens.

The image capturing unit <NUM> is constituted of an image capturing element. The number of the image capturing elements constituting the image capturing unit <NUM> may be one (namely, single plate type) or two or more (namely, multiple plate type). When the image capturing unit <NUM> has a multiple plate type, for example, image capturing elements may generate image signals respectively corresponding to RGB, and the generated image signals may be synthesized with each other so as to obtain a color image. Or the image capturing unit <NUM> may be configured to include a pair of image capturing elements for acquiring respective image signals for the right eye and the left eye, which are corresponding to three-dimensional (3D) display. When 3D display is performed, the operator <NUM> is able to more precisely grasp a depth of a biological tissue in an operative part. Note that when the image capturing unit <NUM> has a multiple plate type, the plurality of lens units <NUM> may be also provided in accordance with the respective image capturing elements.

The image capturing unit <NUM> may be not always arranged in the camera head <NUM>. For example, the image capturing unit <NUM> may be arranged just behind an objective lens in the lens barrel <NUM>.

The drive unit <NUM> is constituted of an actuator, and caused by the control of the camera-head controlling unit <NUM>, moves a zoom lens and a focus lens of the lens unit <NUM> along an optical axis by a predetermined distance. Thus, a magnification and a focus of a captured image captured by the image capturing unit <NUM> are appropriately adjusted.

The communication unit <NUM> is constituted of a communication device for transmitting and receiving various kinds of information to and from the CCU <NUM>. The communication unit <NUM> transmits, in a form of RAW data, an image signal obtained from the image capturing unit <NUM> to the CCU <NUM> via the transmission cable <NUM>.

The communication unit <NUM> receives, from the CCU <NUM>, a control signal for controlling operations of the camera head <NUM>, and supplies the received control signal to the camera-head controlling unit <NUM>. The control signal includes information on capturing conditions such as information that specifies a frame rate of a captured image, information that specifies an exposure value in capturing, and/or information that specifies a magnification and a focus of a captured image.

The above-mentioned capturing conditions of a frame rate, an exposure value, a magnification, a focus, and the like may be appropriately specified by a user, or may be automatically set by the control unit <NUM> of the CCU <NUM> on the basis of an acquired image signal. In the latter case, i.e. an Auto Exposure (AE) function and an Auto White Balance (AWB) function may be implemented to the endoscope <NUM>.

The camera-head controlling unit <NUM> controls operations of the camera head <NUM> on the basis of a control signal transmitted from the CCU <NUM> and received via the communication unit <NUM>.

The communication unit <NUM> is constituted of a communication device that transmits and receives various kinds of information to and from the camera head <NUM>. The communication unit <NUM> receives an image signal transmitted via the transmission cable <NUM> from the camera head <NUM>.

The communication unit <NUM> transmits, to the camera head <NUM>, a control signal for controlling operations of the camera head <NUM>. The image signal and the control signal are transmitted by using electrical communication or light communication.

The image processing unit <NUM> executes various kinds of image processing on an image signal of RAW data transmitted from the camera head <NUM>.

The control unit <NUM> executes various kinds of control on capturing of an operative part and the like by the endoscope <NUM> and displaying of a captured image obtained by the capturing of the operative part and the like. For example, the control unit <NUM> generates a control signal for controlling operations of the camera head <NUM>.

On the basis of an image signal on which image processing is executed by the image processing unit <NUM>, the control unit <NUM> causes the display <NUM> to display a captured image on which an operative part and the like appears. In this case, the control unit <NUM> may recognize various objects in a captured image by using various image recognizing technologies. For example, the control unit <NUM> may detect shapes, colors, etc. of an edge of an object included in a captured image so as to recognize a surgical instrument such as a forceps, a specific biological part, hemorrhage, mist in using the energy processing apparatus <NUM>. When causing the display <NUM> to display the captured image, the control unit <NUM> may display an image of the operative part while superposing thereon various kinds of operation assisting information by using the recognition result. When the operation assisting information is displayed in a superposed manner to be presented to the operator <NUM>, a burden of the operator <NUM> is able to be reduced, and further the operator <NUM> is able to reliably carry out his/her operation.

The transmission cable <NUM> connecting the camera head <NUM> and the CCU <NUM> to each other may be an electric signal cable corresponding to communication of electrical signals, an optical fiber corresponding to optical communication, or a combination cable thereof.

In the illustrated example, communication is carried out in a wired manner by using the transmission cable <NUM>; however, communication between the camera head <NUM> and the CCU <NUM> may be wirelessly carried out.

Hereinafter, one example of an endoscopic surgery system has been explained, to which the technology according to the present disclosure is applied. The technology according to the present disclosure may be applied to, for example, the endoscope <NUM>, the image capturing unit <NUM> of the camera head <NUM>, the image processing unit <NUM> of the CCU <NUM>, and the like among the above-mentioned configurations. Specifically, the solid-state image capturing device <NUM> according to the present disclosure may be applied to the endoscope <NUM>, the image capturing unit <NUM> of the camera head <NUM>, the image processing unit <NUM> of the CCU <NUM>, and the like. When the technology according to the present disclosure is applied thereto, performance of the endoscopic surgery system is able to be improved. For example, when the solid-state image capturing device <NUM> is employed whose dynamic range is enlarged, it is possible to obtain high-definition captured images. Specifically, an object is easily recognized in capturing the inside and the outside of a living body even in a case of a position whose difference in brightness is large. Moreover, fast operation of the solid-state image capturing device is realized, so that it is possible to shorten a time interval needed for operation from a time when an object is detected to a time when the camera head <NUM> and the like are controlled.

The technology (present technology) according to the present disclosure may be applied to various products. For example, the technology according to the present disclosure may be realized as a device that is provided in a moving body such as an automobile, an electric automobile, a hybrid electric automobile, a motorcycle, a bicycle, a personal mobility, an airplane, a drone, a vessel, and a robot.

<FIG> is a block diagram illustrating one example of a schematic configuration of a vehicle controlling system that is one example of a moving-body control system to which the technology according to the present disclosure is to be applied.

A vehicle controlling system <NUM> includes a plurality of electrical control units that is connected to each other via a communication network <NUM>. In the example illustrated in <FIG>, the vehicle controlling system <NUM> includes a drive-system controlling unit <NUM>, a body-system controlling unit <NUM>, an out-of-vehicle information detecting unit <NUM>, an in-vehicle information detecting unit <NUM>, and an integrated control unit <NUM>. As a functional configuration of the integrated control unit <NUM>, there are illustrated a microcomputer <NUM>, an audio/video outputting unit <NUM>, and an on-vehicle network I/F (interface) <NUM>.

The drive-system controlling unit <NUM> controls operations of devices related to a drive system of a vehicle in accordance with various programs. For example, the drive-system controlling unit <NUM> functions as a controller of a driving-force generating device that generates driving force of a vehicle such as an internal combustion engine and a drive motor, a driving-force transmitting mechanism that transmits driving force to wheels, a steering mechanism that adjusts a steering angle of a vehicle, a braking device that generates braking force of a vehicle, and the like.

The body-system controlling unit <NUM> controls operations of various devices provided in a vehicle body in accordance with various programs. For example, the body-system controlling unit <NUM> functions as a controller of a keyless entry system, a smart key system, a power window device, or various lamps such as a headlight, a back lamp, a brake lamp, a blinker, or a fog lamp. In this case, an electrical wave sent from a portable device substituted for a key or signals of various switches may be input to the body-system controlling unit <NUM>. The body-system controlling unit <NUM> receives input of these electrical waves or signals so as to control a door locking device, a power window device, a lamp, etc. of a vehicle.

The out-of-vehicle information detecting unit <NUM> detect information on the outside of a vehicle in which the vehicle controlling system <NUM> is provided. For example, the out-of-vehicle information detecting unit <NUM> is connected to an image capturing unit <NUM>. The out-of-vehicle information detecting unit <NUM> causes the image capturing unit <NUM> to capture images of the outside of a vehicle, and further receives the captured images. On the basis of the received images, the out-of-vehicle information detecting unit <NUM> may execute a detection process for detecting an object such as a human being, an automobile, an obstacle, a sign, and letters on a road surface; or a distance detecting process.

The image capturing unit <NUM> is a light sensor that receives light, and outputs an electrical signal according to a light amount of the received light. The image capturing unit <NUM> may output the electrical signal as an image, or as information on a measured distance. The light received by the image capturing unit <NUM> may be visible light or invisible light such as infrared ray.

The in-vehicle information detecting unit <NUM> detects information on the inside of a vehicle. The in-vehicle information detecting unit <NUM> is connected to a driver-state detecting unit <NUM> that detects a state of a driver, for example. The driver-state detecting unit <NUM> may include a camera that captures a driver, for example, and on the basis of detection information input from the driver-state detecting unit <NUM>, the in-vehicle information detecting unit <NUM> may calculate fatigue degree or concentration degree of the driver, or may determine whether or not the driver takes a nap.

On the basis of information on the outside and/or the inside of a vehicle that is acquired by the out-of-vehicle information detecting unit <NUM> or the in-vehicle information detecting unit <NUM>, the microcomputer <NUM> may calculate a control target value of a driving-force generating device, a steering mechanism, or a braking device so as to output a control command to the drive-system controlling unit <NUM>. For example, the microcomputer <NUM> is capable of executing cooperative control for realizing function of an Advanced Driver Assistance System (ADAS) including collision avoidance or shock absorbing of a vehicle, follow-up driving based on an inter-vehicle distance, constant-speed driving, collision warning of a vehicle, or lane departure warning of a vehicle.

The microcomputer <NUM> controls a driving-force generating device, a steering mechanism, a braking device, or the like on the basis of information on the periphery of a vehicle which is acquired by the out-of-vehicle information detecting unit <NUM> or the in-vehicle information detecting unit <NUM>, so as to execute cooperative control for autonomous driving that autonomously drives independent from operations of a driver.

On the basis of information on the outside of a vehicle which is acquired by the out-of-vehicle information detecting unit <NUM>, the microcomputer <NUM> outputs a control command to the body-system controlling unit <NUM>. For example, in accordance with a position of a preceding vehicle or an oncoming car which is detected by the out-of-vehicle information detecting unit <NUM>, the microcomputer <NUM> controls a headlight so as to execute cooperative control for antiglare such as changing high beam into low beam.

The audio/video outputting unit <NUM> transmits, to an output device capable of visually or aurally providing information to an occupant of a vehicle or the outside of a vehicle, an output signal in a form of one of sounds and images. In the example illustrated in <FIG>, an audio speaker <NUM>, a display <NUM>, and an instrument panel <NUM> are exemplified as the output devices. The display <NUM> may include at least one of an onboard display and a head up display, for example.

<FIG> is a diagram illustrating one example of arrangement positions of the image capturing units <NUM>.

In <FIG>, a vehicle <NUM> includes image capturing units <NUM>, <NUM>, <NUM>, <NUM>, and <NUM> as the image capturing units <NUM>.

For example, the image capturing units <NUM>, <NUM>, <NUM>, <NUM>, and <NUM> are arranged at positions in the vehicle <NUM>, for example, a front nose, a side mirror, a rear bumper, a back door, an upper part of a windscreen in the interior of a vehicle, etc. The image capturing unit <NUM> arranged at a front nose and the image capturing unit <NUM> arranged at an upper part of a windscreen in the interior of a vehicle mainly acquire images in front of the vehicle <NUM>. The image capturing units <NUM> and <NUM> arranged at side mirrors mainly acquire side images of the vehicle <NUM>. The image capturing unit <NUM> arranged at a rear bumper or a back door acquires an image in the rear of the vehicle <NUM>. Front images acquired by the image capturing units <NUM> and <NUM> are mainly used for detecting a preceding vehicle, a pedestrian, an obstacle, a traffic light, a road sign, a lane, or the like.

In <FIG>, there is illustrated one example of capturing ranges of the image capturing units <NUM> to <NUM>. A capturing range <NUM> indicates a capturing range of the image capturing unit <NUM> arranged at a front nose, capturing ranges <NUM> and <NUM> indicate capturing ranges of the image capturing units <NUM> and <NUM> arranged at respective side mirrors, and a capturing range <NUM> indicates a capturing range of the image capturing unit <NUM> arranged at a rear bumper or a back door. For example, image data obtained by the image capturing units <NUM> to <NUM> are overlapped with each other to obtain a bird's-eye image viewed from above the vehicle <NUM>.

At least one of the image capturing units <NUM> to <NUM> may have function for acquiring distance information. For example, at least one of the image capturing units <NUM> to <NUM> may be a stereo camera constituted of a plurality of image capturing elements, or may be an image capturing element including pixels for detecting phase difference.

For example, on the basis of distance information obtained from the image capturing units <NUM> to <NUM>, the microcomputer <NUM> obtains distances up to a three-dimensional object in the capturing ranges <NUM> to <NUM> and temporal changes (relative velocity with respect to vehicle <NUM>) in the distances to be able to extract, as a preceding vehicle, a three-dimensional object that is present on a travelling road of the vehicle <NUM> and closest to the vehicle <NUM>, and is traveling at a predetermined velocity (for example, equal to or more than <NUM>/h) in a direction similar to that of the vehicle <NUM>. Moreover, the microcomputer <NUM> may set an inter-vehicle distance to be kept up to a preceding vehicle so as to execute automatic braking control (including follow-up stopping control), automatic acceleration control (including follow-up starting control), and the like. As described above, cooperative control is able to be executed for autonomous driving aimed at autonomous traveling independent from operation of a driver and the like.

For example, on the basis of distance information obtained from the image capturing units <NUM> to <NUM>, the microcomputer <NUM> classifies and extracts three-dimensional object data on three-dimensional objects into a motorcycle, a standard-sized vehicle, a large-sized vehicle, a pedestrian, another three-dimensional object such as a utility pole to be able to use it for automatic avoidance of an obstacle. For example, the microcomputer <NUM> determines an obstacle in the periphery of the vehicle <NUM> into an obstacle that is able to be visually recognized by a driver of the vehicle <NUM> or an obstacle whose visual recognition is difficult. The microcomputer <NUM> determines a collision risk indicating a risk of collision with each obstacle, and in a case of a situation where a collision risk is equal to or more than a set value and there presents possibility of collision, causes the audio speaker <NUM> or the display <NUM> to output a warning to a driver, or causes the drive-system controlling unit <NUM> to execute forced deceleration or avoidance steering, so as to assist in driving for collision avoidance.

At least one of the image capturing units <NUM> to <NUM> may be an infrared camera that detects infrared ray. For example, the microcomputer <NUM> determines whether or not there presents a pedestrian in captured images of the image capturing units <NUM> to <NUM> to be able to recognize a pedestrian. The above-mentioned recognition of a pedestrian is constituted of a procedure for extracting feature points from captured images of the image capturing units <NUM> to <NUM> as infrared cameras, and a procedure for executing a pattern matching process on a series of feature points indicating an outline of an object so as to determine whether or not there presents a pedestrian, for example. When the microcomputer <NUM> determines that there presents a pedestrian in captured images of the image capturing units <NUM> to <NUM> to recognize the pedestrian, the audio/video outputting unit <NUM> controls the display <NUM> such that a rectangular outline for emphasis is displayed in a superposed manner on the recognized pedestrian. The audio/video outputting unit <NUM> may control the display <NUM> such that an icon and the like indicating a pedestrian is displayed at a desired position.

Claim 1:
A solid-state image capturing system (<NUM>) comprising:
a solid-state image capturing device (<NUM>); and
an information processing device (<NUM>), wherein
the solid-state image capturing device (<NUM>) includes:
a first Deep-Neural-Network (DNN) processing unit (<NUM>) that executes, on image data, a part of a DNN algorithm by a first DNN to generate a first result, and
the information processing device includes:
a second DNN processing unit (<NUM>) that executes, on the first result acquired from the solid-state image capturing device (<NUM>), remaining of the DNN algorithm by a second DNN to generate a second result,
wherein
the solid-state image capturing device (<NUM>) further includes:
a first control unit (<NUM>) that controls the first DNN processing unit (<NUM>), and
the information processing device (<NUM>) further includes:
a second control unit (<NUM>) that controls the second DNN processing unit (<NUM>),
the first control unit (<NUM>) generating first control information including information on the first DNN, and transmitting the generated first control information to the second control unit (<NUM>), and
the second control unit (<NUM>) generating second control information including information on the second DNN, and transmitting the generated second control information to the first control unit (<NUM>),
wherein
the first control unit (<NUM>) controls the first DNN processing unit based on the second control information, and
the second control unit (<NUM>) controls the second DNN processing unit based on the first control information.