Patent Description:
Conventionally, flat-type image sensors in which chips such as a sensor chip, a memory chip, and a digital signal processor (DSP) chip are connected in parallel with a plurality of bumps exist as imaging devices that acquire still images and moving images.

In recent years, one-chip image sensors having a stack structure in which a plurality of dies are stacked have been developed for the purpose of miniaturization of imaging devices.

Patent Literature <NUM>: <CIT>
<CIT> discloses a semiconductor device, a solid-state image sensor and a camera system capable of reducing the influence of noise at a connection between chips without a special circuit for communication and reducing the cost as a result. The semiconductor device includes a first chip, a second chip, wherein the first chip and the second chip are bonded to have a stacked structure, the first chip has a high-voltage transistor circuit mounted thereon, the second chip has mounted thereon a low-voltage transistor circuit having lower breakdown voltage than the high-voltage transistor circuit, and wiring between the first chip and the second chip is connected through a wire formed in the first chip. <CIT> discloses a camera module including an objective lens housing with an optical system, an image sensor chip with wire bonding connections, and a printed circuit board for contacting the image sensor chip. <CIT> discloses a vehicle-mounted camera including a camera main assembly including a lens assembly, an image sensor located behind the lens assembly, and a camera circuit board located behind the image sensor and mounted with the image sensor, a tabular main circuit board located on the lower side of the camera main assembly and extending in the front-back direction and the left-right direction. A connection flexible board that electrically connects the camera circuit board and the main circuit board is provided. <CIT> discloses a camera module for a vision system of a vehicle which includes a housing, a lens and at least one circuit board having circuitry disposed in the housing with an imaging array sensor disposed at the circuit board. A stacked image sensor comprising a first and a second substrate is known from <CIT>. The first substrate comprises a pixel array, and the second substrate comprises reconfigurable components that can be programmed. <CIT> discloses a small, high-performance imaging device and its application to products at low cost by preventing noise superimposed on a timing pulse feed line from affecting the output of an imaging chip. The imaging device includes two chips: An imaging chip including a sensor and an image processing chip including an image processing circuit. The transistors of all circuits in the imaging chip are formed as either nMOS or pMOS transistors. The imaging chip is stacked on the image processing chip.

Unless explicitly indicated as "embodiment(s) according to the claimed invention", any embodiment [example, aspect, implementation,. ] in the description may include some but not all features as literally defined in the claims and are present for illustration purposes only.

In recent years, more advanced processing in image sensor chips is desired in terms of increasing the variety and the speed of image processing and protection of private information, for example.

The present disclosure develops a stacked light-receiving sensor and an in-vehicle imaging device capable of performing more advanced processing in a chip.

According to the claimed invention, a stacked light-receiving sensor and an in-vehicle imaging device as defined in the appended claims are provided.

Embodiments of the present disclosure will be described in detail below with reference to the drawings. In the following embodiments, the same parts are denoted by the same reference signs and an overlapping description will be omitted.

The present disclosure will be described in the order of items below.

First of all, a first embodiment will be described in detail with reference to the drawings.

<FIG> is a block diagram illustrating an overall configuration example of an imaging device as an electronic device according to the first embodiment. As illustrated in <FIG>, an imaging device <NUM> includes an image sensor <NUM> that is a solid-state imaging device and an application processor <NUM>. The image sensor <NUM> includes an imager <NUM>, a controller <NUM>, a converter (analog-to-digital converter, hereinafter referred to as ADC) <NUM>, a signal processor <NUM>, a digital signal processor (DSP) <NUM>, a memory <NUM>, and a selector (also referred to as output module) <NUM>.

The controller <NUM> controls each part in the image sensor <NUM> in accordance with user's operation or a set operation mode, for example.

The imager <NUM> includes, for example, an optical system <NUM> including a zoom lens, a focus lens, and an aperture, and a pixel array <NUM> having a configuration in which unit pixels (unit pixels 101a in <FIG>) including light-receiving elements such as photodiodes are arranged in a two-dimensional matrix. External incident light passes through the optical system <NUM> to form an image on a light-receiving surface that is an array of light-receiving elements in the pixel array <NUM>. Each unit pixel 101a in the pixel array <NUM> converts light incident on its light-receiving element into electricity to accumulate charge in accordance with the quantity of incident light so that the charge can be read out.

The ADC <NUM> converts an analog pixel signal for each unit pixel 101a read from the imager <NUM> to a digital value to generate digital image data and outputs the generated image data to the signal processor <NUM> and/or the memory <NUM>. The ADC <NUM> may include a voltage generating circuit that generates a drive voltage for driving the imager <NUM> from power supply voltage and the like.

The signal processor <NUM> performs a variety of signal processing for digital image data input from the ADC <NUM> or digital image data read from the memory <NUM> (hereinafter referred to as process target image data). For example, when the process target image data is a color image, the signal processor <NUM> converts the format of this image data to YUV image data, RGB image data, or the like. The signal processor <NUM> performs, for example, processing such as noise removal and white balance adjustment for the process target image data, if necessary. In addition, the signal processor <NUM> performs a variety of signal processing (also referred to as pre-processing) for the process target image data in order for the DSP <NUM> to process the image data.

The DSP <NUM> executes, for example, a computer program stored in the memory <NUM> to function as a processing unit that performs a variety of processing using a pre-trained model (also referred to as neural network calculation model) created by machine learning using a deep neural network (DNN). This pre-trained model (neural network calculation model) may be designed based on parameters generated by inputting, to a predetermined machine learning model, training data in which an input signal corresponding to output of the pixel array <NUM> is associated with a label for the input signal. The predetermined machine learning model may be a learning model using a multi-layer neural network (also referred to as multi-layer neural network model).

For example, the DSP <NUM> performs a computation process based on the pre-trained model stored in the memory <NUM> to perform a process of combining image data with a dictionary coefficient stored in the memory <NUM>. The result obtained through such a computation process (computation result) is output to the memory <NUM> and/or the selector <NUM>. The computation result may include image data obtained by performing a computation process using the pre-trained model and a variety of information (metadata) obtained from the image data. A memory controller for controlling access to the memory <NUM> may be embedded in the DSP <NUM>.

The image data to be processed by the DSP <NUM> may be image data normally read out from the pixel array <NUM> or may be image data having a data size reduced by decimating pixels of the image data normally read out. Alternatively, the image data to be processed may be image data read out in a data size smaller than normal obtained by performing readout from the pixel array <NUM> with pixels decimated. As used herein "normal readout" may be readout without decimating pixels.

The memory <NUM> stores image data output from the ADC <NUM>, image data subjected to signal processing by the signal processor <NUM>, the computation result obtained from the DSP <NUM>, and the like, if necessary. The memory <NUM> stores an algorithm of the pre-trained model to be executed by the DSP <NUM>, in the form of a computer program and a dictionary coefficient.

The DSP <NUM> can perform the computation process described above by training a learning model by changing the weights of a variety of parameters in the learning model using training data, by preparing a plurality of learning models and changing a learning model to be used in accordance with a computation process, or by acquiring a pre-trained learning model from an external device.

The selector <NUM>, for example, selectively outputs image data output from the DSP <NUM>, or image data or a computation result stored in the memory <NUM>, in accordance with a select control signal from the controller <NUM>. When the DSP <NUM> does not process image data output from the signal processor <NUM> and the selector <NUM> outputs the image data output from the DSP <NUM>, the selector <NUM> outputs the image data output from the signal processor <NUM> as it is.

As described above, the image data or the computation result output from the selector <NUM> is input to the application processor <NUM> that processes display and user interface. The application processor <NUM> is configured, for example, with a central processing unit (CPU) and executes an operating system and a variety of application software. This application processor <NUM> may be equipped with functions such as a graphics processing unit (GPU) and a baseband processor. The application processor <NUM> performs a variety of processes for the input image data or the computation result as necessary, or performs display to users, or transmits the input image data or the computation result to an external cloud server <NUM> through a predetermined network <NUM>.

For example, a variety of networks such as the Internet, a wired local area network (LAN) or a wireless LAN, a mobile communication network, or Bluetooth (registered trademark) can be applied to the predetermined network <NUM>. The image data or the computation result may be transmitted not only to the cloud server <NUM> but also to a variety of information processing devices (systems) having a communication function, such as a server operating on its own, a file server storing a variety of data, and a communication terminal such as a mobile phone.

An example of the chip configuration of the image sensor <NUM> illustrated in <FIG> will now be described in detail below with reference to the drawings.

<FIG> is a diagram illustrating a chip configuration of the image sensor according to the present embodiment. As illustrated in <FIG>, the image sensor <NUM> has a stack structure in which a first substrate (die) <NUM> shaped like a quadrangular flat plate and a second substrate (die) <NUM> similarly shaped like a quadrangular flat plate are bonded together.

The first substrate <NUM> and the second substrate may have the same size, for example. The first substrate <NUM> and the second substrate <NUM> each may be a semiconductor substrate such as a silicon substrate.

In the first substrate <NUM>, in the configuration of the image sensor <NUM> illustrated in <FIG>, the pixel array <NUM> of the imager <NUM> is provided. A part or the whole of the optical system <NUM> may be provided on a chip in the first substrate <NUM>.

In the second substrate <NUM>, in the configuration of the image sensor <NUM> illustrated in <FIG>, the ADC <NUM>, the controller <NUM>, the signal processor <NUM>, the DSP <NUM>, the memory <NUM>, and the selector <NUM> are arranged. A not-illustrated interface circuit, driver circuit, and the like may be arranged in the second substrate <NUM>.

The first substrate <NUM> and the second substrate <NUM> may be bonded together by chip-on-chip (CoC) technology in which the first substrate <NUM> and the second substrate <NUM> are individually diced into chips, and these diced first substrate <NUM> and second substrate <NUM> are bonded together, or by chip-on-wafer (CoW) technology in which one of the first substrate <NUM> and the second substrate <NUM> (for example, the first substrate <NUM>) is diced into a chip, and the diced first substrate <NUM> is bonded to the second substrate <NUM> before dicing (that is, in a wafer state), or by wafer-on-wafer (WoW) technology in which the first substrate <NUM> and the second substrate <NUM> both in a wafer state are bonded together.

For example, plasma joining can be used as a joining process between the first substrate <NUM> and the second substrate <NUM>. However, the present invention is not limited thereto and a variety of joining processes may be used.

When the DSP <NUM> operates as a processing unit that performs a computation process based on a pre-trained model as described above, implementation of its operation algorithm is software implementation by running computer programs. Operation algorithms for pre-trained models are updated day by day. It is therefore difficult to grasp in advance, for example, at which timing the DSP <NUM> performing a computation process based on a pre-trained model performs a process or at which timing a process of the DSP <NUM> peaks.

As illustrated in <FIG>, in the case where the DSP <NUM> operates as a processing unit that performs computation based on a pre-trained model in a chip configuration in which the pixel array <NUM> is mounted on the first substrate <NUM> and the DSP <NUM> is mounted on the second substrate <NUM>, if the DSP <NUM> starts a computation process or a process in the DSP <NUM> reaches a peak during resetting of the pixel array <NUM>, during exposure of the pixel array <NUM>, or during readout of a pixel signal from each unit pixel 101a of the pixel array <NUM>, noise is superimposed on a pixel signal read out from the pixel array <NUM>, and consequently, the quality of the image acquired by the image sensor <NUM> is deteriorated.

The present embodiment then reduces intrusion of noise resulting from the signal processing by the DSP <NUM> into the pixel array <NUM>, by adjusting the positional relation between the pixel array <NUM> and the DSP <NUM>. Accordingly, an image with less deterioration in quality can be acquired even when the DSP <NUM> operates as a processing unit that performs computation based on a pre-trained model.

The positional relation between the pixel array <NUM> and the DSP <NUM> according to the present embodiment will now be described in detail below with reference to the drawings. In the following, the positional relation between the pixel array <NUM> and the DSP <NUM> will be described by taking several examples of the layout (also referred to as floor map) of layers (the first substrate <NUM> and the second substrate <NUM>).

<FIG> are diagrams for explaining a first layout example according to the present embodiment. <FIG> illustrates a layout example of the first substrate <NUM>, and <FIG> illustrates a layout example of the second substrate <NUM>.

As illustrated in <FIG>, in the first substrate <NUM>, in the configuration of the image sensor <NUM> illustrated in <FIG>, the pixel array <NUM> of the imager <NUM> is provided. When a part or the whole of the optical system <NUM> is mounted on the first substrate <NUM>, it is provided at a position corresponding to the pixel array <NUM>.

The pixel array <NUM> is provided off-center to one side L101 among four sides L101 to L104 of the first substrate <NUM>. In other words, the pixel array <NUM> is provided such that its center O101 is more proximate to the side L101 than the center O100 of the first substrate <NUM>. When the surface having the pixel array <NUM> in the first substrate <NUM> is rectangular, the side L101 may be, for example, a shorter side. However, the present invention is not limited thereto, and the pixel array <NUM> may be provided off-center to a longer side.

In a region proximate to the side L101 among four sides of the pixel array <NUM>, in other words, a region between the side L101 and the pixel array <NUM>, a TSV array <NUM> is provided, in which a plurality of through silicon vias (hereinafter referred to as TSVs) passing through the first substrate <NUM> are arranged as wiring for electrically connecting each unit pixel 101a in the pixel array <NUM> to the ADC <NUM> provided in the second substrate <NUM>. In this way, the TSV array <NUM> is provided in proximity to the side L101 proximate to the pixel array <NUM> to ensure a space for each part such as the ADC <NUM> in the second substrate <NUM>.

The TSV array <NUM> may also be provided in a region proximate to one side L104 (or may be the side L103) of two sides L103 and L104 intersecting the side L101, in other words, in a region between the side L104 (or the side L103) and the pixel array <NUM>.

A pad array <NUM> having a plurality of pads arranged linearly is provided on each of the sides L102 and L103, on which the pixel array <NUM> is not provided off-center, among four sides L101 to L104 of the first substrate <NUM>. The pads included in the pad array <NUM> include, for example, a pad (also referred to as power supply pin) receiving power supply voltage for analog circuits such as the pixel array <NUM> and the ADC <NUM>, a pad (also referred to as power supply pin) receiving power supply voltage for digital circuits such as the signal processor <NUM>, the DSP <NUM>, the memory <NUM>, the selector <NUM>, and the controller <NUM>, a pad (also referred to as signal pin) for interfaces such as a mobile industry processor interface (MIPI) and a serial peripheral interface (SPI), and a pad (also referred to as signal pin) for input/output of clock and data. Each pad is electrically connected to, for example, an external power supply circuit or an interface circuit through a wire. It is preferable that each pad array <NUM> and the TSV array <NUM> are sufficiently spaced apart to such a degree that influences of reflection of signals from the wire connected to each pad in the pad array <NUM> can be ignored.

On the other hand, as illustrated in <FIG>, in the second substrate <NUM>, in the configuration of the image sensor <NUM> illustrated in <FIG>, the ADC <NUM>, the controller <NUM>, the signal processor <NUM>, the DSP <NUM>, and the memory <NUM> are arranged. In the first layout example, the memory <NUM> is divided into two regions: a memory 15A and a memory 15B. Similarly, the ADC <NUM> is divided into two regions: an ADC 17A and a digital-to-analog converter (DAC) 17B. The DAC 17B supplies a reference voltage for AD conversion to the ADC 17A and, broadly speaking, is included in a part of the ADC <NUM>. Although not illustrated in <FIG>, the selector <NUM> is also provided on the second substrate <NUM>.

The second substrate <NUM> also has wiring <NUM> in contact with and electrically connected to the TSVs in the TSV array <NUM> passing through the first substrate <NUM> (hereinafter simply referred to as TSV array <NUM>), and a pad array <NUM>, in which a plurality of pads electrically connected to the pads in the pad array <NUM> of the first substrate <NUM> are arranged linearly.

For the connection between the TSV array <NUM> and the wiring <NUM>, for example, the following technology can be employed: twin TSV technology in which two TSVs, namely, a TSV provided in the first substrate <NUM> and a TSV provided from the first substrate <NUM> to the second substrate <NUM> are connected with the chip facing out, or shared TSV technology in which a shared TSV provided from the first substrate <NUM> to the second substrate <NUM> provides connection. However, the present invention is not limited thereto, and a variety of connection modes can be employed. Examples include Cu-Cu bonding in which copper (Cu) exposed on the joint surface of the first substrate <NUM> and Cu exposed on the joint surface of the second substrate <NUM> are joined.

The connection mode between the pads in the pad array <NUM> on the first substrate <NUM> and the pads in the pad array <NUM> of the second substrate <NUM> is, for example, wire bonding. However, the present invention is not limited thereto, and connection modes such as through holes and castellation may be employed.

In a layout example of the second substrate <NUM>, for example, the ADC 17A, the signal processor <NUM>, and the DSP <NUM> are arranged in order from the upstream side along the flow of a signal read out from the pixel array <NUM>, where the upstream side is the vicinity of the wiring <NUM> connected to the TSV array <NUM>. That is, the ADC 17A to which a pixel signal read out from the pixel array <NUM> is initially input is provided in the vicinity of the wiring <NUM> on the most upstream side, next the signal processor <NUM> is provided, and the DSP <NUM> is provided in a region farthest from the wiring <NUM>. Such a layout in which the ADC <NUM> to the DSP <NUM> are arranged from the upstream side along the flow of a signal can shorten the wiring connecting the parts. This layout leads to reduction in signal delay, reduction in signal propagation loss, improvement of the S/N ratio, and lower power consumption.

The controller <NUM> is provided, for example, in the vicinity of the wiring <NUM> on the upstream side. In <FIG>, the controller <NUM> is provided between the ADC 17A and the signal processor <NUM>. Such a layout leads to reduction in signal delay, reduction in signal propagation loss, improvement of the S/N ratio, and lower power consumption when the controller <NUM> controls the pixel array <NUM>. Advantageously, the signal pin and the power supply pin for analog circuits can be collectively arranged in the vicinity of the analog circuits (for example, in the lower side of <FIG>), the remaining signal pin and power supply pin for digital circuits can be collectively arranged in the vicinity of the digital circuits (for example, in the upper side of <FIG>), or the power supply pin for analog circuits and the power supply pin for digital circuits can be sufficiently spaced apart from each other.

In the layout illustrated in <FIG>, the DSP <NUM> is provided on the side opposite to the ADC 17A on the most downstream side. With such a layout, in other words, the DSP <NUM> can be provided in a region not overlapping with the pixel array <NUM> in the stacking direction of the first substrate <NUM> and the second substrate <NUM> (hereinafter simply referred to as top-bottom direction).

In this way, in the configuration in which the pixel array <NUM> and the DSP <NUM> are not superimposed in the top-bottom direction, intrusion of noise produced due to signal processing by the DSP <NUM> into the pixel array <NUM> can be reduced. As a result, even when the DSP <NUM> operates as a processing unit that performs computation based on a pre-trained model, intrusion of noise resulting from signal processing by the DSP <NUM> into the pixel array <NUM> can be reduced, and consequently, an image with less deterioration in quality can be acquired.

The DSP <NUM> and the signal processor <NUM> are connected by an interconnect 14a configured with a part of the DSP <NUM> or a signal line. The selector <NUM> is provided, for example, in the vicinity of the DSP <NUM>. When the interconnect 14a is a part of the DSP <NUM>, the DSP <NUM> may partially overlap with the pixel array <NUM> in the top-bottom direction. However, even in such a case, compared with when the whole of the DSP <NUM> is superimposed on the pixel array <NUM> in the top-bottom direction, intrusion of noise into the pixel array <NUM> can be reduced.

Memories 15A and 15B are arranged, for example, so as to surround the DSP <NUM> from three directions. In such an arrangement of the memories 15A and 15B surrounding the DSP <NUM>, the distance of wiring between each memory element in the memory <NUM> and the DSP <NUM> can be averaged while the distance can be reduced as a whole. Consequently, signal delay, signal propagation loss, and power consumption can be reduced when the DSP <NUM> accesses the memory <NUM>.

The pad array <NUM> is provided, for example, at a position on the second substrate <NUM> corresponding to the pad array <NUM> of the first substrate <NUM> in the top-bottom direction. Here, among the pads included in the pad array <NUM>, a pad positioned in the vicinity of the ADC 17A is used for propagation of power supply voltage for analog circuits (mainly the ADC 17A) or an analog signal. On the other hand, a pad positioned in the vicinity of the controller <NUM>, the signal processor <NUM>, the DSP <NUM>, or the memories 15A and 15B is used for propagation of power supply voltage for digital circuits (mainly, the controller <NUM>, the signal processor <NUM>, the DSP <NUM>, the memories 15A and 15B) and a digital signal. Such a pad layout can reduce the distance of wiring connecting the pads to the parts. This layout leads to reduction in signal delay, reduction in propagation loss of signals and power supply voltage, improvement of the S/N ratio, and lower power consumption.

A second layout example will now be described. In the second layout example, the layout example of the first substrate <NUM> may be similar to the layout example described with reference to <FIG> in the first layout example.

<FIG> is a diagram illustrating a layout example of the second substrate in the second layout example. As illustrated in <FIG>, in the second layout example, in a layout similar to the first layout example, the DSP <NUM> is provided at the center of a region in which the DSP <NUM> and the memory <NUM> are arranged. In other words, in the second layout example, the memory <NUM> is provided so as to surround the DSP <NUM> from four directions.

In such an arrangement of the memories 15A and 15B surrounding the DSP <NUM> from four directions, the distance of wiring between each memory element in the memory <NUM> and the DSP <NUM> can be further averaged while the distance can be further reduced as a whole. Consequently, signal delay, signal propagation loss, and power consumption can be further reduced when the DSP <NUM> accesses the memory <NUM>.

In <FIG>, the DSP <NUM> and the pixel array <NUM> are arranged so as not to be superimposed on each other in the top-bottom direction. However, the present invention is not limited thereto, and the DSP <NUM> may be partially superimposed on the pixel array <NUM> in the top-bottom direction. Even in such a case, compared with when the whole of the DSP <NUM> is superimposed on the pixel array <NUM> in the top-bottom direction, intrusion of noise into the pixel array <NUM> can be reduced.

The other layout may be similar to the first layout example and is not further elaborated here.

A third layout example will now be described. In the third layout example, the layout example of the first substrate <NUM> may be similar to the layout example described with reference to <FIG> in the first layout example.

<FIG> is a diagram illustrating a layout example of the second substrate in the third layout example. As illustrated in <FIG>, in the third layout example, in a layout similar to the first layout example, the DSP <NUM> is provided adjacent to the signal processor <NUM>. In such a configuration, the signal line from the signal processor <NUM> to the DSP <NUM> can be shortened. This layout leads to reduction in signal delay, reduction in propagation loss of signals and power supply voltage, improvement of the S/N ratio, and lower power consumption.

In the third layout example, the memory <NUM> is provided so as to surround the DSP <NUM> from three directions. Consequently, signal delay, signal propagation loss, and power consumption can be reduced when the DSP <NUM> accesses the memory <NUM>.

In the third layout example, the DSP <NUM> is partially superimposed on the pixel array <NUM> in the top-bottom direction. Even in such a case, compared with when the whole of the DSP <NUM> is superimposed on the pixel array <NUM> in the top-bottom direction, intrusion of noise into the pixel array <NUM> can be reduced.

The other layout may be similar to the other layout examples and is not further elaborated here.

A fourth layout example will now be described. In the fourth layout example, the layout example of the first substrate <NUM> may be similar to the layout example described with reference to <FIG> in the first layout example.

<FIG> is a diagram illustrating a layout example of the second substrate in the fourth layout example. As illustrated in <FIG>, in the fourth layout example, in a layout similar to the third layout example, that is, in a layout in which the DSP <NUM> is provided adjacent to the signal processor <NUM>, the DSP <NUM> is provided at a position far from both of two TSV arrays <NUM>.

In such an arrangement of the DSP <NUM> at a position far from both of two TSV arrays <NUM>, because the ADC 17A to the DSP <NUM> can be provided more faithfully to the signal flow, the signal line from the signal processor <NUM> to the DSP <NUM> can be further shortened. As a result, signal delay, signal propagation loss, and power consumption can be further reduced.

In the fourth layout example, the memory <NUM> is provided so as to surround the DSP <NUM> from two directions. Consequently, signal delay, signal propagation loss, and power consumption can be reduced when the DSP <NUM> accesses the memory <NUM>.

Also in the fourth layout example, the DSP <NUM> is partially superimposed on the pixel array <NUM> in the top-bottom direction. Even in such a case, compared with when the whole of the DSP <NUM> is superimposed on the pixel array <NUM> in the top-bottom direction, intrusion of noise into the pixel array <NUM> can be reduced.

A fifth layout example will now be described. In the fifth layout example, the layout example of the first substrate <NUM> may be similar to the layout example described with reference to <FIG> in the first layout example.

<FIG> is a diagram illustrating a layout example of the second substrate in the fifth layout example. As illustrated in <FIG>, in the fifth layout example, in a layout similar to the first layout example, that is, in a layout in which the DSP <NUM> is provided on the most downstream side, the DSP <NUM> is provided at a position far from both of two TSV arrays <NUM>.

Even in such a layout, because the ADC 17A to the DSP <NUM> can be arranged more faithfully to the signal flow, the signal line from the signal processor <NUM> to the DSP <NUM> can be further shortened. As a result, signal delay, signal propagation loss, and power consumption can be further reduced.

A sixth layout example will now be described. In the sixth layout example, the layout example of the first substrate <NUM> may be similar to the layout example described with reference to <FIG> in the first layout example.

<FIG> is a diagram illustrating a layout example of the second substrate in the sixth layout example. As illustrated in <FIG>, in the sixth layout example, the DSP <NUM> is sandwiched in the top-bottom direction in the drawing between memories 15C and 15D divided into two regions.

In such an arrangement of the memories 15C and 15D sandwiching the DSP <NUM>, the distance of wiring between each memory element in the memory <NUM> and the DSP <NUM> can be averaged while the distance can be reduced as a whole. Consequently, signal delay, signal propagation loss, and power consumption can be further reduced when the DSP <NUM> accesses the memory <NUM>.

A seventh layout example will now be described. In the seventh layout example, the layout example of the first substrate <NUM> may be similar to the layout example described with reference to <FIG> in the first layout example.

<FIG> is a diagram illustrating a layout example of the second substrate in the seventh layout example. As illustrated in <FIG>, in the seventh layout example, the memory <NUM> is sandwiched in the top-bottom direction in the drawing between DSPs 14A and 14B divided into two regions.

In such an arrangement of the DSPs 14A and 14B sandwiching the memory <NUM>, the distance of wiring between each memory element in the memory <NUM> and the DSP <NUM> can be averaged while the distance can be reduced as a whole. Consequently, signal delay, signal propagation loss, and power consumption can be further reduced when the DSP <NUM> accesses the memory <NUM>.

An eighth layout example will now be described. In the eighth layout example, the layout example of the first substrate <NUM> may be similar to the layout example described with reference to <FIG> in the first layout example.

<FIG> is a diagram illustrating a layout example of the second substrate in the eighth layout example. As illustrated in <FIG>, in the eighth layout example, the DSP <NUM> is sandwiched in the left-right direction in the drawing between memories 15E and 15F divided into two regions.

A ninth layout example will now be described. In the ninth layout example, the layout example of the first substrate <NUM> may be similar to the layout example described with reference to <FIG> in the first layout example.

<FIG> is a diagram illustrating a layout example of the second substrate in the ninth layout example. As illustrated in <FIG>, in the ninth layout example, the memory <NUM> is sandwiched in the left-right direction in the drawing between DSPs 14C and 14D divided into two regions.

In such an arrangement of the DSPs 14C and 14D sandwiching the memory <NUM>, the distance of wiring between each memory element in the memory <NUM> and the DSP <NUM> can be averaged while the distance can be reduced as a whole. Consequently, signal delay, signal propagation loss, and power consumption can be further reduced when the DSP <NUM> accesses the memory <NUM>.

As described above, according to the present embodiment, the positional relation between the pixel array <NUM> and the DSP <NUM> is adjusted such that at least a part of the DSP <NUM> of the second substrate <NUM> is not superimposed on the pixel array <NUM> in the stacking direction (the top-bottom direction) of the first substrate <NUM> and the second substrate <NUM>. This configuration can reduce intrusion of noise resulting from signal processing by the DSP <NUM> into the pixel array <NUM> and therefore can provide an image with less deteriorated quality even when the DSP <NUM> operates as a processing unit that performs computation based on a pre-trained model.

A second embodiment will now be described in detail with reference to the drawings. In the following description, a configuration similar to the first embodiment is denoted by the same reference sign and an overlapping description thereof is omitted.

An imaging device as an electronic device according to the second embodiment may be similar to, for example, the imaging device <NUM> described in the first embodiment with reference to <FIG>, which is hereby referred to and will not be further elaborated.

An example of the chip configuration of an image sensor according to the present embodiment will now be described in detail below with reference to the drawings. <FIG> is a layout diagram illustrating an overall configuration example of the first substrate in the image sensor according to the present embodiment. <FIG> is a diagram illustrating a chip configuration example of the image sensor according to the present embodiment.

As illustrated in <FIG>, in the present embodiment, the size of a first substrate <NUM> is smaller than the size of the second substrate <NUM>. For example, the size of the first substrate <NUM> is reduced in accordance with the size of the pixel array <NUM>. With such size reduction of the first substrate <NUM>, many first substrates <NUM> can be fabricated from a single semiconductor wafer. Furthermore, the chip size of the image sensor <NUM> can be further reduced.

For the bonding between the first substrate <NUM> and the second substrate <NUM>, chip-on-chip (CoC) technology in which the first substrate <NUM> and the second substrate <NUM> are individually diced into chips and then bonded, or chip-on-wafer (CoW) technology in which the diced first substrate <NUM> is bonded to the second substrate <NUM> in a wafer state can be employed.

The layout of the first substrate <NUM> may be similar to, for example, the layout of the first substrate <NUM> illustrated in the first embodiment, excluding the upper portion. The layout of the second substrate <NUM> may be similar to, for example, the second substrate <NUM> illustrated in the first embodiment. The bonding place of the first substrate <NUM> to the second substrate <NUM> may be a position where at least a part of the pixel array <NUM> does not overlap the DSP <NUM> of the second substrate <NUM> in the top-bottom direction, in the same manner as in the first embodiment.

As described above, even when the first substrate <NUM> is downsized in accordance with the size of the pixel array <NUM>, intrusion of noise resulting from signal processing by the DSP <NUM> into the pixel array <NUM> can be reduced, in the same manner as in the first embodiment. Consequently, an image with less deterioration in quality can be acquired even when the DSP <NUM> operates as a processing unit that performs computation based on a pre-trained model. The other configuration (including the layout example of the second substrate <NUM>) and effects may be similar to those of the first embodiment and will not be further elaborated here.

A third embodiment will now be described in detail with reference to the drawings. In the following description, a configuration similar to the first or second embodiment is denoted by the same reference sign and an overlapping description thereof is omitted.

An imaging device as an electronic device according to the third embodiment may be similar to, for example, the imaging device <NUM> described in the first embodiment with reference to <FIG>, which is hereby referred to and will not be further elaborated.

An example of the chip configuration of an image sensor according to the present embodiment will now be described in detail below with reference to the drawings. <FIG> is a layout diagram illustrating an overall configuration example of the first substrate in the image sensor according to the present embodiment. <FIG> is a layout diagram illustrating an overall configuration example of the second substrate in the image sensor according to the present embodiment. <FIG> is a diagram illustrating a chip configuration example of the image sensor according to the present embodiment.

As illustrated in <FIG>, in the present embodiment, the size of a first substrate <NUM> is reduced in accordance with the size of the pixel array <NUM>. In the present embodiment, the size of a second substrate <NUM> is reduced to the same degree as the size of the first substrate <NUM>. With such a configuration, in the present embodiment, a surplus region of the first substrate <NUM> can be reduced, and the chip size of the image sensor <NUM> can be further reduced accordingly.

However, in the present embodiment, the pixel array <NUM> and the DSP <NUM> are superimposed on each other in the stacking direction of the first substrate <NUM> and the second substrate <NUM> (hereinafter simply referred to as top-bottom direction). Because of this, noise resulting from the DSP <NUM> may be superimposed on a pixel signal read out from the pixel array <NUM> in some cases and may reduce the quality of an image acquired by the image sensor <NUM>.

Then, in the present embodiment, the ADC 17A and the DSP <NUM> are spaced apart from each other. Specifically, for example, the ADC 17A is provided closer to one end L321 of the second substrate <NUM>, while the DSP <NUM> is provided closer to an end L322 on the side opposite to the end L321 at which the ADC 17A is disposed.

With such an arrangement, noise produced by intrusion of noise produced in the DSP <NUM> into the ADC 17A can be reduced, thereby suppressing deterioration in quality of an image acquired by the image sensor <NUM>. The end L321 proximate to the ADC 17A may be an end on which the wiring <NUM> connected to the TSV array <NUM> is provided.

With such an arrangement, for example, the ADC 17A, the signal processor <NUM>, and the DSP <NUM> are arranged in order from the upstream side along the flow of a signal read out from the pixel array <NUM>, where the upstream side is the vicinity of the wiring <NUM> connected to the TSV array <NUM>, in the same manner as in the foregoing embodiments. The wiring connecting the parts therefore can be shortened. Consequently, reduction in signal delay, reduction in signal propagation loss, improvement in the S/N ratio, and lower power consumption can be achieved.

As described above, when the first substrate <NUM> and the second substrate <NUM> are downsized in accordance with the size of the pixel array <NUM>, the ADC 17A and the DSP <NUM> are spaced apart from each other, thereby reducing noise produced by intrusion of noise produced in the DSP <NUM> into the ADC 17A. Consequently, deterioration in quality of an image acquired by the image sensor <NUM> can be suppressed.

The other configuration and effects are similar to those of the foregoing embodiments and will not be further elaborated here.

In a fourth embodiment according to the claimed invention, a specific configuration example of the image sensor <NUM> and the imaging device <NUM> according to the foregoing embodiments and an attachment example thereof will be described. In the following description, an example based on the first embodiment is illustrated. However, the present embodiment may be based on any other embodiments rather than the first embodiment. In the present embodiment, the imaging device <NUM> is mounted on a vehicle, that is, the imaging device <NUM> is an in-vehicle camera, by way of example. However, the imaging device <NUM> is not necessarily attached to a vehicle but may be attached to various equipment, devices, and locations.

<FIG> is a diagram illustrating an overall configuration example of an imaging device according to the present embodiment. In <FIG>, for simplicity of explanation, the first substrate <NUM> in the image sensor <NUM> is not illustrated.

As illustrated in <FIG>, the chip of the image sensor <NUM> according to the foregoing embodiments may be connected to a circuit board <NUM> provided with the application processor <NUM> and the like, for example, through connection wiring that is flexible and deformable, such as a flexible cable <NUM>.

In this case, one end of the flexible cable <NUM> may be connected to a side other than the side on which the ADC 17A in the second substrate <NUM> is provided in proximity, for example, to the side on which the DSP <NUM> is provided in proximity. This configuration can reduce the wiring length from a data output end in the DSP <NUM> or the memory <NUM> provided in vicinity thereof to the flexible cable <NUM>, thereby suppressing size increase of the chip of the image sensor <NUM> and facilitating designing of this wiring layout. Reducing the wiring length can lead to suppression of signal delay, reduction in signal propagation loss, and lower power consumption.

Some of attachment examples of the imaging device <NUM> according to the present embodiment will now be described.

First, an attachment example in which the imaging device <NUM> is mounted as a front camera for capturing an image in front of a vehicle will be described as a first example. <FIG> is a diagram illustrating an attachment example of the imaging device according to the first example of the present embodiment.

As illustrated in <FIG>, when the imaging device <NUM> is mounted as a front camera on a vehicle <NUM>, the imaging device <NUM> is provided, for example, in the vicinity of a windshield <NUM> inside the vehicle <NUM>, at a location that does not interrupt the field of view of a driver D.

In a state in which the imaging device <NUM> is attached to the inside of the vehicle <NUM>, the image sensor <NUM> is inclined at about <NUM>° to the front direction of the vehicle <NUM>, for example, such that a light-receiving surface of the pixel array <NUM> of the first substrate <NUM> faces substantially the front side of the vehicle <NUM>. On the other hand, the circuit board <NUM> connected to the image sensor <NUM> through the flexible cable <NUM> is installed, for example, such that its main plane is substantially horizontal. The main plane of the circuit board <NUM> may be, for example, a surface parallel to the surface provided with an electronic circuit such as the application processor <NUM>. As used herein the horizontal direction may be a horizontal direction in the interior space of the vehicle <NUM>.

The image sensor <NUM> and the circuit board <NUM> may be arranged, for example, such that the circuit board <NUM> is positioned below the image sensor <NUM> and at least the front-side end of the circuit board <NUM> protrudes toward the front direction of the vehicle relative to the front-side end of the image sensor <NUM>. Thus, the imaging device <NUM> can be provided in closer proximity to the windshield <NUM> inclined toward the front direction of the vehicle <NUM>, so that the space occupied by the imaging device <NUM> can be reduced.

In this case, the respective end portions of the image sensor <NUM> and the circuit board <NUM> are arranged in proximity to each other so as to form the L shape, and the end portions arranged in proximity to each other are connected by the flexible cable <NUM>, whereby the length of the flexible cable <NUM> can be reduced.

In such an arrangement example, for example, when the flexible cable <NUM> is connected to the end on which the DSP <NUM> is provided in proximity in the second substrate <NUM>, the image sensor <NUM> is installed such that the ADC 17A in the second substrate <NUM> is positioned on the upper side in the vertical direction and the DSP14 is positioned on the lower side in the vertical direction. The flexible cable <NUM> is attached so as to connect the lower-side end of the image sensor <NUM> and the rear-side end of the circuit board <NUM>. Furthermore, the application processor <NUM> in the circuit board <NUM> is provided, for example, on the front side of the vehicle <NUM> relative to the image sensor <NUM>.

Next, an attachment example in which the imaging device <NUM> is mounted as a rear camera for capturing an image behind a vehicle will be described as a second example. <FIG> is a diagram illustrating an installation example of the imaging device according to the second example of the present embodiment.

As illustrated in <FIG>, when the imaging device <NUM> is mounted as a rear camera on the vehicle <NUM>, the imaging device <NUM> is provided, for example, in the vicinity of a rear window <NUM> inside the vehicle <NUM>, at a location that does not interrupt the field of view of a driver D through a rearview mirror.

In a state in which the imaging device <NUM> is attached to the inside of the vehicle <NUM>, the image sensor <NUM> is inclined at about <NUM>° to the rear direction of the vehicle <NUM>, for example, such that the light-receiving surface of the pixel array <NUM> of the first substrate <NUM> faces substantially the rear side of the vehicle <NUM>. On the other hand, the circuit board <NUM> connected to the image sensor <NUM> through the flexible cable <NUM> is installed, for example, such that its main plane is substantially parallel to the light-receiving surface of the image sensor <NUM>. The flexible cable <NUM> may connect the upper ends of the image sensor <NUM> and the circuit board <NUM> in the installed state, or may connect the lower ends thereof, or may connect the side ends thereof.

However, the embodiment is not limited to such an attachment example. Even when the imaging device <NUM> is installed as a rear camera, the image sensor <NUM> and the circuit board <NUM> may be arranged, for example, such that the circuit board <NUM> is positioned below the image sensor <NUM> and at least the rear-side end of the circuit board <NUM> protrudes to the rear direction of the vehicle relative to the rear-side end of the image sensor <NUM>, similarly to when the imaging device <NUM> is installed as a front camera in the first example. Thus, the imaging device <NUM> can be provided in closer proximity to the rear window <NUM> inclined toward the rear direction of the vehicle <NUM>, so that the space occupied by the imaging device <NUM> can be reduced.

As described above, in the present embodiment, one end of the flexible cable <NUM> is connected to a side other than the side on which the ADC 17A in the second substrate <NUM> is provided in proximity, for example, to the side on which the DSP <NUM> is provided in proximity. This configuration can reduce the wiring length from a data output end in the DSP <NUM> or the memory <NUM> provided in vicinity thereof to the flexible cable <NUM>, thereby suppressing size increase of the chip of the image sensor <NUM> and facilitating designing of this wiring layout. Reducing the wiring length can lead to suppression of signal delay, reduction in signal propagation loss, and lower power consumption.

The image sensor <NUM> and the circuit board <NUM> are not necessarily connected by a flexible and deformable connection cable, such as the flexible cable <NUM>, as described above but may be connected using a connection part that does not have flexibility, such as solder balls, bonding pads, and connection pins.

Other configurations and effects may be similar to those in the foregoing embodiments and will not be further elaborated here.

In the foregoing embodiments, the technique according to the present disclosure is applied to a solid-state imaging device (image sensor <NUM>) that acquires a two-dimensional image. However, the application of the technique according to the present disclosure is not limited to a solid-state imaging device. For example, the technique according to the present disclosure can be applied to a variety of light-receiving sensors such as Time of Flight (ToF) sensors, infrared (IR) sensors, and dynamic vision sensors (DVS). That is, when the chip structure of light-receiving sensors is of the stacked type, reduction of noise included in sensor results and miniaturization of sensor chips can be achieved.

The technique according to the present disclosure (the present technique) is applicable to a variety of products. For example, the technique according to the present disclosure may be implemented as a device mounted on any type of movable bodies, such as automobiles, electric vehicles, hybrid electric vehicles, motorcycles, bicycles, personal mobility devices, airplanes, drones, vessels and ships, and robots.

<FIG> is a block diagram illustrating an example of the overall configuration of a vehicle control system that is an example of a movable body control system to which the technique according to the present disclosure is applicable.

A vehicle control system <NUM> includes a plurality of electronic control units connected through a communication network <NUM>. In the example illustrated in <FIG>, the vehicle control system <NUM> includes a drive control unit <NUM>, a body control unit <NUM>, a vehicle exterior information detection unit <NUM>, a vehicle interior information detection unit <NUM>, and a central control unit <NUM>. As a functional configuration of the central control unit <NUM>, a microcomputer <NUM>, a sound image output module <NUM>, and an in-vehicle network I/F (interface) <NUM> are illustrated.

The drive control unit <NUM> controls operation of devices related to a drive system of a vehicle in accordance with a variety of computer programs. For example, the drive control unit <NUM> functions as a control device for a drive force generating device for generating drive force of the vehicle, such as an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting drive force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, and a braking device for generating braking force of the vehicle.

The body control unit <NUM> controls operation of a variety of devices installed in the vehicle body in accordance with a variety of computer programs. For example, the body control unit <NUM> functions as a control device for a keyless entry system, a smart key system, a power window device, or a variety of lamps such as head lamps, rear lamps, brake lamps, turn signals, and fog lamps. In this case, the body control unit <NUM> may receive radio waves transmitted from a portable device alternative to a key or signals from a variety of switches. The body control unit <NUM> accepts input of the radio waves or signals and controls a door lock device, a power window device, a lamp, and the like of the vehicle.

The vehicle exterior information detection unit <NUM> detects information on the outside of the vehicle equipped with the vehicle control system <NUM>. For example, an imager <NUM> is connected to the vehicle exterior information detection unit <NUM>. The vehicle exterior information detection unit <NUM> allows the imager <NUM> to capture an image of the outside of the vehicle and receives the captured image. The vehicle exterior information detection unit <NUM> may perform an object detection process or a distance detection process for persons, vehicles, obstacles, signs, or characters on roads, based on the received image.

The imager <NUM> is an optical sensor that receives light and outputs an electrical signal corresponding to the quantity of received light of the light. The imager <NUM> may output an electrical signal as an image or output as information on a measured distance. Light received by the imager <NUM> may be visible light or invisible light such as infrared rays.

The vehicle interior information detection unit <NUM> detects information on the inside of the vehicle. The vehicle interior information detection unit <NUM> is connected to, for example, a driver state detector <NUM> that detects a state of the driver. The driver state detector <NUM> includes, for example, a camera for taking an image of the driver, and the vehicle interior information detection unit <NUM> may calculate the degree of fatigue or the degree of concentration of the driver or may determine whether the driver falls asleep, based on detection information input from the driver state detector <NUM>.

The microcomputer <NUM> can compute a control target value for the drive force generating device, the steering mechanism, or the braking device, based on information on the inside and outside of the vehicle acquired by the vehicle exterior information detection unit <NUM> or the vehicle interior information detection unit <NUM>, and output a control command to the drive control unit <NUM>. For example, the microcomputer <NUM> can perform coordination control for the purpose of function implementation of advanced driver assistance systems (ADAS), including collision avoidance or shock mitigation of the vehicle, car-following drive based on the distance between vehicles, vehicle speed-keeping drive, vehicle collision warning, or lane departure warning.

The microcomputer <NUM> can perform coordination control for the purpose of, for example, autonomous driving, in which the drive force generating device, the steering mechanism, or the braking device is controlled based on information on the surroundings of the vehicle acquired by the vehicle exterior information detection unit <NUM> or the vehicle interior information detection unit <NUM> to enable autonomous driving without depending on the operation by the driver.

The microcomputer <NUM> can output a control command to the body control unit <NUM>, based on information on the outside of the vehicle acquired by the vehicle exterior information detection unit <NUM>. For example, the microcomputer <NUM> can perform coordination control for the antidazzle purpose, for example, by controlling the head lamps in accordance with the position of a vehicle ahead or an oncoming vehicle detected by the vehicle exterior information detection unit <NUM> to switch high beams to low beams.

The sound image output module <NUM> transmits an output signal of at least one of sound and image to an output device capable of visually or aurally giving information to a passenger in the vehicle or the outside of the vehicle. In the example in <FIG>, an audio speaker <NUM>, a display <NUM>, and an instrument panel <NUM> are illustrated as the output device. The display <NUM> may include, for example, at least one of an on-board display and a head-up display.

<FIG> is a diagram illustrating an example of the installation position of the imager <NUM>.

In <FIG>, imagers <NUM>, <NUM>, <NUM>, <NUM>, and <NUM> are provided as the imager <NUM>.

The imagers <NUM>, <NUM>, <NUM>, <NUM>, and <NUM> are provided, for example, at positions such as front nose, side mirrors, rear bumper, back door of a vehicle <NUM>, and an upper portion of the front glass inside the vehicle. The imager <NUM> provided at the front nose and the imager <NUM> provided at the upper portion of the front glass inside the vehicle mainly acquire an image in front of the vehicle <NUM>. The imagers <NUM> and <NUM> provided at the side mirrors mainly acquire images on the sides of the vehicle <NUM>. The imager <NUM> provided at the rear bumper or the back door mainly acquires an image behind the vehicle <NUM>. The imager <NUM> provided at the upper portion of the front glass in the vehicle interior is mainly used for detecting a vehicle ahead, pedestrians, obstacles, traffic signs, road signs, traffic lanes, and the like.

<FIG> illustrates an example of the imaging ranges of the imagers <NUM> to <NUM>. An imaging range <NUM> indicates the imaging range of the imager <NUM> provided at the front nose, imaging ranges <NUM> and <NUM> indicate the imaging ranges of the imagers <NUM> and <NUM> provided at the side mirrors, and an imaging range <NUM> indicates the imaging range of the imager <NUM> provided at the rear bumper or the back door. For example, a bird's eye view of the vehicle <NUM> viewed from above can be obtained by superimposing image data captured by the imagers <NUM> to <NUM>.

At least one of the imagers <NUM> to <NUM> may have a function of acquiring distance information. For example, at least one of the imagers <NUM> to <NUM> may be a stereo camera including a plurality of image sensors or may be an image sensor having a pixel for phase difference detection.

For example, the microcomputer <NUM> can obtain the distance to a three-dimensional object within the imaging ranges <NUM> to <NUM> and a temporal change of this distance (relative speed to the vehicle <NUM>), based on distance information obtained from the imagers <NUM> to <NUM>, to specifically extract a three-dimensional object closest to the vehicle <NUM> on the path of travel and traveling at a predetermined speed (for example, <NUM>/h or more) in substantially the same direction as the vehicle <NUM>, as a vehicle ahead. In addition, the microcomputer <NUM> can preset a distance between vehicles to be kept behind a vehicle ahead and perform, for example, automatic braking control (including car-following stop control) and automatic speed-up control (including car-following startup control). In this way, coordination control can be performed for the purpose of, for example, autonomous driving in which the vehicle runs autonomously without depending on the operation by the driver.

For example, the microcomputer <NUM> can classify three-dimensional object data on a three-dimensional object into "two-wheel vehicle", "standard-sized vehicle", "heavy vehicle", "pedestrian", "utility pole", or "any other three-dimensional object", based on the distance information obtained from the imagers <NUM> to <NUM> and extract the data, and can use the extracted data for automatic avoidance of obstacles. For example, the microcomputer <NUM> identifies an obstacle in the surroundings of the vehicle <NUM> as an obstacle visible to the driver of the vehicle <NUM> or as an obstacle hardly visible. The microcomputer <NUM> then determines a collision risk indicating the degree of risk of collision with each obstacle and, when the collision risk is equal to or higher than a setting value and there is a possibility of collision, outputs an alarm to the driver through the audio speaker <NUM> or the display <NUM>, or performs forced deceleration or avoidance steering through the drive control unit <NUM>, thereby implementing drive assistance for collision avoidance.

At least one of the imagers <NUM> to <NUM> may be an infrared camera that detects infrared rays. For example, the microcomputer <NUM> can recognize a pedestrian by determining whether a pedestrian exists in the image captured by the imagers <NUM> to <NUM>. Such recognition of pedestrians is performed, for example, through the procedure of extracting feature points in the image captured by the imagers <NUM> to <NUM> serving as an infrared camera and the procedure of performing pattern matching with a series of feature points indicating the outline of an object to determine whether the object is a pedestrian. When the microcomputer <NUM> determines that a pedestrian exists in the image captured by the imagers <NUM> to <NUM> and recognizes a pedestrian, the sound image output module <NUM> controls the display <NUM> such that a rectangular outline for highlighting the recognized pedestrian is superimposed. The sound image output module <NUM> may control the display <NUM> such that an icon indicating a pedestrian appears at a desired position.

An example of the vehicle control system to which the technique according to the present disclosure is applicable has been described above. The technique according to the present disclosure is applicable to the imager <NUM> and the like in the configuration described above. When the technique according to the present disclosure is applied to the imager <NUM> and the like, miniaturization of the imager <NUM> and the like can be achieved, thereby facilitating design of the interior and the exterior of the vehicle <NUM>. When the technique according to the present disclosure is applied to the imager <NUM> and the like, a clear image with reduced noise can be acquired to provide a driver with a more visible image. Consequently, the driver's fatigue can be alleviated.

The technique according to the present disclosure (the present technique) is applicable to a variety of products. For example, the technique according to the present disclosure may be applied to an endoscopic surgery system.

<FIG> is a diagram illustrating an example of the overall configuration of an endoscopic surgery system to which the technique according to the present disclosure (the present technique) is applicable.

<FIG> illustrates a situation in which an operator (doctor) <NUM> uses an endoscopic surgery system <NUM> to perform an operation on a patient <NUM> on a patient bed <NUM>. As illustrated in the drawing, the endoscopic surgery system <NUM> includes an endoscope <NUM>, other surgical instruments <NUM> such as an insufflation tube <NUM> and an energy treatment tool <NUM>, a support arm device <NUM> supporting the endoscope <NUM>, and a cart <NUM> carrying a variety of devices for endoscopic surgery.

The endoscope <NUM> includes a barrel <NUM> having a region of a predetermined length from its tip end to be inserted into the body cavity of the patient <NUM>, and a camera head <NUM> connected to the base end of the barrel <NUM>. In the example illustrated in the drawing, the endoscope <NUM> is a rigid borescope having a rigid barrel <NUM>. However, the endoscope <NUM> may be configured as a soft borescope having a soft barrel.

The tip end of the barrel <NUM> has an opening having an objective lens fitted therein. A light source device <NUM> is connected to the endoscope <NUM>. Light generated by the light source device <NUM> is propagated to the tip end of the barrel through a light guide extending inside the barrel <NUM> and irradiates an observation target in the body cavity of the patient <NUM> through the objective lens. The endoscope <NUM> may be a forward-viewing endoscope or may be a forward-oblique viewing endoscope or a side-viewing endoscope.

An optical system and an image sensor are provided inside the camera head <NUM>. Reflected light (observation light) from an observation target is collected by the optical system onto the image sensor. The observation light is converted to electricity by the image sensor to generate an electrical signal corresponding to the observation light, that is, an image signal corresponding to an observation image. The image signal is transmitted as RAW data to a camera control unit (CCU) <NUM>.

The CCU <NUM> is configured with a central processing unit (CPU), a graphics processing unit (GPU), or the like to centrally control the operation of the endoscope <NUM> and a display device <NUM>. The CCU <NUM> receives an image signal from the camera head <NUM> and performs a variety of image processing on the image signal, for example, a development process (demosaicing) for displaying an image based on the image signal.

The display device <NUM> displays an image based on the image signal subjected to image processing by the CCU <NUM>, under the control of the CCU <NUM>.

The light source device <NUM> is configured with, for example, a light source such as a light emitting diode (LED) and supplies the endoscope <NUM> with radiation light in imaging a surgery site.

An input device <NUM> is an input interface with the endoscopic surgery system <NUM>. The user can input a variety of information and instructions to the endoscopic surgery system <NUM> through the input device <NUM>. For example, the user inputs an instruction to change the imaging conditions by the endoscope <NUM> (the kind of radiation light, magnification, focal length, etc.).

A treatment tool control device <NUM> controls actuation of the energy treatment tool <NUM> for cauterization of tissues, incision, or sealing of blood vessels. An insufflator <NUM> feeds gas into the body cavity through the insufflation tube <NUM> to insufflate the body cavity of the patient <NUM> in order to ensure the field of view with the endoscope <NUM> and ensure a working space for the operator. A recorder <NUM> is a device capable of recording a variety of information on surgery. A printer <NUM> is a device capable of printing a variety of information on surgery in a variety of forms such as text, image, or graph.

The light source device <NUM> that supplies the endoscope <NUM> with radiation light in imaging a surgery site can be configured with, for example, a white light source such as an LED, a laser light source, or a combination thereof. When a white light source is configured with a combination of RGB laser light sources, the output power and the output timing of each color (each wavelength) can be controlled accurately, and, therefore, the white balance of the captured image can be adjusted in the light source device <NUM>. In this case, an observation target is irradiated time-divisionally with laser light from each of the RGB laser light sources, and actuation of the image sensor in the camera head <NUM> is controlled in synchronization with the radiation timing, whereby an image corresponding to each of R, G, and B can be captured time-divisionally. According to this method, a color image can be obtained even without color filters in the image sensor.

The actuation of the light source device <NUM> may be controlled such that the intensity of output light is changed every certain time. In synchronization with the timing of changing the intensity of light, the actuation of the image sensor in the camera head <NUM> is controlled to acquire images time-divisionally, and the images are combined to generate an image with a high dynamic range free from blocked-up shadows and blown out highlights.

The light source device <NUM> may be configured to supply light in a predetermined wavelength band corresponding to specific light observation. In specific light observation, for example, narrow band imaging is performed, which uses the wavelength dependency of light absorption in body tissues and applies light in a narrow band, compared with radiation light (that is, white light) in normal observation, to capture a high-contrast image of predetermined tissues such as blood vessels in the outermost surface of mucosa. Alternatively, in specific light observation, fluorescence observation may be performed in which an image is acquired by fluorescence generated by radiation of excitation light. In fluorescence observation, for example, excitation light is applied to body tissues and fluorescence from the body tissues is observed (autofluorescence imaging), or a reagent such as indocyanine green (ICG) is locally injected to body tissues and excitation light corresponding to the fluorescence wavelength of the reagent is applied to the body tissues to obtain a fluorescence image. The light source device <NUM> may be configured to supply narrow-band light and/or excitation light corresponding to such specific light observation.

<FIG> is a block diagram illustrating an example of the functional configuration of the camera head <NUM> and the CCU <NUM> illustrated in <FIG>.

The camera head <NUM> includes a lens unit <NUM>, an imager <NUM>, a driver <NUM>, a communication module <NUM>, and a camera head controller <NUM>. The CCU <NUM> includes a communication module <NUM>, an image processor <NUM>, and a controller <NUM>. The camera head <NUM> and the CCU <NUM> are connected to communicate with each other through a transmission cable <NUM>.

The lens unit <NUM> is an optical system provided at a connection portion to the barrel <NUM>. Observation light taken in from the tip end of the barrel <NUM> is propagated to the camera head <NUM> and enters the lens unit <NUM>. The lens unit <NUM> is configured with a combination of a plurality of lenses including a zoom lens and a focus lens.

The imager <NUM> may be configured with one image sensor (called single sensor-type) or a plurality of image sensors (called multi sensor-type). When the imager <NUM> is a multi-sensor construction, for example, image signals corresponding to R, G, and B may be generated by image sensors and combined to produce a color image. Alternatively, the imager <NUM> may have a pair of image sensors for acquiring image signals for right eye and for left eye corresponding to three-dimensional (3D) display. The 3D display enables the operator <NUM> to more accurately grasp the depth of living tissues in a surgery site. When the imager <NUM> is a multi-sensor construction, several lines of lens units <NUM> may be provided corresponding to the image sensors.

The imager <NUM> is not necessarily provided in the camera head <NUM>. For example, the imager <NUM> may be provided immediately behind the objective lens inside the barrel <NUM>.

The driver <NUM> is configured with an actuator and moves the zoom lens and the focus lens of the lens unit <NUM> by a predetermined distance along the optical axis under the control of the camera head controller <NUM>. The magnification and the focal point of an image captured by the imager <NUM> thus can be adjusted as appropriate.

The communication module <NUM> is configured with a communication device for transmitting/receiving a variety of information to/from the CCU <NUM>. The communication module <NUM> transmits an image signal obtained from the imager <NUM> as RAW data to the CCU <NUM> through the transmission cable <NUM>.

The communication module <NUM> receives a control signal for controlling actuation of the camera head <NUM> from the CCU <NUM> and supplies the received signal to the camera head controller <NUM>. The control signal includes, for example, information on imaging conditions, such as information specifying a frame rate of the captured images, information specifying an exposure value in imaging, and/or information specifying a magnification and a focal point of the captured image.

The image conditions such as frame rate, exposure value, magnification, and focal point may be specified as appropriate by the user or may be automatically set by the controller <NUM> of the CCU <NUM> based on the acquired image signal. In the latter case, the endoscope <NUM> is equipped with an auto exposure (AE) function, an auto focus (AF) function, and an auto white balance (AWB) function.

The camera head controller <NUM> controls actuation of the camera head <NUM>, based on a control signal received from the CCU <NUM> through the communication module <NUM>.

The communication module <NUM> is configured with a communication device for transmitting/receiving a variety of information to/from the camera head <NUM>. The communication module <NUM> receives an image signal transmitted from the camera head <NUM> through the transmission cable <NUM>.

The communication module <NUM> transmits a control signal for controlling actuation of the camera head <NUM> to the camera head <NUM>. The image signal and the control signal can be transmitted via electrical communication or optical communication.

The image processor <NUM> performs a variety of image processing on the image signal that is RAW data transmitted from the camera head <NUM>.

The controller <NUM> performs a variety of control on imaging of a surgery site and the like by the endoscope <NUM> and display of a captured image obtained by imaging of a surgery site and the like. For example, the controller <NUM> generates a control signal for controlling actuation of the camera head <NUM>.

The controller <NUM> displays a captured image visualizing a surgery site and the like on the display device <NUM>, based on the image signal subjected to image processing by the image processor <NUM>. In doing so, the controller <NUM> may recognize a variety of objects in the captured image using a variety of image recognition techniques. For example, the controller <NUM> can detect the shape of edge, color, and the like of an object included in the captured image to recognize a surgical instrument such as forceps, a specific living body site, bleeding, and mist in use of the energy treatment tool <NUM>. When displaying the captured image on the display device <NUM>, the controller <NUM> may use the recognition result to superimpose a variety of surgery assisting information on the image of the surgery site. The surgery assisting information superimposed and presented to the operator <NUM> can alleviate burden on the operator <NUM> or ensure the operator <NUM> to proceed surgery.

The transmission cable <NUM> connecting the camera head <NUM> and the CCU <NUM> is an electrical signal cable corresponding to communication of electrical signals, an optical fiber corresponding to optical communication, or a composite cable thereof.

In the example illustrated in the drawing, the transmission cable <NUM> is used for wired communication. However, communication between the camera head <NUM> and the CCU <NUM> may be wireless.

An example of the endoscopic surgery system to which the technique according to the present disclosure is applicable has been described above. The technique according to the present disclosure is applicable to, for example, the imager <NUM> and the like in the camera head <NUM> among the configurations described above. When the technique according to the present disclosure is applied to the camera head <NUM>, the camera head <NUM> and the like can be miniaturized, resulting in the compact endoscopic surgery system <NUM>. When the technique according to the present disclosure is applied to the camera head <NUM> and the like, a clear image with reduced noise can be acquired to provide the operator with a more visible image. Consequently, the operator's fatigue can be alleviated.

Although the endoscopic surgery system has been described here by way of example, the technique according to the present disclosure may be applied to, for example, a microscopic surgery system.

The technique according to the present disclosure is applicable to a variety of products. For example, the technique according to the present disclosure may be applied to a pathology diagnosis system to allow doctors to diagnose pathological changes by observing cells and tissues sampled from patients, and an assistance system therefor (hereinafter referred to as diagnostic assistance system). This diagnostic assistance system may be a whole slide imaging (WSI) system for diagnosing pathological changes based on an image acquired using digital pathology technology, and assisting the diagnosis.

<FIG> is a diagram illustrating an example of the overall configuration of a diagnostic assistance system <NUM> to which the technique according to the present disclosure is applied. As illustrated in <FIG>, the diagnostic assistance system <NUM> includes one or more pathology systems <NUM>. The diagnostic assistance system <NUM> may further include a medical information system <NUM> and a derivation device <NUM>.

Each of one or more pathology systems <NUM> is a system mainly used by pathologists and introduced into, for example, a research laboratory or a hospital. The pathology systems <NUM> may be introduced into different hospitals and are connected to the medical information system <NUM> and the derivation device <NUM> through a variety of networks such as wide area networks (WANs) (including the Internet), local area networks (LAN), public networks, and mobile communication networks.

Each pathology system <NUM> includes a microscope <NUM>, a server <NUM>, a display control device <NUM>, and a display device <NUM>.

The microscope <NUM> has the function of an optical microscope and captures an image of an observation target on a glass slide to acquire a pathological image that is a digital image. The observation target is, for example, tissues or cells sampled from a patient and may be a piece of organ, saliva, or blood.

The server <NUM> stores and saves the pathological image acquired by the microscope <NUM> in a not-illustrated storage unit. When accepting an inspection request from the display control device <NUM>, the server <NUM> searches the not-illustrated storage unit for a pathological image and sends the retrieved pathological image to the display control device <NUM>.

The display control device <NUM> sends an inspection request for a pathological image accepted from the user to the server <NUM>. The display control device <NUM> then displays the pathological image accepted from the server <NUM> on the display device <NUM> using liquid crystal, electro-luminescence (EL), cathode ray tube (CRT), or the like. The display device <NUM> may support <NUM> or <NUM>, and one or more display devices <NUM> may be provided.

Here, when the observation target is a solid matter such as a piece of organ, the observation target may be, for example, a stained slice. The slice may be prepared, for example, by slicing a block cut out from a specimen such as an organ. When sliced, the block may be fixed by, for example, paraffin.

In staining the slice, a variety of staining can be employed, such as common staining such as hematoxylin-eosin (HE) staining for defining the form of tissue, and immunostaining such as immunohistochemistry (IHC) staining for identifying the immune state of tissue. In doing so, one slice may be stained using different kinds of reagents, or two or more slices (also referred to as adjacent slices) continuously cut out from the same block may be stained using different reagents.

The microscope <NUM> may include a low-resolution imager for capturing an image at low resolution and a high-resolution imager for capturing an image at high resolution. The low-resolution imager and the high-resolution imager may be different optical systems or the same optical system. In the case of the same optical system, the microscope <NUM> may have a resolution changed according to an imaging target.

A glass slide having an observation target is placed on a stage positioned in the angle of view of the microscope <NUM>. The microscope <NUM> first acquires the entire image in the angle of view using the low-resolution imager and specifies the region of the observation target from the acquired entire image. Subsequently, the microscope <NUM> divides the region including the observation target into a plurality of division regions with a predetermined size and successively captures images of the division regions using the high-resolution imager to acquire high-resolution images of the division regions. In switching the target division regions, the stage may be moved, the imaging optical system may be moved, or both may be moved. Each division region may be overlapped with the adjacent division region in order to prevent occurrence of an imaging-missed region due to unintended slippage of the glass slide. The entire image may include identification information for associating the entire image with the patient. Examples of the identification information include a character string and a QR code (registered trademark).

The high-resolution images acquired by the microscope <NUM> are input to the server <NUM>. The server <NUM> divides each high-resolution image into partial images (hereinafter referred to as tile images) with a smaller size. For example, the server <NUM> vertically and horizontally divides one high-resolution image into <NUM>×<NUM>, in total, <NUM> tile images. In doing so, if adjacent division regions are overlapping, the server <NUM> may perform a stitching process for the high-resolution images adjacent to each other using such technology as template matching. In this case, the server <NUM> may divide the stitched high-resolution images as a whole to generate tile images. However, the generation of tile images from a high-resolution image may precede the stitching process.

The server <NUM> may further divide a tile image to generate tile images with a smaller size. The generation of such tile images may be repeated until a tile image set as a minimum unit is generated.

Upon generating a tile image as a minimum unit, the server <NUM> executes a tile combining process for all the tile images to combine a predetermined number of adjacent tile images and generate one tile image. This tile combining process may be repeated until finally one tile image is generated. Through such a process, a tile image group having a pyramid structure is generated, in which each layer is configured with one or more tile images. In this pyramid structure, a tile image on a certain layer and a tile image on a layer different from this layer have the same pixel count, but their resolutions are different. For example, when <NUM>×<NUM>, in total, four tile images are combined into one tile image on a higher layer, the resolution of the tile image on the higher layer is half the resolution of the tile images on the lower layer used in combining.

When such a tile image group having a pyramid structure is constructed, the level of detail of the observation target appearing on the display device can be changed according to the layer to which a tile image to be displayed belongs to. For example, when the tile image on the lowest layer is used, a narrow region of the observation target can be displayed in detail, and as the tile image on a higher layer is used, a wide region of the observation target can be displayed more coarsely.

The generated tile image group having a pyramid structure is, for example, stored in a not-illustrated storage unit together with identification information uniquely identifying each tile image (referred to as tile identification information). When accepting an acquisition request for a tile image including tile identification information from another device (for example, the display control device <NUM> or the derivation device <NUM>), the server <NUM> transmits the tile image corresponding to the tile identification information to another device.

The tile image that is a pathological image may be generated for each imaging condition such as focal length and staining condition. When a tile image is generated for each imaging condition, a certain pathological image as well as another pathological image corresponding to an imaging condition different from a certain imaging condition and in the same region as the certain pathological image may be displayed side by side. The certain imaging condition may be designated by an inspector. When the inspector designates a plurality of imaging conditions, pathological images in the same region corresponding to the respective imaging conditions may be displayed side by side.

The server <NUM> may store the tile image group having a pyramid structure in a storage device other than the server <NUM>, for example, a cloud server. A part or the whole of the tile image generating process as described above may be performed, for example, by a cloud server.

The display control device <NUM> extracts a desired tile image from the tile image group having a pyramid structure in accordance with input operation from the user and outputs the same to the display device <NUM>. Through such a process, the user can attain a sense of viewing the observation target while changing the observation magnification. That is, the display control device <NUM> functions as a virtual microscope. The virtual observation magnification here actually corresponds to a resolution.

High-resolution images can be captured by any methods. A high-resolution image may be acquired by capturing images of division regions while the stage is repeatedly stopped and moved, or a high-resolution image on a strip may be acquired by capturing images of division regions while the stage is moved at a predetermined speed. The process of generating tile images from a high-resolution image is not an essential configuration, and the resolution of the stitched high-resolution images as a whole may be changed stepwise to generate an image with resolution changing stepwise. Also in this case, a low-resolution image in a wide area to a high-resolution image in a narrow area can be presented stepwise to the user.

The medical information system <NUM> is an electronic health record system and stores information related to diagnosis, such as information identifying patients, disease information of patients, examination information and image information used in diagnosis, diagnosis results, and prescribed drugs. For example, a pathological image obtained by imaging an observation target of a patient may be saved once through the server <NUM> and thereafter displayed on the display device <NUM> by the display control device <NUM>. The pathologist using the pathology system <NUM> conducts pathology diagnosis based on the pathological image appearing on the display device <NUM>. The result of pathology diagnosis conducted by the pathologist is stored in the medical information system <NUM>.

The derivation device <NUM> may perform analysis of a pathological image. In this analysis, a learning model created by machine learning can be used. The derivation device <NUM> may derive the classification result of a certain region, the identification result of tissues, and the like, as the analysis result. The derivation device <NUM> may further derive the identification result such as cell information, count, position, and brightness information, scoring information therefor, and the like. These pieces of information derived by the derivation device <NUM> may be displayed as diagnostic assistance information on the display device <NUM> of the pathology system <NUM>.

The derivation device <NUM> may be a server system including one or more servers (including a cloud server). The derivation device <NUM> may be a configuration incorporated into, for example, the display control device <NUM> or the server <NUM> in the pathology system <NUM>. That is, a variety of analysis for a pathological image may be performed in the pathology system <NUM>.

The technique according to the present disclosure may be preferably applied, for example, to the microscope <NUM> among the configurations described above. Specifically, the technique according to the present disclosure can be applied to the low-resolution imager and/or the high-resolution imager in the microscope <NUM>. The application of the technique according to the present disclosure to the low-resolution imager and/or the high-resolution imager leads to miniaturization of the low-resolution imager and/or the high-resolution imager, and thus miniaturization of the microscope <NUM>. The miniaturization facilitates transportation of the microscope <NUM> and thereby facilitates system introduction and system replacement. In addition, when the technique according to the present disclosure is applied to the low-resolution imager and/or the high-resolution imager, a part or the whole of the process from acquisition of a pathological image to analysis of the pathological image can be performed on the fly in the microscope <NUM>, so that diagnostic assistance information can be output more promptly and appropriately.

The configuration described above is not limited to a diagnostic assistance system and may be applied generally to biological microscopes such as a confocal microscope, a fluorescent microscope, and a video microscope. Here, the observation target may be a biological sample such as cultured cell, fertilized egg, and sperm, a biological material such as cell sheet and three-dimensional cell tissue, and a living body such as zebrafish and mouse. The observation target may be observed in a microplate or a petri dish, rather than on a glass slide.

A moving image may be generated from still images of the observation target acquired using the microscope. For example, a moving image may be generated from still images captured successively for a predetermined period of time, or an image sequence may be generated from still images captured at predetermined intervals. With a moving image generated from still images, dynamic features of an observation target, such as motion such as pulsation, expansion, and migration of cancer cell, nerve cell, cardiac muscle tissue, sperm, and the like, and a division process of cultured cell and fertilized egg, can be analyzed using machine learning.

Although embodiments of the present disclosure have been described above, the technical scope of the present disclosure is not limited to the foregoing embodiments as they are and may be susceptible to various modifications without departing from claimed subject-matter. The constituent elements in different embodiments and modifications may be combined as appropriate.

The effects in the embodiments described in the present description are only by way of illustration and are not intended to be limitative, and any other effects may be achieved.

Claim 1:
A stacked light-receiving sensor comprising:
a first substrate (<NUM>; <NUM>; <NUM>);
a second substrate (<NUM>; <NUM>) bonded to the first substrate (<NUM>; <NUM>; <NUM>); and
connection wiring (<NUM>) attached to the second substrate for connecting the light-receiving sensor (<NUM>) with a circuit board (<NUM>),
the first substrate (<NUM>; <NUM>; <NUM>) including a pixel array (<NUM>) in which a plurality of unit pixels (101a) are arranged in a two-dimensional matrix,
the second substrate (<NUM>; <NUM>) including
a converter (17A; 17B) configured to convert an analog pixel signal output from the pixel array (<NUM>) to digital image data and
a digital signal processor (<NUM> ) configured to perform a process based on a neural network calculation model for data based on the image data, wherein
at least a part of the converter (17A; 17B) is disposed on a first side of the second substrate in plan view,
the digital signal processor (<NUM>) is disposed on a second side opposite to the first side on the second substrate in plan view (<NUM>; <NUM>), wherein the first side is upstream of the second side along the flow of a signal read out from the pixel array, wherein the digital signal processor(<NUM>) is not superimposed on the pixel array (<NUM>) in the stacking direction of the first substrate (<NUM>) and the second substrate (<NUM>), and
the connection wiring (<NUM>) is attached to the second side of the second substrate