Patent Description:
In the related art, a harvest prediction device is proposed in which the types of crops are recognized, and a yield amount is predicted with respect to each of the crops, from an image in which various crops cultivated in a wide region are photographed (for example, refer to PTL <NUM>).

In addition, an imaging element detecting light in a predetermined narrow wavelength band (a narrow band) (hereinafter, also referred to as narrow band light) by using a plasmon filter is proposed (for example, refer to PTL <NUM>).

Patent Literature <CIT> discloses a method and system for using one or more sensors configured to capture two-dimensional and/or three dimensional image data of one or more objects.

However, as described above, in object recognition processing of recognizing an object from an image, such as recognition of the type of crop photographed in the image, in general, it is necessary to perform machine learning in advance by using a large quantity of training data. In the related art, for example, the training data is generated by visually determining and labeling the object, and thus, enormous man-hour is necessary for preparing a large quantity of training data.

The present disclosure has been made in consideration of the circumstances described above, and is capable of automating the labeling of an object.

According to the present disclosure, there is provided an imaging system as defined in the appended set of claims.

According to one aspect of the present disclosure, it is possible to automate the labeling of an object.

Hereinafter, aspects for carrying out the disclosure (hereinafter, referred to as an "embodiment") will be described in detail by using the drawings. Furthermore, the embodiments will be described in the following sequence.

First, an embodiment of an imaging device of the present technology will be described with reference to <FIG>.

<FIG> is a block diagram illustrating an embodiment of an imaging device, which is one type of electronic devices to which the present technology is applied.

An imaging device <NUM> of <FIG>, for example, is formed of a digital camera which is capable of imaging both of a still image and a moving image. In addition, the imaging device <NUM>, for example, is formed of a multispectral camera which is capable of detecting light (multi-spectrum) of four or more wavelength bands (four or more bands) greater than three wavelength bands (three bands) of the related art of R (red), G (green), and B (blue) or Y (yellow), M (magenta), and C (cyan), based on three primary colors or a color-matching function.

The imaging device <NUM> includes an optical system <NUM>, an imaging element <NUM>, a memory <NUM>, a signal processing unit <NUM>, an output unit <NUM>, and a control unit <NUM>.

The optical system <NUM>, for example, includes a zoom lens, a focus lens, a diaphragm, and the like, which are not illustrated, and allows light from the outside to be incident on the imaging element <NUM>. In addition, as necessary, various filters such as a polarization filter are disposed on the optical system <NUM>.

The imaging element <NUM>, for example, is formed of a complementary metal oxide semiconductor (CMOS) image sensor. The imaging element <NUM> receives incident light from the optical system <NUM>, and performs photoelectric conversion, and thus, outputs image data corresponding to the incident light.

The memory <NUM> temporarily stores the image data which is output from the imaging element <NUM>.

The signal processing unit <NUM> performs signal processing (for example, processing such as elimination of a noise and adjustment of a white balance) using the image data stored in the memory <NUM>, and thus, supplies the image data to the output unit <NUM>.

The output unit <NUM> outputs the image data from the signal processing unit <NUM>. For example, the output unit <NUM> includes a display (not illustrated) configured of a liquid crystal or the like, and displays a spectrum (an image) corresponding to the image data from the signal processing unit <NUM> as a so-called through image. For example, the output unit <NUM> includes a driver (not illustrated) driving a recording medium such as a semiconductor memory, a magnetic disk, and an optical disk, and records the image data from the signal processing unit <NUM> in a recording medium. For example, the output unit <NUM> functions as a communication interface for performing communication with respect to an external device (not illustrated), and transmits the image data from the signal processing unit <NUM> to the external device in a wireless manner or a wired manner.

The control unit <NUM> controls each of the units of the imaging device <NUM> according to an operation or the like of a user.

<FIG> is a block diagram illustrating a configuration example of a circuit of the imaging element <NUM> of <FIG>.

The imaging element <NUM> includes a pixel array <NUM>, a row scanning circuit <NUM>, a phase locked loop (PLL) <NUM>, a digital analog converter (DAC) <NUM>, a column analog digital converter (ADC) circuit <NUM>, a column scanning circuit <NUM>, and a sense amplifier <NUM>.

A plurality of pixels <NUM> are two-dimensionally arranged in the pixel array <NUM>.

The pixel <NUM> includes a horizontal signal line H which is connected to the row scanning circuit <NUM>, a photodiode <NUM> which is disposed in each point where the photodiode <NUM> intersects with a perpendicular signal line V connected to the column ADC circuit <NUM>, and performs photoelectric conversion, and several types of transistors for reading out an accumulated signal. That is, the pixel <NUM>, as enlargedly illustrated on the right side of <FIG>, includes the photodiode <NUM>, a transfer transistor <NUM>, a floating diffusion <NUM>, an amplification transistor <NUM>, a selection transistor <NUM>, and a reset transistor <NUM>.

An electric charge accumulated in the photodiode <NUM> is transferred to the floating diffusion <NUM> through the transfer transistor <NUM>. The floating diffusion <NUM> is connected to a gate of the amplification transistor <NUM>. In a case where the pixel <NUM> is a target from which a signal is read out, the selection transistor <NUM> is turned on from the row scanning circuit <NUM> through the horizontal signal line H, and the amplification transistor <NUM> is subjected to source follower driving according to the signal of the selected pixel <NUM>, and thus, the signal is read out to the perpendicular signal line V as a pixel signal corresponding to an accumulation electric charge amount of the electric charge accumulated in the photodiode <NUM>. In addition, the pixel signal is reset by turning on the reset transistor <NUM>.

The row scanning circuit <NUM> sequentially outputs a driving (for example, transferring, selecting, resetting, or the like) signal for driving the pixel <NUM> of the pixel array <NUM> for each row.

The PLL <NUM> generates and outputs a clock signal of a predetermined frequency which is necessary for driving each of the units of the imaging element <NUM>, on the basis of the clock signal supplied from the outside.

The DAC <NUM> generates and outputs a lamp signal in the shape of being returned to a predetermined voltage value after a voltage drops from a predetermined voltage value at a certain slope (in the shape of approximately a saw).

The column ADC circuit <NUM> includes a comparator <NUM> and a counter <NUM> as many as the number corresponding to the number of columns of the pixel <NUM> of the pixel array <NUM>, extracts a signal level from the pixel signal output from the pixel <NUM> by a correlated double sampling (CDS) operation, and outputs pixel data. That is, the comparator <NUM> compares the lamp signal supplied from the DAC <NUM> with the pixel signal (a brightness value) output from the pixel <NUM>, and supplies a comparison result signal obtained as the result thereof to the counter <NUM>. Then, the counter <NUM> counts a counter clock signal of a predetermined frequency according to the comparison result signal output from the comparator <NUM>, and thus, the pixel signal is subjected to A/D conversion.

The column scanning circuit <NUM> sequentially supplies a signal of outputting the pixel data to the counter <NUM> of the column ADC circuit <NUM> at a predetermined timing.

The sense amplifier <NUM> amplifies the pixel data which is supplied from the column ADC circuit <NUM>, and outputs the pixel data to the outside of the imaging element <NUM>.

<FIG> schematically illustrates a configuration example of a sectional surface of an imaging element 12A, which is a first embodiment of the imaging element <NUM> of <FIG>. <FIG> illustrates sectional surfaces of four pixels of a pixel <NUM>-<NUM> to a pixel <NUM>-<NUM> of the imaging element <NUM>. Furthermore, hereinafter, in a case where it is not necessary to distinguish the pixel <NUM>-<NUM> to the pixel <NUM>-<NUM> from each other, the pixel will be simply referred to as the pixel <NUM>.

An on-chip microlens <NUM>, an interlayer film <NUM>, a narrow band filter layer <NUM>, an interlayer film <NUM>, a photoelectric conversion element layer <NUM>, and a signal wiring layer <NUM> are laminated in each of the pixels <NUM>, in this order from the above. That is, the imaging element <NUM> is formed of a back-side illumination type CMOS image sensor in which the photoelectric conversion element layer <NUM> is disposed on an incident side of light from the signal wiring layer <NUM>.

The on-chip microlens <NUM> is an optical element for condensing light into the photoelectric conversion element layer <NUM> of each of the pixels <NUM>.

The interlayer film <NUM> and the interlayer film <NUM> are formed of a dielectric body such as SiO2. As described below, it is desirable that dielectric constants of the interlayer film <NUM> and the interlayer film <NUM> are as low as possible.

In the narrow band filter layer <NUM>, a narrow band filter NB, which is an optical filter transmitting narrow band light in a predetermined narrow wavelength band (a narrow band), is disposed in each of the pixels <NUM>. For example, a plasmon filter using front plasmon, which is one type of metal thin film filters using a thin film formed of a metal such as aluminum, is used in the narrow band filter NB. In addition, a transmission band of the narrow band filter NB is set for each of the pixels <NUM>. The type (the number of bands) of the transmission band of the narrow band filter NB is arbitrary, and for example, the number of bands is set to be greater than or equal to <NUM>.

Here, the narrow band, for example, is a wavelength band which is narrower than a transmission band of a color filter of the related art of red (R), green (G), and blue (B) or yellow (Y), magenta (M), and cyan (C), based on three primary colors or a color-matching function. In addition, hereinafter, a pixel receiving the narrow band light transmitted through the narrow band filter NB will be referred to as a multispectral pixel or a MS pixel.

The photoelectric conversion element layer <NUM>, for example, includes the photodiode <NUM> or the like of <FIG>, receives the light transmitted through the narrow band filter layer <NUM> (the narrow band filter NB) (the narrow band light), and converts the received light into an electric charge. In addition, the photoelectric conversion element layer <NUM> is configured such that the pixels <NUM> are electrically separated from each other by an element separating layer.

Wiring or the like for reading the electric charge which is accumulated in the photoelectric conversion element layer <NUM> is disposed on the signal wiring layer <NUM>.

Next, the plasmon filter which can be used in the narrow band filter NB will be described with reference to <FIG>.

<FIG> illustrates a configuration example of a plasmon filter 121A having a pore array structure.

The plasmon filter 121A is configured of a plasmon resonator in which holes 132A are arranged in a metal thin film (hereinafter, referred to as a conductor thin film) 131A in the shape of a honeycomb.

Each of the holes 132A penetrates through the conductor thin film 131A, and functions as a waveguide. In general, the waveguide has a cutoff frequency and a cutoff wavelength which are determined according to a shape such as a length of a side or a diameter, and has properties of not allowing light of a frequency less than or equal to the cutoff frequency (a wavelength less than or equal to the cutoff wavelength) to propagate. A cutoff wavelength of the hole 132A mainly depends on an opening diameter D1, and the cutoff wavelength shortens as the opening diameter D1 decreases. Furthermore, the opening diameter D1 is set to a value which is smaller than the wavelength of the transmitted light.

On the other hand, in a case where light is incident on the conductor thin film 131A in which holes 132A are periodically formed during a short period less than or equal to the wavelength of the light, a phenomenon occurs in which light at a wavelength which is longer than the cutoff wavelength of the hole 132A is transmitted. Such a phenomenon will be referred to as an abnormal transmission phenomenon of the plasmon. Such a phenomenon occurs due to the excitation of front plasmon on a boundary between the conductor thin film 131A and the interlayer film <NUM>, which is an upper layer of the conductor thin film 131A.

Here, occurrence conditions of the abnormal transmission phenomenon of the plasmon (a front plasmon resonance) will be described with reference to <FIG>.

<FIG> is a graph illustrating a dispersion relationship of the front plasmon. In the graph, a horizontal axis represents an angular wave number vector k, and a vertical axis represents an angular frequency ω. ωp represents a plasma frequency of the conductor thin film 131A. ωsp represents a front plasma frequency on a boundary surface between the interlayer film <NUM> and the conductor thin film 131A, and is represented by formula (<NUM>) described below. <NUM>] <MAT>.

εd represents a dielectric constant of a dielectric body configuring the interlayer film <NUM>.

According to formula (<NUM>), the front plasma frequency ωsp increases as the plasma frequency ωp increases. In addition, the front plasma frequency ωsp increases as the dielectric constant εd decreases.

A line L1 represents a dispersion relationship of the light (a write line), and is represented by formula (<NUM>) described below. <NUM>] <MAT>.

A line L2 represents a dispersion relationship of the front plasmon, and is represented by formula (<NUM>) described below. <NUM>] <MAT>.

εm represents a dielectric constant of the conductor thin film 131A.

The dispersion relationship of the front plasmon represented by the line L2 is close to the write line represented by the line L1 in a range where the angular wave number vector k is small, and is close to the front plasma frequency ωsp as the angular wave number vector k increases.

Then, when formula (<NUM>) described below is established, the abnormal transmission phenomenon of the plasmon occurs. <NUM>] <MAT>.

λ represents the wavelength of the incident light. θ represents an incident angle of the incident light. Gx and Gy are represented by formula (<NUM>) described below.

a0 represents a lattice constant of a pore array structure formed of the hole 132A of the conductor thin film 131A.

In formula (<NUM>), the left member represents an angular wave number vector of the front plasmon, and the right member represents an angular wave number vector of the conductor thin film 131A during a pore array period. Accordingly, when the angular wave number vector of the front plasmon is identical to the angular wave number vector of the conductor thin film 131A during the pore array period, the abnormal transmission phenomenon of the plasmon occurs. Then, at this time, the value of λ is a resonance wavelength of the plasmon (the transmission wavelength of the plasmon filter 121A).

Furthermore, in formula (<NUM>), the angular wave number vector of the front plasmon in the left member is determined according to the dielectric constant εm of the conductor thin film 131A and the dielectric constant εd of the interlayer film <NUM>. On the other hand, the angular wave number vector during the pore array period in the right member is determined according to the incident angle θ of the light and a pitch (a hole pitch) P1 between the adjacent holes 132A of the conductor thin film 131A. Accordingly, the resonance wavelength and the resonance frequency of the plasmon are determined according to the dielectric constant εm of the conductor thin film 131A, the dielectric constant εd of the interlayer film <NUM>, the incident angle θ of the light, and the hole pitch P1. Furthermore, in a case where the incident angle of the light is <NUM>°, the resonance wavelength and the resonance frequency of the plasmon are determined according to the dielectric constant εm of the conductor thin film 131A, the dielectric constant εd of the interlayer film <NUM>, and the hole pitch P1.

Accordingly, the transmission band of the plasmon filter 121A (the resonance wavelength of the plasmon) is changed according to a material and a film thickness of the conductor thin film 131A, a material and a film thickness of the interlayer film <NUM>, a pattern period of the pore array (for example, the opening diameter D1 and the hole pitch P1 of the hole 132A), and the like. In particular, in a case where the material and the film thickness of the conductor thin film 131A and the interlayer film <NUM> are determined, the transmission band of the plasmon filter 121A is changed according to the pattern period of the pore array, in particular, the hole pitch P1. That is, the transmission band of the plasmon filter 121A is shifted to a short wavelength side as the hole pitch P1 narrows, and the transmission band of the plasmon filter 121A is shifted to a long wavelength side as the hole pitch P1 widens.

<FIG> is a graph illustrating an example of spectral characteristics of the plasmon filter 121A in a case where the hole pitch P1 is changed. In the graph, a horizontal axis represents a wavelength (the unit is nm), and a vertical axis represents sensitivity (the unit is an arbitrary unit). A line L11 represents spectral characteristics in a case where the hole pitch P1 is set to <NUM>, a line L12 represents spectral characteristics in a case where the hole pitch P1 is set to <NUM>, and a line L13 represents spectral characteristics in a case where the hole pitch P1 is set to <NUM>.

In a case where the hole pitch P1 is set to <NUM>, the plasmon filter 121A mainly transmits light in a wavelength band of a blue color. In a case where the hole pitch P1 is set to <NUM>, the plasmon filter 121A mainly transmits light in a wavelength band of a green color. In a case where the hole pitch P1 is set to <NUM>, the plasmon filter 121A mainly transmits light in a wavelength band of a red color. However, in a case where the hole pitch P1 is set to <NUM>, the plasmon filter 121A transmits a great amount of light in a low wavelength band of a red color according to a waveguide mode described below.

<FIG> is a graph illustrating another example of the spectral characteristics of the plasmon filter 121A in a case where the hole pitch P1 is changed. In the graph, a horizontal axis represents a wavelength (the unit is nm), and a vertical axis represents sensitivity (the unit is an arbitrary unit). This example illustrates an example of spectral characteristics of sixteen types of plasmon filters 121A in a case where the hole pitch P1 is changed by being divided by <NUM> from <NUM> to <NUM>.

Furthermore, the transmittance of the plasmon filter 121A is mainly determined according to the opening diameter D1 of the hole 132A. The transmittance increases as the opening diameter D1 increases, but color mixture easily occurs. In general, it is desirable that the opening diameter D1 is set such that an opening rate is <NUM>% to <NUM>% of the hole pitch P1.

In addition, as described above, each of the holes 132A of the plasmon filter 121A functions as a waveguide. Accordingly, in the spectral characteristics, there is a case where not only a wavelength component transmitted by the front plasmon resonance (a wavelength component in a plasmon mode) but also a wavelength component transmitted through the hole 132A (the waveguide) (a wavelength component in a waveguide mode) increases, according to a pattern of the pore array of the plasmon filter 121A.

For a given hole pitch P1 of the plasmon filter there is a range of desirable thicknesses of the plasmon filter to maximize light transmittance of the filter for those wavelengths that are transmitted. For instance, a range of desirable thicknesses of the plasmon filter may range between <NUM>% and <NUM>% of the size of the hole pitch P1, or between <NUM>% and <NUM>% of the size of the hole pitch, or between <NUM>% and <NUM>% of the size of the hole pitch.

For example, in a case where the plasmon filter is formed from Aluminum, a desirable range of thicknesses of the plasmon filter for a <NUM> hole pitch is between <NUM> and <NUM>, with a preferred thickness of <NUM>. For an Aluminum plasmon filter with a <NUM> hole pitch, a desirable range of thicknesses of the plasmon filter is between <NUM> and <NUM>, with a preferred thickness of <NUM>.

For a given peak transmission wavelength of the plasmon filter there is a range of desirable thicknesses of the plasmon filter to maximize light transmittance of the filter for those wavelengths that are transmitted. For instance, a range of desirable thicknesses of the plasmon filter may range between <NUM>% and <NUM>% of the peak transmission wavelength, or between <NUM>% and <NUM>% of the peak transmission wavelength, or between <NUM>% and <NUM>% of the peak transmission wavelength.

For example, in a case where the plasmon filter is formed from Aluminum, a desirable range of thicknesses of the plasmon filter when desirable a peak transmission wavelength of <NUM> is between <NUM> and <NUM>, with a preferred thickness of <NUM>. For an Aluminum plasmon filter with a peak transmission wavelength of <NUM>, a desirable range of thicknesses of the plasmon filter is between <NUM> and <NUM>, with a preferred thickness of <NUM>.

<FIG> illustrates the spectral characteristics of the plasmon filter 121A in a case where the hole pitch P1 is set to <NUM>, as with the spectral characteristics represented by the line L13 of <FIG>. In this example, a wavelength side which is longer than the cutoff wavelength in the vicinity of <NUM> is the wavelength component in the plasmon mode, and a wavelength side which is shorter than the cutoff wavelength is the wavelength component in the waveguide mode.

As described above, the cutoff wavelength mainly depends on the opening diameter D1 of the hole 132A, and the cutoff wavelength decreases as the opening diameter D1 decreases. Then, wavelength resolution characteristics of the plasmon filter 121A are improved as a difference between the cutoff wavelength and the peak wavelength in the plasmon mode increases.

In addition, as described above, the front plasma frequency ωsp of the conductor thin film 131A increases as the plasma frequency ωp of the conductor thin film 131A increases. In addition, the front plasma frequency ωsp increases as the dielectric constant εd of the interlayer film <NUM> decreases. Then, it is possible to set the resonance frequency of the plasmon to be higher as the front plasma frequency ωsp increases, and to set the transmission band of the plasmon filter 121A (the resonance wavelength of the plasmon) to a shorter wavelength band.

Accordingly, in a case where a metal having a smaller plasma frequency ωp is used in the conductor thin film 131A, it is possible to set the transmission band of the plasmon filter 121A to a shorter wavelength band. For example, aluminum, silver, gold, or the like is preferable as the metal. Here, in a case where the transmission band is set to a long wavelength band of infrared light or the like, copper or the like can also be used.

In addition, in a case where a dielectric body having a small dielectric constant εd is used in the interlayer film <NUM>, it is possible to set the transmission band of the plasmon filter 121A to a shorter wavelength band. For example, SiO2, Low-K, or the like is preferable as the dielectric body.

In addition, <FIG> is a graph illustrating propagation characteristics of the front plasmon on an interface between conductor thin film 131A and the interlayer film <NUM> in a case where aluminum is used in the conductor thin film 131A, and SiO2 is used in the interlayer film <NUM>. In the graph, a horizontal axis represents the wavelength of the light (the unit is nm), and a vertical axis represents a propagation distance (the unit is µm). In addition, a line L21 represents propagation characteristics in an interface direction, a line L22 represents propagation characteristics in a depth direction of the interlayer film <NUM> (a direction perpendicular to the interface), and a line L23 represents propagation characteristics in a depth direction of the conductor thin film 131A (a direction perpendicular to the interface).

A propagation distance ASPP (λ) in a depth direction of the front plasmon is represented by formula (<NUM>) described below. <NUM>] <MAT>.

kSPP represents an absorption coefficient of a substance propagated by the front plasmon. εm (λ) represents a dielectric constant of the conductor thin film 131A with respect to light at a wavelength of λ. εd (λ) represents a dielectric constant of the interlayer film <NUM> with respect to light at the wavelength of λ.

Accordingly, as illustrated in <FIG>, front plasmon with respect to light at a wavelength of <NUM> propagates in the depth direction from a front surface of the interlayer film <NUM> formed of SiO2 to approximately <NUM>. Accordingly, the thickness of the interlayer film <NUM> is set to be greater than or equal to <NUM>, and thus, the front plasmon on the interface between the interlayer film <NUM> and the conductor thin film 131A is prevented from being affected by a substance laminated on a surface of the interlayer film <NUM> on a side opposite to the conductor thin film 131A.

In addition, front plasmon with respect to light at a wavelength of <NUM> propagates in the depth direction from a front surface of the conductor thin film 131A formed of aluminum to approximately <NUM>. Accordingly, the thickness of the conductor thin film 131A is set to be greater than or equal to <NUM>, and thus, the front plasmon on the interface between the interlayer film <NUM> and the conductor thin film 131A is prevented from being affected by the interlayer film <NUM>.

Next, other examples of the plasmon filter will be described with reference to <FIG>.

A plasmon filter 121B of <FIG> is configured of a plasmon resonator in which holes 132B are arranged in a conductor thin film 131B in the shape of an orthogonal matrix. In the plasmon filter 121B, for example, a transmission band is changed according to a pitch P2 between adjacent holes 132B.

In addition, in the plasmon resonator, it is not necessary that all of the holes penetrate through the conductor thin film, and even in a case where a part of the holes is configured as a non-through which does not penetrate through the conductor thin film, the plasmon resonator functions as a filter.

For example, in <FIG>, a plan view and a sectional view (a sectional view taken along A-A' of the plan view) of a plasmon filter 121C configured of a plasmon resonator in which holes 132C formed of through holes and holes 132C' formed of non-through holes are arranged in the conductor thin film 131C in the shape of a honeycomb. That is, holes 132C formed of through holes and holes 132C' formed of non-through holes are periodically arranged in the plasmon filter 121C.

Further, a plasmon resonator of a single layer is basically used as the plasmon filter, and for example, the plasmon filter can be configured of a two-layer plasmon resonator.

For example, a plasmon filter 121D illustrated in <FIG> is configured of two layers of a plasmon filter 121D-<NUM> and a plasmon filter 121D-<NUM>. The plasmon filter 121D-<NUM> and the plasmon filter 121D-<NUM> have a structure in which holes are arranged in the shape of a honeycomb, as with the plasmon resonator configuring the plasmon filter 121A of <FIG>.

In addition, it is preferable that an interval D2 between the plasmon filter 121D-<NUM> and the plasmon filter 121D-<NUM> is approximately <NUM>/<NUM> of a peak wavelength of a transmission band. In addition, in consideration of the freedom in design, it is preferable that the interval D2 is less than or equal to <NUM>/<NUM> of the peak wavelength of the transmission band.

Furthermore, as with the plasmon filter 121D, the holes are arranged in the same pattern in the plasmon filter 121D-<NUM> and the plasmon filter 121D-<NUM>, and for example, the holes may be arranged in patterns similar to each other in a two-layer plasmon resonator structure. In addition, in the two-layer plasmon resonator structure, holes and dots may be arranged in a pattern in which a pore array structure and a dot array structure (described below) are inversed from each other. Further, the plasmon filter 121D has the two-layer structure, and is capable of being multilayered to be three or more layers.

In addition, in the above description, the configuration example of the plasmon filter using the plasmon resonator having the pore array structure has been described, but a plasmon resonator having a dot array structure may be adopted as the plasmon filter.

A plasmon filter having a dot array structure will be described with reference to <FIG>.

A plasmon filter 121A' of <FIG> has a structure which is negatively and positively inversed with respect to the plasmon resonator of the plasmon filter 121A of <FIG>, that is, is configured of a plasmon resonator in which dots 133A are arranged in a dielectric layer 134A in the shape of a honeycomb. A space between the respective dots 133A is filled with the dielectric layer 134A.

The plasmon filter 121A' absorbs light in a predetermined wavelength band, and thus, is used as a complementary color filter. The wavelength band of the light which is absorbed by the plasmon filter 121A' (hereinafter, referred to as an absorption band) is changed according to a pitch (hereinafter, referred to as a dot pitch) P3 between the adjacent dots 133A. In addition, a diameter D3 of the dot 133A is adjusted according to the dot pitch P3.

A plasmon filter 121B' of <FIG> has a structure which is negatively and positively inversed with respect to the plasmon resonator of the plasmon filter 121B of <FIG>, that is, is configured of a plasmon resonator structure in which dots 133B are arranged in a dielectric layer 134B in the shape of an orthogonal matrix. A space between the respective dots 133B is filled with the dielectric layer 134B.

An absorption band of the plasmon filter 121B' is changed according to a dot pitch P4 or the like between the adjacent dots 133B. In addition, a diameter D3 of the dot 133B is adjusted according to the dot pitch P4.

<FIG> is a graph illustrating an example of spectral characteristics in a case where the dot pitch P3 of the plasmon filter 121A' of <FIG> is changed. In the graph, a horizontal axis represents a wavelength (the unit is nm), and a vertical axis represents transmittance. A line L31 represents spectral characteristics in a case where the dot pitch P3 is set to <NUM>, a line L32 represents spectral characteristics in a case where the dot pitch P3 is set to <NUM>, and a line L33 represents spectral characteristics in a case where the dot pitch P3 is set to <NUM>.

As illustrated in the drawing, the absorption band of the plasmon filter 121A' is shifted to a short wavelength side as the dot pitch P3 narrows, and the absorption band of the plasmon filter 121A' is shifted to a long wavelength side as the dot pitch P3 widens.

Furthermore, in both of the plasmon filters having the pore array structure and the dot array structure, the transmission band or the absorption band can be adjusted by only adjusting the pitch between the holes or the dots in a planar direction. Accordingly, for example, the transmission band or the absorption band can be individually set with respect to each pixel by only adjusting the pitch between the holes or the dots in a lithography process, and the filter can be multicolored through a fewer process.

In addition, the thickness of the plasmon filter is approximately <NUM> to <NUM>, which is approximately similar to that of a color filter of an organic material, and a process affinity is excellent.

In addition, a plasmon filter <NUM> using a guided mode resonant (GMR) illustrated in <FIG> can also be used in the narrow band filter NB.

A conductor layer <NUM>, an SiO2 film <NUM>, an SiN film <NUM>, and an SiO2 substrate <NUM> are laminated in the plasmon filter <NUM>, in this order from the above. The conductor layer <NUM>, for example, is included in the narrow band filter layer <NUM> of <FIG>, and the SiO2 film <NUM>, the SiN film <NUM>, and the SiO2 substrate <NUM>, for example, are included in the interlayer film <NUM> of <FIG>.

For example, rectangular conductor thin films 161A formed of aluminum are arranged in the conductor layer <NUM> such that long sides of the conductor thin films 161A are adjacent to each other at a predetermined pitch P5. Then, a transmission band of the plasmon filter <NUM> is changed according to the pitch P5 or the like.

<FIG> is a graph illustrating an example of spectral characteristics of the plasmon filter <NUM> in a case where the pitch P5 is changed. In the graph, a horizontal axis represents a wavelength (the unit is nm), and a vertical axis represents transmittance. This example illustrates an example of spectral characteristics in a case where the pitch P5 is changed to six types of pitches by being divided by <NUM> from <NUM> to <NUM>, and a width of a slit between the adjacent conductor thin films 161A is set to be <NUM>/<NUM> of the pitch P5. In addition, a waveform having the shortest peak wavelength of the transmission band represents spectral characteristics in a case where the pitch P5 is set to <NUM>, and the peak wavelength elongates as the pitch P5 widens. That is, the transmission band of the plasmon filter <NUM> is shifted to a short wavelength side as the pitch P5 narrows, and the transmission band of the plasmon filter <NUM> is shifted to a long wavelength side as the pitch P5 widens.

The plasmon filter <NUM> using GMR has excellent affinity with respect to a color filter of an organic material, as with the plasmon filters having the pore array structure and the dot array structure described above.

Next, a second embodiment of the imaging element <NUM> of <FIG> will be described with reference to <FIG>.

<FIG> schematically illustrates a configuration example of a sectional surface of an imaging element 12B which is the second embodiment of the imaging element <NUM>. Furthermore, in the drawing, the same reference numerals are applied to portions corresponding to the imaging element 12A of <FIG>, and the description thereof will be suitably omitted.

The imaging element 12B is different from the imaging element 12A in that a color filter layer <NUM> is laminated between the on-chip microlens <NUM> and the interlayer film <NUM>.

In the narrow band filter layer <NUM> of the imaging element 12B, the narrow band filter NB is disposed in a part of the pixels <NUM> but not all of the pixels <NUM>. The type of the transmission band of the narrow band filter NB (the number of bands) is arbitrary, and for example, the number of bands is set to be greater than or equal to <NUM>.

In the color filter layer <NUM>, a color filter is disposed in each of the pixels <NUM>. For example, in the pixel <NUM> where the narrow band filter NB is not disposed, any one of a general red color filter R, a general green color filter G, and a general blue color filter B (not illustrated) is disposed. Accordingly, for example, an R pixel in which the red color filter R is disposed, a G pixel in which the green color filter G is disposed, a B pixel in which the blue color filter is disposed, and an MS pixel in which in which the narrow band filter NB is disposed, are arranged in the pixel array <NUM>.

In addition, in the pixel <NUM> where the narrow band filter NB is disposed, a transmission filter P is disposed on the color filter layer <NUM>. As described below, the transmission filter P is configured of an optical filter transmitting light in a wavelength band including the transmission band of the narrow band filter NB of the same pixel <NUM> (a low pass filter, a high pass filter, or a band pass filter).

Furthermore, the color filter disposed on the color filter layer <NUM> may be color filters of both of an organic material and an inorganic material.

Examples of the color filter of the organic material include a dyeing and coloring color filter of a synthetic resin or natural protein, and a color filter containing a dye using a pigment dye or a colorant dye.

Examples of the color filter of the inorganic material include materials such as TiO2, ZnS, SiN, MgF2, SiO2, and Low-k. In addition, for example, a method such as vapor deposition, sputtering, and chemical vapor deposition (CVD) film formation is used for forming the color filter of the inorganic material.

In addition, as described above with reference to <FIG>, the interlayer film <NUM> is set to have a film thickness which is capable of preventing the influence of the color filter layer <NUM> on the front plasmon on an interface between the interlayer film <NUM> and the narrow band filter layer <NUM>.

Here, the occurrence of flare is suppressed by the transmission filter P disposed on the color filter layer <NUM>. This will be described with reference to <FIG> and <FIG>.

<FIG> schematically illustrates an aspect of the occurrence of the flare of the imaging device <NUM> using the imaging element 12A of <FIG> in which the color filter layer <NUM> is not disposed.

In this example, the imaging element 12A is disposed on a semiconductor chip <NUM>. Specifically, the semiconductor chip <NUM> is mounted on a substrate <NUM>, and is surrounded by seal glass <NUM> and a resin <NUM>. Then, light transmitted through a lens <NUM>, an IR cut filter <NUM>, and the seal glass <NUM>, which are disposed in the optical system <NUM> of <FIG>, is incident on the imaging element 12A.

Here, in a case where the narrow band filter NB of the narrow band filter layer <NUM> of the imaging element 12A is formed of a plasmon filter, a conductor thin film formed of metal is formed in the plasmon filter. The conductor thin film has a high reflection rate, and thus, light at a wavelength other than the transmission band is easily reflected. Then, a part of the light reflected on the conductor thin film, for example, as illustrated in <FIG>, is reflected on the seal glass <NUM>, the IR cut filter <NUM>, or the lens <NUM>, and is incident again on the imaging element 12A. The flare occurs due to the re-incident light. In particular, a plasmon filter using a pore array structure has a low opening rate, and thus, the flare easily occurs.

In order to prevent the reflection light, for example, it is considered that an antireflection film formed of a metal or a material having a high dielectric constant, which is different from the conductor thin film, is used. However, in a case where the plasmon filter uses a front plasmon resonance, and such an antireflection film is in contact with the front surface of the conductor thin film, there is a possibility that the characteristics of the plasmon filter are degraded, and desired characteristics are not obtained.

On the other hand, <FIG> schematically illustrates an aspect of the occurrence of the flare of the imaging device <NUM> using the imaging element 12B of <FIG>, in which the color filter layer <NUM> is disposed. Furthermore, in the drawing, the same reference numerals are applied to portions corresponding to those of <FIG>.

The example of <FIG> is different from the example of <FIG> in that a semiconductor chip <NUM> is disposed instead of the semiconductor chip <NUM>. The semiconductor chip <NUM> is different from the semiconductor chip <NUM> in that the imaging element 12B is disposed instead of the imaging element 12A.

As described above, in the imaging element 12B, the transmission filter P is disposed on an upper side from the narrow band filter NB (an incident side of light). Accordingly, the light incident on the imaging element 12B is incident on the narrow band filter NB, in which a predetermined wavelength band is cutoff, by the transmission filter P, and thus, a light amount of the incident light with respect to the narrow band filter NB is suppressed. As a result thereof, a light amount of the reflection light on the conductor thin film of the narrow band filter NB (the plasmon filter) is also reduced, and thus, the flare is reduced.

<FIG> illustrate examples of the spectral characteristics of the narrow band filter NB and the spectral characteristics of the transmission filter P disposed on the upper side of the narrow band filter NB. Furthermore, in the graphs of <FIG>, a horizontal axis represents a wavelength (the unit is nm), and a vertical axis represents sensitivity (the unit is an arbitrary unit).

In <FIG>, a line L41 represents the spectral characteristics of the narrow band filter NB. A peak wavelength of the spectral characteristics of the narrow band filter NB is approximately in the vicinity of <NUM>. A line L42 represents the spectral characteristics of a low pass type transmission filter P. A line L43 represents the spectral characteristics of a high pass type transmission filter P. A line L44 represents the spectral characteristics of a band pass type transmission filter P. The sensitivities of all of the transmission filters P are greater than the sensitivity of the narrow band filter NB in a predetermined wavelength band including the peak wavelength of the spectral characteristics of the narrow band filter NB. Accordingly, it is possible to reduce the light amount of the incident light which is incident on the narrow band filter NB without substantially attenuating the light in the transmission band of the narrow band filter NB, by using any transmission filter P.

In <FIG>, a line L51 represents the spectral characteristics of narrow band filter NB. A peak wavelength of the spectral characteristics of the narrow band filter NB is approximately in the vicinity of <NUM>. A line L52 represents the spectral characteristics of the low pass type transmission filter P. A line L53 represents the spectral characteristics of the high pass type transmission filter P. A line L54 represents the spectral characteristics of the band pass type transmission filter P. The sensitivities of all of the transmission filters are greater than the sensitivity of the narrow band filter NB in a predetermined wavelength band including the peak wavelength of the spectral characteristics of the narrow band filter NB. Accordingly, it is possible to reduce the light amount of the incident light which is incident on the narrow band filter NB without substantially attenuating the light in the transmission band of the narrow band filter NB, by using any transmission filter P.

In <FIG>, a line L61 represents the spectral characteristics of narrow band filter NB. A peak wavelength of the spectral characteristics of the narrow band filter NB in a plasmon mode is approximately in the vicinity of <NUM>. A line L62 represents the spectral characteristics of the low pass type transmission filter P. A line L63 represents the spectral characteristics of the high pass type transmission filter P. A line L64 represents the spectral characteristics of the band pass type transmission filter P. The sensitivities of all of the transmission filters are greater than the sensitivity of the narrow band filter NB in a predetermined wavelength band including the peak wavelength in the plasmon mode of greater than or equal to <NUM>, which is the cutoff wavelength of the spectral characteristics of the narrow band filter NB. Accordingly, it is possible to reduce the light amount of the incident light which is incident on the narrow band filter NB without substantially attenuating the light in the transmission band of the narrow band filter NB in the plasmon mode, by using any transmission filter P. Here, using the high pass type transmission filter P or the band pass type transmission filter P is desirable as the characteristics of a narrow band filter since light in a wavelength band of the narrow band filter NB in a waveguide mode can be cutoff.

Furthermore, in a case where the transmission band of the red color filter R, the green color filter G, or the blue color filter B includes a transmission band of the narrow band filter NB of a lower layer, such filters may be used in the transmission filter P.

In addition, in the example of <FIG>, an example is described in which the narrow band filter NB is disposed only in a part of the pixels <NUM>, and the narrow band filter NB is capable of being disposed in all of the pixels <NUM>. In this case, in each of the pixels <NUM>, the transmission filter P having a transmission band which includes the transmission band of the narrow band filter NB of the pixel <NUM> may be disposed on the color filter layer <NUM>.

Further, a combination of the colors of the color filters in the color filter layer <NUM> is not limited to the example described above, and can be arbitrarily changed.

In addition, in a case where a countermeasure against the flare described above is not necessary, for example, the transmission filter P may be disposed on an upper layer of the narrow band filter NB, or a dummy filter transmitting light at all wavelengths may be disposed.

Next, a third embodiment of the imaging element <NUM> of <FIG> will be described with reference to <FIG>.

<FIG> schematically illustrates a configuration example of a sectional surface of an imaging element 12C, which is the third embodiment of the imaging element <NUM>. Furthermore, in the drawing, the same reference numerals are applied to portions corresponding to the imaging element 12A of <FIG>, and the description thereof will be suitably omitted.

The imaging element 12C is different from the imaging element 12A in that a filter layer <NUM> is disposed instead of the narrow band filter layer <NUM>. In addition, the imaging element 12C is different from the imaging element 12B of <FIG> in that the narrow band filter NB and the color filter (for example, the red color filter R, the green color filter G, and the blue color filter B) are disposed in the same filter layer <NUM>.

Accordingly, in a case where the R pixel, the G pixel, the B pixel, and the MS pixel are arranged in the pixel array <NUM> of the imaging element 12C, the color filter layer <NUM> can be omitted.

Furthermore, in a case where the color filter of the organic material is used, in order to prevent a damage or the like of the color filter due to heat, for example, the narrow band filter NB is formed first, and final heat processing such as sinter processing is performed at a high temperature, and then, the color filter is formed. On the other hand, in a case where the color filter of the inorganic material is used, basically, there is no necessity to restrict the formation sequence described above.

In addition, in a case where the countermeasure against the flare is performed as in the imaging element 12B of <FIG>, as with the imaging element 12B, the color filter layer may be laminated between the on-chip microlens <NUM> and the interlayer film <NUM>. In this case, in the pixel <NUM> where the narrow band filter NB is disposed on the filter layer <NUM>, the transmission filter P described above is disposed on the color filter layer. On the other hand, in the pixel <NUM> where the color filter is disposed on the filter layer <NUM>, a filter may be disposed on the color filter layer, or a dummy filter transmitting light in all wavelengths or a color filter of the same color as that of the filter layer <NUM> may be disposed.

Next, processing performed by using an image output from the imaging element <NUM> of <FIG> (hereinafter, referred to as a multispectral image) will be described with reference to <FIG>.

<FIG> illustrates a configuration example of a training data generating system performing processing of automatically generating training data to be used for machine learning, as a first usage example of processing which is performed by using the multispectral image.

As illustrated in <FIG>, a training data generating system <NUM> is configured of a multispectral camera <NUM>, an RGB camera <NUM>, a storage device <NUM>, and a training data generating processing device <NUM>.

The multispectral camera <NUM> includes the imaging element <NUM> of <FIG>, and supplies the multispectral image obtained by imaging an object (for example, an apple, a face of a person, or the like), which is a target generating the training data, to the training data generating processing device <NUM>.

The RGB camera <NUM> is an imaging device which is capable of imaging a color image (so-called an RGB image), images the same object as that imaged by the multispectral camera <NUM>, and supplies a color image obtained as a result thereof to the training data generating processing device <NUM>.

The storage device <NUM>, for example, is configured of a hard disk drive, a semiconductor memory, or the like, and stores the training data supplied from the training data generating processing device <NUM>.

The training data generating processing device <NUM> performs the processing of automatically generating the training data to be used for the machine learning on the basis of the multispectral image supplied from the multispectral camera <NUM> and the color image supplied from the RGB camera <NUM>.

In general, spectrum information representing specific spectral characteristics with respect to each type of the object, which is a subject, can be extracted from the multispectral image acquired by imaging the object with the light dispersed into a plurality of wavelength bands. Accordingly, the object photographed in the multispectral image can be recognized with a high accuracy, on the basis of the spectral characteristics with respect to each of the objects. On the other hand, the color image is obtained by only imaging the object with light in three wavelength bands (for example, R, G, and B), and thus, in order to recognize the object photographed in the color image with a high accuracy, it is necessary to perform the machine learning using a large quantity of training data in advance. For this reason, it is necessary to automatically generate the training data to be used in the machine learning for recognizing the object of the color image, for example, a color image to which a name representing the type of the object is added.

As illustrated in the drawing, the training data generating processing device <NUM> includes a spectrum information retaining unit <NUM>, a recognition processing unit <NUM>, and a labeling unit <NUM>.

The spectrum information retaining unit <NUM> retains the spectrum information which is obtained in advance from the multispectral image in which various objects are photographed and represents specific spectral characteristics with respect to each type of the object, in association with the name representing each type of the object.

The recognition processing unit <NUM> extracts spectral characteristics in a region where the object, which is the target generating the training data in the multispectral image supplied from the multispectral camera <NUM>, is photographed, and thus, acquires the spectrum information representing the spectral characteristics of the object. Further, the recognition processing unit <NUM> obtains a similarity ratio with respect to a plurality of spectrum information items retained in the spectrum information retaining unit <NUM>, with respect to the spectrum information obtained from the multispectral image.

Then, the recognition processing unit <NUM> supplies a name associated with spectrum information having the highest similarity ratio as a recognition result of the object, which is the target generating the training data, from the plurality of spectrum information items retained in the spectrum information retaining unit <NUM>, to the labeling unit <NUM>. Furthermore, in a case where the highest similarity ratio obtained here is less than or equal to a predetermined defined value (a threshold value which can be determined as the same type), the recognition processing unit <NUM> sets the effect that the object photographed in the multispectral image is not capable of being recognized (unrecognizable) as the recognition result.

The labeling unit <NUM> performs labeling processing with respect to the color image supplied from the RGB camera <NUM>, that is, processing of adding the recognition result supplied from the recognition processing unit <NUM> to the color image supplied from the RGB camera <NUM>. Then, the labeling unit <NUM> supplies the color image to which the recognition result is added, to the storage device <NUM> for storage as the training data to be used for the machine learning for performing the object recognition using the color image.

The training data generating system <NUM> is configured as described above, and is capable of generating the color image to which the recognition result of the object photographed in the multispectral image is added, as the training data to be used for the machine learning for performing the object recognition using the color image. Accordingly, various objects are imaged by the multispectral camera <NUM> and the RGB camera <NUM>, by using the training data generating system <NUM>, and thus, the training data with respect to such objects can be automatically generated.

Accordingly, in the related art, for example, a huge labor of visually determining the type of the object, which is the target generating the training data and manually performing the labeling is necessary, but the labor can be reduced due to the automation of the training data generating system <NUM>. That is, a large quantity of training data can be rapidly generated by the training data generating system <NUM>.

Then, it is possible to perform the machine learning for performing the object recognition using the color image by using a large quantity of training data which is automatically generated by the training data generating system <NUM>.

<FIG> illustrates a configuration example of the object recognition system <NUM> performing machine learning by using the training data generated in the training data generating system <NUM>, and performing the object recognition by using a learning result thereof.

As illustrated in <FIG>, the object recognition system <NUM> includes a storage device <NUM>, a learning tool <NUM>, an RGB camera <NUM>, and an output device <NUM>. Furthermore, when the learning is performed in the object recognition system <NUM>, the object recognition system <NUM> may include at least the storage device <NUM> and the learning tool <NUM>. In addition, when the learning result is used in the object recognition system <NUM>, the object recognition system <NUM> may include at least the learning tool <NUM>, the RGB camera <NUM>, and the output device <NUM>.

The storage device <NUM> stores a large quantity of training data which is automatically generated in the training data generating system <NUM> of <FIG>, that is, the color image to which the recognition result with respect to various objects (the name representing the type of the object) is added.

The learning tool <NUM> sequentially reads out a large quantity of training data which is stored in the storage device <NUM> at the time of performing the learning, and for example, performs the learning of extracting the common characteristics for the objects with respect to a plurality of images in which the same type of the object is photographed. Then, when the object recognition is performed by using the learning result, the learning tool <NUM> compares the characteristics of the object photographed in the color image which is imaged by the RGB camera <NUM> with the characteristics which are subjected to the learning in advance. Accordingly, the learning tool <NUM> recognizes the type of the object photographed in the color image which is imaged by the RGB camera <NUM>, and supplies the recognition result (the name representing the type of the object) to the output device <NUM>.

The RGB camera <NUM> is an imaging device which is capable of imaging a general color image, and supplies the color image obtained by imaging the object, which is the target to be subjected to the object recognition, to the learning tool <NUM>.

The output device <NUM>, for example, is configured of a display such as a liquid crystal panel or an organic electro luminescence (EL) panel, and displays the recognition result supplied from the learning tool <NUM> by superimposing the recognition result on the color image which is imaged by the RGB camera <NUM>. Furthermore, in a case where the output device <NUM> is configured of a speaker, and a synthetic audio expressing the recognition result may be output.

Thus, the object recognition system <NUM> performs the machine learning by using the training data which is generated in the training data generating system <NUM>, and is capable of performing the object recognition with a higher accuracy by using the learning result.

<FIG> is a flowchart illustrating training data generating processing of the training data generating system <NUM>.

For example, in a state where the multispectral camera <NUM> and the RGB camera <NUM> are directed towards the object, which is the target generating the training data, in the case of performing an operation of instructing the training data of the object to be generated, the processing is started. In step S11, the multispectral camera <NUM> supplies the multispectral image in which the object as the target is photographed, to the recognition processing unit <NUM>, and the RGB camera <NUM> supplies the color image in which the object as the target is photographed, to the labeling unit <NUM>.

In step S12, the recognition processing unit <NUM> extracts the spectral characteristics from the multispectral image which is supplied from the multispectral camera <NUM> in step S11, and acquires the spectrum information representing the spectral characteristics of the object, which is the target generating the training data.

In step S13, the recognition processing unit <NUM> obtains the similarity ratio with respect to the plurality of spectrum information items which are retained in the spectrum information retaining unit <NUM>, with respect to the spectrum information acquired in step S12. Then, the recognition processing unit <NUM> obtains the name associated with the spectrum information in which the highest similarity ratio is obtained, as the recognition result of the object, which is the target generating the training data, and supplies the name to the labeling unit <NUM>.

In step S14, the labeling unit <NUM> labels the recognition result supplied from the recognition processing unit <NUM> in step S13 with respect to the color image supplied from the RGB camera <NUM> in step S11, and generates the training data of the object as the target. Then, the labeling unit <NUM> supplies the generated training data to the storage device <NUM>.

In step S15, the storage device <NUM> stores the training data supplied from the labeling unit <NUM> in step S14, and then, ends the training data generating processing. After that, for example, in the case of performing an operation of instructing the training data to be generated by using the next object as the target, hereinafter, similar processing is repeated.

As described above, in the training data generating system <NUM>, it is possible to automate the labeling processing with respect to the object, which is the target generating the training data, and to easily generate a large quantity of training data.

<FIG> illustrates a configuration example of an evaluation index presenting system performing processing of presenting a suitable evaluation index with respect to the object as the target, as a second usage example of the processing which is performed by using the multispectral image.

As illustrated in <FIG>, the evaluation index presenting system <NUM> includes a multispectral camera <NUM>, an RGB camera <NUM>, an output device <NUM>, and an evaluation index acquisition processing device <NUM>. Furthermore, the multispectral camera <NUM> and the RGB camera <NUM> have a similar configuration to that of the training data generating system <NUM> of <FIG>, and the detailed description will be omitted.

The output device <NUM>, for example, is configured of a display such as a liquid crystal panel or an organic EL panel, and displays the evaluation index supplied from the evaluation index acquisition processing device <NUM> by superimposing the evaluation index on the color image which is imaged by the RGB camera <NUM>. Furthermore, in a case where the output device <NUM> is configured of a speaker, a synthetic audio expressing the evaluation index may be output.

The evaluation index acquisition processing device <NUM> recognizes the object as the target, on the basis of the multispectral image supplied from the multispectral camera <NUM> and the color image supplied from the RGB camera <NUM>, and performs processing of acquiring the evaluation index quantitatively evaluating the object.

As illustrated in the drawing, the evaluation index acquisition processing device <NUM> includes a spectrum information retaining unit <NUM>, a recognition processing unit <NUM>, a labeling unit <NUM>, and an evaluation index calculating unit <NUM>. Furthermore, the spectrum information retaining unit <NUM>, the recognition processing unit <NUM>, and the labeling unit <NUM> have a similar configuration to that of the training data generating processing device <NUM> of <FIG>, and the detailed description will be omitted.

A recognition result of the recognition processing unit <NUM> is supplied to the evaluation index calculating unit <NUM>, along with the multispectral image imaged by the multispectral camera <NUM>. Then, the evaluation index calculating unit <NUM> automatically selects the evaluation index suitable for the type of the object according to the recognition result supplied from the recognition processing unit <NUM>, that is, the name representing the type of the object, which is the target calculating the evaluation index. For example, the evaluation index calculating unit <NUM> retains the type of the object and the optimal evaluation index with respect to the type of the object in association with each other. Then, as described below with reference to <FIG>, in a case where a tomato is the target, a sugar content is selected as the evaluation index, and in a case where a cabbage is the target, freshness is selected as the evaluation index.

Further, the evaluation index calculating unit <NUM> automatically selects an index calculation formula and a coefficient necessary for calculating the evaluation index suitable for the type of the object as the target, calculates the evaluation index on the basis of the multispectral image, and supplies the evaluation index to the labeling unit <NUM>.

Accordingly, the labeling unit <NUM> performs the labeling processing with respect to the color image supplied from the RGB camera <NUM>, that is, processing of adding the recognition result supplied from the recognition processing unit <NUM> and the evaluation index supplied from the evaluation index calculating unit <NUM> to the color image supplied from the RGB camera <NUM>. Then, the labeling unit <NUM> supplies the color image to which the recognition result and the evaluation index are added, to the output device <NUM> for display as the evaluation result.

The evaluation index presenting system <NUM> is configured as described above, and thus, it is possible to display the color image to which the recognition result of the object photographed in the multispectral image and the evaluation index are added, on the display of the output device <NUM>. Accordingly, the evaluation index presenting system <NUM> is capable of automatically presenting the evaluation index suitable for the object by only imaging a desired object with the multispectral camera <NUM> and the RGB camera <NUM>.

Accordingly, in the related art, for example, an evaluation index suitable for each food is different, and thus, in a case where it is necessary for the user to activate dedicated software with respect to each of the foods or to set an operation mode, a coefficient, or the like, such a labor can be reduced by the automation of the evaluation index presenting system <NUM>. That is, the user may only activate the software presenting the evaluation index regardless of the object as the target, and thus, more excellent user experience can be provided by the evaluation index presenting system <NUM>.

<FIG> illustrate usage examples of an information processing terminal on which the evaluation index presenting system <NUM> is mounted.

For example, as illustrated in <FIG>, in a tomato selling space, it is automatically recognized that the target presenting the evaluation index is a tomato, by only imaging the tomato with the multispectral camera <NUM> and the RGB camera <NUM> of the information processing terminal. Then, a sugar content is selected as the evaluation index applied to the tomato, and an evaluation result of "SUGAR CONTENT OF THIS TOMATO IS <NUM>" with respect to the tomato as the target is presented on the display, which is the output device <NUM> of the information processing terminal, by being superimposed on the color image imaged by the RGB camera <NUM>. Furthermore, for example, a message of "DURING RECOGNITION OF TYPE OF FOOD" representing that it is in the middle of the processing is displayed during object recognition processing or the acquisition of the evaluation index.

Similarly, for example, as illustrated in <FIG>, in a cabbage selling space, it is automatically recognized that the target presenting the evaluation index is a cabbage by only imaging the cabbage with the multispectral camera <NUM> and the RGB camera <NUM> of the information processing terminal. Then, freshness is selected as the evaluation index applied to the cabbage, and an evaluation result of "FRESHNESS OF CABBAGE IS <NUM>%" with respect to the cabbage as the target is presented on the display, which is the output device <NUM> of the information processing terminal by being superimposed on the color image imaged by the RGB camera <NUM>. Furthermore, for example, a message of "DURING RECOGNITION OF TYPE OF FOOD" representing that it is in the middle of the processing is displayed during the object recognition processing and the acquisition of the evaluation index.

In addition, the evaluation index presenting system <NUM>, for example, is capable of obtaining the tastiness of the food as described below with reference to <FIG>, the moisture of the fruit as described below with reference to <FIG>, and the like as the evaluation index.

<FIG> is a flowchart illustrating evaluation index presenting processing of the evaluation index presenting system <NUM>.

For example, in the case of performing an operation of instructing the evaluation index of the object to be presented in a state where the multispectral camera <NUM> and the RGB camera <NUM> are directed towards the object, which is the target presenting the evaluation index, the processing is started. Then, in steps S21 to S23, similar processing to that in steps S11 to S13 of <FIG> is performed.

In step S24, the evaluation index calculating unit <NUM> automatically selects the optimal evaluation index with respect to the type of the object as the target, according to a recognition result supplied from the recognition processing unit <NUM> in step S23.

In step S25, the evaluation index calculating unit <NUM> automatically selects an index calculation formula and a coefficient necessary for calculating the evaluation index selected in step S24, calculates the evaluation index, and supplies the evaluation index to the labeling unit <NUM>.

In step S26, the labeling unit <NUM> labels the recognition result supplied from the recognition processing unit <NUM> in step S23 and the evaluation index supplied from the evaluation index calculating unit <NUM> in step S25, on the color image supplied from the RGB camera <NUM> in step S21, and generates the evaluation result of the object as the target. Then, the labeling unit <NUM> supplies the generated evaluation result to the output device <NUM>.

In step S27, the output device <NUM> outputs the evaluation result supplied from the labeling unit <NUM> in step S26, and then, the evaluation index presenting processing is ended. After that, for example, in the case of performing an operation of instructing the evaluation index to be presented by using the next object as the target, hereinafter, similar processing is repeated.

As described above, in the evaluation index presenting system <NUM>, it is possible to automatically present the evaluation result applied to the object by only imaging the object, which is the target presenting the evaluation index, and for example, it is possible to provide the user with more excellent experience.

<FIG> is a block diagram illustrating a configuration example of hardware of a computer which executes a set of processings described above (the training data generating processing of <FIG> and the evaluation index presenting processing of <FIG>) by a program.

In the computer, a central processing unit (CPU) <NUM>, a read only memory (ROM) <NUM>, a random access memory (RAM) <NUM>, and an electronically erasable and programmable read only memory (EEPROM) <NUM> are connected to each other through a bus <NUM>. An input/output interface <NUM> is further connected to the bus <NUM>, and the input/output interface <NUM> is connected to the outside (for example, the multispectral camera <NUM>, the RGB camera <NUM>, or the like).

In the computer configured as described above, the CPU <NUM>, for example, loads a program stored in the ROM <NUM> and the EEPROM <NUM> on the RAM <NUM> through the bus <NUM>, and executes the program, and thus, performs the set of processings described above. In addition, the program executed by the computer (the CPU <NUM>) is written in advance in the ROM <NUM>, and can be installed in the EEPROM <NUM> from the outside through the input/output interface <NUM> or can be updated.

Furthermore, the training data generating system <NUM> and the evaluation index presenting system <NUM> include two cameras of the multispectral camera <NUM> and the RGB camera <NUM>, and for example, may include one camera which is capable of detecting a multiwavelength.

In addition, the training data generating system <NUM> and the evaluation index presenting system <NUM> may be incorporated in the imaging device <NUM> including the imaging element <NUM> of <FIG>, and for example, the training data, the evaluation result, or the like can be output from the imaging device <NUM>.

Furthermore, the training data generating system <NUM> and the evaluation index presenting system <NUM> are not limited to recognizing the object by only using the multispectral camera <NUM> as described above, and may recognize the object by using other types of sensors. For example, a brightness sensor acquiring brightness information, a time-of-flight (ToF) sensor acquiring distance information, an ambient sensor acquiring various environmental information items, an infrared ray sensor acquiring infrared ray information, a thermal sensor acquiring temperature information, and the like can be used, and an image configured of the information acquired by such sensors can be used for recognizing the object.

Further, the labeling processing which is automated by using the multispectral image output from the imaging element <NUM> of <FIG> is not limited to the recognition result or the evaluation result of the object, and the labeling of various information items can be performed.

Hereinafter, a modification example of the embodiments of the present technology described above will be described.

For example, the number of types of the film thicknesses of the conductor thin film may be set to be greater than or equal to <NUM>, according to the hole pitch (the transmission band).

In addition, in the plasmon filter having the dot array structure, the film thickness of the conductor thin film (the dot) may be changed according to the dot pitch (the absorption band).

Specifically, as illustrated in <FIG>, as the dot pitch narrows and the absorption band is shifted to a short wavelength, a peak width and a half width of the absorption band basically narrow, but an absorption rate (a negative peak value of the absorption band) decreases. In contrast, as the dot pitch widens and the absorption band is shifted to a long wavelength, the absorption rate (the negative peak value of the absorption band) is basically improved, but the peak width and the half width of the absorption band widen.

In addition, as the conductor thin film configuring the dot becomes thin, the absorption rate basically decreases, but the peak width and the half width of the absorption band narrow. In contrast, as the conductor thin film configuring the dot becomes thick, the peak width and the half width of the absorption band basically widen, but the absorption rate is improved.

Accordingly, for example, it is desirable that as the dot pitch of the plasmon filter narrows and the absorption band is shifted to the short wavelength, the conductor thin film becomes thick and the absorption rate increases, even though the peak width and the half width of the absorption band slightly widen. In contrast, it is desirable that as the dot pitch of the plasmon filter widens and the absorption band is shifted to the long wavelength, the conductor thin film becomes thin and the peak width and the half width of the transmission band narrow, even though the absorption rate slightly decreases.

Further, for example, in the plasmon filter of the same transmission band (the same hole pitch) or the same absorption band (the same dot pitch), the film thickness of the conductor thin film may be changed for each pixel. Accordingly, it is possible to provide pixels of which the transmission bands or the absorption bands are identical to each other, but the sensitivities or the absorption rates are different from each other. Accordingly, for example, it is possible to improve a detection accuracy of narrow band light in a part of the pixels.

In addition, the present technology is not limited only to the back-side illumination type CMOS image sensor described above, but can be applied to other imaging elements using the plasmon filter. For example, the present technology can be applied to a surface irradiation type CMOS image sensor, a charge coupled device (CCD) image sensor, an image sensor having a photoconductor structure in which an organic photoelectric conversion film, a quantum dot structure, or the like is embedded, and the like.

In addition, the present technology, for example, can be applied to a laminated solid imaging device illustrated in <FIG>.

<FIG> illustrates a schematic configuration example of a non-laminated solid imaging device. As illustrated in <FIG>, a solid imaging device <NUM> includes one die (a semiconductor substrate) <NUM>. A pixel region <NUM> in which the pixels are arranged in the shape of an array, a control circuit <NUM> performing various controls other than the driving of the pixel, and logic circuit <NUM> for signal processing are mounted on the die <NUM>.

<FIG> illustrate schematic configuration examples of a laminated solid imaging device. As illustrated in <FIG>, two dies of a sensor die <NUM> and a logic die <NUM> are laminated on a solid imaging device <NUM>, are electrically connected to each other, and are configured as one semiconductor chip.

In <FIG>, the pixel region <NUM> and the control circuit <NUM> are mounted on the sensor die <NUM>, and the logic circuit <NUM> including a signal processing circuit which performs the signal processing is mounted on the logic die <NUM>.

In <FIG>, the pixel region <NUM> is mounted on the sensor die <NUM>, and the control circuit <NUM> and the logic circuit <NUM> are mounted on the logic die <NUM>.

Further, the present technology can be applied to a metal thin film filter using a metal thin film other than the plasmon filter, and a possibility that the present technology is applied to photonic crystals using a semiconductor material is considered as an application example.

Next, an application example of the present technology will be described.

For example, as illustrated in <FIG>, the present technology can be applied to various cases of sensing light such as visible light, infrared light, ultraviolet light, and an X ray.

Hereinafter, a more detailed application example will be described.

For example, the transmission band of the narrow band filter NB of each of the pixels <NUM> of the imaging device <NUM> of <FIG> is adjusted, and thus, a wavelength band of light which is detected by each of the pixels <NUM> of the imaging device <NUM> (hereinafter, referred to as a detection band) can be adjusted. Then, the detection band of each of the pixels <NUM> is suitably set, and thus, the imaging device <NUM> can be used for various applications.

For example, <FIG> illustrates an example of a detection band in a case where the tastiness or the freshness of the food is detected.

For example, a peak wavelength of a detection band in the case of detecting myoglobin representing a tastiness component of tuna, beef, or the like is in a range of <NUM> to <NUM>, and a half width is in a range of <NUM> to <NUM>. A peak wavelength of a detection band in the case of detecting an oleic acid representing the freshness of the tuna, the beef, or the like is <NUM>, and a half width is in a range of <NUM> to <NUM>. A peak wavelength of a detection band in the case of detecting chlorophyll representing the freshness of leaf vegetable such as brassica rapa is in a range of <NUM> to <NUM>, and a half width is in a range of <NUM> to <NUM>.

<FIG> illustrates an example of a detection band in a case where a sugar content or the moisture of a fruit is detected.

For example, a peak wavelength of a detection band in the case of detecting a flesh light path length representing a sugar content of Raiden, which is one breed of melon, is <NUM>, and a half width is in a range of <NUM> to <NUM>. A peak wavelength of a detection band in the case of detecting sucrose representing the sugar content of Raiden is <NUM>, and a half width is in a range of <NUM> to <NUM>. A peak wavelength of a detection band in the case of detecting sucrose representing a sugar content of Raiden Red, which is another breed of melon, is <NUM>, and a half width is in a range of <NUM> to <NUM>. A peak wavelength of a detection band in the case of detecting moisture representing the sugar content of Raiden Red is <NUM>, and a half width is in a range of <NUM> to <NUM>.

A peak wavelength of a detection band in the case of detecting sucrose representing a sugar content of an apple is <NUM>, and a half width is in a range of <NUM> to <NUM>. A peak wavelength of a detection band in the case of detecting water representing the moisture of a mandarin orange is <NUM>, and a half width is <NUM>. A peak wavelength of a detection band in the case of detecting sucrose representing a sugar content of the mandarin orange is <NUM>, and a half width is in a range of <NUM> to <NUM>.

<FIG> illustrates an example of a detection band in a case where plastics are sorted.

For example, a peak wavelength of a detection band in the case of detecting poly ethylene terephthalate (PET) is <NUM>, and a half width is in a range of <NUM> to <NUM>. A peak wavelength of a detection band in the case of detecting poly styrene (PS) is <NUM>, and a half width is in a range of <NUM> to <NUM>. A peak wavelength of a detection band in the case of detecting poly ethylene (PE) is <NUM>, and a half width is in a range of <NUM> to <NUM>. A peak wavelength of a detection band in the case of detecting poly vinyl chloride (PVC) is in a range of <NUM> to <NUM>, and a half width is in a range of <NUM> to <NUM>. A peak wavelength of a detection band in the case of detecting polypropylene (PP) is in a range of <NUM> to <NUM>, and a half width is in a range of <NUM> to <NUM>.

In addition, for example, the present technology can be applied to freshness management of plucked flower.

Further, for example, the present technology can be applied to an inspection of foreign substances which are mixed into the food. For example, the present technology can be applied to the detection of the foreign substances, such as a shell, a hull, a stone, a leaf, a branch, and a wood chip, which are mixed into nuts, such as an almond, a blueberry, and a walnut, or fruits. In addition, for example, the present technology can be applied to the detection of the foreign substances such as plastic pieces mixed into processed food, beverage, or the like.

Further, for example, the present technology can be applied to the detection of a normalized difference vegetation index (NDVI), which is an index of vegetation.

In addition, for example, the present technology can be applied to the detection of a human body on the basis of any one or both of a spectral shape in the vicinity of a wavelength of <NUM>, derived from Hemoglobin of the human skin and a spectral shape in the vicinity of a wavelength of <NUM>, derived from a melanin dye contained in the human skin.

Further, for example, the present technology can be applied to biological detection (biological authentication), fabrication prevention, monitoring, and the like of a user interface and a sign.

In addition, for example, a technology according to an embodiment of the present disclosure (the present technology) may be applied to an endoscopic surgery system.

<FIG> is a diagram illustrating an example of a schematic configuration of the endoscopic surgery system to which the technology according to an embodiment of the present disclosure (the present technology) is applied.

<FIG> illustrates an aspect in which an operator (a medical doctor) <NUM> performs a surgery with respect to a patient <NUM> on a patient bed <NUM> by using an endoscopic surgery system <NUM>. As illustrated in the drawing, the endoscopic surgery system <NUM> is configured of an endoscope <NUM>, other surgical tools <NUM> such as a pneumoperitoneum tube <NUM> or an energy treatment tool <NUM>, a support arm device <NUM> supporting the endoscope <NUM>, and a cart <NUM> on which various devices for the surgery under the endoscope are mounted.

The endoscope <NUM> is configured of a lens barrel <NUM> in which a region having a predetermined length from a tip end is inserted into a body cavity of the patient <NUM>, and a camera head <NUM> connected to a base end of the lens barrel <NUM>. In the illustrated example, the endoscope <NUM> configured as a so-called rigid scope including a rigid lens barrel <NUM> is illustrated, and the endoscope <NUM> may be configured as a so-called flexible scope including a flexible lens barrel.

An opening portion embedded with an objective lens is disposed on the tip end of the lens barrel <NUM>. A light source device <NUM> is connected to the endoscope <NUM>, and light generated by the light source device <NUM> is guided to the tip end of the lens barrel by a light guide extending in the lens barrel <NUM>, and is emitted towards an observation target in the body cavity of the patient <NUM> through the objective lens. Furthermore, the endoscope <NUM> may be a direct view mirror, or may be a perspective view mirror or a side view mirror.

An optical system and an imaging element are disposed on the camera head <NUM>, and reflection light from the observation target (observation light) is condensed on the imaging element by the optical system. The observation light is subjected to photoelectric conversion by the imaging element, and thus, an electric signal corresponding to the observation light, that is, an image signal corresponding to the observation image is generated. The image signal is transmitted to a camera control unit (CCU) <NUM> as RAW data.

The CCU <NUM> is configured of a central processing unit (CPU), a graphics processing unit (GPU), or the like, and integrally controls the operations of the endoscope <NUM> and a display device <NUM>. Further, the CCU <NUM> receives the image signal from the camera head <NUM>, and performs various image processings for displaying an image based on the image signal with respect to the image signal, such as developing processing (demosaic processing).

The display device <NUM> displays the image based on the image signal, which is subjected to the image processing by the CCU <NUM>, according to the control from the CCU <NUM>.

The light source device <NUM>, for example, is configured of a light source such as a light emitting diode (LED), and supplies irradiation light at the time of shooting a surgical site or the like to the endoscope <NUM>.

An input device <NUM> is an input interface with respect to the endoscopic surgery system <NUM>. It is possible for the user to perform various information inputs or instruction inputs with respect to the endoscopic surgery system <NUM> through the input device <NUM>. For example, the user inputs an instruction or the like to the effect of changing imaging conditions of the endoscope <NUM> (the type of irradiation light, a magnification, a focal point distance, and the like).

A treatment tool control device <NUM> controls the drive of the energy treatment tool <NUM>, such as the cauterization of tissues, and the incision or the sealing of a blood vessel. A pneumoperitoneum device <NUM> feeds gas in the body cavity through the pneumoperitoneum tube <NUM>, in order to inflate the body cavity of the patient <NUM> to ensure a visual field of the endoscope <NUM> and an operation space of the operator. A recorder <NUM> is a device which is capable of recording various information items relevant to the surgery. A printer <NUM> is a device which is capable of printing various information items relevant to the surgery in various formats such as a text, an image, or a graph.

Furthermore, the light source device <NUM> supplying the irradiation light at the time of shooting the surgical site to the endoscope <NUM>, for example, can be configured of a white light source which is configured of an LED, a laser light source, or a combination thereof. In a case where the white light source is configured of a combination of RGB laser light sources, an output intensity and an output timing of each color (each wavelength) can be controlled with a high accuracy, and thus, a white balance of the imaged image can be adjusted in the light source device <NUM>. In addition, in this case, the RGB laser light source irradiates the observation target with each laser light ray in time division, and controls the driving of the imaging element of the camera head <NUM> in synchronization with the irradiation timing, and thus, it is also possible to image an image corresponding to each of RGB in time division. According to the method described above, it is possible to obtain a color image even in a case where the color filter is not disposed in the imaging element.

In addition, the light source device <NUM> may control the driving such that the light intensity to be output is changed for each predetermined time. The driving of the imaging element of the camera head <NUM> is controlled in synchronization with a timing at which the light intensity is changed, an image is acquired in time division, and the image is synthesized, and thus, it is possible to generate an image in a high dynamic range without having so-called black defects and overexposure.

In addition, the light source device <NUM> may be configured to be capable of supplying light in a predetermined wavelength band corresponding to special light observation. In the special light observation, for example, light in a narrow band, compared to the irradiation light (that is, white light) at the time of normal observation, is emitted by using wavelength dependency of light absorption in the body tissues, and thus, so-called narrow band light observation (narrow band imaging) shooting a predetermined tissue of the blood vessel or the like on a surface layer of a mucous membrane with a high contrast is performed. Alternatively, in the special light observation, fluorescent light observation may be performed in which an image is obtained by fluorescent light generated by emitting excitation light. In the fluorescent light observation, the body tissues are irradiated with the excitation light, and thus, the fluorescent light from the body tissues can be observed (self-fluorescent light observation), or a reagent such as indocyanine green (ICG) is locally injected into the body tissues, and the body tissues are irradiated with excitation light corresponding to the wavelength of the fluorescent light of the reagent, and thus, a fluorescent image can be obtained. The light source device <NUM> can be configured to be capable of supplying the narrow band light and/or the excitation light corresponding to the special light observation.

<FIG> is a block diagram illustrating an example of functional configurations of the camera head <NUM> and the CCU <NUM> illustrated in <FIG>.

The camera head <NUM> includes a lens unit <NUM>, an imaging unit <NUM>, a driving unit <NUM>, a communication unit <NUM>, and a camera head control unit <NUM>. The CCU <NUM> includes a communication unit <NUM>, an image processing unit <NUM>, and a control unit <NUM>. The camera head <NUM> and the CCU <NUM> are connected to each other to communicate with each other by a transmission cable <NUM>.

The lens unit <NUM> is an optical system which is disposed in a connection portion with respect to the lens barrel <NUM>. The captured observation light from the tip end of the lens barrel <NUM> is guided to the camera head <NUM>, and is incident on the lens unit <NUM>. The lens unit <NUM> is configured of a combination of a plurality of lenses including a zoom lens and a focus lens.

The imaging element configuring the imaging unit <NUM> may be one imaging element (a so-called single-plate type imaging element), or may be a plurality of imaging elements (a so-called multi-plate type imaging element). In a case where the imaging unit <NUM> is configured of the multi-plate type imaging element, for example, image signals corresponding to each of RGB are generated by each of the imaging elements, and are synthesized, and thus, a color image may be obtained. Alternatively, the imaging unit <NUM> may be configured to include a pair of imaging elements for acquiring image signals for a right eye and a left eye, which correspond to three-dimensional (3D) display. By performing the 3D display, it is possible for the operator <NUM> to more accurately grasp the depth of the body tissues in the surgical site. Furthermore, in a case where the imaging unit <NUM> is configured of the multi-plate type imaging element, a plurality of lens units <NUM> can also be disposed corresponding to each of the imaging elements.

In addition, the imaging unit <NUM> may not be necessarily disposed on the camera head <NUM>. For example, the imaging unit <NUM> may disposed in the lens barrel <NUM> immediately behind the objective lens.

The driving unit <NUM> is configured of an actuator, and moves the zoom lens and the focus lens of the lens unit <NUM> along an optical axis by a predetermined distance, according to the control from the camera head control unit <NUM>. Accordingly, the magnification and the focal point of the imaged image obtained by the imaging unit <NUM> can be suitably adjusted.

The communication unit <NUM> is configured of a communication device for transmitting and receiving various information items with respect to the CCU <NUM>. The communication unit <NUM> transmits the image signal obtained from the imaging unit <NUM> to the CCU <NUM> through the transmission cable <NUM>, as RAW data.

In addition, the communication unit <NUM> receives a control signal for controlling the driving the camera head <NUM> from the CCU <NUM>, and supplies the control signal to the camera head control unit <NUM>. The control signal, for example, includes information relevant to the imaging conditions, such as information to the effect of designating a frame rate of the imaged image, information to the effect of designating an exposure value at the time of imaging, and/or information to the effect of designating the magnification and the focal point of the imaged image.

Furthermore, the imaging conditions such as the frame rate or the exposure value, the magnification, and the focal point, described above, may be suitably designated by the user, or may be automatically set by the control unit <NUM> of the CCU <NUM> on the basis of the acquired image signal. In the latter case, a so-called auto exposure (AE) function, an auto focus (AF) function, and an auto white balance (AWB) function are mounted on the endoscope <NUM>.

The camera head control unit <NUM> controls the driving of the camera head <NUM> on the basis of the control signal from the CCU <NUM>, which is received through the communication unit <NUM>.

The communication unit <NUM> is configured of a communication device for transmitting and receiving various information items with respect to the camera head <NUM>. The communication unit <NUM> receives the image signal transmitted through the transmission cable <NUM> from the camera head <NUM>.

In addition, the communication unit <NUM> transmits the control signal for controlling the driving of the camera head <NUM> to the camera head <NUM>. The image signal or the control signal can be transmitted by telecommunication, light communication, or the like.

The image processing unit <NUM> performs various image processings with respect to the image signal, which is the RAW data transmitted from the camera head <NUM>.

The control unit <NUM> performs various controls relevant to the imaging of the surgical site or the like using the endoscope <NUM> and the display of the imaged image obtained by imaging the surgical site or the like. For example, the control unit <NUM> generates the control signal for controlling the driving of the camera head <NUM>.

In addition, the control unit <NUM> displays the imaged image, on which the surgical site or the like is reflected, on the display device <NUM>, on the basis of the image signal which is subjected to the image processing by the image processing unit <NUM>. At this time, the control unit <NUM> may recognize various objects in the imaged image by using various image recognition technologies. For example, the control unit <NUM> detects the shape, the color, or the like of the edge of the object which is included in the imaged image, and thus, is capable of recognizing a surgical tool such as forceps, a specific organic site, bleed, mist at the time of using the energy treatment tool <NUM>, or the like. The control unit <NUM> may display various surgery assistance information items by superimpose the information on the image of the surgical site, by using the recognition result, at the time of displaying the imaged image on the display device <NUM>. The surgery assistance information is displayed by being superimposed, and is presented to the operator <NUM>, and thus, it is possible to reduce a load on the operator <NUM>, and it is possible for the operator <NUM> to reliably perform the surgery.

The transmission cable <NUM> connecting the camera head <NUM> and the CCU <NUM> to each other is an electric signal cable corresponding to the communication of the electric signal, an optical fiber corresponding to the light communication, or a composite cable thereof.

Here, in the illustrated example, the communication is performed in a wired manner by using the transmission cable <NUM>, and the communication between the camera head <NUM> and the CCU <NUM> may be performed in a wireless manner.

As described above, an example of the endoscopic surgery system which can be obtained by applying the technology according to an embodiment of the present disclosure thereto has been described. In the configurations described above, the technology according to an embodiment of the present disclosure, for example, can be obtained by being applied to the camera head <NUM> or the imaging unit <NUM> of the camera head <NUM>. Specifically, for example, the imaging element <NUM> of <FIG> can be applied to the imaging unit <NUM>. It is possible to obtain a more specific and high accurate surgical site image by applying the technology according to an embodiment of the present disclosure to the imaging unit <NUM>, and thus, it is possible for the operator to reliably confirm the surgical site.

Furthermore, here, the endoscopic surgery system has been described as an example, but the technology according to an embodiment of the present disclosure, for example, may be applied to a microscope surgery system or the like in addition to the endoscopic surgery system.

In addition, for example, the technology according to an embodiment of the present disclosure may be realized as a device mounted on any type of movable body such as an automobile, an electric automobile, a hybrid electric automobile, a motorcycle, a bicycle, a personal mobility, an airplane, a drone, a ship, and a robot.

<FIG> is a block diagram illustrating a schematic configuration example of a vehicle control system, which is an example of a movable body control system obtained by applying the technology according to an embodiment of the present disclosure thereto.

A vehicle control system <NUM> includes a plurality of electronic control units connected to each other through a communication network <NUM>. In the example illustrated in <FIG>, the vehicle control system <NUM> includes a driving system control unit <NUM>, a body system control unit <NUM>, an outdoor information detection unit <NUM>, an in-vehicle information detection unit <NUM>, and an integral control unit <NUM>. In addition, a microcomputer <NUM>, an audio image output unit <NUM>, and an in-vehicle network interface (I/F) <NUM> are illustrated as a functional configuration of the integral control unit <NUM>.

The driving system control unit <NUM> controls an operation of a device relevant to a driving system of the vehicle according to various programs. For example, the driving system control unit <NUM> functions as a control device of a driving force generating device for generating a driving force of a vehicle, such as an internal-combustion engine or a driving motor, a driving force transfer mechanism for transferring the driving force to a wheel, a steering mechanism adjusting a rudder angle of the vehicle, a braking device generating a braking force of the vehicle, and the like.

The body system control unit <NUM> controls the operations of various devices mounted on a vehicle body according to various programs. For example, the body system control unit <NUM> functions as a control device of a keyless entry system, a smart key system, an electric window device, and various lamps such as a head lamp, a back lamp, a brake lamp, a winker lamp, or a fog lamp. In this case, an electric wave transmitted from a portable machine instead of a key or signals of various switches can be input into the body system control unit <NUM>. The body system control unit <NUM> receives the input of the electric wave or the signal, and controls the door lock device, the electric window device, the lamp, and the like of the vehicle.

The outdoor information detection unit <NUM> detects the outside information of the vehicle on which the vehicle control system <NUM> is mounted. For example, an imaging unit <NUM> is connected to the outdoor information detection unit <NUM>. The outdoor information detection unit <NUM> images the outdoor image by the imaging unit <NUM>, and receives the imaged image. The outdoor information detection unit <NUM> may perform object detection processing or distance detection processing of a person, a car, an obstacle, a sign, characters on a road surface, or the like, on the basis of the received image.

The imaging unit <NUM> is an optical sensor which receives light and outputs an electric signal according to the amount of the received light. The imaging unit <NUM> is capable of outputting the electric signal as an image, and is capable of outputting the electric signal as distance measuring information. In addition, the light received by the imaging unit <NUM> may be visible light, or may be non-visible light such as an infrared ray.

The in-vehicle information detection unit <NUM> detects in-vehicle information. For example, a driver state detecting unit <NUM> detecting the state of the driver is connected to the in-vehicle information detection unit <NUM>. The driver state detecting unit <NUM>, for example, includes a camera imaging the driver, and the in-vehicle information detection unit <NUM> may calculate a fatigue degree or a concentration degree of the driver, or may determine whether or not the driver dozes off, on the basis of detection information input from the driver state detecting unit <NUM>.

The microcomputer <NUM> calculates a control target value of the driving force generating device, the steering mechanism, or the braking device on the basis of the in-vehicle information and the outdoor information, which are acquired in the outdoor information detection unit <NUM> or the in-vehicle information detection unit <NUM>, and is capable of outputting a control command to the driving system control unit <NUM>. For example, the microcomputer <NUM> is capable of performing cooperative control for realizing the function of an advanced driver assistance system (ADAS) including collision avoidance or impact relaxation of the vehicle, following running based on an inter-vehicle distance, vehicle speed maintaining running, collision warning of the vehicle, lane departure warning of the vehicle, and the like.

In addition, the microcomputer <NUM> controls driving force generating device, the steering mechanism, the braking device, or the like, on the basis of the information around the vehicle, which is acquired in the outdoor information detection unit <NUM> or the in-vehicle information detection unit <NUM>, and is capable of performing cooperative control for automated driving in which the vehicle autonomously runs without depending on the operation of the driver.

In addition, the microcomputer <NUM> is capable of outputting the control command to the body system control unit <NUM>, on the basis of the outdoor information, which is acquired in the outdoor information detection unit <NUM>. For example, the microcomputer <NUM> controls the head lamp according to the position of a leading vehicle or an oncoming vehicle, which is detected by the outdoor information detection unit <NUM>, and thus, is capable of performing cooperative control for glare-proof such as switching the high beam with a low beam.

The audio image output unit <NUM> transmits at least one output signal of an audio and an image to an output device which is capable of visually or auditorily notifying a person on board or the outdoor of the vehicle of the information. In the example of <FIG>, an audio speaker <NUM>, a display unit <NUM>, and an instrument panel <NUM> are exemplified as the output device. The display unit <NUM>, for example, may include at least one of an on-board display and a head-up display.

<FIG> is a diagram illustrating an example of a disposition position of the imaging unit <NUM>.

In <FIG>, the imaging unit <NUM> includes imaging units <NUM>, <NUM>, <NUM>, <NUM>, and <NUM>.

The imaging units <NUM>, <NUM>, <NUM>, <NUM>, and <NUM>, for example, are disposed in positions such as a front nose, a side mirror, a rear bumper, a back door of a vehicle <NUM>, and an upper portion of a front glass of a vehicle interior. The imaging unit <NUM> provided in the front nose and the imaging unit <NUM> provided in the upper portion of the front glass of the vehicle interior mainly acquire a front image of the vehicle <NUM>. The imaging units <NUM> and <NUM> provided in the side mirror mainly acquire a side image of the vehicle <NUM>. The imaging unit <NUM> provided in the rear bumper or the back door mainly acquires a rear image of the vehicle <NUM>. The imaging unit <NUM> provided in the upper portion of the front glass of the vehicle interior is mainly used for detecting a leading vehicle, a pedestrian, an obstacle, a traffic light, a traffic sign, a traffic lane, or the like.

Furthermore, <FIG> illustrates an example of shooting ranges of the imaging units <NUM> to <NUM>. The imaging range <NUM> illustrates an imaging range of the imaging unit <NUM> provided in the front nose, imaging ranges <NUM> and <NUM> illustrate imaging ranges of the imaging units <NUM> and <NUM> respectively provided in the side mirror, and the imaging range <NUM> illustrates an imaging range of the imaging unit <NUM> provided in the rear bumper or the back door. For example, image data items imaged in the imaging units <NUM> to <NUM> are superimposed on each other, and thus, an overhead image is obtained in which the vehicle <NUM> is viewed from the upper side.

At least one of the imaging units <NUM> to <NUM> may have a function of acquiring the distance information. For example, at least one of the imaging units <NUM> to <NUM> may be a stereo camera formed of a plurality of imaging elements, or may be an imaging element including a pixel for detecting a phase difference.

For example, the microcomputer <NUM> obtains a distance to each solid object in the imaging ranges <NUM> to <NUM>, and a temporal change of the distance (a relative speed with respect to the vehicle <NUM>), on the basis of the distance information obtained from the imaging units <NUM> to <NUM>, and thus, in particular, it is possible to extract the solid object running at a predetermined speed (for example, greater than or equal to <NUM>/h) in approximately the same direction as that of the vehicle <NUM> as the leading vehicle, in the closest solid object on a traveling path of the vehicle <NUM>. Further, the microcomputer <NUM> sets the inter-vehicle distance to be ensured in advance immediately before the leading vehicle, and thus, is capable of performing automatic brake control (also including following stop control), automatic acceleration control (also including following start control), or the like. Thus, it is possible to perform the cooperative control for the automated driving in which the vehicle autonomously runs without depending on the operation of the driver.

For example, it is possible for the microcomputer <NUM> to extract solid object data relevant to the solid object by sorting the data into other solid objects such as a two-wheeled vehicle, an ordinary vehicle, a large vehicle, a pedestrian, and a telegraph pole, on the basis of the distance information obtained from the imaging units <NUM> to <NUM>, and to use the data for automatically avoiding the obstacle. For example, the microcomputer <NUM> distinguishes the obstacle around the vehicle <NUM> between an obstacle which is visible to the driver of the vehicle <NUM> and an obstacle which is not visible. Then, the microcomputer <NUM> determines collision risk representing a dangerous extent of the collision with respect to each of the obstacles, and in the case of a situation in which the collision risk is greater than or equal to a set value, that is, there is a possibility of the collision, an alarm is output to the driver through the audio speaker <NUM> or the display unit <NUM>, or forced deceleration and avoidance steering is performed through the driving system control unit <NUM>, and thus, it is possible to perform driving assistance for avoiding the collision.

At least one of the imaging units <NUM> to <NUM> may be an infrared ray camera detecting an infrared ray. For example, the microcomputer <NUM> determines whether or not the pedestrian exists in the imaged images of the imaging units <NUM> to <NUM>, and thus, it is possible to recognize the pedestrian. Such recognition of the pedestrian, for example, is performed in the order of extracting a characteristic point in the imaged images of the imaging units <NUM> to <NUM> as the infrared ray camera and the order of determining whether or not there is the pedestrian by performing pattern matching processing with respect to a set of characteristic points representing the outline of the object. The microcomputer <NUM> determines that the pedestrian exists in the imaged images of the imaging units <NUM> to <NUM>, and in a case where the pedestrian is recognized, the audio image output unit <NUM> controls the display unit <NUM> such that a rectangular outline for emphasis is displayed by being superimposed on the recognized pedestrian. In addition, the audio image output unit <NUM> may control the display unit <NUM> such that an icon or the like representing the pedestrian is displayed in a desired position.

Claim 1:
An imaging system, comprising:
a multispectral camera (<NUM>) configured to capture a multispectral image of an object;
an RGB camera (<NUM>) configured to capture a color image of the object; at least one storage device (<NUM>) configured to store spectrum information for each of a plurality of labeled objects; and
processing circuitry (<NUM>) configured to:
determine, based on the captured multispectral image, spectrum information associated with the object;
wherein the spectrum information associated with the object represent specific spectral characteristics with respect to the object, which are extracted from the multispectral image, which is acquired by imaging the object with light dispersed into a plurality of wavelength bands;
associate, based at least in part, on the spectrum information associated with the object and the stored spectrum information for each of the plurality of objects, a label with the color image of the object, which comprises:
determining a similarity measure of the spectrum information of the object with the stored spectrum information for each of the plurality of labeled objects;
identifying, based on the determined similarity measures, the labeled object having a highest similarity measure; and
associating a label associated with the labeled object having the highest similarity measure with the color image of the object;
wherein the similarity measure comprises a similarity ratio;
wherein the similarity ratio is obtained with respect to a plurality of spectrum information items retained in a spectrum information retaining unit (<NUM>) and with respect to the spectrum information obtained from the multispectral image; and
store, on the at least one storage device, the color image and the associated label;
train an object classifier using a plurality of stored color images and a plurality of associated labels as training data to generate a trained object classifier and classify an object in a received color image using the trained object classifier.