Preparation element set, preparation, manufacturing method of preparation, imaging apparatus, and imaging method

A preparation element set including an image sensor including a sensor surface, a sensor back surface, and a board; a package including a front surface, a back surface, and terminals on the back surface, the front surface touching or facing the sensor back surface; and a transparent plate facing the sensor surface with a subject placed therebetween, wherein the board includes a board surface and a board back surface, a distance between the board surface and the sensor surface is less than a distance between the board back surface and the sensor surface, a distance between the board surface and the sensor back surface is more than a distance between the board back surface and the sensor back surface, conductive holes pierce the board from the board surface to the board back surface, and conductors on the board surface are electrically connected to terminals by using the conductive holes.

BACKGROUND

1. Technical Field

The present disclosure relates to a preparation element set, a preparation, a manufacturing method of preparation, an imaging apparatus, and an imaging method.

2. Description of the Related Art

In pathological diagnosis, a tissue is cut out from an organ or a tumor of a body of a patient and is examined to confirm diagnosis of a disease and to determine the extent of the disease. In this process, the cut tissue is sectioned into a slice having a thickness of several microns and sandwiched between glass plates as a pathological slide (specimen) to be examined under a microscope. Since pathological diagnosis is an examination that is typically performed to determine whether a tumor is benign or malignant, several hundred specimens could be produced a day in each hospital. Unlike radiograph, pathological specimens are difficult to store in the form of data. For this reason, the produced specimens themselves are typically stored in a semi-permanent fashion for later reference.

Microscopes have been used to observe a microstructure of a living tissue. The microscope enlarges light transmitted through an observation target or light reflected from the observation target through a lens. An observer directly views an enlarged image of light. A digital microscope photographs a microscope image through a camera, and indicates the image on a display to be observed. Multiple persons are thus enabled to view the image concurrently or a from a remote place. The camera is located at an imaging plane and photographs an image that is enlarged through the lens of the microscope.

Japanese Unexamined Patent Application Publication No. 4-316478 discloses a technique that allows a user to view a microstructure of a tissue through a contact image sensing (CIS) system. In the CIS system, an observation target is directly placed on the surface of an image sensor and then photographed. The CIS system is free from using an image enlarged through the lens, and a magnification ratio is thus determined by the pixel size of the image sensor. In other words, the smaller the pixel size is, the finer microstructure is photographed.

SUMMARY

One non-limiting and exemplary embodiment provides the implementation of photographing through the CIS system.

In one general aspect, the techniques disclosed here feature a preparation element set. The preparation element set includes an image sensor including a sensor surface, a sensor back surface opposite to the sensor surface, and a board, a package including a front surface, a back surface opposite to the front surface, and a plurality of terminals on the back surface, the front surface touching or facing the sensor back surface, and a transparent plate facing the sensor surface with a subject placed therebetween. The board includes a board surface and a board back surface opposite to the board surface. A distance between the board surface and the sensor surface is less than a distance between the board back surface and the sensor surface. A distance between the board surface and the sensor back surface is more than a distance between the board back surface and the sensor back surface. A plurality of conductive holes pierces the board from the board surface to the board back surface. A plurality of conductors on the board surface is electrically connected to the plurality of terminals by using the plurality of conductive holes.

Comprehensive and exemplary embodiments may be implemented by a preparation, a manufacturing method of the preparation, an imaging apparatus, and an imaging method. The comprehensive and exemplary embodiments may be implemented using a system, an integrated circuit, a computer program, or a non-transitory recording medium such as a compact disk read-only memory (CD-ROM). The comprehensive and exemplary embodiments may also be implemented by any combination of the preparation element set, the preparation, the manufacturing method of the preparation, the imaging apparatus, and the imaging method.

In accordance with the disclosure, a microscope is constructed in a space-saving and low-cost manner without a lens, and CIS photographing is thus implemented.

DETAILED DESCRIPTION

Microscopes are used to observe cells of an organ of a subject in the medical field. Observing the shape of the cells helps determine whether the subject suffers from any disease. If the subject suffers from a disease, the degree of malignancy of the disease may be determined. In an examination called pathological diagnosis, a specimen taken from a patient is sectioned into a slice having a thickness of about 4 μm and thin enough to be observed. Cells are transparent. Since a microscope image features a lower contrast, the cells are stained such that the structure of the cells are easy to be observed.

An example of a manufacturing method of a preparation A01for pathological diagnosis is described with reference toFIG. 1.

As illustrated inFIG. 1, a sectioned slice A02is mounted on a slide glass plate (transparent plate) A03. The slide glass plate A03has a size of 1 mm in thickness, 76 mm in a longer-side direction, and 26 mm in a shorter-side direction. The slice A02is immersed together with the slide glass plate A03into a stain fluid A04for staining. If the slice A02is stained with the stain fluid, the slice A02becomes a stained slice A05. To protect and fix the stained slice A05, a sealing agent A06is placed on the slide glass plate A03, and a cover glass plate A07is mounted to complete a preparation A01.

FIG. 2is a cross-sectional view diagrammatically illustrating the preparation A01in an observable state under a microscope.

As illustrated inFIG. 2, the stained slice A05is mounted of the slide glass plate A03. The cover glass plate A07is fixed on the slide glass plate A03with the sealing agent A06inserted therebetween. The stained slice A05is enclosed by the sealing agent A06and inserted between the cover glass plate A07and the slide glass plate A03.

When the preparation A01is set up and observed under an optical microscope, a light source G01emits light from below the preparation A01. Illumination light G02is transmitted through the slide glass plate A03, the stained slice A05, the sealing agent A06, and the cover glass plate A07and is then incident on an objective lens G03of the microscope.

When the preparation A01is observed under the optical microscope, it takes time to set a magnification and an observation area.

The principle of the CIS system observation method is described with reference toFIG. 3.

Referring toFIG. 3, a preparation E01includes an image sensor B02in place of the cover glass plate A07. More specifically, the preparation E01includes an ordinary slide glass plate A03, the image sensor B02fixed on the slide glass plate A03with the sealing agent A06inserted therebetween, and the stained slice A05(an object) enclosed in the sealing agent A06. The image sensor B02may be a solid-state imaging device including a large number of photoelectric converters arranged in a matrix of rows and columns in an imaging plane. The photoelectric converter is typically a photodiode formed on a semiconductor layer or a semiconductor board, and generates charge in response to incident light. The resolution of a two-dimensional image sensor depends on a layout pitch or a layout density of the photoelectric converters on an imaging plane. The layout pitch of the photoelectric converters is approximately as short as the wavelength of visible light. The image sensor B02is typically a charge-coupled device (CCD) image sensor or a metal-oxide semiconductor (MOS) image sensor.

During a photographing operation, illumination light G02passes through the slide glass plate A03, the stained slice A05, and the sealing agent A06and is incident on the image sensor B02. The image sensor B02is electrically connected to a circuit (not illustrated), and performs the photographing operation. The image sensor B02outputs an image signal in accordance with a light transmittance distribution (also referred to as optical density distribution), and acquires an image of the stained slice A05.

Through the CIS observation method, no optical system, such as a lens, is present between an element that performs the photographing operation and the stained slice A05. However, since fine optical detector elements (photodiodes) are arranged at a high density on the imaging plane of the image sensor B02, a miniature structure of the stained slice A05is acquired as an image.

Referring toFIG. 4, a manufacturing method of a preparation11C as a comparative example is described.

As illustrated inFIG. 4, the slice A02is mounted on the slide glass plate (transparent plate) A03. The slice A02is immersed together with the slide glass plate A03in the stain fluid A04for staining. When the stain fluid A04sticks to the slice A02, the slice A02becomes the stained slice A05. To protect and fix the stained slice A05, the sealing agent A06is disposed on the slide glass plate A03, and then the image sensor B02is placed in place of the cover glass plate A07(seeFIG. 1). In the comparative example ofFIG. 4, the image sensor B02is connected to a package12at the back surface thereof. A preparation11C is thus complete.

FIG. 5diagrammatically illustrates a cross-sectional structure of the preparation11C as a comparative example including the image sensor B02and the package12. In the illustrated example, the image sensor B02is contained in the package12, and the image sensor B02is electrically connected to the package12via wire-like electrodes F01. The package12includes a bottom face and a wall face (side wall) forming a space that receives the image sensor B02.

The electrodes F01electrically connecting the image sensor B02to the package12as illustrated inFIG. 6Aare fine metal wires in the comparative example, and are arranged densely around the image sensor B02. Since the wire electrodes F01illustrated inFIG. 6Amay be deformed, broken, or disconnected (D03) or adjacent electrodes F01may touch and may be shorted to each other (D02) if the slide glass plate A03touches the electrodes F01(D01) as illustrated inFIG. 6B.

The preparation element set, the preparation, the manufacturing method of the preparation, the imaging apparatus, and the imaging method of the disclosure are directed to solving these problem.

The disclosure is described in the following aspects.

According to an aspect of the disclosure, there is provided a preparation element set. The preparation element set includes an image sensor including a sensor surface, a sensor back surface opposite to the sensor surface, and a board, a package including a front surface, a back surface opposite to the front surface, and a plurality of terminals on the back surface, the front surface touching or facing the sensor back surface, and a transparent plate facing the sensor surface with a subject placed therebetween. The board includes a board surface and a board back surface opposite to the board surface. A distance between the board surface and the sensor surface is less than a distance between the board back surface and the sensor surface. A distance between the board surface and the sensor back surface is more than a distance between the board back surface and the sensor back surface. A plurality of conductive holes pierces the board from the board surface to the board back surface. A plurality of conductors on the board surface is electrically connected to the plurality of terminals by using the plurality of conductive holes.

The board may be manufactured of a semiconductor, and include a plurality of electrodes on the board back surface connected to the plurality of conductors on the board surface. The electrodes may be electrically connected to the terminals of the package.

The image sensor may have a through silicon via (TSV) structure.

The electrodes may be covered with an insulator disposed on the front surface of the package.

The transparent plate may be a slide glass plate having a size of 76 mm in a first direction and 26 mm in a second direction perpendicular to the first direction.

According to another aspect, there is provided a preparation. The preparation includes an image sensor including a sensor surface, a sensor back surface opposite to the sensor surface, and a board, a package including a front surface, a back surface opposite to the front surface, and a plurality of terminals on the back surface, the front surface touching or facing the sensor back surface, the terminals being connected to the image sensor electrically, and a transparent plate facing the sensor surface with a subject placed therebetween. The board includes a board surface and a board back surface opposite to the board surface. A distance between the board surface and the sensor surface is less than a distance between the board back surface and the sensor surface. A distance between the board surface and the sensor back surface is more than a distance between the board back surface and the sensor back surface. A plurality of conductive holes pierces the board from the board surface to the board back surface. A plurality of conductors on the board surface is electrically connected to the plurality of terminals by using the plurality of conductive holes.

The board may be manufactured of a semiconductor, and include a plurality of electrodes on the board back surface connected to the conductors on the board surface. The electrodes may be electrically connected to the terminals of the package.

The image sensor may have a through silicon via (TSV) structure.

The electrodes may be covered with an insulator disposed on the front surface of the package.

The transparent plate may be a slide glass plate having a size of 76 mm in a first direction and 26 mm in a second direction perpendicular to the first direction.

According to another aspect of the disclosure, there is provided a manufacturing method of a preparation. The manufacturing method includes making a front surface of a package including the front surface, a back surface opposite to the front surface, and a plurality of terminals on the back surface be in touch with or face a sensor back surface of an image sensor including a sensor surface, the sensor back surface opposite to the sensor surface, and a board, placing a subject on a transparent plate or the sensor surface, and fixing the transparent plate and the image sensor in a manner such that the transparent plate faces the sensor surface with the subject placed therebetween. The board includes a board surface and a board back surface opposite to the board surface. A distance between the board surface and the sensor surface is less than a distance between the board back surface and the sensor surface. A distance between the board surface and the sensor back surface is more than a distance between the board back surface and the sensor back surface. A plurality of conductive holes pierces the board from the board surface to the board back surface. A plurality of conductors on the board surface is electrically connected to the plurality of terminals by using the plurality of conductive holes.

The fixing may include dipping the image sensor into a liquid, and placing the subject onto the sensor surface. The manufacturing method may further include pulling the image sensor with the subject placed on the sensor surface out of the liquid.

The manufacturing method may further include, subsequent to placing the subject on the transparent plate or the sensor surface, staining the subject, and drying the subject.

According to another aspect of the disclosure, there is provided an imaging apparatus. The imaging apparatus includes a socket that is loaded with the preparation, and is electrically connected to an image sensor via a plurality of terminals. The preparation includes the image sensor including a sensor surface, a sensor back surface opposite to the sensor surface, and a board, a package including a front surface, a back surface opposite to the front surface, and the plurality of terminals on the back surface, the front surface touching or facing the sensor back surface, and a transparent plate facing the sensor surface with a subject placed therebetween. The board includes a board surface and a board back surface opposite to the board surface. A distance between the board surface and the sensor surface is less than a distance between the board back surface and the sensor surface. A distance between the board surface and the sensor back surface is more than a distance between the board back surface and the sensor back surface. A plurality of conductive holes pierces the board from the board surface to the board back surface. A plurality of conductors on the board surface is electrically connected to the plurality of terminals by using the plurality of conductive holes. The imaging apparatus further includes a light source unit that emits light on the image sensor via the transparent plate, and a control device that causes the image sensor to photograph the subject by controlling the light source unit and the image sensor on the preparation loaded into the socket.

The light source unit may include a plurality of light sources or a moving light source. The control device may emit the light onto the subject with an angle of the light changed by plural times to photograph the subject at different angles.

According to another aspect of the disclosure, there is provided an imaging method. The imaging method includes loading a preparation into a socket of an imaging apparatus and electrically connecting the socket to an image sensor through a plurality of terminals. The preparation includes the image sensor including a sensor surface, a sensor back surface opposite to the sensor surface, and a board, a package including a front surface, a back surface opposite to the front surface, and the plurality of terminals on the back surface, the front surface touching or facing the sensor back surface, and a transparent plate facing the sensor surface with a subject placed therebetween. The board includes a board surface and a board back surface opposite to the board surface. A distance between the board surface and the sensor surface is less than a distance between the board back surface and the sensor surface. A distance between the board surface and the sensor back surface is more than a distance between the board back surface and the sensor back surface. A plurality of conductive holes pierces the board from the board surface to the board back surface. A plurality of conductors on the board surface is electrically connected to the plurality of terminals by using the plurality of conductive holes. The imaging method further includes emitting light from a light source unit to the image sensor via the transparent plate, and causing the image sensor to photograph the subject by controlling the light source unit and the mage sensor on the preparation loaded into the socket.

The light source unit may include a plurality of light sources or a moving light source. The causing may include emitting the light onto the subject with an angle of the light changed by plural times to photograph the subject at different angles.

According to another aspect of the disclosure, there is provided a preparation element set including an image sensor chip including a semiconductor board having a plurality of through-holes, a plurality of photoelectric converters disposed on a front side of the semiconductor board, and a signal pickup unit disposed on a back surface opposite to the front side and electrically connected to a circuit disposed on the front side via the through-holes, and a transparent preparation, the image sensor chip being glued to the transparent preparation with a subject placed therebetween.

Embodiments of the disclosure are described in detail with reference to the drawings.

The embodiments described below are comprehensive and specific examples of the disclosure. Values, shapes, materials, elements, mounting locations, connection form, steps, and order of steps in the embodiments are described for exemplary purposes only, and are not intended to limit the disclosure. Among the elements in the embodiment, elements not described in the independent claims indicative of higher concepts may be any arbitrary element. The embodiments may also be combined.

First Embodiment

FIG. 7Adiagrammatically illustrates a cross-sectional structure of a preparation11used in a first embodiment of the disclosure and part of an imaging unit that detachably supports the preparation11. The entire structure of the imaging unit is described below. The imaging unit includes the socket C03configured to receive the preparation11. The socket C03is electrically connected to a circuit board C05. The socket C03is electrically connected to the circuit board C05by setting multiple terminals disposed on the back surface of the socket C03to be in touch with wirings or electrodes pads disposed on the circuit board C05. The circuit board C05may have a structure available in the related art, and may be a multi-layered printed circuit board. The socket C03may be mounted on the circuit board C05in one of related art mounting methods to mount electronic components onto a circuit board. The package12includes on the back surface thereof terminals13to electrically connect an image sensor B01to an external circuit. The socket C03includes multiple terminals C04disposed to electrically connect to the terminals13of the package12.

FIG. 7Bis a cross-sectional view partially illustrating the image sensor B01in the embodiment of the disclosure. The image sensor B01includes a semiconductor board400and a wiring layer402disposed on a surface400aof the semiconductor board400. The semiconductor board400includes a via410that pierces therethrough from the surface400ato a back surface400b.FIG. 7Billustrates the single via410, but the actual semiconductor board400has a large number of vias410. Each of the multiple vias410includes a conductor420. The conductors420electrically connect the wiring layer402to the multiple terminals13on the package12. The conductors420is electrically insulated from the semiconductor board400by an insulation layer (not illustrated). The conductors420is manufactured of a metal having a sufficiently low electrical resistance. A structure in which a conductor in a via piercing through a semiconductor board leads to the back surface of the semiconductor board is typically applied to a silicon board, and is thus called through silicon via (TSV) structure. The semiconductor board400is typically manufactured of single crystal silicon, but may be manufactured of another semiconductor.

In the TSV structure, a signal pickup unit may be disposed on the back surface of the semiconductor board (the surface opposite to the front surface having photoelectric converters), and a circuit (a power supply, a driver circuit, and/or a signal processor circuit) disposed on the front surface of an image sensor chip is connected to the signal pickup unit through through-vias. The image sensor chip having the TSV structure may be referred to “backside illumination type”.

Referring toFIG. 7B, the conductors420filling the via410is connected to one of the multiple electrodes P07disposed on the back surface400bof the semiconductor board400. The image sensor B01is electrically connected to the package12via the electrodes P07in place of the wire-like electrodes F01described with reference to the comparative case.

As illustrated inFIG. 7B, the image sensor B01includes the semiconductor board400, photodiodes (PDs)40disposed in the front surface of the semiconductor board400, wiring layer402supported by the semiconductor board400, light shielding layer42covering the wiring layer402, transparent layer406covering a light incident surface of the semiconductor board400, and insulation layer P01covering the back surface of the semiconductor board400.

If the image sensor B01is a CCD image sensor, the semiconductor board400includes beneath the wiring layer402an impurity diffusion layer (not illustrated) that functions as a vertical or horizontal charge transfer path. The wiring layer402is connected to an electrode (not illustrated) disposed on the charge transfer path. If the image sensor B01is a MOS image sensor, a MOS transistor (not illustrated) is formed on a per pixel basis on the semiconductor board400. The MOS transistor functions as a switching element that reads a charge on the photodiode40. An organic semiconductor film or inorganic semiconductor film formed in the upper portion of the semiconductor board400may be used as a photoelectric conversion film that converts light into charge in place of the photodiode40formed in the semiconductor board400.

FIG. 7Cis a plan view diagrammatically illustrating part of an imaging plane of a charge-coupled device (CCD) image sensor as an example of the image sensor B01. As illustrated inFIG. 7C, multiple photodiodes (photoelectric converters)40are arranged in a matrix of rows and columns on the imaging plane. As illustrated inFIG. 7C, a single pixel50is represented a broken-lined rectangle inFIG. 7C. A large number of pixels50is densely arranged in a matrix of rows and columns in the imaging plane.

Light incident on the photodiode40generates charge in the photodiode40. An amount of charge generated varies depending an amount of light incident on the photodiode40. Charges generated in each photodiode40move along a vertical charge transfer path44extending vertically. The generated charges thus move along the vertical charge transfer paths44and then reach a horizontal transfer path46. The charges are transferred along the horizontal charge transfer path46and then are output from one end of the horizontal charge transfer path46to the outside of the image sensor B01as a pixel signal. Transfer electrodes (not illustrated) are disposed on the vertical charge transfer paths44and the horizontal charge transfer path46. The image sensor B01used in the imaging apparatus of the disclosure is not limited to the structure described above. The MOS image sensor may be used in place of the CCD image sensor.

FIG. 7Dis a plan view diagrammatically illustrating a single pixel50. A region covered with the light shielding layer42is hatched. Elements other than the photodiode40in the image sensor B01are covered with the light shielding layer42. In the embodiment of the disclosure, a micro lens that increases the aperture ratio of each photodiode40is not arranged in the image sensor B01. Parallel light rays are incident on the photodiode40. In accordance with the embodiment of the disclosure, the size of the photodiode40determines the resolution. The larger in size the photodiode40is, the lower the resolution becomes.

FIG. 7Eis a cross-sectional view illustrating a wider portion of the image sensor B01. In this example, an electrode is hemispherical. Referring toFIG. 7F, a large number of electrodes (bumps) P07are disposed on the back surface of the image sensor B01. The electrodes P07are isolated from each other by the insulation layer P01.

FIG. 7Gdiagrammatically illustrates the preparation11loaded in the socket C03. The preparation11is temporarily fixed to the socket C03by means of the structure of the socket C03and other mechanism. The terminal C04of the socket C03is electrically connected to the image sensor B01via the terminal13of the package12by loading. The structure of the socket C03is not limited to the structure described herein. The electrical connection between the socket C03and the image sensor B01is not limited to the connection described herein.

Illumination light is directed to the preparation11from above in the state illustrated inFIG. 7G, and transmitted through the stained slice A05, and then incident on the image sensor B01. A photographing operation is then performed by multiple times. After the preparation11as a photographing target is photographed, the preparation11is removed from the socket C03, and another preparation11is then loaded in the socket C03as a next photographing target.

The imaging apparatus10includes a lighting system C09that emits light to the image sensor B01via a slide glass plate A03in the preparation11loaded in the socket C03. The structure and operation of the lighting system C09are described below. In the example illustrated inFIG. 8, the lighting system C09is located above the preparation11supported by an imaging unit90. The embodiment of the disclosure is not limited to this arrangement. Alternatively, the preparation11is located above the lighting system C09, or a line connecting the lighting system C09and the preparation11may be tilted from a vertical alignment inFIG. 8.

The imaging apparatus10includes a control device (computer) C06, and the control device C06includes a controller121, an image processor122, and a memory145. The controller121controls the lighting system C09and the image sensor B01of the preparation11loaded in the socket C03. The controller121thus causes the image sensor B01to photograph the stained slice in the preparation11.

As described with reference toFIG. 7G, the package12, if loaded in the socket C03, is electrically connected to the socket C03. The socket C03is connected to the control device C06ofFIG. 8via the circuit board C05ofFIG. 7G.

The image data obtained as a result of photographing is combined and pixel-interpolated through the image processor122. These processes result in a photographed image of the stained slice at a higher resolution. The photographed image is displayed on a display C07, for example, and stored on the memory145or the database148.

FIG. 9is a plan view diagrammatically illustrating a layout of light source elements in the lighting system C09used in the first embodiment of the disclosure. As illustrated inFIG. 9, 25light source elements20are arranged in a matrix of rows and columns. More specifically, the light source elements20are arranged in a matrix of five rows and five columns on the light output side of the lighting system C09.

As illustrated inFIG. 10, the lighting system C09including the light source elements20arranged in a matrix causes illumination light rays to be incident on the image sensor B01in the preparation at a different angle. The illumination light rays emitted from the light source elements20are substantially parallel and incident on the image sensor B01. If the lighting system C09includes at least four light source elements20, the illumination light rays are caused to be incident on the image sensor B01in the preparation with the lighting direction successively changed one by one, from at least four different directions. The light source elements20in the lighting system C09may include a combination of light emitting elements, such as LEDs, and a color filter. Each light source element20may also include an optical element to adjust the divergence of a light beam and a reflecting mirror.

The manufacturing method of the preparation11of the first embodiment is described with reference toFIG. 11.

The sectioned slice A02is mounted on the slide glass plate A03. The location of the slice A02on the slide glass plate A03is not necessarily in the center of the slide glass plate A03. In the example ofFIG. 11, the slice A02is closer to one end of the slide glass plate A03. The slice A02together with the slide glass plate A03is immersed in the stain fluid A04for staining. The slice A02thus stained with the stain fluid A04becomes the stained slice A05. To protect and fix the stained slice A05, a sealing agent is applied on the slide glass plate A03. The package12having the image sensor thereon is mounted on the slide glass plate A03. In this operation, the package12is adjusted in location to be in alignment with the stained slice A05.FIG. 12illustrates the stained slice A05at a different location on the slide glass plate A03. In this example, as well, the package12is adjusted in location to be in alignment with the stained slice A05.

Referring toFIG. 11andFIG. 12, the back surface of the package12is seen. Although the package12has a thickness in practice, the package12is illustrated as a thin film inFIG. 11andFIG. 12.

When the package12with the image sensor B01fixed thereto is mounted on the slide glass plate A03, the package12may be electrically connected to the imaging unit to perform the photographing operation. In this photographing operation, the positional relationship between the image sensor B01and the stained slice A05may be detected. Sine no optical system is present between the image sensor B01and the stained slice A05, the image of the stained slice A05acquired by the image sensor B01is blurred, but the location of the stained slice A05is detectable.

Referring toFIG. 13, the manufacturing method of a preparation11A of another embodiment is described below.

The sectioned slice A02is mounted on the imaging plane of the image sensor B01. The image sensor B01may be in a state mounted on a package (not illustrated) or in a state prior to being mounted on the package. The electrodes of the image sensor B01are waterproofed. The waterproofing operation may be performed by covering the electrodes on the front surface of the package. The slice A02is immersed together with the image sensor B01into the stain fluid A04for staining. The slice A02, if stained with the stain fluid A04, becomes the stained slice A05. The slice A02in the stain fluid A04may be lifted using a package (not illustrated), for example. To protect and fix the stained slice A05, the sealing agent is applied on the image sensor B01. The image sensor B01is fixed onto the slide glass plate A03with the stained slice A05disposed on the slide glass plate A03.

In practice, the slice A02prior to staining is covered with paraffin (not illustrated) in the stain fluid A04. When the slice A02is lifted up with the package12(seeFIG. 5) from the stain fluid A04, paraffin may spread out of the image sensor B01in the package12. In such a case, the image sensor B01is connected with the package12via bonding wires, the bonding wires may be damage by paraffin. The embodiment of the disclosure controls such a problem.

In the embodiment of the disclosure, the magnification and point of view may be changed without switching lenses and moving the slide glass plate. The imaging apparatus of the embodiment may be applied to a specimen management apparatus.

An example of the specimen management apparatus that is implemented using the imaging apparatus is described below.

FIG. 14illustrates an example of the entire configuration of the specimen management apparatus300.

The specimen management apparatus300includes a specimen image acquisition apparatus110and an information processing apparatus230. The specimen image acquisition apparatus110may be the imaging apparatus10that has been described with reference toFIG. 8. The specimen image acquisition apparatus110may acquire the image of the pathological specimen30including a preparation of the embodiment (such as the preparation11) at one of multiple resolutions (magnifications).

The information processing apparatus230is connected to the specimen image acquisition apparatus110in a wired or wireless fashion, and receives information acquired by the specimen image acquisition apparatus110. The information processing apparatus230determines a feature quantity of an image acquired by the specimen image acquisition apparatus110, and causes the output apparatus170to output the patient information of the pathological specimen30in response to the feature quantity. More in detail, the information processing apparatus230references a database (not illustrated inFIG. 14) that associates the feature quantity calculated from a specimen image of the patient with patient information, and searches for the patient information matching the feature quantity of the image of the pathological specimen30.

The information processing apparatus230is connected to an input apparatus160and an output apparatus170. With the input apparatus160, the user may input data and an instruction to the information processing apparatus230. The input apparatus160may be a keyboard, a mouse, or a touchscreen. The output apparatus170may be a display that is configured to display an image and characters, a printer, or a loudspeaker. The input apparatus160and the output apparatus170may be a unitary module into which the functions thereof are integrated. If the specimen management apparatus300includes the input apparatus160and the output apparatus170, the imaging apparatus as the specimen image acquisition apparatus110may not necessarily have to include the control device (computer) C06and the display C07(seeFIG. 8).

If one piece of patient information matching the feature quantity of the image is stored on the database, the information processing apparatus230outputs the patient information to the output apparatus170. If multiple pieces of patient information matching the image are stored on the database, the information processing apparatus230acquires a high-resolution image having a resolution higher than the resolution of the image, and searches the database for patient information matching the feature quantity of the high-resolution image. If no piece of patient information matching the feature quantity of the high-resolution image is stored on the database, the information processing apparatus230receives patient information from the input apparatus160, associates the feature quantity calculated from the image with the patient information and stores the associated information onto the database. The specimen image acquisition apparatus110acquires a high-resolution image having a resolution higher than the resolution of the first acquired image, and the information processing apparatus230stores the feature quantity calculated from each image and patient information in association with each other.

FIG. 15is a block diagram illustrating the configuration of the specimen management apparatus300. As illustrated inFIG. 15, the specimen management apparatus300ofFIG. 14includes the socket C03, specimen image acquisition apparatus110, image feature quantity calculator120, information searching unit130, patient information database (hereinafter referred to as simply referred to as a database)140, magnification adjuster150, input apparatus160, and output apparatus170.

The pathological specimen30, the patient information of which is to be acquired or updated, is placed on the socket C03. The pathological specimen30may be one of the preparations described with reference to the embodiment. The pathological specimen30is herein the preparation11.

The specimen image acquisition apparatus110captures an image of the specimen in the preparation11(the stained slice A05) at one of predetermined multiple different magnifications. The image feature quantity calculator120calculates an image feature quantity from the image acquired by the specimen image acquisition apparatus110. The information searching unit130searches the database140that stores the patient information and the image feature quantity in association with each other for patient information matching the image feature quantity calculated by the image feature quantity calculator120. If multiple hits acquired by the information searching unit130are present, the magnification adjuster150changes a magnification for acquisition to a higher magnification (a higher resolution), the specimen image acquisition apparatus110acquires an image again, and performs a search operation based on information acquired at the higher magnification (higher resolution).

If the patient information matching the image feature quantity is not hit by the information searching unit130, the input apparatus160receives the patient information as a specimen as a new patient. If the patient information matching the image feature quantity is hit by the information searching unit130, the output apparatus170outputs the acquired patient information.

The operation and configuration of each element of the specimen management apparatus300are described more in detail.

Operation of Specimen Management Apparatus

Refer toFIG. 16.FIG. 16is a flowchart illustrating a process of a specimen management method.

In step S10, the preparation11, the patient information of which is to be referenced or updated, is placed on the socket C03. The socket C03is configured to receive the preparation11. More specifically, the socket C03may have a recess sized to precisely receive the pathological specimen30. Such socket C03may control a position deviation in the pathological specimen30that could occur when the image is captured. Standardized pathological specimens having a size of 76 mm by 26 mm are typically used in Japan. The socket C03is thus shaped to receive such pathological specimen30.

In step S11, the specimen image acquisition apparatus110acquires an image of the pathological specimen30at one of predetermined multiple different magnifications.FIG. 17is a block diagram illustrating the configuration of the specimen image acquisition apparatus110. As illustrated inFIG. 17, the specimen image acquisition apparatus110includes a lighting direction adjuster200, and a lighting device210. The lighting device210includes the light source G01that causes light to be incident on the image sensor B01of the preparation11. The specimen image acquisition apparatus110acquires an image (such as a whole picture) at any magnification specified by the information processing apparatus230.

When an image is acquired at a different magnification, a high-resolution enhancement process may be performed by an inverse matrix calculator240and a matrix storage unit250. As illustrated inFIG. 17, the inverse matrix calculator240and the matrix storage unit250may be included in the information processing apparatus230. Alternatively, the inverse matrix calculator240and/or the matrix storage unit250may be included in the specimen image acquisition apparatus110. The operation of the inverse matrix calculator240and the matrix storage unit250is described below in detail.

An image acquisition process is described with reference toFIG. 18.

In step S110, the lighting direction adjuster200adjusts an angle of parallel illumination light incident on the pathological specimen30. To adjust the lighting direction, multiple light sources may be arranged to emit light at predetermined angles as illustrated inFIG. 19A(including light sources G01-1, G01-2, and G01-3), or as illustrated inFIG. 19B, a single light source G01-0may be directed to a specified angle.

In step S111, the lighting device210emits parallel light rays to a specimen as a photographing target as an angle adjusted in step S110.FIG. 20AandFIG. 20Billustrate how the lighting direction changes. The pathological specimen30and the image sensor B01have a two-dimensional layout relationship as illustrated inFIG. 21. For simplicity of explanation,FIG. 20AandFIG. 20Billustrate a cross-section of a pixel region including a single photodiode PD. Light incident on the photodiode PD is converted into an electrical signal through photoelectric conversion. The size of the line of each arrow diagrammatically illustrates an amount of light incident on the photodiode PD, and the larger the size of the arrow line is, the higher amount of light is incident as illustrated inFIG. 20AandFIG. 20B.

Referring toFIG. 20A, parallel light rays enter right from above. In this case, light rays transmitted through regions S2 and S3 of the pathological specimen30are incident on the photodiode PD. Referring toFIG. 20B, parallel light rays emitted at an angle, and transmitted through regions S2, S3, and S4 of the pathological specimen30are incident on the photodiode PD. More specifically, as illustrated inFIG. 20B, half amount of each of the light rays transmitted through the regions S2 and S4 of the pathological specimen30is incident on the photodiode PD, and all amount of the light ray transmitted through the region S3 is incident on the photodiode PD. A pixel value different from the pixel value ofFIG. 20Ais thus output from the photodiode PD.

As illustrated inFIG. 20AandFIG. 20B, it is difficult to determine a pixel value at each of the regions S1, S2, S3, and S4 from a single image photographed at a single lighting direction alone. The specimen management apparatus300described above determines pixel values from multiple images with the lighting direction changed in response to the light rays transmitted through the regions S1, S2, S3, and S4 as illustrated inFIGS. 20A and 20B. The regions S1, S2, S3, and S4 are smaller in size than a single pixel, and correspond to sub-pixel regions. This is described in detail below.

A light ray may now be incident on the pathological specimen30at four different directions 1, 2, 3, and 4. Four images result with the light ray is incident at each of the four different directions 1, 2, 3, and 4. One pixel at the same location from among pixels forming the four images is now studied. The outputs of the photodiode PD included in the studied pixel are referred to as A1, A2, A3, and A4 in response to the lighting directions 1, 2, 3, and 4. Light transmittances of at the regions S1, S2, S3, and S4 in the pathological specimen30are designated S1, S2, S3, and S4. In the example ofFIG. 20A, formula A1=0×S1+1×S2+1×S3+0×S4 holds. In the example ofFIG. 20B, formula A2=0×S1+(½)×S2+1×S3+(½)×S4 holds. In the lighting direction S3 (not illustrated), formula A3=0×S1+0×S2+(½)×S3+1×S4 holds. In the lighting direction S4 (not illustrated), formula A4=(½)×S1+1×S2+(½)×S3+0×S4 holds.

In the above example, light transmittances S1, S2, S3, and S4 depend on the tissue structure of the pathological specimen30, and are not known. The light transmittances S1, S2, S3, and S4 are obtained by acquiring the four images of the outputs A1, A2, A3, and A4 of the photodiode PD. Simultaneous equations having the light transmittances S1, S2, S3, and S4 as four unknown quantities are determined, and the light transmittances S1, S2, S3, and S4 are thus calculated.

FIG. 22Aillustrates elements of a matrix as coefficients of the simultaneous equations. By calculating a vector having as components the outputs A1, A2, A3, and A4 from an inverse matrix of the matrix, the light transmittances S1, S2, S3, and S4 of regions narrower than one pixel (sub-pixel region) are thus determined. As a result, an image having a resolution four times as high as the original image results. In other words, a high resolution image having an image density four times as high as the image density of the image sensor B01is obtained.

The values of the matrix elements ofFIG. 22Ado not depend on the tissue structure of the pathological specimen30but depend on the structure of the image sensor B01and the lighting directions. Given the same image sensor B01, the value of the matrix elements vary if the lighting directions change.FIG. 22Billustrates the values of the matrix elements when light is emitted in different lighting directions 1 through 8. The number of sub-pixel regions is eight, and light is directed to the pathological specimen30in at least eight different lighting directions 1 through 8, and eight outputs are obtained on each pixel. Light transmittances of eight sub-pixel regions as unknown quantities are determined. As a result, an image having a resolution eight times higher thus results. In other words, a high resolution image having a pixel density eight times as high as the pixel density of the image sensor B01is obtained.

In this way, a higher resolution image results. In other words, the photographing operation is made with the lighting direction changed, and images at different resolutions (magnifications) are obtained. This operation is free from focusing using the objective lens.

Referring toFIG. 18, the pathological specimen30is photographed by the image sensor B01in step S112. A line sensor is typically used in a standard scanner. The use of an area sensor, such as a CCD image sensor, as the image sensor B01allows the specimen to be photographed at a high speed in a wide area of the image used to identify the specimen. The specimen image acquisition apparatus110ofFIG. 14dispenses with a lens to control magnification, and generates an image at any magnification from among multiple images obtained with lighting directions changed.

In step S113, all images used to generate a specimen image at a specified magnification are prepared. If all the images are prepared, processing proceeds to step S114. If not all the images are prepared, processing returns to step S110to capture an image lit at a desired lighting direction.

In step S114, the information processing apparatus230(seeFIG. 14) generates an image at a specified magnification from multiple images photographed at different lighting directions from step S110through S112. To generate the image at the specified magnification, a matrix of a pre-calculated relationship between a lighting direction and an amount of light incident on a photodiode PD is stored on the matrix storage unit250(seeFIG. 17).FIG. 22AandFIG. 22Billustrate the examples of matrices indicating the relationships between the lighting directions and the light incident on the sensor. These matrices may be calculated based on the lighting direction, the size of the photodiode PD, and the desired size of the pixel. A test specimen having a known pixel value may be used. For example, the matrix may be experimentally calculated by measuring what percent of light is incident on the photodiode PD after the light is transmitted through a given region of the test specimen at an incident angle.

Let M represent a matrix that represents a relationship between the lighting direction and light incident on an imaging element, let A represent a pixel value vector, and let S represent a vector of desired pixel values, and a relationship of MS=A holds in each pixel. Since matrix M and vector A are known, matrix S may be determined through inverse matrix calculation. In step S114ofFIG. 18, a matrix representing the relationship between the lighting direction and the light incident on the photodiode PD is obtained from the matrix storage unit250, and the inverse matrix calculator240(seeFIG. 17) calculates each pixel value. Using the specimen management apparatus300thus constructed, the whole picture of the specimen is photographed at any magnification. The process described above may be performed by the specimen image acquisition apparatus110.

In step S12ofFIG. 16, the image feature quantity calculator120calculates the image feature quantity identifying the specimen from the specimen image acquired in step S11. The image feature quantity may be color information of average luminance, shape feature, such as roundness, scale-invariant feature transform (SIFT), histogram of oriented gradient (HOG), or higher-order local autocorrelation (HLAC). A distance between a cell and its nucleus, or a ratio of color of nucleus to color of cell may serve as a feature quantity specific to a pathological image.

FIG. 23andFIG. 24illustrate pathological images.FIG. 23illustrates a pathological specimen observed at a high magnification (such as a magnification of 200 times or more), andFIG. 24illustrates a pathological specimen observed at a low magnification (such as a magnification less than 10 times). A magnification of N times means that a resolution of an image (the number of pixels or a pixel density per image) increases to N×N times. The magnification on a display screen of a display included in the output apparatus170is defined by a ratio of a pixel pitch of the imaging element to a screen pitch of the display.

If the pathological specimen is observed at a high magnification as illustrated inFIG. 23, cells and nucleuses are observable. Since the layout of and distance between a cell and its nucleus are different depending on specimen, the average distance between cells and nucleuses may be used as a feature identifying a specimen. Since the tissue as an observation target in the pathological specimen is transparent as it is, the pathological specimen is typically stained for easier observation. The staining method includes hematoxylin and eosin stain as a basic staining method, or a variety of immunostaining in which stain is performed in view of purposes of a particular examination. The ratio of cells to nucleus differently stained may be used as a feature. For example, in Ki-67 as one of immunostaining, growing cells are stained reddish brown, and other cells are stained blue. Such ratios not only serve as a measure in diagnosis, but also are useful as identification information of the pathological specimen. In step S12, the image feature quantity may be modified depending on the magnification of the pathological specimen image. In the pathological specimen, image features greatly change in accordance with the magnification in use. When the specimen is observed at a high magnification as illustrated inFIG. 23, the cells and nucleuses are observable. At a low magnification, the entire shape of the pathological specimen is recognizable as illustrated inFIG. 24. In view of these features, feature quantities appropriate for general shape recognition, such as roundness, SIFT, HOG, or HLAC, may be mainly used in the image at a low magnification. Features characteristic of the pathological specimen, such as the distance between the cell and nucleus, or the ratio of stained colors may be used in the image at a high magnification. More specifically, if the resolution of image is lower than a criterion, at least one of roundness, SIFT, HOG, and HLAC is calculated and acquired. If the resolution of image is equal to or above the criterion, the average distance between the cells and the nucleuses and/or the ratio of the differently stained colors may be calculated in addition to the feature quantity.

In step S13(seeFIG. 16), the information searching unit130retrieves from the database140patient data matching the image feature quantity calculated in step S12(seeFIG. 15).FIG. 25illustrates an example of the database. The database includes an image feature quantity calculated from the pathological specimen image, an imaging magnification of the specimen image from which the image feature quantity is calculated, and patient data associated with the patient information. Since the patient information is stored in such a format, the patient data having the image feature quantity matching the image quantity calculated in step S12is searched for in the database. The match condition in the search may be full match of the image feature quantities. If the image feature quantity is expressed in a vector, the images are determined to be matched if an Euclidean distance between vectors is equal to or below a predetermined threshold value. The database may be in a format as illustrated inFIG. 26. In the format ofFIG. 26, information of the specimen of the same patient with different stains is associated and stored by attaching identification (ID) to the patient. In the pathological examination (tissue diagnosis) today, immunostaining in which staining is performed in view of purposes of a particular examination is typically performed in addition to hematoxylin and eosin stain as a basic stain method. The specimens of the same patient with different stains are different in color but generally similar in shape as illustrated inFIG. 27. This is because when multiple stain samples are produced from the same patient, the samples are often taken from continued slices. The characteristics of the pathological specimens may be used. Since the specimen image of the specimen is acquired as an image in the disclosure, the different stain specimens of the same patient may be automatically associated by comparing the shape feature quantities of the acquired images.

In step S14, it is determined whether the search results in step S13indicate that the patient data having the same image feature quantity as the image feature quantity calculated in step S12is present in the database140. If the patient data having the same image feature quantity is not present, processing proceeds to step S15. If the patient data having the same image feature quantity is present, processing proceeds to step S17.

In step S15, the input apparatus160requests the patient information corresponding to the pathological specimen loaded in step S10to be input. In step S16, the patient information input in step S15is stored on the database140in association with the magnification of the specimen image acquired in step S11and the image feature quantity calculated in step S12.

In step S17, it is determined whether the search results in step S13indicates that multiple pieces of the patient information having the same image feature quantity as the image feature quantity calculated in step S12are present in the database140. If multiple pieces of the patient information having the same image feature quantity are present in the database140and it is difficult to identify a single piece of the patient information, processing proceeds to step S18. If a single piece of the patient information having the same image feature quantity is present, processing proceeds to step S19.

If the patient information is not identified, the specimen image acquisition apparatus110changes the magnification in step S18, and returns to step S11. The pathological specimen has a feature that if the specimens are similar in shape at a low magnification, a difference therebetween is definitely recognizable at cell and nucleus level at a high magnification. There is a trade-off between time to capture the specimen images in step S11and the magnification. For this reason, the specimen image is identified at a lower magnification first. In an efficient way, if the specimen image is not identified at a lower magnification, the magnification may be increased. More specifically, operations in steps S11through S17are repeated with the magnification increased until a single piece of patient information is identified. When patient information of a new specimen is added to the database, the information searching unit130searches the database for a case matching a shape feature quantity not dependent on color, from among the feature quantities of the image. If a matching case is hit, the case may be associated with the different stained specimen of the same patient.

In step S19, the output apparatus170outputs the patient information acquired in step S13. The output apparatus170may not necessarily have to include a display or a printer. Alternatively, the output apparatus170may be connected to an external display or an external printer, and may output the signal to the external display or the external printer.

As described above, the pathological specimen is managed precisely without imposing an excessive workload on an operator. The specimen management of the embodiment is free from attaching a bar code or an IC tag onto the pathological slide.

Another configuration of the specimen management apparatus is described with reference toFIG. 28andFIG. 29.

A specimen management apparatus300A photographs the pathological specimen30placed on the socket C03while moving the pathological specimen30as illustrated inFIG. 28. The specimen management apparatus300A thus photographs multiple images to generate a specimen image at a higher magnification. The configuration except the specimen image acquisition apparatus is similar to the configuration of the specimen management apparatus300.

FIG. 29is a block diagram illustrating an example of the specimen image acquisition apparatus110A included in the specimen management apparatus300A. Referring toFIG. 29, the specimen image acquisition apparatus110A is different from the specimen image acquisition apparatus110ofFIG. 17in that the specimen image acquisition apparatus110A includes a specimen mover260in place of the lighting direction adjuster200. The specimen image acquisition apparatus110A acquires multiple images in order to obtain an image at a higher magnification by photographing the specimen with the specimen moved rather than acquiring multiple images with the parallel light rays changed in lighting direction. The matrix storage unit250stores a matrix that represents a relationship of a moving direction, a distance of movement, and light incident on the imaging element in place of the matrix representing the relationship between the lighting direction and the light incident on the imaging element. The specimen image acquisition apparatus110A implements a function of acquiring an image at any magnification in operations similar to those in steps S110through S114described with reference toFIG. 18. In step S110, however, the specimen placed on the socket C03is moved with the lighting direction of the parallel illumination light rays unchanged. In the example, the direction of the parallel light rays incident on the pathological specimen may be fixed. With the operations similar to the operations of the specimen image acquisition apparatus110in steps S111through S114, an image at a higher magnification results from multiple images at a lower magnification.

The disclosure finds applications in a specimen management apparatus that manages specimens.