Patent Description:
Japanese Patent Application Laid-Open (<CIT>, for example, describes a microscope that captures an image of a sample to be observed by a camera function of a portable information terminal and displays the image on a display. The microscope includes a microscope body and a placing table that is detachably connectable to the microscope body at plural connection positions and on which a portable information terminal is placed in a state of being connected to the microscope body. The microscope body includes a sample placement section for placing a sample, a light source for irradiating the sample placed on the sample placement section with light, and an optical system disposed inside the microscope body. The optical system includes an objective lens on which light from the sample placed on the sample placement section is incident, and an eye lens on which light from the objective lens is incident and that emits the incident light to the outside of the microscope body. The placing table includes plural see-through windows formed at positions coinciding with the camera lens of any one of the various mobile information terminals when any one of the plural types of mobile information terminal is placed thereon, and the placing table and the microscope body are configured to be connectable such that the see-through window coincides with a position of the eye lens in a state where the camera lens of the mobile information terminal and the see-through window are coincident.

In order to observe the sample well, it is necessary to adjust an optical axis of an eyepiece lens of the microscope and an optical axis of the camera to match them as accurately as possible. However, fine operation is required for position adjustment with respect to the optical axis of the eyepiece lens provided in a high-magnification microscope.

Further background art is provided in a paper by <NPL>), which discloses that quantitative microscopy with micron-scale spatial resolution can be carried out with mobile phones, and that image linearity, distortion and color can be corrected as needed. It is also stated that automatic focus, exposure, and color gain standard on mobile phones can degrade image resolution and reduce accuracy of color capture if uncorrected, and procedures are proposed to avoid these barriers to quantitative imaging.

An object of the present disclosure is to provide a guidance program, a guidance method, an imaging device, an information processing device, and a microscope device that can easily adjust a position of a camera with respect to an eyepiece lens of a microscope.

More particularly, the present invention provides a guidance program as defined in Claim <NUM> of the appended claims, for guiding a camera, which is held by a holder and which captures a field of view of a microscope through an eyepiece lens of the microscope, to an appropriate holding position with respect to an optical axis of the eyepiece lens. Details of certain embodiments are set out in the dependent claims. Also provided is a guidance method as defined in Claim <NUM>, an imaging device as defined in Claim <NUM>, an information processing device as defined in Claim <NUM>, and a microscope device as defined in Claim <NUM>.

According to the present disclosure, it is possible to easily adjust the position of the camera with respect to the eyepiece lens of the microscope.

Hereinafter, an example of a mode for carrying out the technology of the present disclosure will be described in detail with reference to the drawings. Note that components and processes having the same operation, action, and function are denoted by the same reference signs throughout the drawings, and redundant description may be omitted as appropriate. Each drawing is only schematically illustrated to the extent that the technology of the disclosure can be sufficiently understood. Therefore, the technology of the disclosure is not limited only to the illustrated example. Furthermore, in the present embodiment, description of configurations that are not directly related to the disclosure or well-known configurations may be omitted.

<FIG> is a side view schematically illustrating an example of a microscope device <NUM> according to a first embodiment.

As illustrated in <FIG>, the microscope device <NUM> according to the present embodiment includes an imaging device <NUM>, a holder <NUM>, and a microscope <NUM>. The imaging device <NUM> includes a display unit <NUM> and a camera <NUM>. The camera <NUM> is, for example, a camera using a charge coupled device (CCD), and the camera <NUM> is provided with a lens (hereinafter, referred to as a "camera lens") 19A. As the imaging device <NUM>, for example, a portable device such as a smartphone, a tablet terminal, or a digital still camera is applied. The display unit <NUM> is a display integrally provided with a touch panel, and displays a captured image obtained by capturing by the camera <NUM>.

Note that a portable microscope (for example, Handy Microscope DX (Distributor: Raymay Fujii Corporation, model number: RXT300N), Reference HP: https://www. jp/nature/contents/micro/item/RXT300/) can be cited as a reference example of a microscope device capable of capturing a field of view of a microscope with a smartphone camera.

The microscope <NUM> includes a housing lower portion <NUM>, a housing support portion <NUM>, a housing upper portion <NUM>, a light source <NUM>, a stage <NUM>, and a lens <NUM>. The housing lower portion <NUM> is connected to a lower end of the housing support portion <NUM>, and the housing upper portion <NUM> is connected to an upper end thereof. The light source <NUM> is disposed in the housing lower portion <NUM>, and the lens <NUM> is disposed in the housing upper portion <NUM>. The stage <NUM> on which a sample is placed is disposed between the light source <NUM> and the lens <NUM>. The lens <NUM> includes an eyepiece lens 36A and an objective lens 36B. The eyepiece lens 36A is disposed in the housing upper portion <NUM> such that an optical axis of the eyepiece lens 36A faces upward. Light from the light source <NUM> is applied to the stage <NUM>, the transmitted light transmitted through the stage <NUM> is incident on the eyepiece lens 36A through the objective lens 36B, and the light incident on the eyepiece lens 36A is emitted toward the imaging device <NUM>.

The holder <NUM> includes a through hole <NUM> penetrating an upper surface and a lower surface of the holder <NUM> in a vertical direction, and is disposed on the housing upper portion <NUM> of the microscope <NUM> so that the optical axis of the eyepiece lens 36A is exposed to an upper surface side of the holder <NUM> through the through hole <NUM>. The holder <NUM> detachably holds the imaging device <NUM> such that an optical axis of the camera lens 19A of the imaging device <NUM> faces downward. Specifically, the imaging device <NUM> is placed on the upper surface of the holder <NUM>. The camera lens 19A of the imaging device <NUM> and the eyepiece lens 36A of the microscope <NUM> are provided to face each other via the through hole <NUM> of the holder <NUM>. As a result, the imaging device <NUM> can capture a field of view of the microscope <NUM>. The holder <NUM> holds the camera <NUM>, that is, the imaging device <NUM> such that the optical axis of the camera lens 19A and the optical axis of the eyepiece lens 36A are parallel to each other. The term "parallel" as used herein is not limited to perfect parallel, and is allowed to include a predetermined error. The holder <NUM> has a structure capable of moving the held imaging device <NUM> in a horizontal direction (that is, a direction parallel to a display surface of the display unit <NUM>), and can adjust a two-dimensional position of the camera <NUM>.

<FIG> is a block diagram illustrating an example of an electrical configuration of the imaging device <NUM> according to the first embodiment.

As illustrated in <FIG>, the imaging device <NUM> according to the present embodiment includes a central processing unit (CPU) <NUM>, a read only memory (ROM) <NUM>, a random access memory (RAM) <NUM>, an input/output interface (I/O) <NUM>, a storage unit <NUM>, the display unit <NUM>, an operation unit <NUM>, a communication unit <NUM>, and the camera <NUM>.

The CPU <NUM>, the ROM <NUM>, the RAM <NUM>, and the I/O <NUM> constitute a control unit. Each unit of the CPU <NUM>, the ROM <NUM>, the RAM <NUM>, and the I/O <NUM> is connected via a bus.

Each functional unit including the storage unit <NUM>, the display unit <NUM>, the operation unit <NUM>, the communication unit <NUM>, and the camera <NUM> is connected to the I/O <NUM>. These functional units can communicate with the CPU <NUM> via the I/O <NUM>.

The control unit may be configured as a sub-control unit that controls a part of the operation of the imaging device <NUM>, or may be configured as a part of a main control unit that controls the entire operation of the imaging device <NUM>. For some or all of the blocks of the control unit, for example, an integrated circuit such as a large scale integration (LSI) or an integrated circuit (IC) chip set is used. An individual circuit may be used for each of the blocks, or a circuit in which some or all of the blocks are integrated may be used. The blocks may be provided integrally with each other, or some of the blocks may be provided separately. Furthermore, a part of each of the blocks may be provided separately. The integration of the control unit is not limited to LSI, and a dedicated circuit or a general-purpose processor may be used.

As the storage unit <NUM>, for example, a hard disk drive (HDD), a solid state drive (SSD), a flash memory, or the like is used. The storage unit <NUM> stores a guidance program 15A according to the present embodiment. Note that the guidance program 15A may be stored in the ROM <NUM>.

The guidance program 15A is a program for guiding the camera <NUM> held by the holder <NUM> to an appropriate holding position with respect to the optical axis of the eyepiece lens 36A, for example, a holding position (Hereinafter, the position is referred to as an "appropriate position". ) where the optical axis of the eyepiece lens 36A and the optical axis of the camera <NUM> (camera lens 19A) are matched. The guidance program 15A may be installed in advance in the imaging device <NUM>, for example. The guidance program 15A may be realized by being stored in a non-volatile non-transitory storage medium or distributed via a network line and installed or upgraded, as appropriate, in the imaging device <NUM>. Note that, as an example of the non-volatile non-transitory storage medium, a compact disc read only memory (CD-ROM), a magneto-optical disk, an HDD, a digital versatile disc read only memory (DVD-ROM), a flash memory, a memory card, and the like are assumed. Note that "the optical axis of the eyepiece lens 36A and the optical axis of the camera <NUM> are matched" includes not only that the optical axis of the eyepiece lens 36A and the optical axis of the camera <NUM> are perfectly matched, but also that the optical axis of the eyepiece lens 36A and the optical axis of the camera <NUM> are substantially matched so that the camera <NUM> can satisfactorily capture the field of view of the microscope <NUM>.

As the display unit <NUM>, for example, a liquid crystal display (LCD), an organic electro luminescence (EL) display, or the like is used. The display unit <NUM> integrally includes a touch panel. The operation unit <NUM> is provided with, for example, a power button, a volume button, and the like. The display unit <NUM> displays a captured image obtained by capturing by the camera <NUM>.

The communication unit <NUM> is connected to a network line such as the Internet, a local area network (LAN), or a wide area network (WAN), and can communicate with an external device via the network line.

As described above, the camera <NUM> is, for example, a camera using a CCD, and captures a field of view of the microscope <NUM> through the eyepiece lens 36A and the objective lens 36B of the microscope <NUM>.

By the way, as described above, in order to observe the sample well, it is necessary to adjust the optical axis of the eyepiece lens of the microscope and the optical axis of the camera and to match them as accurately as possible. However, fine operation is required for position adjustment with respect to the optical axis of the eyepiece lens provided in the high-magnification microscope. Furthermore, since an image seen from the eyepiece lens is often seen to be reversed vertically and horizontally, the position adjustment becomes more difficult.

On the other hand, in the case of guiding the camera <NUM> held by the holder <NUM> to the appropriate position, the imaging device <NUM> according to the present embodiment acquires the captured image captured by the camera <NUM> held by the holder <NUM>, specifies a visual field area corresponding to the field of view of the microscope <NUM> from the acquired captured image, obtains a center of gravity coordinate of the visual field area, and creates and outputs guidance information to the appropriate position on the basis of a comparison between the center of gravity coordinate and a reference coordinate corresponding to a center of gravity of the visual field area obtained from the captured image captured by the camera <NUM> held at the appropriate position. Note that, as described above, the appropriate position is a holding position when the optical axis of the eyepiece lens 36A and the optical axis of the camera <NUM>, that is, the camera lens 19A are matched.

Specifically, the CPU <NUM> of the imaging device <NUM> according to the present embodiment functions as each unit illustrated in <FIG> by writing and executing the guidance program 15A stored in the storage unit <NUM> or the ROM <NUM> in the RAM <NUM>.

<FIG> is a block diagram illustrating an example of a functional configuration of the imaging device <NUM> according to the first embodiment.

As illustrated in <FIG>, the CPU <NUM> of the imaging device <NUM> according to the present embodiment functions as an image acquisition unit 11A, a specification unit 11B, a center of gravity coordinate acquisition unit 11C, a guidance information creation unit 11D, and an output unit 11E.

The image acquisition unit 11A acquires, from the camera <NUM>, a captured image obtained by capturing a field of view of the microscope <NUM> by the camera <NUM> held by the holder <NUM>. The captured image includes an area (visual field area) in which the field of view of the microscope <NUM> is captured in the captured image. In other words, there is a visual field area corresponding to the field of view of the microscope <NUM> in the captured image.

The specification unit 11B specifies a visual field area corresponding to the field of view from the captured image acquired by the image acquisition unit 11A. That is, an area (visual field area) corresponding to the field of view included in the captured image is extracted. Since the field of view of the microscope <NUM> appears brighter in the captured image than a portion other than the field of view, for example, the captured image is binarized by binarization processing, and a high luminance region of the binarized image is specified as a visual field area. In addition to the binarization processing, a known method such as Snake/Active Contour method, Mean Shift method, Graph Cuts method, Region Growing method, or Otsu's binarization method is used for the area extraction.

The center of gravity coordinate acquisition unit 11C obtains a center of gravity coordinate of the visual field area specified by the specification unit 11B. Specifically, for example, a center of gravity coordinate of the high luminance region specified by the specification unit 11B is calculated as the center of gravity coordinate of the visual field area. For example, it is assumed that a weight of each coordinate (each pixel) of the visual field area is the same, and a simple average of the visual field area is obtained as the center of gravity coordinate. Specifically, X coordinate values of all the coordinates (respective pixels) included in the visual field area are summed, and a value obtained by dividing the obtained total value by the number of coordinates included in the visual field area is set as an X coordinate of the center of gravity coordinate. Similarly, Y-coordinate values of all the coordinates (respective pixels) included in the visual field area are summed, and a value obtained by dividing the obtained total value by the number of coordinates included in the visual field area is set as a Y-coordinate of the center of gravity coordinate. A known method (See, e.g., <NUM>. https://www. higashisalary. com/entry/cv2-calc-moment, <NUM>. https://plantraspberrypi3. hatenablog. com/entry/<NUM>/<NUM>/<NUM>/<NUM>) is used for the center of gravity coordinate calculation processing. Note that the center of gravity coordinate is a coordinate in the coordinate system of the captured image.

The guidance information creation unit 11D creates guidance information to the appropriate position on the basis of a comparison between the center of gravity coordinate obtained by the center of gravity coordinate acquisition unit 11C and a reference coordinate corresponding to a center of gravity of the visual field area obtained from the captured image captured by the camera <NUM> held at the appropriate position.

Here, each of the center of gravity coordinate and the reference coordinate is represented by a first coordinate and a second coordinate representing a two-dimensional position on the captured image. The first coordinate is, for example, an X coordinate, and the second coordinate is, for example, a Y coordinate. The X coordinate in the captured image corresponds to a coordinate in the right-left direction with the current position of the camera <NUM> as an origin, and the Y coordinate in the captured image corresponds to a coordinate in the back-front direction with the current position of the camera <NUM> as an origin. That is, the X and Y coordinates in the captured image correspond to a position of the camera <NUM>. Therefore, the position in the vertical direction and the position in the horizontal direction of the camera <NUM> can be adjusted based on a comparison between the X coordinate and the Y coordinate of the center of gravity coordinate and the X coordinate and the Y coordinate of the reference coordinate.

The output unit 11E outputs the guidance information created by the guidance information creation unit 11D. An output destination of the guidance information is, for example, the display unit <NUM>.

Next, a method of obtaining the reference coordinate corresponding to the center of gravity of the visual field area obtained from the image captured by the camera <NUM> held at the appropriate position will be specifically described with reference to <FIG> and <FIG>.

<FIG> is a plan view and a side view schematically illustrating an example of a positional relationship between the imaging device <NUM> held at the appropriate position and the microscope <NUM> according to the present embodiment. Note that, in order to simplify the description, the holder <NUM> is not illustrated, and also, the microscope <NUM> is not illustrated in the plan view. Furthermore, <FIG> is a diagram illustrating an example of a captured image captured at the appropriate position, a binarized image, and a reference coordinate.

As illustrated in <FIG>, a display surface of the display unit <NUM> of the imaging device <NUM> is disposed at the appropriate position on the holder <NUM> along the horizontal direction. The holding position of the imaging device <NUM> in the horizontal direction when the optical axis of the eyepiece lens 36A and the optical axis of the camera lens 19A are matched is the appropriate position of the imaging device <NUM>. The captured image obtained by capturing the field of view of the microscope with the imaging device <NUM> at the appropriate position includes a circular visual field area. Therefore, a position of the imaging device <NUM> is adjusted while viewing the captured image displayed on the display unit <NUM>, and the position of the imaging device <NUM> at which the captured image including the circular visual field area is obtained is set as the appropriate position. In the present embodiment, when a user views the display unit <NUM> from above, a longitudinal direction of the imaging device <NUM> is defined as the back-front direction as viewed from the user, and a lateral direction is defined as the right-left direction as viewed from the user. Here, a direction from the display unit <NUM> toward the camera lens 19A is defined as a backward direction. Note that the method of disposing the imaging device <NUM> at the appropriate position is not limited thereto, and for example, a position of the eyepiece lens 36A in the microscope <NUM> can be measured, and the appropriate position can be determined on the basis of the measurement result.

A captured image illustrated in <FIG> is the captured image captured at the appropriate position illustrated in <FIG>. The visual field area is an area corresponding to the field of view of the microscope <NUM>, and is specified from the captured image captured by the camera <NUM> and displayed on the display unit <NUM>. In the example of <FIG>, a circular portion at a center of the image represents the field of view (visual field area). A binarized image is an image obtained by binarizing the captured image, and a high luminance region that is a circular white portion at the center of the image is specified as the visual field area. A reference coordinate P1(Xs, Ys) can be obtained by calculating a center of gravity coordinate of the high luminance region specified from the binarized image and using the obtained center of gravity coordinate as a center of gravity coordinate of the visual field area.

Next, a method of obtaining a direction in which the camera <NUM> is moved on the basis of the center of gravity coordinate corresponding to the center of gravity of the visual field area included in the captured image captured by the camera <NUM> held at a position shifted from the appropriate position and the reference coordinate will be specifically described with reference to <FIG>.

<FIG> is a plan view and a side view schematically illustrating an example of a positional relationship between the imaging device <NUM> held at a position shifted from the appropriate position and the microscope <NUM> according to the present embodiment. Note that, in order to simplify the description, the holder <NUM> is not illustrated, and also, the microscope <NUM> is not illustrated in the plan view. Furthermore, <FIG> and <FIG> are diagrams illustrating examples of a captured image captured at a position shifted from the appropriate position, a binarized image, and a center of gravity coordinate.

As illustrated in <FIG>, the display surface of the display unit <NUM> of the imaging device <NUM> is disposed along the horizontal direction. When the user looks at the display unit <NUM>, the longitudinal direction of the imaging device <NUM> is defined as the back-front direction as viewed from the user, and the lateral direction is defined as the right-left direction as viewed from the user. Here, a direction from the display unit <NUM> toward the camera lens 19A is defined as a backward direction. A "rightward shift position" is a position where the holding position of the imaging device <NUM> in the horizontal direction is shifted rightward from the appropriate position. A "leftward shift position" is a position where the holding position of the imaging device <NUM> in the horizontal direction is shifted leftward from the appropriate position. A "backward shift position" is a position where the holding position of the imaging device <NUM> in the horizontal direction is shifted backward from the appropriate position. A "frontward shift position" is a position where the holding position of the imaging device <NUM> in the horizontal direction is shifted frontward from the appropriate position. The imaging device <NUM> is disposed at the "rightward shift position", the "leftward shift position", the "backward shift position", and the "frontward shift position".

The captured image illustrated in <FIG> is a captured image captured at the rightward shift position and the leftward shift position illustrated in <FIG>. In the captured image at the rightward shift position, a dark portion (vignetting) is seen on the left side of the field of view. In other words, the captured image captured by the imaging device <NUM> at the rightward shift position includes a visual field area having a shape in which the left side of a circle is missing. On the other hand, in the captured image at the leftward shift position, a dark portion (vignetting) is seen on the right side of the field of view. In other words, the captured image captured by the imaging device <NUM> at the leftward shift position includes a visual field area having a shape in which the right side of a circle is missing. Each center of gravity coordinate P2(Xi, Yi) is calculated as a center of gravity coordinate of the high luminance region specified from a binarized image. Note that, in a relationship between the center of gravity coordinate P2 of the visual field area at the rightward shift position and the center of gravity coordinate P1 (reference coordinate P1) of the visual field area at the appropriate position, Yi = Ys and Xi > Xs, and in a relationship between the center of gravity coordinate P2 of the visual field area at the leftward shift position and the center of gravity coordinate P1 (reference coordinate P1) of the visual field area at the appropriate position, Yi = Ys and Xi < Xs. In other words, in the case of Xi > Xs, since the imaging device <NUM> is shifted rightward from the appropriate position, the imaging device <NUM> is moved leftward. In the case of Xi < Xs, since the imaging device <NUM> is shifted leftward from the appropriate position, the imaging device <NUM> is moved rightward.

A captured image illustrated in <FIG> is a captured image captured at the backward shift position and the frontward shift position illustrated in <FIG>. In the captured image at the backward shift position, a dark portion (vignetting) is seen on the lower side of the field of view. In other words, the captured image captured by the imaging device <NUM> at the backward shift position includes a visual field area having a shape in which the lower side of a circle is missing. On the other hand, in the captured image at the frontward shift position, a dark portion (vignetting) is seen on the upper side of the field of view. In other words, the captured image captured by the imaging device <NUM> at the frontward shift position includes a visual field area having a shape in which the upper side of a circle is missing. Each center of gravity coordinate P2(Xi, Yi) is calculated as a center of gravity coordinate of the high luminance region specified from a binarized image. Note that, in a relationship between the center of gravity coordinate P2 of the visual field area of the backward shift position and the center of gravity coordinate P1 (reference coordinate P1) of the visual field area of the appropriate position, Yi > Ys and Xi = Xs, and in a relationship between the center of gravity coordinate P2 of the visual field area of the frontward shift position and the center of gravity coordinate P1 (reference coordinate P1) of the visual field area of the appropriate position, Yi < Ys and Xi = Xs. In other words, in the case of Yi > Ys, since the imaging device <NUM> is shifted backward from the appropriate position, the imaging device <NUM> is moved frontward. In a case of Yi < Ys, since the imaging device <NUM> is shifted frontward from the appropriate position, the imaging device <NUM> is moved backward.

<FIG> is a diagram illustrating an example of a relationship between the reference coordinate P1 and the center of gravity coordinate P2 according to the present embodiment. Note that the X and Y coordinate systems illustrated in <FIG> indicate a two-dimensional coordinate system set in the image displayed on the display unit <NUM>.

As illustrated in <FIG>, the guidance information creation unit 11D creates guidance information in a case where a distance L1 between the center of gravity coordinate P2(Xi, Yi) and the reference coordinate P1(Xs, Ys) is outside a predetermined range, and does not create guidance information in a case where the distance between the center of gravity coordinate P2(Xi, Yi) and the reference coordinate P1(Xs, Ys) is within the predetermined range. The distance L1 is calculated using the following Equation (<NUM>). Note that the predetermined range is a range in which an appropriate captured image can be obtained, and an appropriate value is determined according to the specifications of the imaging device <NUM> and the microscope <NUM>, and the like. [Equation <NUM>] <MAT>.

Furthermore, the guidance information creation unit 11D may create the guidance information in a case where at least one of a first value representing an absolute value of a difference between the X coordinate Xi of the center of gravity coordinate P2(Xi, Yi) and the X coordinate Xs of the reference coordinate P1(Xs, Ys) and a second value representing an absolute value of a difference between the Y coordinate Yi of the center of gravity coordinate P2(Xi, Yi) and the Y coordinate Ys of the reference coordinate P1(Xs, Ys) is outside the predetermined range, and may not create the guidance information in a case where both the first value and the second value are within the predetermined range.

Here, in a case where both of a first value representing an absolute value of a difference between the X coordinate Xi of the center of gravity coordinate P2(Xi, Yi) and the X coordinate Xs of the reference coordinate P1(Xs, Ys) and a second value representing an absolute value of a difference between the Y coordinate Yi of the center of gravity coordinate P2(Xi, Yi) and the Y coordinate Ys of the reference coordinate P1(Xs, Ys) are outside the predetermined range, for example, as illustrated in <FIG>, the guidance information creation unit 11D creates the guidance information for separately adjusting the position of the camera <NUM> corresponding to the first value and the position of the camera <NUM> corresponding to the second value.

<FIG> is a diagram illustrating an example of guidance information displayed together with the captured image according to the present embodiment.

As illustrated in <FIG>, the output unit 11E outputs a direction in which the camera <NUM> is guided to the display unit <NUM> as guidance information together with the captured image acquired by the image acquisition unit 11A. The direction to guide is represented by, for example, at least one of a character and a figure. In the example of <FIG>, in the case of the "appropriate position", characters "position OK" are displayed. In the case of the "rightward shift position", characters "Please move the camera to the left" and a leftward arrow are displayed as the guidance information. The leftward arrow is an example of a figure. The figure referred to herein is not limited to an arrow, and may be a figure indicating a direction. Similarly, in the case of the "leftward shift position", characters "Please move the camera to the right" and a rightward arrow are displayed as the guidance information. In the case of the "backward shift position", characters "Please move the camera frontward" and a frontward arrow are displayed as the guidance information. In the case of the "frontward shift position", characters "Please move the camera backward" and a backward arrow are displayed as the guidance information. However, the back, front, left, and right directions here correspond to the back, front, left, and right directions illustrated in <FIG> and <FIG>.

Furthermore, in a case where both of a first value representing an absolute value of a difference between the X coordinate Xi of the center of gravity coordinate P2(Xi, Yi) and the X coordinate Xs of the reference coordinate P1(Xs, Ys) and a second value representing an absolute value of a difference between the Y coordinate Yi of the center of gravity coordinate P2(Xi, Yi) and the Y coordinate Ys of the reference coordinate P1(Xs, Ys) are outside the predetermined range, the guidance information creation unit 11D may create guidance information for simultaneously adjusting the position of the camera <NUM> corresponding to the first value and the position of the camera <NUM> corresponding to the second value. For example, in the case of a position shifted diagonally backward to the right, characters such as "Please move the camera diagonally frontward to the left" and a left diagonal frontward arrow are displayed as the guidance information. The guidance information is displayed in a similar relationship for other diagonal directions. Note that, in this case, the holder <NUM> has a structure in which the imaging device <NUM> is movable in a diagonal direction in addition to back, front, left, and right with respect to the microscope <NUM>.

Furthermore, in a case where at least one of a first value representing an absolute value of a difference between the X coordinate Xi of the center of gravity coordinate P2(Xi, Yi) and the X coordinate Xs of the reference coordinate P1(Xs, Ys) and a second value representing an absolute value of a difference between the Y coordinate Yi of the center of gravity coordinate P2(Xi, Yi) and the Y coordinate Ys of the reference coordinate P1(Xs, Ys) is outside the predetermined range, for example, as illustrated in <FIG>, the guidance information creation unit 11D may change a length of the figure according to the first value or the second value.

<FIG> is a diagram illustrating another example of the guidance information displayed together with the captured image according to the present embodiment.

For example, in the case of the "rightward shift position", the captured image and the guidance information illustrated in <FIG> are displayed. The guidance information is indicated by characters "Please move the camera to the left" and a leftward arrow. In the example of <FIG>, a length of the arrow in a case where the first value is large is longer than a length of the arrow in a case where the first value is small. By changing the length of the arrow, it is possible to visually grasp a degree of an adjustment amount of the camera position.

Next, actions of the imaging device <NUM> according to the first embodiment will be described with reference to <FIG>.

<FIG> is a flowchart illustrating an example of a flow of reference coordinate deriving processing by the guidance program 15A according to the first embodiment.

When execution of the reference coordinate deriving processing by the guidance program 15A is instructed, the CPU <NUM> of the imaging device <NUM> writes the guidance program 15A stored in the ROM <NUM> or the storage unit <NUM> into the RAM <NUM> to execute the processing.

In step S101 of <FIG>, the CPU <NUM> captures the field of view of the microscope <NUM> as illustrated in <FIG> described above as an example using the camera <NUM> of the imaging device <NUM> held at the appropriate position by the holder <NUM>.

In step S102, as an example, as illustrated in <FIG> described above, the CPU <NUM> binarizes the captured image obtained by capturing in step S101 to generate a binarized image.

In step S103, as an example, as illustrated in <FIG> described above, the CPU <NUM> specifies a high luminance region from the binarized image generated in step S102 and specifies a visual field area.

In step S104, as an example, as illustrated in <FIG> described above, the CPU <NUM> derives a reference coordinate P1(Xs, Ys) corresponding to a center of gravity of the visual field area specified in step S103, and ends the reference coordinate deriving processing by the guidance program 15A.

<FIG> is a flowchart illustrating an example of a flow of guidance information display processing by the guidance program 15A according to the first embodiment.

When execution of the guidance information display processing by the guidance program 15A is instructed, the CPU <NUM> of the imaging device <NUM> writes the guidance program 15A stored in the ROM <NUM> or the storage unit <NUM> into the RAM <NUM> to execute the processing.

In step S111 of <FIG>, the CPU <NUM> captures the field of view of the microscope <NUM> as illustrated in <FIG> and <FIG> described above as an example using the camera <NUM> of the imaging device <NUM> held by the holder <NUM>.

In step S112, as an example, as illustrated in <FIG> and <FIG> described above, the CPU <NUM> binarizes the captured image obtained by capturing in step S111 to generate a binarized image.

In step S113, as an example, as illustrated in <FIG> and <FIG> described above, the CPU <NUM> specifies a high luminance region from the binarized image generated in step S112 and specifies a visual field area.

In step S114, as an example, as illustrated in <FIG> and <FIG> described above, the CPU <NUM> derives a center of gravity coordinate P2(Xi, Yi) corresponding to a center of gravity of the visual field area specified in step S113.

In step S115, as an example, as illustrated in <FIG> described above, the CPU <NUM> determines whether or not |Xi-Xs|, which is a first value representing an absolute value of a difference between the X coordinate Xi of the center of gravity coordinate P2(Xi, Yi) and the X coordinate Xs of the reference coordinate P1(Xs, Ys), is within a predetermined range. In a case where it is determined that |Xi-Xs| is within the predetermined range (in the case of positive determination), the processing proceeds to step S116, and in a case where it is determined that |Xi-Xs| is outside the predetermined range (in the case of negative determination), the processing proceeds to step S119.

In step S116, as an example, as illustrated in <FIG> described above, the CPU <NUM> determines whether or not |Yi - Ys|, which is a second value representing an absolute value of a difference between the Y coordinate Yi of the center of gravity coordinate P2(Xi, Yi) and the Y coordinate Ys of the reference coordinate P1(Xs, Ys), is within a predetermined range. In a case where it is determined that |Yi - Ys| is within the predetermined range (in the case of positive determination), the processing proceeds to step S117, and in a case where it is determined that |Yi - Ys| is outside the predetermined range (in the case of negative determination), the processing proceeds to step S118.

In step S117, as an example, as illustrated in <FIG> described above, the CPU <NUM> displays information (for example, "position OK") indicating that the imaging device <NUM> is at the appropriate position on the display unit <NUM>, and ends the guidance information display processing by the guidance program 15A.

On the other hand, in step S118, the CPU <NUM> creates the Y-direction guidance information. A Y-direction guidance information creation subroutine will be described with reference to <FIG>.

<FIG> is a flowchart illustrating an example of a flow of processing by a Y-direction guidance information creation subroutine executed in step S118 of <FIG>.

In step S131 of <FIG>, the CPU <NUM> determines whether or not Yi - Ys > <NUM>. In a case where it is determined that Yi - Ys > <NUM> (in the case of positive determination), the processing proceeds to step S132, and in a case where it is determined that Yi - Ys > <NUM> is not (in the case of negative determination), the processing proceeds to step S133.

In step S132, the CPU <NUM> sets the center of gravity coordinates P2(Xi, Yi) as the backward shift position, creates guidance information for moving the camera <NUM> frontward as illustrated in <FIG> described above as an example, and returns to step S118 in <FIG>.

On the other hand, in step S133, the CPU <NUM> determines whether or not Yi - Ys < <NUM>. In a case where it is determined that Yi - Ys < <NUM> (in the case of positive determination), the processing proceeds to step S134, and in a case where it is determined that Yi - Ys < <NUM> is not (in the case of negative determination), the processing proceeds to the return.

In step S134, the CPU <NUM> sets the center of gravity coordinates P2(Xi, Yi) as the frontward shift position, creates guidance information for moving the camera <NUM> backward as illustrated in <FIG> described above as an example, and returns to step S118 in <FIG>.

Returning to <FIG>, in step S119, as an example, the CPU <NUM> determines whether or not |Yi - Ys|, which is the second value, is within a predetermined range as illustrated in <FIG> described above. In a case where it is determined that |Yi - Ys| is within the predetermined range (in the case of positive determination), the processing proceeds to step S120, and in a case where it is determined that |Yi - Ys| is outside the predetermined range (in the case of negative determination), the processing proceeds to step S121.

In step S120, the CPU <NUM> creates X-direction guidance information. An X-direction guidance information creation subroutine will be described with reference to <FIG>.

<FIG> is a flowchart illustrating an example of a flow of processing by an X-direction guidance information creation subroutine executed in step S120 of <FIG>.

In step S141 of <FIG>, the CPU <NUM> determines whether or not Xi - Xs > <NUM>. In a case where it is determined that Xi - Xs > <NUM> (in the case of positive determination), the processing proceeds to step S142, and in a case where it is determined that Xi - Xs > <NUM> is not (in the case of negative determination), the processing proceeds to step S143.

In step S142, the CPU <NUM> sets the center of gravity coordinates P2(Xi, Yi) to the rightward shift position, creates guidance information for moving the camera <NUM> leftward as illustrated in <FIG> described above as an example, and returns to step S120 in <FIG>.

On the other hand, in step S143, the CPU <NUM> determines whether or not Xi - Xs < <NUM>. In a case where it is determined that Xi - Xs < <NUM> (in the case of positive determination), the processing proceeds to step S144, and in a case where it is determined that Xi - Xs < <NUM> is not (in the case of negative determination), the processing proceeds to the return.

In step S144, the CPU <NUM> sets the center of gravity coordinates P2(Xi, Yi) to the leftward shift position, creates guidance information for moving the camera <NUM> rightward as illustrated in <FIG> described above as an example, and returns to step S120 in <FIG>.

Returning to <FIG>, in step S121, the CPU <NUM> creates the X-direction guidance information and the Y-direction guidance information. An X-direction guidance information and Y-direction guidance information creation subroutine will be described with reference to <FIG>.

<FIG> is a flowchart illustrating an example of a flow of processing by an X-direction guidance information and Y-direction guidance information creation subroutine executed in step S121 of <FIG>.

In step S151 of <FIG>, the CPU <NUM> determines whether or not Yi - Ys > <NUM>. In a case where it is determined that Yi - Ys > <NUM> (in the case of positive determination), the processing proceeds to step S152, and in a case where it is determined that Yi - Ys > <NUM> (in the case of negative determination), the processing proceeds to step S153.

In step S152, the CPU <NUM> creates guidance information for moving the camera <NUM> frontward as illustrated in <FIG> described above as an example with the center of gravity coordinate P2(Xi, Yi) as the backward shift position.

On the other hand, in step S153, the CPU <NUM> determines whether or not Yi - Ys < <NUM>. In a case where it is determined that Yi - Ys < <NUM> (in the case of positive determination), the processing proceeds to step S154, and in a case where it is determined that Yi - Ys < <NUM> is not (in the case of negative determination), the processing proceeds to the return.

In step S154, the CPU <NUM> creates guidance information for moving the camera <NUM> backward as illustrated in <FIG> described above as an example with the center of gravity coordinate P2(Xi, Yi) as the frontward shift position.

Next, in step S155, the CPU <NUM> determines whether or not Xi - Xs > <NUM>. In a case where it is determined that Xi - Xs > <NUM> (in the case of positive determination), the processing proceeds to step S156, and in a case where it is determined that Xi - Xs > <NUM> is not (in the case of negative determination), the processing proceeds to step S157.

In step S156, the CPU <NUM> sets the center of gravity coordinate P2(Xi, Yi) to the rightward shift position, creates guidance information for moving the camera <NUM> leftward as illustrated in <FIG> described above as an example, and returns to step S121 in <FIG>.

On the other hand, in step S157, the CPU <NUM> determines whether or not Xi - Xs < <NUM>. In a case where it is determined that Xi - Xs < <NUM> (in the case of positive determination), the processing proceeds to step S158, and in a case where it is determined that Xi - Xs < <NUM> is not (in the case of negative determination), the processing proceeds to the return.

In step S158, the CPU <NUM> sets the center of gravity coordinate P2(Xi, Yi) to the leftward shift position, creates guidance information for moving the camera <NUM> rightward as illustrated in <FIG> described above as an example, and returns to step S121 in <FIG>.

Returning to <FIG>, in step S122, the CPU <NUM> displays the guidance information created in step S118, step S120, or step S121 on the display unit <NUM>, and ends the guidance information display processing by the guidance program 15A.

As described above, according to the present embodiment, in a case where the optical axis of the eyepiece lens of the microscope and the optical axis of the camera lens are mismatched, the direction in which the camera is moved to the appropriate position is displayed as the guidance information. The user only needs to move the camera according to the guidance information, and can easily adjust the position of the camera with respect to the eyepiece lens of the microscope.

In the first embodiment, a mode in which the user moves the camera to the appropriate position according to the guidance information has been described. In the second embodiment, a mode in which the camera is automatically moved to the appropriate position by controlling a motor by the guidance information will be described.

<FIG> is a diagram schematically illustrating configurations of an imaging device <NUM> and motors 40A and 40B according to the second embodiment.

As illustrated in <FIG>, the output unit 11E according to the present embodiment outputs the guidance information to the motors 40A and 40B that adjust the position of the camera <NUM>. The motor 40A is a motor that moves the imaging device <NUM> in the back-front direction, and the motor 40B is a motor that moves the imaging device <NUM> in the right-left direction. The guidance information output to the motors 40A and 40B is information including a moving direction and a moving amount based on a shift amount between the reference coordinate P1(Xs, Ys) and the center of gravity coordinate P2(Xi, Yi).

Next, the action of the imaging device <NUM> according to the second embodiment will be described with reference to <FIG>.

<FIG> is a flowchart illustrating an example of a flow of guidance information output processing by the guidance program 15A according to the second embodiment.

When execution of the guidance information output processing by the guidance program 15A is instructed, the CPU <NUM> of the imaging device <NUM> writes the guidance program 15A stored in the ROM <NUM> or the storage unit <NUM> into the RAM <NUM> to execute the processing.

Since the processing of steps S161 to S171 of <FIG> is similar to the processing of steps S111 to S121 illustrated in <FIG> described above, the repeated description thereof will be omitted. In the flow of <FIG>, the guidance information is displayed and output on the display unit <NUM>, whereas in the flow of <FIG>, the guidance information is output to the motors 40A and 40B.

That is, in step S172 of <FIG>, the CPU <NUM> outputs the guidance information created in step S168, step S170, or step S171 to the motors 40A and 40B as illustrated in <FIG> described above as an example, and ends the guidance information output processing by the guidance program 15A.

As described above, according to the present embodiment, in a case where the optical axis of the eyepiece lens of the microscope and the optical axis of the camera lens are mismatched, the camera can be automatically moved to the appropriate position by controlling the motors by the guidance information. Therefore, it is possible to easily adjust the position of the camera with respect to the eyepiece lens of the microscope.

In a third embodiment, a form in which a guidance program is provided not in an imaging device but in an information processing device connected to the imaging device via a network will be described.

<FIG> is a diagram schematically illustrating an example of a microscope device 100A and an information processing device <NUM> according to the third embodiment.

As illustrated in <FIG>, the microscope device 100A includes an imaging device 10A. The imaging device 10A and the information processing device <NUM> are connected via a network N, and the information processing device <NUM> can be accessed by the imaging device <NUM> via the network N. As the information processing device <NUM>, for example, a general-purpose computer device such as a server computer or a personal computer is applied.

<FIG> is a block diagram illustrating an example of an electrical configuration of an information processing device <NUM> according to the third embodiment.

As illustrated in <FIG>, the information processing device <NUM> according to the present embodiment includes a CPU <NUM>, a ROM <NUM>, a RAM <NUM>, an I/O <NUM>, a storage unit <NUM>, a display unit <NUM>, an operation unit <NUM>, and a communication unit <NUM>.

Each functional unit including the storage unit <NUM>, the display unit <NUM>, the operation unit <NUM>, and the communication unit <NUM> is connected to the I/O <NUM>. These functional units can communicate with the CPU <NUM> via the I/O <NUM>.

The control unit may be configured as a sub-control unit that controls a part of the operation of the information processing device <NUM>, or may be configured as a part of a main control unit that controls the entire operation of the information processing device <NUM>.

As the storage unit <NUM>, for example, an HDD, an SSD, a flash memory, or the like is used. The storage unit <NUM> stores the guidance program 15A according to the present embodiment. Note that the guidance program 15A may be stored in the ROM <NUM>.

As the display unit <NUM>, for example, a liquid crystal display (LCD), an organic EL display, or the like is used. The display unit <NUM> may integrally include a touch panel. The operation unit <NUM> is provided with an operation input device such as a keyboard and a mouse. The display unit <NUM> and the operation unit <NUM> receive various instructions from a user of the information processing device <NUM>. The display unit <NUM> displays various types of information such as a result of processing executed in response to an instruction received from the user and a notification for the processing.

The communication unit <NUM> is connected to a network N such as the Internet, a LAN, or a WAN, and can communicate with the imaging device 10A via the network N.

The guidance program 15A according to the present embodiment is stored not in the imaging device 10A but in the information processing device <NUM>. In this case, a captured image obtained by capturing the field of view of the microscope <NUM> is transmitted from the imaging device 10A to the information processing device <NUM>.

The CPU <NUM> of the information processing device <NUM> according to the present embodiment functions as each unit illustrated in <FIG> described above by writing and executing the guidance program 15A stored in the storage unit <NUM> or the ROM <NUM> into the RAM <NUM>. That is, the CPU <NUM> of the information processing device <NUM> functions as the image acquisition unit 11A, the specification unit 11B, the center of gravity coordinate acquisition unit 11C, the guidance information creation unit 11D, and the output unit 11E. The image acquisition unit 11A according to the present embodiment acquires a captured image obtained by capturing the field of view of the microscope <NUM>, which is transmitted from the imaging device 10A, and the output unit 11E outputs guidance information to the imaging device 10A. Since the specification unit 11B, the center of gravity coordinate acquisition unit 11C, and the guidance information creation unit 11D are as described above with reference to <FIG>, repeated description is omitted. However, the reference coordinate P1(Xs, Ys) corresponding to the appropriate position is transmitted from the imaging device 10A to the information processing device <NUM> in advance and held in the information processing device <NUM>.

As described above, according to the present embodiment, the guidance program may not be provided in each imaging device, and the guidance information can be acquired from the information processing device via the network.

Note that, in each of the above embodiments, a processor refers to a processor in a broad sense, and includes a general-purpose processor (for example, a central processing unit (CPU), and the like) or a dedicated processor (for example, a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable logic device, and the like).

Furthermore, the operation of the processor in each of the above embodiments may be performed not only by one processor but also by cooperation of a plurality of processors existing at physically separated positions. Furthermore, the order of each operation of the processor is not limited to the order described in each of the embodiments, and may be changed if appropriate.

The imaging device, the microscope device, and the information processing device according to the embodiments have been described above by way of example. The embodiments may be in the form of a non-transitory computer-readable storage medium storing the guidance program for causing a computer to execute a function of each unit included in the imaging device or the information processing device.

In addition, the configurations of the imaging device, the microscope device, and the information processing device described in the embodiments are merely examples, and may be changed according to the situation without departing from the scope of the invention as defined in the appended claims.

Furthermore, the flow of processing of the program described in the embodiments is also an example, and unnecessary steps may be deleted, new steps may be added, or the processing order may be changed within the scope of the invention as defined in the appended claims.

Claim 1:
A guidance program (15A) for guiding a camera (<NUM>), which is held by a holder (<NUM>) and which captures a field of view of a microscope (<NUM>) through an eyepiece lens (36A) of the microscope, to an appropriate holding position with respect to an optical axis of the eyepiece lens, the guidance program (15A) causing a computer to function as:
an image acquisition unit (11A) that acquires, from the camera, an image of the field of view captured by the camera held by the holder;
a specification unit (11B) that specifies a visual field area in the image, the visual field area corresponding to the field of view;
a center of gravity coordinate acquisition unit (11C) that obtains a luminance center of gravity coordinate of the visual field area;
a guidance information creation unit (11D) that creates guidance information to guide the camera to the appropriate holding position based on a distance between the luminance center of gravity coordinate and a reference coordinate corresponding to a center of gravity of the visual field area obtained from an image captured by the camera held at the appropriate holding position; and
an output unit (11E) that outputs the guidance information,
wherein:
the appropriate holding position is a holding position of the camera (<NUM>) at which an optical axis of the eyepiece lens and an optical axis of the camera (<NUM>) are matched,
each of the luminance center of gravity coordinate and the reference coordinate is represented by a first coordinate and a second coordinate representing a two-dimensional position on an image, and
the guidance information creation unit (11D) creates the guidance information in a case in which at least one of a first value representing an absolute value of a difference between the first coordinate of the luminance center of gravity coordinate and the first coordinate of the reference coordinate, or a second value representing an absolute value of a difference between the second coordinate of the luminance center of gravity coordinate and the second coordinate of the reference coordinate, is outside a predetermined range, and
the guidance information creation unit (11D) does not create the guidance information in a case in which both of the first value and the second value are within the predetermined range.