Patent Description:
The present disclosure relates to an image processing apparatus, an image processing program, and an image processing method that performs an image analysis on a moving image obtained by imaging an analysis target with time.

In preparation for drug discovery and regenerative medicine, there have been problems of functional evaluation of cells, and various analytical techniques have been studied therefor. One of the analytical techniques is an image analysis to acquire information of a cell by analyzing a moving image, which is obtained by imaging a cell to be analyzed with time.

For example, Patent Literature <NUM> discloses a method of analyzing neurite outgrowth by fluorescent dyeing and image processing. Further, Patent Literature <NUM> discloses a method of evaluating a neural network of neuron by an image analysis.

<CIT> describes a method for analyzing the beating of a derived human cardiomyocyte (CM). Said method comprises obtaining the derived human CM, capturing a sequence of images depicting the derived human CM over an analysis period, and determining at least one signal descriptive of the beating of the derived human CM during the analysis period on basis of the sequence of captured images. Furthermore, apparatuses, a computer program for analyzing the beating of a derived human CM and use thereof are provided, the apparatuses, the computer program and the use configured to provide for obtaining the sequence of images depicting the derived human CM over an analysis period, obtaining information indicative of the region of the images depicting the derived human CM, obtaining information indicative of two or more sub-regions within said region, and determining two or more signals characterizing the displacement within said region of the images depicting the derived human cardiomyocyte, each signal characterizing the extent of displacement within a respective sub-region of said region as a function of time.

There are some cells having motions that are detectable in an image. For example, cardiac myocytes contract and relax, and neurons perform axonal transport, outgrowth, and the like. If such motions can be detected by an image analysis, it is effective for functional evaluation of cells.

However, for the functional evaluation of cells, a mere extraction of a motion is not sufficient. This is because a motion of a cell includes a motion of a cell itself (expansion and contraction etc.), a motion due to a surrounding fluid flow, and the like. Additionally, in the motion of a cell, its direction is important in some cases. For example, in the case of cardiac myocytes, a motion along an expansion and contraction direction of cardiac muscle is important, and for the functional evaluation, it is necessary to extract a motion along that direction. Further, in the case of a neuron, there are various motions including an oscillation of a cell body, axonal transport, outgrowth, and the like. In such motions, directions and properties of the respective motions are different from one another.

In view of the circumstances as described above, it is desirable to provide an image processing apparatus, an image processing program, and an image processing method that are capable of evaluating a motion of an analysis target.

Particular and preferred aspects of the present invention are described in the appended claims.

As described above, according to the present disclosure, it is possible to provide an image processing apparatus, an image processing program, and an image processing method that are capable of evaluating a motion of an analysis target.

Description will be given on an image processing apparatus according to an embodiment of the present disclosure.

<FIG> is a schematic diagram showing a functional configuration of an image processing apparatus <NUM> according to this embodiment. As shown in <FIG>, the image processing apparatus <NUM> includes a moving image acquiring unit <NUM>, a range specifying unit <NUM>, a motion analyzing unit <NUM>, an axial-direction setting unit <NUM>, an information extracting unit <NUM>, and a result outputting unit <NUM>.

The moving image acquiring unit <NUM> acquires an "analysis-target moving image". The analysis-target moving image is a moving image obtained by imaging an analysis target with time. The analysis-target moving image includes moving images formed of a plurality of successively imaged frames, and still images obtained by time-lapse imaging. The analysis target is a cell, a cell group, a biological tissue, or the like. If the analysis target can move, this embodiment can be applied to such an analysis target. An imaging speed of the analysis-target moving image can be appropriately set in accordance with the analysis target.

The analysis-target moving image is assumed to be imaged by various types of optical imaging methods including bright-field imaging, dark-field imaging, phase-difference imaging, fluorescence imaging, confocal imaging, multiphoton-excited fluorescence imaging, absorption spectrum imaging, and scattered-light imaging.

<FIG> is an exemplary analysis-target moving image and is a moving image including a neuron. <FIG> is a schematic diagram showing an example of a motion of a neuron. The neuron has various motions including oscillations of a cell body as indicated by A of <FIG>, axonal transport (transport for intracellular minute organ in axon) as indicated by B, axonal outgrowth indicated by C, and the like. In addition, similar to the neuron, there are cells having various motions. The cells are moving by themselves or moving by extracellular environments (surrounding water flow and the like) in some cases. In this embodiment, those motions can be discriminated.

The moving image acquiring unit <NUM> may acquire an analysis-target moving image from an imaging apparatus (microscope imaging apparatus) (not shown). Alternatively, the moving image acquiring unit <NUM> may also acquire, as an analysis-target moving image, a moving image stored in storage or a moving image supplied from a network. At that time, the moving image acquiring unit <NUM> may perform sampling on previously-captured moving images at predetermined cycles in accordance with the type of the analysis target, to acquire an analysis-target moving image.

The range specifying unit <NUM> specifies an analytical range in the analysis-target moving image. <FIG> is a schematic diagram showing an example of the analytical range. As shown in <FIG>, the range specifying unit <NUM> specifies an analytical range R in the analysis-target moving image. The analytical range R may be a range including, for example, one cell or a cell group. Further, the analytical range R may be the entire range of the analysis-target moving image.

The range specifying unit <NUM> may specify a range, which is instructed by an operation input by a user, as the analytical range R. Alternatively, the range specifying unit <NUM> may detect the analysis target by image processing performed on the analysis-target moving image and specify a range of the analysis target as the analytical range R.

The motion analyzing unit <NUM> calculates a motion vector from the analytical range R. The motion analyzing unit <NUM> sets a plurality of calculation sections within the analytical range R. <FIG> is a schematic diagram showing an example of the calculation sections. As shown in <FIG>, the motion analyzing unit <NUM> sections the analytical range R into a plurality of calculation sections D.

The number of calculation sections D and the sizes thereof may be arbitrarily set. The motion analyzing unit <NUM> can set the calculation sections D according to a specification by the user or the size of the analytical range R. It should be noted that the motion analyzing unit <NUM> may set the entire analytical range R as one calculation section D.

Subsequently, the motion analyzing unit <NUM> calculates a motion vector of each of the calculation sections D. The motion analyzing unit <NUM> can calculate a motion vector of each of the calculation sections D by block matching. <FIG> is a schematic diagram showing a state of calculation of the motion vector.

As shown in <FIG>, the motion analyzing unit <NUM> compares a pixel group, which is included in a specific calculation section D (in <FIG>, calculation section D1) in one frame that forms the analysis-target moving image, with a pixel group of a range in a previous frame, to specify the best matched range of the pixel groups (in <FIG>, pixel range D2). It should be noted that the previous frame may be located one frame before the current frame or some frames before the current frame. Pixel groups between different image frames may be identified as best matched when there is a large degree of matching and/or similarity between the pixel groups. For example, a comparison between a pixel group in a first image to pixel groups in a second image, where the second image is acquired prior to the first image, may allow for determining a pixel group in the second image having a largest degree of matching with the first pixel group.

The motion analyzing unit <NUM> can calculate a vector that extends from the pixel range D2 specified in the previous frame to the calculation section D1 of the current frame, as a motion vector of the calculation section D1 (hereinafter, referred to as calculation section vector B). Similarly, the motion analyzing unit <NUM> calculates a calculation section vector for each of the calculation sections D included in the analytical range R.

<FIG> is a schematic diagram showing an example of the calculation section vectors B calculated in the respective calculation sections D. In <FIG>, the direction of an arrow represents the direction of the calculation section vector B, and the length of the arrow represents the magnitude of the calculation section vector B (a dot represents a motion vector <NUM>). The calculation section vectors B are calculated for each of the frames of the analysis-target moving image and thus vary with time of the analysis-target moving image. It should be noted that the motion analyzing unit <NUM> may calculate the calculation section vectors B by a technique other than the block matching.

The axial-direction setting unit <NUM> sets an axial direction for the analytical range R. <FIG> are schematic diagrams each showing an axial direction S that is set for the analytical range R. The axial-direction setting unit <NUM> can set a direction specified by the user as an axial direction. The user can set a direction, in which a motion is intended to be extracted in the analysis-target moving image, as an axial direction.

For example, in <FIG>, the axial direction S is set along the axon of the neuron. Further, the axial-direction setting unit <NUM> may set a plurality of axial directions. <FIG> shows the axial directions S that are radially set from the center of the cell body.

Additionally, the axial-direction setting unit <NUM> may set an axial direction by image processing for the analysis-target moving image. For example, the axial-direction setting unit <NUM> may detect an analysis target included in the analysis-target moving image, and set a long-side direction thereof as an axial direction or set an axial direction radially from the center of the analysis target. Alternatively, the axial-direction setting unit <NUM> may set a direction of a motion vector of an analytical range calculated for the analytical range, which will be described later, as an axial direction.

The information extracting unit <NUM> extracts information on the motion of the analysis target with respect to the axial direction in the analysis-target moving image (hereinafter, motion information), based on the calculation section vector and the axial direction. The information extracting unit <NUM> can calculate a motion vector for the analytical range R (hereinafter, analytical range vector) from the calculation section vectors B of the respective calculation sections D supplied from the motion analyzing unit <NUM>.

<FIG> is a schematic diagram showing a state in which the analytical range vector is calculated according to an example not covered by the appended claims. As shown in <FIG>, the information extracting unit <NUM> can add the calculation section vectors B of the respective calculation sections D included in the analytical range R, to calculate an analytical range vector C.

In line with the claims, the information extracting unit <NUM> sets a calculation section vector B having the largest motion amount (length) as an analytical range vector. <FIG> is a schematic diagram showing the analytical range vector C calculated for the analytical range R, and the axial direction S. As described above, since each calculation section vector B varies with time of the analysis-target moving image, the analytical range vector C also varies with time.

The information extracting unit <NUM> extracts motion information based on the analytical range vector C and the axial direction S. <FIG> is a schematic diagram showing a relationship between the analytical range vector C and the axial direction S. The information extracting unit <NUM> projects the analytical range vector C onto the axial direction S.

<FIG> shows a projected analytical range vector (hereinafter, projected vector) C'. As shown in <FIG>, the information extracting unit <NUM> can convert the coordinate system (x, y) of the analytical range vector C into the coordinate system (x', y') of the axial direction S. <FIG> shows an example of a conversion equation of the coordinate systems. As shown in <FIG>, an angle formed by the axial direction S and the analytical range vector C is referred to as a degree θ.

The information extracting unit <NUM> can extract an angle of the analytical range vector C with respect to the axial direction S, as motion information. <FIG> is a graph showing an example of a time change of the angle of the analytical range vector C with respect to the axial direction S. As shown in <FIG>, the degree θ varies with an imaging time (frame) of the analysis-target moving image.

In the case where the value of the degree θ (absolute value) is large, this shows that the motion of the analysis target included in the analytical range R is not matched with the axial direction S. In the case where the value of the degree θ is small, this shows that the motion of the analysis target included in the analytical range R is matched with the axial direction S. In other words, whether a motion direction of the analysis target is matched with the axial direction S or not can be grasped from this motion information.

Additionally, the information extracting unit <NUM> can extract the motion amount of the analytical range vector C with respect to the axial direction S, as motion information. The information extracting unit <NUM> can set the motion amount of the projected vector C' (i.e., the length of the projected vector C' represented by (x', y')) as the motion amount of the analytical range vector C with respect to the axial direction S. <FIG> is a graph showing an example of a time change of the motion amount (motion) of the analytical range vector C with respect to the axial direction S.

As shown in <FIG>, the motion amount with respect to the axial direction S varies with an imaging time (frame) of the analysis-target moving image. As described above, the motion amount with respect to the axial direction S is the motion amount of the projected vector C', which is projected onto the axial direction S. Even when the motion amount of the analytical range vector C is large, if the motion direction thereof differs from the axial direction S, the motion amount with respect to the axial direction is can be grasped from this motion information.

Additionally, the information extracting unit <NUM> can extract the motion amount having a motion direction within a predetermined angle from the axial direction S, as motion information. The information extracting unit <NUM> can extract the motion amount of the analytical range vector C in which the degree θ formed together with the axial direction S is within the predetermined angle (i.e., the length of the analytical range vector C represented by (x, y)). The predetermined angle (for example, +/- <NUM> degrees) is arbitrarily set, and may be preset or may be set by the user.

When the angle range of a motion direction to be extracted is small, the user can set the angle to be small. When the angle range of a motion direction to be extracted is large, the user can set the angle to be large. <FIG> is a graph showing an example of a time change of the motion amount (motion) of the analytical range vector C within a predetermined angle from the axial direction S. Since the motion amount of the analytical range vector C, which is out of the predetermined angle or has a direction different from the axial direction S, is not extracted, the user can grasp the motion amount of a desired motion direction.

It should be noted that in the above description, the information extracting unit <NUM> extracts the motion information of the analytical range vector C calculated for the analytical range R, but may similarly calculate motion information of the calculation section vector B calculated for each of the calculation sections D (see <FIG>). In other words, the motion information as shown in <FIG> may be extracted for each of the calculation sections D.

The result outputting unit <NUM> outputs the motion information supplied from the information extracting unit <NUM> and presents the motion information to the user. As shown in <FIG>, for example, the result outputting unit <NUM> can generate an image including a graph of the motion information and shows the image on a display.

Additionally, the result outputting unit <NUM> may perform mapping on the motion information in the analysis-target moving image and generate a motion-information presenting image. <FIG> are examples of the motion-information presenting image. <FIG> is a motion-information presenting image at time <NUM>. <FIG> is a motion-information presenting image at time <NUM>.

<FIG> is an analysis-target moving image that is a source of the motion-information presenting images shown in <FIG> and is an image of cardiac myocytes. As described above, the axial-direction setting unit <NUM> sets the axial direction S in one direction of the analysis-target moving image.

As shown in <FIG>, the result outputting unit <NUM> can express the motion amount of the calculation section vector B with respect to the axial direction S, in shades of gray, color coding, or the like, for each of the calculation sections D. In <FIG>, white sections represent the calculation sections D that have large motion amounts with respect to the axial direction S, and black sections represent the calculation sections D that have small motion amounts with respect to the axial direction S.

The motion-information presenting image at time <NUM> of <FIG> shows cardiac myocytes in contraction. The motion-information presenting image at time <NUM> of <FIG> shows cardiac myocytes in relaxation. Comparing <FIG> with <FIG>, the distribution of the white sections and the black sections is different between those figures. This shows that sections having large motion amounts with respect to the axial direction differ with time. It is found that the motion direction of the cardiac myocytes can be grasped from those images.

Further, the result outputting unit <NUM> may present the analytical range vector C to the user. <FIG> is a schematic diagram showing an example in which the analytical range vectors C are presented. As shown in <FIG>, the result outputting unit <NUM> may present the analytical range vectors C in one graph, the analytical range vectors C being calculated for the analytical range R at different times of the analysis-target moving image.

Additionally, the result outputting unit <NUM> may present a line D that connects the tip ends of the analytical range vectors C, which are calculated for the analytical range R at different times of the analysis-target moving image. The shape of the line D indicates a tendency of a time change of the analytical range vector C. For example, the line D of <FIG> is extended in the lower left direction, and this indicates that the motion in that direction is large in the analytical range R.

If the motion of the analysis-target moving image in the analytical range R is isotropic, the line D has a shape close to a circle. In other words, the user can intuitively grasp the motion direction in the analytical range R by referring to the shape of the line D. Further, the result outputting unit <NUM> may present the calculation section vectors B calculated for the respective calculation sections D, similarly to the analytical range vectors C.

Additionally, the result outputting unit <NUM> may generate an image in which the analytical range vector C is superimposed on the analysis-target moving image, to present the image to the user. <FIG> is an example of an image in which the analytical range vector C is superimposed on the analysis-target moving image.

As shown in <FIG>, the result outputting unit <NUM> may present the calculation section vectors B together with the analytical range vector C and may present only calculation section vectors B having large motion amounts. Further, the result outputting unit <NUM> may present calculation section vectors B having a motion direction that is the same as the analytical range vector C or having an angle within a predetermined angle.

Additionally, the result outputting unit <NUM> may superimpose the calculation section vector B of a predetermined calculation section D on the analysis-target moving image and generate an image. <FIG> is an example of an image in which the calculation section vector B is superimposed on the analysis-target moving image. In the example shown in <FIG>, the calculation section vector B of the calculation section D, which corresponds to the tip end of the axon of the neuron, is superimposed on the analysis-target moving image.

The calculation section D presenting the calculation section vector B may be specified by the user or may be determined by image processing performed on the analysis-target moving image. For example, when an object that is extended by image processing performed on the analysis-target moving image is detected, the result outputting unit <NUM> can present the calculation section vector B of the calculation section D, which is located at the tip end of the object.

Moreover, the result outputting unit <NUM> may superimpose the calculation section vectors B of different frames in the analysis-target moving image on the analysis-target moving image and generate an image. <FIG> is an example of an image in which the calculation section vectors B of different frames are superimposed on the analysis-target moving image.

The result outputting unit <NUM> can superimpose a calculation section vector B2 and a calculation section vector B1 on the analysis-target moving image. The calculation section vector B2 is the largest in a certain frame. The calculation section vector B1 is the largest in a frame that is one frame before the certain frame. The result outputting unit <NUM> may present the calculation section vectors B of a larger number of different frames. This allows the user to grasp a transition of the motion in the analysis-target moving image.

The result outputting unit <NUM> can also calculate a distance of the motion by integrating the calculation section vector B or the analytical range vector C, or calculate an acceleration rate of the motion by differentiating the calculation section vector B or the analytical range vector C, for presentation of the result.

The image processing apparatus <NUM> has the functional configuration as described above. In the image processing apparatus <NUM>, as described above, the motion in the analysis-target moving image is presented in accordance with the motion direction. This allows an evaluation of the motion of the analysis target based on not only the motion amount but also the motion direction.

It should be noted that the range specifying unit <NUM> specifies the analytical range R in the analysis-target moving image, but may move the analytical range R with the elapse of time of the analysis-target moving image. <FIG> is a schematic diagram showing a movement of the analytical range R. An analytical range R1 is an analytical range that is set at a predetermined time in the analysis-target moving image. An analytical range R2 is an analytical range that is set at a predetermined time after the analytical range R1 is set.

As shown in <FIG>, the range specifying unit <NUM> can move the analytical range R1 in the direction of the analytical range vector C and set the analytical range R2 as a new analytical range. The amount of movement of the analytical range may be a predetermined amount or may be adjusted in accordance with the magnitude of the analytical range vector C.

The motion analyzing unit <NUM> can analyze the motion of the newly-set analytical range R2 as described above, to calculate a calculation section vector. The information extracting unit <NUM> can extract motion information based on the calculation section vector or an analytical range vector and an axial direction.

By such processing, since the analytical range R moves in a direction of the analytical range vector at predetermined time intervals, the motion can be evaluated by following the motion of the analysis target. For example, in the axonal transport of a neuron (see <FIG>), the analytical range R is moved together with the transport object, and the transport can be evaluated.

The operation of the image processing apparatus <NUM> will be described. <FIG> is a flowchart showing the operation of the image processing apparatus <NUM>.

As shown in <FIG>, the moving image acquiring unit <NUM> acquires an analysis-target moving image (St101), and the range specifying unit <NUM> specifies an analytical range (St102). Subsequently, the motion analyzing unit <NUM> calculates a calculation section vector in the analytical range (St103), and the information extracting unit <NUM> calculates an analytical range vector (St104).

The axial-direction setting unit <NUM> then sets an axial direction in the analytical range (St105). At that time, the axial-direction setting unit <NUM> may set the axial direction based on the analytical range vector. Subsequently, the information extracting unit <NUM> extracts motion information based on the calculation section vector or the analytical range vector and the axial direction (St106), and the result outputting unit <NUM> presents the motion information (St107).

It should be noted that the result outputting unit <NUM> may generate an image in which the calculation section vector or the analytical range vector is superimposed on the analysis-target moving image. Further, the range specifying unit <NUM> may move the analytical range in accordance with the analytical range vector along with the elapse of time of the analysis-target moving image.

Additionally, the image processing apparatus <NUM> may operate as follows. <FIG> is a flowchart showing another operation of the image processing apparatus <NUM>.

As shown in <FIG>, the moving image acquiring unit <NUM> acquires an analysis-target moving image (St111), and the range specifying unit <NUM> specifies an analytical range (St112). Subsequently, the axial-direction setting unit <NUM> sets an axial direction in the analytical range (St113). The axial-direction setting unit <NUM> can set the axial direction by the user's specification or by image processing performed on the analysis-target moving image.

Subsequently, the motion analyzing unit <NUM> calculates a calculation section vector in the analytical range (St114), and the information extracting unit <NUM> calculates an analytical range vector (St115). Subsequently, the information extracting unit <NUM> extracts motion information based on the calculation section vector or analytical range vector and the axial direction (St116), and the result outputting unit <NUM> presents the motion information (St117).

The functional configuration of the image processing apparatus <NUM> as described above can be achieved by a hardware configuration described below.

<FIG> is a schematic diagram showing a hardware configuration of the image processing apparatus <NUM>. As shown in <FIG>, the image processing apparatus <NUM> includes, as a hardware configuration, a CPU (Central Processing Unit) <NUM>, a GPU (Graphic Processing Unit) <NUM>, a memory <NUM>, storage <NUM>, and an I/O (Input/Output unit) <NUM>. Those components are connected to one another via a bus <NUM>.

The CPU <NUM> controls other configurations according to a program stored in the memory <NUM>, and also performs data processing according to the program, to store a processing result in the memory <NUM>. The CPU <NUM> may be a microprocessor.

The GPU <NUM> is controlled by the CPU <NUM> and executes image processing. The CPU <NUM> can cause the GPU <NUM> to execute parallel computing, to calculate a motion vector and extract motion information at high speed. The GPU <NUM> may be a microprocessor.

The memory <NUM> stores programs and data executed by the CPU <NUM>. The memory <NUM> may be a RAM (Random Access Memory).

The storage <NUM> stores programs and data. The storage <NUM> may be an HDD (Hard disk drive) or an SSD (solid state drive).

The I/O <NUM> receives an input to the image processing apparatus <NUM> or supplies an output of the image processing apparatus <NUM> to the outside. The I/O <NUM> includes input devices such as a keyboard and a mouse, output devices such as a display, and connection interfaces such as a network.

The hardware configuration of the image processing apparatus <NUM> is not limited to those described herein, and may be any configuration as long as the functional configuration of the image processing apparatus <NUM> can be achieved. Additionally, a part or whole of the hardware configuration described above may be present over the network.

Claim 1:
An image processing apparatus (<NUM>) comprising:
at least one processor (<NUM>) configured to receive an image of a cell in a plurality of images of the cell, the cell having various motions including motion of the cell itself and motion due to extracellular environments; and
at least one storage medium configured to store processor-executable instructions that, when executed by the at least one processor, perform a method comprising:
determining an analytical range (R) within the image of the cell, wherein the analytical range includes an analysis target including the cell;
determining a plurality of motion vectors (B) for a plurality of sections (D) of the analytical range;
calculating an analytical range vector (C) from the plurality of motion vectors by identifying a motion vector from among the motion vectors as having a largest motion amount and setting the motion vector as the analytical range vector;
setting at least one axial direction (S) for the analytical range in the image of the cell, the at least one axial direction being a direction in which a motion is to be extracted;; and
determining motion information for the analysis target by analyzing motion of the analysis target using the analytical range vector by projecting the analytical range vector onto the at least one axial direction to identify, with respect to the at least one axial direction, at least one of:
a motion amount of the analysis target; and
a motion direction of the analysis target.