Patent Description:
Knee osteoarthritis is one of main upsetting orthopedic diseases to people. In its clinical surgeries, a high tibial osteotomy (HTO) and a distal femoral osteotomy (DFO) are used to adjust an abnormal mechanical axis of a lower limb, thus prolonging a service life of the knee. Preoperative diagnosis and preoperative planning depend on a lower limb full-length X-ray image of the patient. It is time-consuming for a professional doctor to determine key points in the knee surgery. The workload for planning a surgery scheme limits a number of patients to be treated by the doctor in unit time.

Document <CIT> discloses an image processing method which includes the steps of: Obtain the target key point of the target image; segment the target object in the target image to obtain the segmentation result of the target object; obtain the processing type of the target object; according to the target key point, the segmentation result and the processing type determines at least one processing parameter of the target object.

Further, the selecting a key regional image in the medical image, and performing object detection on the key regional image to extract key points comprises:.

Further, the calculating a corrective angle with coordinates of the extracted key points by labeling the key points and a mechanical axis comprises:.

The automatic planning method and system for a simulated knee surgery based on deep learning provided by the present disclosure has the following advantages over the prior art: The present disclosure provides the planning method and system for a surgery around the knee with coordinate calculation and image processing, and automatically processes the lower limb full-length X-ray image with a regional object detection algorithm and a key point detection algorithm, thereby realizing simulation on eight surgeries around the knee. The present disclosure can effectively shorten manual point selection time by automatically extracting the key points, and can effectively shorten surgery planning time by automatically simulating surgery planning and displaying an effect. Therefore, the present disclosure significantly improves diagnosis time of each patient, greatly reduces the reliance on the professional doctor, alleviates a burden of the doctor, and achieves a higher diagnosis and treatment efficiency.

As a part of the present disclosure, the accompanying drawings of the specification provide further understanding of the present disclosure. The schematic embodiments of the present disclosure and description thereof are intended to explain the present disclosure and are not intended to constitute an improper limitation to the present disclosure. In the drawings:.

It should be noted that the embodiments in the present disclosure and features in the embodiments may be combined in a non-conflicting manner.

It should be understood that in the description of the present disclosure, terms such as "central", "longitudinal", "transverse" "upper", "lower", "front", "rear", "left", "right" "vertical", "horizontal", "top", "bottom", "inside" and "outside" indicate the orientation or positional relationships based on the drawings. They are merely intended to facilitate and simplify the description of the present disclosure, rather than to indicate or imply that the mentioned device or components must have a specific orientation or must be constructed and operated in a specific orientation. Therefore, these terms should not be construed as a limitation to the present disclosure. Moreover, the terms such as "first" and "second" are used only for the purpose of description and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features denoted. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present disclosure, unless otherwise specified, "a plurality of" means at least two.

In the description of the present disclosure, it should be noted that, unless otherwise clearly specified, meanings of terms "install", "connected with", and "connected to" should be understood in a board sense. For example, the connection may be a fixed connection, a removable connection, or an integral connection; may be a mechanical connection or an electrical connection; may be a direct connection or an indirect connection by using an intermediate medium; or may be intercommunication between two components. A person of ordinary skill in the art may understand specific meanings of the above terms in the present disclosure based on a specific situation.

The present disclosure will be described in detail below with reference to the accompanying drawings and the embodiments.

As shown in <FIG>, the present disclosure provides an automatic planning method for a simulated knee surgery based on deep learning, including the following steps.

In step <NUM>, a medical image is acquired, and the pixel pitch in the medical image is converted into an actual length.

An actual size serves as a criterion in actual surgery. Hence, the pixel pitch in the image is converted into the actual length, namely a pixel pitch is obtained. <FIG> shows a ruler image. There are eighteen <NUM>-mm spacings on each ruler.

According to characteristics of the image, a pixel threshold is preset, white pixel points in the image are screened out, and the ruler image is detected with an LSD, as shown in <FIG>. A head endpoint coordinate and a tail endpoint coordinate of the ruler are acquired. Conversion is performed on a pixel difference value between the head endpoint coordinate and the tail endpoint coordinate (XA and XB) of the ruler in an X direction and a millimeter unit to obtain a coefficient of relationship α between the millimeter unit and a coordinate value, as shown in Eq. (<NUM>). Upon the conversion, length information, rather than pixel information, is obtained. According to the criterion in surgery planning, a surgery scheme is selected. The pixel information cannot be used as the criterion: <MAT>.

In step <NUM>, a key regional image in the medical image is selected, and object detection is performed on the key regional image to extract key points.

The present disclosure selects a key region including the key points, and does not perform global processing on the whole image. This downsizes the model and reduces the noise. The present disclosure performs the object detection on a hip region, a knee region and an ankle region with a YOLOX in deep learning. As shown in <FIG>, <FIG> shows an original lower limb X-ray image transmitted to the YoloX, and <FIG> shows a result upon regional selection of the YoloX network. <FIG> show a selected key region, and uses a left lower limb as an example.

In the key regional image, a hip center, a hinge point in the knee region, a Fujisawa point, a surgery point and an ankle center are extracted with a PFDNet. As shown in <FIG>, the extracted key points include the hip center, the hinge point, the surgery point and the ankle center.

In step <NUM>, a corrective angle is calculated with coordinates of the extracted key points by labeling the key points and a mechanical axis, a rotated angle is calculated, bone cutting and bone rotation are simulated to obtain a surgery planning image, and automatic planning is performed on an HTO and a DFO.

The HTO and the DFO are planned with a coordinate of the hip center, a coordinate of the ankle center, a coordinate of the hinge point, a coordinate of the surgery point and a coordinate of the Fujisawa point. As shown in <FIG>, to select a correcting mechanical axis and implement image rotation in four small surgeries in the HTO, based on the surgery point, a lower half part of the image is rotated with the hinge point as a center. The present disclosure takes the medial opening HTO as an example. A connecting line from the hip center to the Fujisawa point and an extended line form a target mechanical axis called a line <NUM>. The hinge point and the ankle center forms a line <NUM>. With the hinge point as a center, and a length of the line <NUM> as a radius, rotation is made to the line <NUM> to obtain a line <NUM>. In this case, the ankle center coincides with the line <NUM> (target mechanical axis). An included angle between the line <NUM> and the line <NUM> is an angle to be corrected in the medial opening HTO.

Specifically, assuming that the line <NUM> (target mechanical axis) determined by the hip center and the Fujisawa center has a linear equation as shown in Eq. (<NUM>), where, (x<NUM>,y<NUM>) is a coordinate of the hinge point, (x<NUM>,y<NUM>) is a coordinate of a parameter point for determining the line <NUM>, and (x<NUM>,y<NUM>) is a coordinate of the original ankle center, then: <MAT>.

The length of the line <NUM> is given by a Euclidean distance Eq. (<NUM>): <MAT>.

As shown in Eq. (<NUM>), assuming that the rotated ankle center has a coordinate (x<NUM>,y<NUM>), a distance from the hinge point (x<NUM>,y<NUM>) to the rotated ankle center is given by: <MAT>.

As shown in Eq. (<NUM>), with the linear equation of the target mechanical axis as a constraint condition, the rotated ankle center has a coordinate given by: <MAT>.

In the present disclosure, the angle is calculated as shown in <FIG>. ∠AHB represents a to-be-calculated angle at any position in a plane, such as an angle formed by the Fujisawa point, the hip center and the ankle center. In order to make two sides of the angle coincide in the surgery, namely the mechanical axis of the lower limb coincide with a connecting line from the hip center to the ankle center, the ∠AHB refers to an angle rotated with the hinge point as an axis.

The ∠AHB is calculated in two parts, including an included angle<NUM> between the segment AH and the x axis, and an included angle<NUM> between the segment BH and the y axis. The ∠AHB is as shown by Eqs. (<NUM>), (<NUM>) and (<NUM>): <MAT> <MAT> and <MAT>.

For surgery planning at a tibial end, regardless of a closed-wedge HTO or an HTO in which the lower half part of the image is rotated, the upper half part and the lower half part of the image are segmented according to a connecting line from the hinge point to the surgery point, as shown in Eq. (<NUM>): <MAT>.

The origin is rotated as shown in <FIG>. A modeling method in a polar coordinate system is used to rotate the image. The point v is rotated around the origin for an angle θ to obtain the point v'. Assuming that the point v has a coordinate (u, w), then the point v' has a coordinate (u', w'). Assuming that a distance from the origin to the point v is r, and an included angle between a vector from the origin to the point v and the x axis is ϕ then v and v' can be expressed as Eq. (<NUM>) and Eq. (<NUM>) respectively, with a matrix form being expressed as Eq. (<NUM>): <MAT> <MAT> and <MAT>.

Assuming that the hinge point has a coordinate (x<NUM>, y<NUM>), then when the image is rotated around the hinge point, the hinge point first moves to the origin, as shown in Eq. (<NUM>), where x,y is a distance from the hinge point (x<NUM>,y<NUM>) to the origin, and after the hinge point (x<NUM>,y<NUM>) moves to the origin, (x'<NUM>, y'<NUM>) is obtained: <MAT>.

Eq. (<NUM>) shows homogeneous coordinates after a translation matrix and a rotation matrix of the image are introduced. The matrix becomes a form <NUM>×<NUM>: <MAT> where, t is an offset coefficient.

After returning to the origin, the hinge point is rotated by θ degrees with the original as a center, as shown in Eq. (<NUM>): <MAT>.

In this case, the hinge point is translated to an original position, as shown in Eq. (<NUM>): <MAT>.

At last, a rotation matrix of the lower half part of the image around the hinge point can be obtained, as shown in Eq. (<NUM>): <MAT>.

According to the corrective angle between the line <NUM> and the line <NUM> in <FIG>, the lower half part of the image is rotated with the hinge point as a center of rotation to obtain a surgery planning image, as shown in <FIG>.

Regarding the DFO, the present disclosure takes a lateral opening DFO as an example. A coordinate of the hip center, a coordinate of the ankle center, a coordinate of the hinge point, a coordinate of the surgery point and a coordinate of the Fujisawa point are obtained first. A connecting line from the ankle center to the Fujisawa point and an extended line form a target mechanical axis. As shown in <FIG>, the target mechanical axis is called a line <NUM>', and the hinge point and the hip center form a line <NUM>'. With the hinge point as a center, and a length of the line <NUM>' as a radius, rotation is made to the line <NUM>' to obtain a line <NUM>'. In this case, the hip center coincides with the line <NUM>'. An included angle between the line <NUM>' and the line <NUM>' is an angle to be corrected. The included angle is solved by Eq. (<NUM>), Eq. (<NUM>) and Eq. (<NUM>).

As shown in Eq. (<NUM>), the line <NUM>' (target mechanical axis) determined by the ankle center and the Fujisawa point has a linear equation: <MAT>.

For surgery planning on a femoral end, an upper half part of the image is rotated. The upper half part and the lower half part of the image are segmented according to a connecting line from the hinge point to the surgery point, as shown in FIG. (<NUM>):<MAT>.

The method for solving the line <NUM>' and the line <NUM>' and the method for rotating the image are the same as those in the medial opening HTO. According to the corrective angle between the line <NUM>' and the line <NUM>' in <FIG>, the upper half part of the image is rotated with the hinge point as a center of rotation to obtain a preoperative planning image in the lateral opening DFO, as shown in <FIG>.

With coordinate calculation and image processing, the present disclosure realizes preoperative planning for various surgeries on the DFO and the HTO. <FIG> shows preoperative planning and effect sketches of the (a) medial closing DFO, (b) medial opening DFO, (c) lateral closing DFO, (d) medial closing HTO, (e) lateral closing HTO and (f) lateral opening HTO.

The present disclosure reduces a key point detection range by segmenting key regions. With image processing, the present disclosure converts size information of the ruler into the pixel pitch, thus providing a reference to determine a size in the surgery. The present disclosure selects the key regions and detects the key points with the YOLOX and the PFDNet. With the coordinate calculation and the image processing, the present disclosure automatically implements bone cutting and bone rotation in the simulated surgery, and displays a final effect.

The present disclosure further provides an automatic planning system for a simulated knee surgery based on deep learning, including: an image preprocessing unit, a key point extraction unit, and an automatic planning unit.

The image preprocessing unit is configured to acquire a medical image, and convert the pixel pitch in the medical image into an actual length.

The key point extraction unit is configured to select a key regional image in the medical image, and perform object detection on the key regional image to extract key points.

Claim 1:
An automatic planning method for a simulated knee surgery based on deep learning, comprising:
acquiring a medical image of a knee joint, and converting a pixel pitch in the medical image into an actual length;
selecting a key regional image in the medical image, and performing object detection on the key regional image to extract key points based on deep learning; and
calculating a corrective angle with coordinates of the extracted key points by labeling the key points and a mechanical axis, calculating a rotated angle, simulating bone cutting and bone rotation to obtain a surgery planning image, and performing automatic planning on a high tibial osteotomy (HTO) and a distal femoral osteotomy (DFO),
characterized in that,
the converting the pixel pitch in the medical image into the actual length comprises:
acquiring a ruler image from the medical image by, screening out white pixels in the medical image, and detecting the ruler image with a line segment detector (LSD); and
acquiring a head endpoint coordinate and a tail endpoint coordinate of a ruler in the ruler image, and performing conversion on a pixel difference value between the head endpoint coordinate and the tail endpoint coordinate (XA and XB) of the ruler in an X direction and a millimeter unit to obtain a coefficient of relationship α between the millimeter unit and the pixel difference value, <MAT>.