Predictive modeling platform for serial casting to correct orthopedic deformities

A system and method are provided herein for modeling of force vectors for serial casts to correct orthopedic deformities includes a camera configured to capture a three-dimensional image of the deformity, a computing device programmed to generate a three-dimensional model of the deformity based on the image of the deformity, determine the boundary conditions for the deformity based on the three-dimensional image of the deformity, and generate force vectors for a series of casts to correct the deformity. In exemplary embodiments, the system can print a series of casts to correct the deformity.

BACKGROUND

Musculoskeletal disorders such as neuromuscular (NM), musculoskeletal (MSK) disorders can be congenital or acquired. Some musculoskeletal disorders can be treated in pediatric and adult populations through serial casting. Conventionally, treating musculoskeletal disorders using serial casting can be difficult because of difficulties with access to care and the relative subjectivity of the treatment. For example, common pediatric orthopedic deformities such as Talipes equinovarus or congenital talipes equinovarus (commonly called clubfoot) have been difficult to treat. In the case of clubfoot, the current standard of care afforded to correct this skeletal deformity is the Ponseti serial casting methodology, in which the deformity is corrected using a weekly series of casts. Limitations of this method include the need for highly trained surgeons proficient in this method and frequent weekly visits to the orthopedic surgeon for placement of the casts. Even when skilled doctors trained in the method are available, there is a lot of variability and subjectivity in determining the next step of the serial cast. Variability in casting technique and inability to predict treatment length lead to difficulties in standardization of the treatment course of serial clubfoot correction. Similar difficulties exist with respect to treating patients with other congenital NM, congenital MSK, acquired NM, and acquired MSK disorders using serial casts.

DETAILED DESCRIPTION

Serial casting corrects a three-dimensional musculoskeletal deformity, neuromuscular deformity or both through periodic manipulation of the deformity. For example, in the case of deformity of clubfoot, the congenital musculoskeletal deformity is corrected through weekly manipulation of the deformity of the foot in a step-wise process using serial casts. Often times correction is in multiple three-directional planes simultaneously. Conventionally, this manipulation is a very manual process, labor intensive and embodies an imprecise prediction of subsequent steps and outcomes. Although, computer modelling for serial casts to correct musculoskeletal deformity, neuromuscular deformity or both exists, the conventional computer modeling requires a linear approach of approximating a series of points and lines to determine a specific direction in which a cast in the series of casts applies a force to the deformity. However, the linear approach does not account for the three-dimensional deformities of the clubfoot within the cavus, adductus, varus, equinus and derotational elements.

Embodiments of the present disclosure include systems and methods for modelling a set of force vectors for a cast or a next cast in a series of casts to correct the musculoskeletal deformity, the neuromuscular deformity or both that overcome the difficulties and problems described herein with respect to conventional serial casting techniques. In exemplary embodiments, the system includes a processor that executes machine readable instructions to receive an image of the deformity. In turn, the processor generates a three-dimensional model of the deformity based on the image of the deformity. Further, the processor determines a deviation between a three-dimensional stereo-typical anatomical alignment and the three-dimensional model of the deformity (i.e., boundary conditions of the deformity) and generates a next intermediate anatomical alignment in a series of intermediate anatomical alignments for correcting the deviation based on a machine learning model trained on a plurality of prior patient records. The processor also simulates a plurality of trial force vectors for a next cast to correct the deformity to the next intermediate anatomical alignment. The simulation generates a projected anatomical alignment post-treatment with the next cast on the deformity using finite element analysis for each of the trial force vectors, and identifies a next force vector from the plurality of trial force vectors that minimizes the difference between the projected anatomical alignment and the next intermediate anatomical alignment, such that using the next force vector for the next cast results in an average difference in the range of 2 mm to 20 mm on a point to point comparison between a point in a point cloud of the next cast and a corresponding point in a point of a point cloud of a prior cast for a similar correction based on prior patient data.

In exemplary embodiments, the camera is an array of cameras, an x-ray imager, an ultrasound scanner, a three-dimensional scanner, an magnetic resonance imaging device, a CT scan or a combination thereof. For example, the camera can be a depth detection camera or include a light detection and ranging (LIDAR) sensor that produces a point cloud of the deformity.

In exemplary embodiments, the system can determine boundary conditions for the deformity based on the three-dimensional image of the deformity and medical data from a plurality of persons. The medical data from the plurality of persons can include images, image data, data and information about stereo-typical anatomical alignment, anatomical alignment of prior patient deformities prior to correction, intermediate anatomical alignment of patient deformities during correction and final alignment of patient deformities post correction. In exemplary embodiments, the medical data for prior patient deformities can be derived from discarded casts of prior patients used during their treatment and scans of the patient deformities during their treatment. In some embodiments, the medical data for prior patient deformities can be derived from medical records generated during patient visits such as notes from the doctor, 2-dimensional images such as x-rays, measurements of force vectors such as spasticity and the like. In exemplary embodiments, the medical data can be converted to a point cloud that represents the three-dimensional model of the anatomical alignment(s) for the patient, the stereo-typical person or both.

In exemplary embodiments, the plurality of prior patient data includes an original image of the deformity of the patient(s), intermediate images of the deformity of the patient(s) and an image of the final corrected deformity of the patient(s). In some exemplary embodiments the plurality of prior patient data includes data generated from a plurality of scans or images of prior discarded casts of patients used during treatment of the deformity.

In exemplary embodiments, to simulate the plurality of trial force vectors for the next cast the system can generate a series of forces at a plurality of nodes in the three-dimensional model of the deformity and determine a resulting change in the anatomical alignment of the deformity after a certain period of time based on the application of the series of forces at the nodes.

In exemplary embodiments, to select the next force vector from the plurality of trial force vectors the system can determine which of the plurality of trial force vectors minimizes the deviation between the projected anatomical alignment and the next intermediate anatomical alignment based on a finite element analysis machine learning model. The finite element analysis machine learning model is trained on medical data that can include next force vectors that correspond to a points on a point cloud of the deformity that were selected form the plurality of trial vectors during treatments in prior patients.

In exemplary embodiments, to select the next force vector the system can minimize the deviation between the projected anatomical alignment and the next intermediate anatomical alignment in multiple directions to reduce the deviation between the three-dimensional stereo-typical anatomical alignment and the three-dimensional model of the deformity in multiple directions simultaneously after the next cast is applied to the deformity.

In exemplary embodiments, to select the next force vector the system can minimize the difference between the projected anatomical alignment and the next intermediate anatomical alignment in a direction with the maximum deviation between the three-dimensional stereo-typical anatomical alignment and the three-dimensional model of the deformity after the next cast is applied to the deformity.

Referring now toFIG.1Awhich illustrates a system100to capture an image of the deformity according to the present disclosure is provided. The system100includes a camera102(shown inFIG.1Aas an array of cameras) configured to capture a three-dimensional image of the deformity and a plurality of light sources106. In an exemplary embodiment the system100can include a calibration target110. Examples of the calibration target110include checkerboard patterns, socks with or without identification patterns and the like. For example, the calibration target110can be a checkerboard pattern that can attached on a flat board that can move relative to the camera102to acquire calibration images of the calibration target110with various poses relative to the camera102. In an exemplary embodiment, the system100can use a calibration target110for subjects where the deformity is kept relatively still. The use of a calibration target with multiple cameras allows the system100to capture a three-dimensional images and compensate for movement.

In an exemplary embodiment, the camera102can be an array of cameras. The system100can generate a three-dimensional image of the deformity by stitching all the images from the array of cameras. In an exemplary embodiment, the camera can be a digital camera, or a video camera, an ultrasound imaging system, MRI or a CT scan. The system100can compensate for movement of the subject using image processing to obtain an accurate representation of the deformity in three-dimensions. In exemplary embodiments, the camera102can capture be a depth sensing camera (e.g., Microsoft Kinect™), which produces a point cloud of the deformity. In some embodiments, the system100can convert a three-dimensional image of the deformity into a point cloud. In exemplary embodiments, the camera102can capture imagery data for generating a point cloud of the deformity based on discarded casts used during treatment of a patient. In an exemplary embodiment, the system100can acquire an image of the deformity from a mobile device such as a phone or tablet camera. The system100can receive an image captured from a mobile device that captures the deformity from different angles. The system100can then stitch the images together to create a three-dimensional image.

In an exemplary embodiment, the system100can determine a deviation or average deviation on a point by point basis between a three-dimensional model of the deformity and the three-dimensional stereo-typical anatomical alignment of the human anatomy in question. For example, the system100can include a database or corpus storing medical data of the individuals. The medical data of the individuals can include three-dimensional models of stereo-typical anatomical alignment of the human anatomy, three-dimensional models of deformities that deviate from the stereo-typical anatomical alignment based on medical records of prior patients with deformities and the like.

In exemplary embodiments, the system100can use one or more point clouds representing a three dimensional model of a cast or deformity to perform shape analysis instead of the more conventional mesh models. Traditionally, shape analysis methods have operated on solid and surface models of objects, especially tessellated models (i.e., triangular mesh surface models). Shape analysis is concerned with understanding the shape of models geometrically, topologically, and relationally. In some embodiments, the system100can use shape analysis to group the deformities in the database based on the type of the deformity, segment the deformities based on the shape into sub-shapes, and find complementary deformities based on the shape of the deformity. For example, the system100can query the database with the medical data using a point cloud of a deformity that has been acquired to return a matching point cloud of a similar deformity.

The system100can generate a next intermediate anatomical alignment in a series of three-dimensional alignments based on a machine learning model116trained on prior patient data from the database or corpus. In order to train the machine learning model, the system100can include a computing device112. The computing device112can include a machine learning trainer114to generate the machine learning model116. In an exemplary embodiment, the system100can generate a machine learning model based on supervised learning, unsupervised learning or reinforcement learning. In an exemplary embodiment, the machine learning trainer114can be implemented as a machine learning computing device that also stores and allows use of the machine learning model116. The machine learning trainer114can analyze a set of training data that includes a classification of the data that the machine learning trainer114can use to calibrate its algorithm to identify what lies within a class or is outside a class. For example, a convolutional neural network or deep learning neural network trained on three-dimensional models of club foot can classify a new three-dimensional model acquired by the system100based on the trained machine learning model116.

The system100can generate the machine learning model116that can be used to generate a next intermediate three-dimensional anatomical alignment in a series of intermediate three-dimensional anatomical alignments to correct the orthopedic deformities. For example, the system100can receive training data that includes medical data of patients who were treated to correct a deformity such as three-dimensional images of the uncorrected deformity, three-dimensional images of the intermediate anatomical alignments achieved during treatment of the deformity, and the three-dimensional images of the final corrected anatomical alignment achieved after treatment. In some embodiments, the training data can also include three-dimensional anatomical alignments of stereo-typical persons. In some embodiments, the training data can be generated from discarded casts of patients who were treated for a deformity. The system100can use the prior discarded casts to approximate the deformity at each stage of the correction process where the three-dimensional images of the foot are not available. When treating clubfoot, the training data can include patient data obtained during the treatment of a patient using the Ponseti method. In an exemplary embodiment, machine learning models analyze data from a plurality of prior patients to identify mean shapes and shape variations. The system100can then determine boundary conditions of the machine learning model116to classify a new three-dimensional surface model of a deformity acquired from a new patient to determine whether the deformity of the new patient falls within the boundary of a particular type of deformity or to identify a similar deformity in the prior medical records that closely match the shape, type or both. The system100can then determine a next three-dimensional intermediate anatomical alignment in a series of anatomical alignments based on the prior identified deformity.

In an exemplary embodiment, the machine learning model116can be trained to output the desired correction angles to correct the deformity to the next intermediate alignment, i.e., the desired angles of correction that will result in the next three-dimensional anatomical alignment for correcting the deformity.

In an exemplary embodiment, the system100can train the machine learning model based on prior patient data for a plurality of patients such an original three-dimensional image of the deformity, images of intermediate stages of correction of the deformity and the final image of the corrected deformity. In exemplary embodiments, the system can use three-dimensional scans of prior discarded casts of patients to determine the original deformity, stages of correction of the deformity and the final corrected deformity.

The system100can simulate trial force vectors for a next cast that can correct the deformity to the next intermediate anatomical alignment using finite element analysis at the nodes of the three-dimensional model of the deformity. The system100can simulate the projected anatomical alignment for each trial force vector post-treatment with the next cast based on finite element analysis of the three-dimensional model of the deformity. For example, the system100can apply finite element analysis with the trial force vectors acting at the nodes of the three-dimensional model of the deformity to determine the projected anatomical alignment post-treatment with the next cast.

In an exemplary embodiment, the system100can generate training data for the machine learning model based on modelling and analysis software such as ANSYS. Modelling and simulation software can be used to deform a 3D three-dimensional CAD model of a normal foot into a plurality of virtually generated CAD models (e.g., 500 models), for example, clubfoot CAD models, with different degrees and angles of deformity potentially seen during the correction sequence. In an embodiment, system100can use supervised learning and the system100can receive inputs from an orthopedic surgeon (e.g., pediatric orthopedic surgeon) to review the models for accuracy. The system100can export the CAD models as point cloud models for anatomical classification/labeling of the generated models for the machine learning model. The system100can use the point cloud model of the foot to identify the severity of the clubfoot deformity by determining the amount of deviation of the foot with respect to normal pose in four different directions as shown inFIG.1B.FIG.1Billustrates the deformities in clubfoot such as the equinus deformity124, the varus deformity126, the calcancopedal derotation128and the horizontal plane deformity relative to hindfoot130. The system100can capture the variation in these deformities using the camera102. In an exemplary embodiment, the machine learning model can be evolved to improve the accuracy of the model over time.

The system100can use a deep learning method such as a PointNet to process the point cloud models. PointNet is an open source platform for classification of point cloud models. Since the point cloud model is randomly oriented, they use a bounding box that fits into the model, and normalizes the point cloud to always align the point cloud model in a certain direction before feeding it into the deep learning network as input datasets. The system100can use PointNet to classify different stages of an orthopedic skeletal deformity, for example, clubfoot deformity. In an exemplary embodiment, the system100can use the point cloud CAD models generated using simulation software to train a deep learning network to objectively classify and label each patient's unique foot deformity compared to a normal foot. The system100can then train the network to predict the cast series for each subject patient in this study. The system100can use supervised learning based on inputs obtained by presenting an orthopedic skeletal deformity model, for example, a clubfoot model, to one or more orthopedic surgeons. In an exemplary embodiment, the models can be presented with a selection of candidate foot correction models (e.g., out of five hundred foot models) that is the next in the correction series based on the Ponseti method. The system100can then receive inputs from the doctors on a consensus basis and select the next correction phase out of the selection of candidate foot models (e.g., 10 models). Over the course of multiple rounds of selection (e.g., 500 rounds) the system100trains the deep learning network to search the training dataset and output the subsequent cast for deformity correction.

In an exemplary embodiment, the system100can generate an STL file for three-dimensional printing using a three-dimensional printer122.

The system100can select a next force vector for the next cast from the trial force vectors by identifying the force vector that minimizes the difference between the projected anatomical alignment and the next intermediate anatomical alignment. In accordance with the teachings herein, using the next force vector in the next cast results in an average difference in the range of 2 mm to 20 mm between a point in a point cloud of the next cast and a corresponding point in a point cloud of a prior cast for a similar correction in a prior patient data. The system100can determine the shape and dimensions of the next cast within a range of 2 mm to 20 mm when compared with a trained and skilled doctor with years of experience performing the casting using subjective parameters.

In exemplary embodiments, the system100can select a next force vector for the next cast from the trial force vectors by identifying the force vector that minimizes the difference between the projected anatomical alignment and the next intermediate anatomical alignment. In accordance with the teachings herein, the next force vector for the next cast results in an average difference in the range of 0.5 degrees to 4 degrees between the angles in a point cloud of the next cast and the corresponding angles in a point cloud of a prior cast for a similar correction in a prior patient data.

Referring toFIG.1Cthe system100can determine the dimensions and shape of the next cast in a series of casts for correcting the deformity to a high degree of accuracy without the subjective judgement of a highly skilled doctor. The system100can generate a point cloud138of the three-dimensional scan of the cast136of a prior patient as shown inFIG.1C. The system100can then determine a simulated angle of derotation140. For a given deformityFIG.1Cillustrates a point cloud of the simulated cast134generated by the system100when compared to the point cloud of a real cast132placed by a highly skilled orthopedic to correct the deformity. When using the system100to generate the next cast, the average difference between the dimensions of the point cloud of the simulated cast134and the point cloud of the real cast132can be in the range of 2 mm to 20 mm on a point to point basis between corresponding points in a point cloud of the simulated next cast134and a point cloud of a prior cast132for a similar correction in a prior patient data. When using the system100to generate the next cast, the average difference can be in the range of 0.5 degrees to 4 degrees between the angles in a point cloud of the simulated next cast134and the corresponding angles in a point cloud of a prior cast132for a similar correction in a prior patient. In an exemplary embodiment, the average difference between the angles of the real132and simulated casts134was 0.93 degrees for equinus and 0.74 degrees for de-rotation. In exemplary embodiments, the average difference between corresponding points of the real cast132and simulated casts134in the respective three-dimensional point clouds was 2.305 mm.

Referring toFIG.1Dthe table represents the difference between the angles on the simulated next cast134and the corresponding angles in a point cloud of a prior cast132for similar correction in a prior patient. For example, the table illustrates that the difference between the angles in a point cloud of the simulated next cast134and the corresponding angles in a point cloud of a prior cast132for Equinus correction136, de-rotation correction138and the average difference140between point clouds.

Referring toFIG.2. the system100can generate a cast or a series of casts as shown inFIG.2to correct the orthopedic deformity. In an exemplary embodiment the system100can generate a cast or a series of casts that can be three-dimensional printed. The system100determines a force vector or a set of force vectors to correct the orthopedic deformity based on three-dimensional imagery of the deformity and the machine learning model116can generate the next intermediate anatomical alignment for correcting the deviation. In an exemplary embodiment, next intermediate anatomical alignment can be the desired corrected angles for correcting the deformity in the next cast. In an exemplary embodiment, the system100can determine the shape and geometry of a cast or a series of casts202-210that exert the determined force vector or set of force vectors that are tailored to the patient. In an exemplary embodiment, the system100can determine the force vectors for the series of casts202-210to correct the deformed foot214with three-dimensional deformities shown along the x, y and z axis to arrive at the corrected foot212. In an exemplary embodiment, the system100selects the force vector or set of force vectors such that the right areas216that are structurally designed in normal foot of children to distribute the load when walking is in the same plane and perform the load bearing function once corrected.

It can be appreciated that, depending on the baseline anatomical shape and arrangement and an anatomical rearrangement goal, or target, an appropriate serial casting strategy can be developed. For instance, not all patients may need the same number of casts. In fact, it may be that a patient requires fewer casts as deformities to the internal anatomy of the foot may be less severe. In other cases, the deformity to the underlying anatomy may be significant and more casts may be prescribed. Being able to combine this internal information, however, with exterior data of the surface of the foot allows for generation of three-dimensional printed ‘corrective’ casts that are patient-specific.

Referring now toFIGS.3A,3B,3C, and3Dthe system100can determine the reference points for generating the corrective plane.FIGS.3A and3Billustrates the reference points for generating the corrective planes.FIG.3Cillustrates before correction cavus image310and after correction cavus image312generated by the system100using finite element analysis.FIG.3Calso illustrates the before correction Adductus image314and the after correction Adductus image316generated by the system100using finite element analysis.FIG.3DandFIG.3Eillustrates the before correction Varus image318and the after correction Varus image320(from two different points of view) generated by the system100using finite element analysis.

In an exemplary embodiment, the system100can determine the reference points for an adductus deformity based on the big toe305, little toe303, and ankle301as shown in a three-dimensional image of the adductus deformity302inFIG.3A. In an exemplary embodiment, the system100can use these reference points to generate the reference plane323as shown inFIG.3B. The system100can use the reference plane323to determine the force vectors to correct the adductus deformity314,316as shown inFIG.3C.

The system100can determine the reference points for correcting an equinus deformity based on the ankle311, heel313, mid plantar of foot315, and thigh307as shown in a three-dimensional image of the equinus deformity304as shown inFIG.3A. In an exemplary embodiment, the system100can use these reference points to generate the reference plane325to determine the force vectors to correct the equinus deformity as shown inFIG.3B. The system100can then determine the force vector or set of force vectors to correct an equinus deformity based on the reference plane327.

The system100can determine the reference points for correcting a cavus deformity based on the big toe319, middle of toes317, pinky toe321and heal315as shown in a three-dimensional image of the cavus deformity306inFIG.3A. The system100can determine the reference plane323based on these reference points as shown inFIG.3B. The system100can then determine the force vector or set of force vectors to correct the cavus deformity310,312as shown inFIG.3C.

The correspondingFIG.3Billustrates identification of four different planes using these reference points shown inFIG.3A. In an exemplary embodiment, system100can determine these reference points using image processing algorithm or a machine learning algorithm that recognizes features of the limb. In an exemplary embodiment, the machine learning model can be trained on deformed foot to identify these features.

With reference toFIGS.4A,4B and4C, the system100can be configured to determine the boundary conditions of the deformity based on a machine learning model. In an exemplary embodiment, the boundary conditions of the deformity can be the desired angles for corrected foot402.FIG.4Aillustrates the process of generating a final model of force vectors for correcting a deformity based on the boundary conditions and the 3d model of the deformity.FIG.4Billustrates the process of generating a solid model based on 3d scans.FIG.4Cillustrates the force vectors at the reference points determined by the system100in accordance with an exemplary embodiment described herein.

The system100can obtain a three-dimensional image or data404of the deformity. In exemplary embodiments, the system100can generate a three-dimensional model of the deformity either as a solid object or as a point cloud.FIG.4Billustrates a method of generating a three-dimensional solid object in a modelling software based on the images of the foot. The system100can convert the three-dimensional scan images of the deformity into an STL file414. The system100can then post-process the STL file414to fill in any missing information using a post-processing tool (e.g., Spacclaim). The system100can then convert the post-processed file into a solid three-dimensional object in the modelling software for further analysis.

Returning toFIG.4A, the system100can determine multiple separate planes for the deformity as shown in406to serve as a reference between the position of the deformity and the expected or normal mean position of the limb or other appendage. The system100can select the number of planes based on the geometry of the deformity, the degrees of freedom of the deformity, the deviation of the deformity from a statistical normal limb or appendage and the like. In an exemplary embodiment, the system100can determine the reference points as described above with reference toFIG.3A-3E. In an exemplary embodiment, the system100can select four different planes based on a machine learning model for club foot. In another example, the four planes can be selected with inputs from a doctor. The system100can use the machine learning trainer114to determine a four plane machine learning model that identifies the appropriate planes to use for correction.

For example, the machine learning trainer114can use data from a plurality of prior patients that includes planes that were selected for correction for the patients compared and the geometry of the deformity and the outcome of the corrective effort. The system100can then fit the three-dimensional data404of the deformity based on the trained four plane machine learning model. Once the four planes are identified the system100can use finite element analysis408to determine the force vectors for correcting the deformity in each plane.

In an exemplary embodiment, the system100can determine the force vectors at the reference points as illustrated inFIG.4AandFIG.4Cusing finite element analysis. The system100can determine the force vectors422as shown inFIG.4Cat the reference points for correcting the deformity. In an exemplary embodiment, the system100can apply a corrective force and determine the predicted correction such as the predicted angles for the corrected foot based on the applied force. The system100can then compare it with the boundary conditions such as desired angles for the corrected foot. The system100can iterate or simulate410for various force corrections then update the force vectors to reduce the error or minimize the error.

In an exemplary embodiment, the system100can use the boundary conditions402and the angle between the four planes and the boundary conditions402to determine the force vectors required during finite element analysis for correcting the deformity in each plane and generating for the next cast in the series of casts. In an exemplary embodiment the system100can determine the angle between the four planes using the modelling tools (e.g., Ansys, Matlab or both). Although theFIG.4Aillustrates the use of two modelling tools (e.g., Ansys and Matlab), to perform the various methods, in an exemplary embodiment the system100can use one or more modelling tools to perform the various methods.

For example, the system100can use the boundary conditions to iterate through a series of force vectors to minimize the error between the boundary condition and the results of applying a particular force vector in a particular plane. The system100can run a series of simulations using a trial correction and then determine the probable corrected deformity. The system100can as shown in theFIG.4Aiterates over a number of simulations until a force vector or a set of force vectors for the next series cast such as final model412is obtained. The system can determine the force vector or set of force vectors with the minimum deviation from the boundary condition using the iterative process. In an exemplary embodiment when the angle between the boundary conditions402and the probable corrected deformity is minimal the error is minimum. In an exemplary embodiment the system100can track the angle between the boundary condition and the predicted or estimated corrected plane if a force vector or set of force vectors is applied for each simulation in real-time412.

In an exemplary embodiment, the system100can generate data for split casts based on the final model412. In an exemplary embodiment, the split cast can include a portion that is not changed during at least a part of the series of casts and a portion that is updated during the next cast in the series of casts.

In exemplary embodiments, the system100can use the machine learning trainer114to determine the finite element analysis machine learning model. The finite element analysis machine learning model can be based force vectors that correspond to points in the point cloud determined from prior simulations for a plurality of prior patients. For example, the data for the plurality of prior patients can include the force vectors generated using simulations for a next cast in a series of casts, the boundary conditions used to arrive at the force vectors and the correction achieved as evident from the subsequent three-dimensional image of the deformity after it was corrected with the cast can be used to train the finite element analysis machine learning model. In exemplary embodiments, the finite element analysis machine learning model can generate a force vector or a set of force vectors for a cast or a series of casts based on the boundary conditions given manually or obtained from the finite element analysis machine learning model.

The system100can run supervised learning, unsupervised learning, reinforcement learning algorithms or any combination thereof. Examples of machine learning algorithms that can be implemented via the computing device112can include, but are not limited to Linear Regression, Logical Regression, Decision Tree, Support Vector Machine, Naïve Bayes, k-Nearest Neighbors, k-Means, Random Forest, Dimensionality Reduction algorithms such as GBM, XGBoost, LightGBM and CatBoost.

Examples of supervised learning algorithms that can be used in the computing device112can include regression, decision tree, random forest, k-Nearest Neighbors, Support Vector Machine, and Logic Regression. Examples of unsupervised learning algorithms that may be used in the computing device112include apriori algorithm and k-means. Examples of reinforcement learning algorithms that may be used in computing device112includes a Markov decision process.

Referring toFIG.5, the system100can apply finite element analysis manually. At step502, the system100can generate a three-dimensional scanned stereolighography (STL) file based on the three-dimensional image from the camera102. At step504, the STL file can be post processed to clean up any irregularities. For example, the system100can remove any imperfections in the STL file such as from motion during capture of the three-dimensional image using image processing algorithms. At step506, the system100can convert the three-dimensional image to a three-dimensional solid model. For example, the system100can use the multiple points present in the STL file and generate a solid shape of the deformity that are connected using extrapolations to generate a surface instead of multiple discrete points. At step508, the system100can create reference points for generating the planes for finite element analysis of the force vector or set of force vectors to be applied to the deformity to correct the deformity. In an exemplary embodiment, the system100can receive a selection of reference points for generating a correction plane from the doctor. In another example, a machine learning algorithm can select the reference points based on a trained machine learning model as described herein above. At step510, the system100can simulate the application of the force vector to the deformity and the effect of the force vector on the points of support for the deformity. At step512, the system100can calculate the deformation and the stresses when the force vector is applied. For example, the system100can determine the deformation of the deformity and the stresses on the deformity when the force vector or set of force vectors is applied via a cast. At step516, the system can calculate the deformation on the reference points for plane creation. For example, the system100can determine the deformation on the chosen reference points in the deformity to determine the effect of the force vectors on the deformity. At step514, the system100can calculate the angle between the planes of the selected reference place and the desired boundary condition502. At step518, the system can convert the generated final model data into a next cast in the series of casts. The system100can convert the final model into an STL file for three-dimensional printing.

Referring now toFIG.6A, the method of the present disclosure will now be described with reference to the flowchart. At step660, of process655, three-dimensional images of the deformity can be acquired. In an embodiment, the three-dimensional images of the deformity can be acquired by a mobile device of a parent. The three-dimensional imaging can include depth mapping of the foot of the patient. In an embodiment, the three-dimensional imaging can include ultrasound for the determination of internal biological structures of the foot. In an exemplary embodiment, the combination of the two above-described three-dimensional imaging modalities allows for improved cast planning by considering the internal structures in addition to the outward appearance.

At sub process665of process655and based upon the acquired three-dimensional images of the patient anatomy, casting stages of patient anatomy movement can be predicted. The casting stages can be predicted via the force vector modeling described herein above. In one instance, this prediction can include computer predictive modeling and finite element analysis of the foot wherein stresses, deformations of the structures of the foot or both are considered from one stage to the next. Sub process665will be further described with reference toFIG.6B. At step670of process655, virtual models of three-dimensional casts can be generated for anticipated patient anatomy movements at each predicted stage. Such virtual models of three-dimensional casts can allow for visualization and modification according to real-world constraints. At step675, of process650, three-dimensional casts, similar to that ofFIG.5, can be generated for each virtual model at each predicted stage of patient anatomy movement.

With reference toFIG.6B, sub process665of process655includes determining the number of stages and trajectory of each predicted stage of movement. At step666, of sub process665, a baseline patient anatomy can be established according to the acquired images of the patient anatomy. Accordingly, at step668of sub process665, a target patient anatomy can be selected, the target being an end goal shape of the structure of the foot.

At step667of sub process665, the patient anatomy movement at each stage can be determined. This determination can include movements of structures of the foot. In an embodiment, such movements can be determined in the context of the Ponseti stages and include, for instance, performing specific angular rotations at specific stages. In an embodiment, such movements can be optimized at each stage such that maximum movement is achieved without creating undue mechanical and/or biological stresses. For instances, each stage may be determined such that von Mises stress, for instance, remain below a threshold value.

At step669, of sub process665, the current position of the patient anatomy can be compared with the target patient anatomy position from step668. If the two values are equal, for instance, only a single stage of casting may be required and the determined patient anatomy movement can be used to generate a virtual model of a necessary three-dimensional cast at step670. If, however, the current position and the target patient anatomy do not match, a successive stage of patient anatomy movement is required and the sub process665returns to step667.

According to an embodiment, in this way, the number of stages, or cast, required to be fitted to a patient is dependent upon the severity of the deformity and the ability to move the patient anatomy at each stage. In the case of clubfoot, this can mean the difference of manufacturing four casts in one instance and six casts in another, thereby allowing each patient to receive only the minimum necessary number of casts.

According to an embodiment, the above described method ofFIG.6AandFIG.6Bcan be performed with only external features gathered via, for instance, depth mapping data. External features can be processed similarly to Schoenecker, et al., Systems and methods for serial treatment of a muscular-skeletal deformity, U.S. Patent Application Publication No. US2017/0091411 A1, incorporated herein by reference.

According to an embodiment, the external features can be applied to a machine learning algorithm in order to generate patient anatomy predictions without need for ultrasound imaging. For instance, a library of corresponding images of a foot may be stored.

The corresponding images can include images of the external features of the foot and corresponding images of the internal features of the foot. In this way, the machine learning algorithm, a convolutional neural network in exemplary embodiment, can be trained to correlate external features with internal features. Therefore, when provided with an external feature of an unknown foot, the machine learning algorithm can generate a corresponding internal feature structure that can be used in determining patient anatomy movements during stage planning. The library of corresponding images can be a corpus of patient data acquired from patients of a similar diagnosis and healthy patients.

FIG.7is a block diagram of an exemplary embodiment of computing device112in accordance with embodiments of the present disclosure. The computing device112can include one or more non-transitory computer-readable media for storing one or more computer-executable instructions or software for implementing exemplary embodiments. The non-transitory computer-readable media can include, but are not limited to, one or more types of hardware memory, non-transitory tangible media (for example, one or more magnetic storage disks, one or more optical disks, one or more flash drives), and the like. For example, memory119included in the computing device112can store computer-readable and computer-executable instructions or software for performing the operations disclosed herein. For example, the memory119can store a software application640which is configured to perform several of the disclosed operations (e.g., the pre-training platform for determining the co-occurrence matrix, the training platform for determining the word vectors and the topic determination platform for determining the plurality of topics and the representative noun). The computing device610can also include configurable, programmable processor120or both and an associated core(s)614, and optionally, one or more additional configurable, programmable processing devices or both, e.g., processor(s)612′ and associated core(s)614′ (for example, in the case of computational devices having multiple processors/cores), for executing computer-readable and computer-executable instructions or software application640stored in the memory119and other programs for controlling system hardware. Processor120and processor(s)612′ can each be a single-core processor or multiple core (614and614′) processor.

Virtualization can be employed in the computing device610so that infrastructure and resources in the computing device can be shared dynamically. A virtual machine624can be provided to handle a process running on multiple processors so that the process appears to be using only one computing resource rather than multiple computing resources. Multiple virtual machines can also be used with one processor.

Memory119can include a computational device memory or random access memory, such as DRAM, SRAM, EDO RAM, and the like. Memory119can include other types of memory as well, or combinations thereof.

A user can interact with the computing device710(shown inFIG.1as112) through a visual display device701, such as a computer monitor, which can display one or more user interfaces742that can be provided in accordance with exemplary embodiments. The computing device710can include other I/O devices for receiving input from a user, for example, a keyboard or any suitable multi-point touch interface718, a pointing device720(e.g., a mouse). The keyboard and the pointing device720can be coupled to the visual display device701. The computing device710can include other suitable conventional I/O peripherals.

The computing device710can also include one or more storage devices such as a hard-drive, CD-ROM, or other computer readable media, for storing data and computer-readable instructions, software that perform operations disclosed herein or both. Exemplary storage device734can also store one or more databases for storing any suitable information required to implement exemplary embodiments. The databases can be updated manually or automatically at any suitable time to add, delete, update one or more items in the databases.

The computing device710can include a communication device744configured to interface via one or more network devices732with one or more networks, for example, Local Area Network (LAN), Wide Area Network (WAN) or the Internet through a variety of connections including, but not limited to, standard telephone lines, LAN or WAN links (for example, 802.11, T1, T3, 56 kb, X.25), broadband connections (for example, ISDN, Frame Relay, ATM), wireless connections, controller area network (CAN), or some combination of any or all of the above. The communication device744can include a built-in network adapter, network interface card, PCMCIA network card, card bus network adapter, wireless network adapter, USB network adapter, modem, radio frequency transceiver, or any other device suitable for interfacing the computing device710to any type of network capable of communication and performing the operations described herein. Moreover, the computing device710can be any computational device, such as a workstation, desktop computer, server, laptop, handheld computer, tablet computer, or other form of computing or telecommunications device that is capable of communication and that has sufficient processor power and memory capacity to perform the operations described herein.

The computing device710can run any operating system726, such as any of the versions of the Microsoft® Windows® operating systems, the different releases of the Unix and Linux operating systems, any version of the MacOS® for Macintosh computers, any embedded operating system, any real-time operating system, any open source operating system, any proprietary operating system, or any other operating system capable of running on the computing device and performing the operations described herein. In exemplary embodiments, the operating system726can be run in native mode or emulated mode. In an exemplary embodiment, the operating system726can be run on one or more cloud machine instances.