Patent Description:
A fluid pump, such as an infusion pump, may be used to administer therapy to a patient by delivering nutrients, medications, blood products, or other substance to the patient. Many types of clinical treatments, such as pain management, blood glucose regulation, and chemotherapy, may require the delivery of precise volumes of fluids to a patient's circulatory system or epidural space via, for example, intravenous infusion, subcutaneous infusion, arterial infusion, epidural infusion, and/or the like. A peristaltic pump (or roller pump) is one example of a fluid pump capable of delivering precise volumes of fluids. For example, a peristaltic pump may be configured to deliver, continuously or intermittently, precisely measured doses of a fluid from a reservoir. The pumping mechanism in the peristaltic pump may include a combination of pumping fingers and occluding fingers that operate in tandem to apply pressure to sequential locations in a tubing (or other conduit) in fluid communication with the reservoir in order to drive the fluid from the reservoir to a patient.

From <CIT> systems and methods are known for anomaly detection of a medical procedure based on obtained images and a trained machine learning model.

Systems and infusion pumps are provided for detecting an improperly loaded infusion set (e.g., misload) on at an infusion pump. The scope of the invention is defined by the independent claims. Preferred embodiments are depicted in the dependent claims, the description and the accompanying Figures. For example, a pump controller may apply a machine learning model trained to detect, based on one or more images of an infusion pump loaded with an intravenous (IV) set, a misload of the intravenous set. The images of the pump loaded with the intravenous set may be captured by one or more cameras having a field of view (FOV) that includes the portion of the pump with the loaded intravenous set. The machine learning model may be trained using images of correctly loaded intravenous sets and is therefore able to determine when the images of the pump loaded with the intravenous set exhibit one or more nonconformities. When the output of the machine learning model indicates a misload of the intravenous set at the infusion pump, the pump controller may perform one or more corrective actions. For instance, the pump controller may prevent the infusion pump from performing an infusion when a misload of the intravenous set is detected at the pump. Furthermore, the pump controller may generate a message indicating a misload of the intravenous set at the infusion pump. In some cases, the message may identify the type and location of the misload at the infusion pump.

In one aspect, there is provided a system that includes at least one processor and at least one memory. The at least one memory includes program code that provides operations when executed by the at least one processor. The operations include: receiving, from a camera at an infusion pump, one or more images of an area of interest of the infusion pump loaded with an infusion set; identifying, within the one or more images, a first component of the infusion pump and a second component of the infusion set in the area of interest, wherein the first component comprises a bezel, a membrane seal, a door, a platen, a locator feature, or an air-in-line detector, and wherein the second component comprises an upper fitment, a lower fitment, or a pumping segment of a tubing; applying a machine learning model trained to detect one or more nonconformities present in the one or more images of the infusion pump loaded with the infusion set based at least on a relative position of the first component and the second component and by at least comparing, to one or more images of a correctly loaded infusion set, the one or more images of the infusion pump loaded with the infusion set; and in response to an output of the machine learning model indicating a presence of a nonconformity in the one or more images of the infusion pump loaded with the infusion set, performing a corrective action.

In some variations, one or more features disclosed herein including the following features can optionally be included in any feasible combination.

In some variations, the first component and the second component may be identified by applying an edge detection technique.

In some variations, the edge detection technique may include a Laplace-Gaussian transformation and/or a Canny edge detection.

In some variations, the first component and the second component may be identified based on a graphical feature disposed on each one of the first component and the second component.

In some variations, the graphical feature may include a pattern, a barcode, and/or an April tag.

In some variations, the one or more nonconformities may include a misload of the infusion set. The misload of the infusion set may include a misplacement of an upper fitment of the infusion set, a lower fitment of the infusion set, and/or a pumping segment of a tubing of the infusion set.

In some variations, the one or more nonconformities may include the intravenous set being reused, past an expiration date, an incorrect type, or a counterfeit product.

In some variations, the one or more nonconformities may include one or more components of the infusion pump and/or the intravenous set being missing.

In some variations, the one or more nonconformities may include one or more components of the infusion pump and/or the intravenous set being damaged.

In some variations, the one or more nonconformities may include a presence of contaminants in the infusion pump and/or the intravenous set.

In some variations, the camera may be mounted to a door of the infusion pump. The camera may be mounted in a location where the camera has a field of view that includes at least a portion of the infusion pump loaded with the intravenous set.

In some variations, the field of view of the camera may exclude one or more areas surveillance is unsuitable, prohibited, or unnecessary.

In some variations, the one or more images may include a first image captured while the door is in an open position and a second image captured while the door is in a partially open position. The one or more nonconformities may be detected based on the first image and the second image.

In some variations, the camera may include a visible light camera, an infrared camera, and/or an ultraviolet camera.

In some variations, the one or more nonconformities may be detected based on a graphical feature that is detectable under visible light, infrared light, or ultraviolet light.

In some variations, the corrective action may include preventing the infusion pump from performing an infusion.

In some variations, the corrective action may include generating a message identifying the one or more nonconformities.

In another aspect, there is provided an infusion pump including a bezel, a camera, and a controller. The bezel includes one or more locator features for receiving an intravenous set. The camera is mounted to a door of the infusion pump. The camera is mounted in a location where the camera has a field of view that includes at least a portion of the infusion pump loaded with the intravenous set. The camera is configured to capture one or more images of the infusion pump loaded with the intravenous set. The controller includes at least data processor and at least one memory storing instructions which, when executed by the at least one data processor, cause the controller to perform operations. The operations include: identifying, within the one or more images, a first component of the infusion pump and a second component of the infusion set within the area of interest, the first component comprising the bezel, the door, the one or more locator features, a membrane seal, a platen, or an air-in-line detector, and wherein the second component comprises an upper fitment, a lower fitment, or a pumping segment of a tubing; applying a machine learning model trained to detect one or more nonconformities present in the one or more images of the infusion pump loaded with the infusion set based at least on a relative position of the first component and the second component and by at least comparing, to one or more images of a correctly loaded infusion set, the one or more images of the infusion pump loaded with the infusion set; and in response to an output of the machine learning model indicating a presence of a nonconformity in the one or more images of the infusion pump loaded with the infusion set, performing a corrective action.

In some variations, the first component and the second component are identified by applying an edge detection technique.

In some variations, the field of view of the camera may further exclude one or more areas surveillance is unsuitable, prohibited, or unnecessary.

By means of example, not part of the claimed invention, methods consistent with the descriptions provided herein as well as articles that comprise a tangibly embodied machine-readable medium operable to cause one or more machines (e.g., computers, etc.) to result in operations implementing one or more of the described features are described. Similarly, computer systems are also described that may include one or more processors and one or more memories coupled to the one or more processors. A memory, which can include a non-transitory computer-readable or machine-readable storage medium, may include, encode, store, or the like one or more programs that cause one or more processors to perform one or more of the operations described herein. Computer implemented methods consistent with one or more implementations of the current subject matter can be implemented by one or more data processors residing in a single computing system or multiple computing systems. Such multiple computing systems can be connected and can exchange data and/or commands or other instructions or the like via one or more connections, including, for example, to a connection over a network (e.g. the Internet, a wireless wide area network, a local area network, a wide area network, a personal area network, a peer-to-peer network, a personal area network, a peer-to-peer network, a mesh network, a wired network, or the like), via a direct connection between one or more of the multiple computing systems, etc..

While certain features of the currently disclosed subject matter are described for illustrative purposes in relation to detecting a tubing misload at an infusion pump, it should be readily understood that such features are not intended to be limiting. The claims that follow this disclosure are intended to define the scope of the protected subject matter.

An infusion pump, such as a peristaltic pump, may fail to operate correctly or as expected (e.g., according to the delivery parameters configuring the device such as rate or volume to be infused) if an intravenous (IV) set is not correctly loaded in the pump. Misloads can occur when the tubing of the intravenous set is placed in an incorrect position relative to various components of the pump including, for example, the pump's pumping mechanism. For example, the proper operation of the pump may require the tubing to be stretched and centered relative to a pumping finger. A misload in which the tubing is not stretched a sufficient amount may prevent the pumping fingers from closing properly against the tubing and cause an uncontrolled flow of fluid from the pump. Meanwhile, a misload that causes an excessive stretching of the tubing may also cause an uncontrolled flow of fluid from the pump. For instance, overstretching the tubing may reduce the thickness of the walls of the tubing and prevent the occluding fingers from closing off the tubing to stop the flow of fluid.

In some cases, a misload of the intravenous set may also introduce inaccuracies in the flow rate of the pump by changing the cross-sectional dimensions of the tubing. For example, the tubing may be twisted along the longitudinal axis. Alternatively, the inner diameter of the tubing may narrow when the upper fitment of the intravenous set is not placed on the base of the pocket in the pump bezel but is placed above the pocket in the pump bezel. In the event the infusion pump includes a platen, either as a part of the door of the pump or as a separate component, a misload may occur if a foreign object, such as a misplaced portion of the intravenous set, is present in an area reserved for the pumping segment of the tubing. In addition to discrepancies in the flow rate of the pump, the presence of the foreign object may cause breakages in the intravenous set and the pump, such as the platen, when excessive force is used to closed the door of the pump.

Any of the aforementioned types of anomalous conditions may cause an infusion pump to deliver incorrect quantities of fluids or malfunction. As such, it may be desirable to detect a misloaded intravenous set at the infusion pump before the infusion commences. However, conventional approaches to detecting a misloaded intravenous set that rely on electromechanical switches pose a number of limitations. For example, electromechanical switches are able to detect nonconformities at a limited quantity of locations within the infusion pump while the flexibility of the tubing in the intravenous set lends multiple degrees of freedom to the placement of the intravenous set. As such, conventional misload detection approaches that rely on electromechanical switches may be neither reliable nor practical. For instance, electromechanical switches may be prone to false negatives as well as false positives. Moreover, the inclusion of electromechanical switches in the infusion pump may introduce complexities in the housing and wiring of the infusion pump. The cavities needed to accommodate the electromechanical switches, for example, may be susceptible to the ingress of contaminants but difficult to clean.

In some example embodiments, a pump controller may be configured to detect a misloaded intravenous set based on one or more images of a pump loaded with an intravenous set. For example, the pump controller may apply a machine learning model that has been trained based on images of correctly loaded intravenous sets to determine when the images of the pump loaded with the intravenous set exhibit one or more nonconformities. When the output of the machine learning model indicates a misload of the intravenous set at the infusion pump, the pump controller may perform one or more corrective actions. For instance, the pump controller may prevent the infusion pump from performing an infusion when a misload of the intravenous set is detected at the pump. Alternatively and/or additionally, the pump controller may generate a message indicating a misload of the intravenous set at the infusion pump. In some cases, the message may provide instructions for correcting the misload including by identifying the type and location of the misload at the infusion pump.

<FIG> depicts a system diagram illustrating an example of an infusion system <NUM>, in accordance with some example embodiments. Referring to <FIG>, the infusion system <NUM> may include a pump controller <NUM>, a pump <NUM>, and a client device <NUM>. As shown in <FIG>, the pump controller <NUM>, the pump <NUM> and the client device <NUM> may be communicatively coupled via a network <NUM>. The client device <NUM> may be a processor-based device including, for example, a point of care unit (PCU), a smartphone, a tablet computer, a wearable apparatus, a desktop computer, a laptop computer, a workstation, and/or the like. Meanwhile, the network <NUM> may be a wired and/or wireless network including, for example, a public land mobile network (PLMN), a local area network (LAN), a virtual local area network (VLAN), a wide area network (WAN), the Internet, and/or the like.

<FIG> depict various views of an example of the pump <NUM> with a door <NUM> open. As shown in <FIG>, the pump <NUM> may be loaded with an intravenous (IV) set <NUM>. The pump <NUM> may act directly on a pump segment <NUM> of the intravenous set <NUM>, which connects an upstream fluid line to a downstream fluid line to form a continuous fluid conduit between a fluid reservoir and a patient. For example, the pump <NUM> may include a pumping mechanism <NUM>, which may act as a flow control device moving fluid through the conduit downstream to the patient. The pump segment <NUM>, the upstream fluid line, and/or the downstream fluid line may be coupled to a pump cassette or cartridge that is configured to be coupled to the pump <NUM>.

Although the type of the pumping mechanism <NUM> may vary, the example of the pumping mechanism <NUM> shown in <FIG> is a multi-finger pumping mechanism that includes an upstream occluding finger, a primary pumping finger, a downstream occluding finger, and a secondary pumping finger. The pumping mechanism <NUM> may therefore operate by the pumping fingers and the occluding fingers alternately applying pressure on the pump segment <NUM> of the fluid conduit. The pressure may be applied at sequential locations in the pump segment <NUM>, beginning at the upstream end of the pumping mechanism <NUM> and continuing through the downstream end of the pumping mechanism <NUM>. At any one point in time, at least one of the fingers of the pumping mechanism <NUM> may apply a pressure that is sufficient to occlude the fluid conduit. Moreover, at least one of the fingers of the pumping mechanism <NUM> may not retract from occluding the pump segment <NUM> until a subsequent finger in the sequence has already occluded the pump segment <NUM>. Accordingly, at no time during the operation of the pump mechanism <NUM> is there a direct fluid path from the reservoir to the patient.

In some example embodiments, the pump controller <NUM> may be configured to detect a misload of the intravenous set <NUM> based at least on one or more images of the pump <NUM> loaded with the intravenous set <NUM>. For example, the pump controller <NUM> may include a machine learning engine <NUM> configured to determine whether the images of the pump <NUM> loaded with the intravenous set <NUM> exhibit one or more nonconformities. Moreover, when the output of the machine learning engine <NUM> indicates a misload of the intravenous set <NUM>, the pump controller <NUM> may perform one or more corrective actions. For instance, the pump controller <NUM> may prevent the pump <NUM> from performing an infusion when a misload of the intravenous set <NUM> is detected at the pump <NUM>. Alternatively and/or additionally, the pump controller <NUM> may generate a message indicating a misload of the intravenous set <NUM> at the pump <NUM>. In some cases, the message may provide instructions for correcting the misload including by identifying the type and location of the misload at the pump <NUM>.

The machine learning engine <NUM> may be include one or more machine learning models that have been trained based on images of correctly loaded intravenous sets. For example, the machine learning engine <NUM> may be implemented using one or more general purpose central processing units (CPUs) and/or machine learning hardware accelerators. The one or more machine learning models may be trained to perform a variety of machine vision tasks including, for example, image enhancement, morphological filtering, blob detection, feature extraction, image segmentation, edge detection, object classification, optical character recognition (OCR), and/or the like. For instance, the one or more machine learning models may be trained based on a training set of reference images depicting correctly loaded intravenous set and/or incorrectly loaded intravenous sets. Moreover, the one or more machine learning models may be update based on data collected from real world clinical settings including, for example, images of intravenous set that clinicians identify as being correctly loaded. Examples of such machine learning models include a neural network, a regression model, an instance-based model, a regularization model, a decision tree, a random forest, a Bayesian model, a clustering model, an associative model, a dimensionality reduction model, and/or an ensemble model.

In some example embodiments, the pump <NUM> may include a camera <NUM> configured to generate one or more images of the pump <NUM> loaded with the intravenous set <NUM>. It should be appreciated that the camera <NUM> may include a variety of image sensors including, for example, a charge-coupled device (CCD), an active-pixel sensor (or complementary metal-oxide-semiconductor (CMOS) sensor), and/or the like. Moreover, the camera <NUM> may be configured to detect wavelengths of light that occupy different portions of the electromagnetic spectrum including, for example, visible light (<NUM>-<NUM> nanometers), infrared light (<NUM>-<NUM>,<NUM> nanometers), ultraviolet light (<NUM>-<NUM> nanometers), and/or the like.

The camera <NUM> may be configured to have a minimum field of view (FOV) such as, for example, <NUM>° or the like. Moreover, the camera <NUM> may be mounted to in a location such that the field of view (FOV) of the camera <NUM> includes one or more areas of interest. In the example shown in <FIG>, the camera <NUM> may be mounted to the door <NUM>, for example, near the edge of the door <NUM>, where the field of view of the camera <NUM> includes at least a portion of the platen <NUM> and the intravenous set <NUM> loaded in the pump <NUM> (e.g., the portion of the intravenous set <NUM> between an upper fitment <NUM> and a lower fitment <NUM>) when the door <NUM> of the pump <NUM> is in an open position. According to some example embodiments, the field of view of the camera <NUM> may be limited to exclude certain areas including, for example, areas where surveillance is unsuitable, prohibited, or unnecessary.

To further illustrate, <FIG> depicts a block diagram illustrating an example of a workflow associated with the infusion system <NUM>, in accordance with some example embodiments. As shown in <FIG>, the machine learning engine <NUM> may receive, from the camera <NUM>, one or more images of the pump <NUM> loaded with the intravenous set <NUM>. The camera <NUM> may be configured to capture images, for example, periodically, while the door <NUM> of the pump <NUM> is in an open position (or partially open position). The machine learning engine <NUM> generate an output indicating whether the one or more images of the pump <NUM> loaded with the intravenous set <NUM> exhibits one or more nonconformities including, for example, a misload of the intravenous set <NUM>.

As shown in <FIG>, the pump controller <NUM> may perform, based at least on the output of the machine learning engine <NUM>, one or more corresponding actions. For example, in the event the output of the machine learning engine <NUM> indicates that the images of the pump <NUM> loaded with the intravenous set <NUM> do not exhibit any nonconformities, the pump controller <NUM> may allow the pump <NUM> to perform an infusion and generate a corresponding output (e.g., illuminate a first colored light at the pump <NUM>). Alternatively, if the output of the machine learning engine <NUM> indicates the presence of a nonconformity in the images of the pump <NUM> loaded with the intravenous set <NUM>, the pump controller <NUM> may prevent the pump <NUM> from performing an infusion.

In the example of the workflow shown in <FIG>, the pump controller <NUM> may interact with a central processing unit <NUM> at a point of care unit (PCU) <NUM> in order to trigger an alarm <NUM>. The alarm <NUM> may include a message indicating a misload of the intravenous set <NUM> at the pump <NUM>. Furthermore, in some cases, the message may provide instructions for correcting the misload including by identifying the type and location of the misload at the pump <NUM>. For instance, the message may include an image within one or more indicators of the type and location of the misload <NUM>. In the event the machine learning engine <NUM> is unable to generate an output based on the images captured by the camera <NUM>, for example, due to various errors at the camera <NUM> (e.g., the door <NUM> not being in a sufficiently open position, poor lighting conditions, obstruction by a foreign object, and/or the like), the pump controller <NUM> may also generate a message with instructions to correct the error.

In some example embodiments, the pump controller <NUM> may track the occurrence of nonconformities, such as misloaded intravenous sets, for individual clinicians and groups of clinicians. If a threshold quantity of nonconformities are detected while a clinician is loading the pump <NUM>, for example, the pump controller <NUM> may generate a message for another clinician to provide assistance with the loading of the pump <NUM>. Alternatively and/or additionally, in the event nonconformities occur at an above threshold frequency, the pump controller <NUM> may generate a recommendation for additional training, for example, for an individual clinician or a group of clinicians.

In some example embodiments, the machine learning engine <NUM> may be configured to detect nonconformities that are present in multiple images captured by the camera <NUM>. For example, the camera <NUM> may capture a series of images while the door <NUM> of the pump <NUM> is being moved from an open position to a closed position including, for example, a first image of the pump <NUM> loaded with the intravenous set <NUM> while the door <NUM> is in an open position and a second image of the pump <NUM> loaded with the intravenous set <NUM> while the door <NUM> is in a partially open (or closed) position. Moreover, the machine learning engine <NUM> may determine, based at least on the first image and the second image, whether a nonconformity is present in the loading of the intravenous set <NUM>. For instance, the first image taken while the door <NUM> is in the open position may permit an analysis of the placement of the upper fitment <NUM> and the lower fitment <NUM> but how the platen <NUM> engages with the pumping mechanism <NUM> and a pumping segment <NUM> of a tubing of the intravenous set <NUM> may not be discernable from the first image. As such, the machine learning engine <NUM> may analyze the first image and the second image in order to detect nonconformities associated with the upper fitment <NUM>, the lower fitment <NUM>, and the platen <NUM>.

The machine learning engine <NUM> may be configured to detect a variety of nonconformities present in the images of the pump <NUM> loaded with the intravenous set <NUM>. A misload of the intravenous set <NUM> in which one or more portions of the intravenous set <NUM> are placed incorrectly in the pump <NUM>, such as the misplacement of the upper fitment <NUM>, the pumping segment <NUM>, and/or the lower fitment <NUM>, is one example of a nonconformity. <FIG> depict one example of a misload in which the upper fitment <NUM> of the intravenous set <NUM> is not in a correct position at the base of a first locator feature 202a (e.g., a pocket) in the bezel of the pump <NUM> but is placed above the first locator feature 202a. When the upper fitment <NUM> of the intravenous set <NUM> is displaced in this manner, the tubing of the intravenous set <NUM>, for example, the pumping segment <NUM>, may stretch to a narrower inner diameter. This change in the cross-sectional dimensions of the tubing may introduce inaccuracies in the flow rate of the pump <NUM> and cause errors such as under infusion or over infusion.

<FIG> depicts another example of a misload in which the upper fitment <NUM> of the intravenous set is placed below the first locator feature 202a. As shown in <FIG>, this type of misload may cause the pumping segment <NUM> of the intravenous set <NUM> to deform and not engage properly with the pumping mechanism <NUM> of the pump <NUM>. The warped portion of the pumping segment <NUM> may also prevent the platen <NUM> from engaging with the pumping mechanism <NUM> of the pump <NUM> when the door <NUM> is in the closed position. Breakages may occur in the intravenous set <NUM> and the pump <NUM> if excessive force is used to close the door <NUM> against the warped section of the pumping segment <NUM>.

Other examples of misloads may include the lower fitment <NUM> of the intravenous set <NUM> not being placed in a correct position relative to a second locator feature 202b, an open flow stop, the tubing of the intravenous set <NUM> not engaged with an air-in-line detector <NUM> of the pump <NUM>, and the presence of foreign objects in the pumping mechanism <NUM>. In some cases, the misload may be the intravenous set <NUM> being of an incorrect type, a counterfeit product, or an expired set. For example, the machine learning engine <NUM> may be trained to identify and differentiate between a standard set and an epidural set, in which case the machine learning engine <NUM> may detect a nonconformity when the intravenous set <NUM> is a standard set but the clinical procedure being performed requires an epidural set instead. The type of the intravenous set <NUM>, the expiration date for the intravenous set <NUM>, and whether the intravenous set <NUM> is a genuine product may be detected based on graphical features on the intravenous set <NUM> (e.g., patterns, barcodes, and/or the like), which may be detectable under visible light or under special lighting conditions (e.g., ultraviolet light). For instance, the barcode (or other unique identifier) associated with the intravenous set <NUM> may be used to query a database and determine whether the intravenous set <NUM> is a genuine product. In some cases, the database may track the usage of the intravenous set <NUM>, in which case the database may be queried to detect when the intravenous set <NUM> is being used impermissibly when the intravenous set <NUM> has already being used and/or is past its expiration date.

In addition to misloads of the intravenous set <NUM>, the machine learning engine <NUM> may be also be trained to identify other nonconformities such as the intravenous set <NUM> not being primed properly of air, a missing component in the pump <NUM> and/or the intravenous set <NUM>, a damaged component in the pump <NUM> and/or the intravenous set <NUM>, and the presence of contaminants in the pump <NUM> and/or the intravenous set <NUM>. Examples of missing or damaged components in the intravenous set <NUM> may include the upper fitment <NUM>, the pumping segment <NUM> of the tubing, and the lower fitment <NUM>. Examples of missing or damaged components in the pump <NUM> may include the bezel, the membrane seal, one or more of the locator features204a and 204b, the door <NUM>, the platen <NUM>, the pumping mechanism <NUM>, and the air-in-line detector <NUM>.

As noted, the machine learning engine <NUM> may include one or more machine learning models that have been trained, based on images of correctly loaded intravenous sets, to detect a misload of the intravenous set <NUM>. To do so, the machine learning engine <NUM> may first enhance the raw images captured by the camera <NUM> including by performing edge detection to determine the relative positions of the various components of the pump <NUM> and the intravenous set <NUM> depicted in the images captured by the camera <NUM>. A Laplace-Gaussian transformation is one example edge detection technique that the machine learning engine <NUM> may apply to identify the edges that are present in the images of the pump <NUM> loaded with the intravenous set <NUM> and generate corresponding line segments. Canny edge detection is another edge detection technique that detects the edges present in the images of the pump <NUM> loaded with the intravenous set <NUM> by subjecting each image to noise reduction, gradient calculation (e.g., Gaussian blur), non-maximum suppression, double thresholding, and edge tracking by hysteresis. The enhanced images may be used for one or more downstream machine vision tasks such as feature extraction, object classification, and optical character recognition. For example, the machine learning engine <NUM> may compare an enhanced image of the pump <NUM> loaded with the intravenous set <NUM> with one or more reference images depicting correctly loaded intravenous sets to determine whether the images of the pump <NUM> exhibits one or more nonconformities. The various components identified within the images of the pump <NUM> loaded with the intravenous set <NUM> may serve as reference points for determining the spatial relationship between the various components. For instance, the machine learning engine <NUM> may identify the top fitment <NUM> of the intravenous set <NUM> as a first reference point and the first locator feature 202a as a second reference point in order to determine the spatial relationship between the top fitment <NUM> and the first locator feature 202a. In cases where at least some of the images captured by the camera <NUM> are stored, data privacy and storage efficiency may be maximized by the machine learning engine <NUM> storing a simplified representation of the images such as, for example, a vector of the coordinates of the various components present in the images.

To further illustrate, <FIG> depicts a raw image illustrating a correctly loaded intravenous set <NUM> with the top fitment <NUM> of the intravenous set <NUM> positioned at the base of the first locator feature 202a (e.g., a pocket) in the bezel of the pump <NUM>. <FIG> depicts an enhanced image that is generated by the machine learning engine <NUM> applying an edge detection technique to identify the position of various components of the pump <NUM> and the intravenous set <NUM> including the top fitment <NUM> and the first locator feature 202a.

<FIG> depicts a raw image illustrating an incorrectly loaded intravenous set <NUM> with the top fitment <NUM> of the intravenous set <NUM> positioned below the first locator feature 202a (e.g., the pocket) in the bezel of the pump <NUM> and in an area of the pump <NUM> occupied by the upper pressure sensor. <FIG> depicts an enhanced image generated by the machine learning engine <NUM> applying an edge detection technique to identify the position of various components of the pump <NUM> and the intravenous set <NUM> including the top fitment <NUM> and the first locator feature 202a.

<FIG> depicts a raw image illustrating the intravenous set <NUM> loaded correctly in the pump <NUM> with the lower fitment <NUM> positioned in a second locator feature 202b and the flow stop in a closed position. The raw image depicted in <FIG> further shows the platen <NUM> (and various features thereon) such that the position of the platen <NUM> may be analyzed relative to that of the pump <NUM> (and various components therein) to ensure that the platen <NUM> will engage properly with the pump <NUM> and the loaded intravenous set <NUM> when the door <NUM> of the pump <NUM> is in a closed position. <FIG> depicts an enhanced image generated by the machine learning engine <NUM> applying an edge detection technique to identify the position of various components of the pump <NUM> and the intravenous set <NUM> including the platen <NUM>, the bottom fitment <NUM>, and the second locator feature 202b.

<FIG> depicts a raw image illustrating the intravenous set <NUM> loaded incorrectly in the pump <NUM> with the lower fitment <NUM> placed backwards in the cavity of the pump <NUM>. An enhanced version of the raw image shown in <FIG> is depicted in <FIG>. The enhanced image shown in <FIG> may be generated by the machine learning engine <NUM> applying an edge detection technique to identify the position of various components of the pump <NUM> and the intravenous set <NUM> including the platen <NUM>, the bottom fitment <NUM>, and the second locator feature 202b.

In some example embodiments, to identity various components of the pump <NUM> and the intravenous set <NUM>, the machine learning engine <NUM> may be configured to recognize one or more graphical features such as barcodes, April Tags, and/or the like. <FIG> depict the pump <NUM> with various examples of a graphical feature <NUM>, which may be placed on different components of the pump <NUM> and the intravenous set <NUM> to provide reference points for determining the spatial relationships between, for example, the upper fitment <NUM>, the platen <NUM>, and/or the like. It should be appreciated that the graphical feature <NUM> may enable a <NUM> degree-of-freedom localization of the components that are present in the image. For example, the location and orientation of the graphical feature <NUM> on the upper fitment <NUM> relative to the location and orientation of the graphical feature <NUM> on the first locator feature 202a may indicate whether the upper fitment <NUM> is in a correct position relative to the first locator feature 202a. Where the graphical feature <NUM> is on one side (or surface) of a component such as the upper fitment <NUM>, the quantity of the graphical feature <NUM> that is visible (or obscured) in the image may indicate whether, for example, the upper fitment <NUM> is placed in a correct orientation or is rotated in an incorrect position.

In some cases, the position of one or more the graphical features <NUM> in an image of the pump <NUM> loaded with the intravenous set <NUM> may also be used to correct for various forms of distortions that may be present in the image. For example, the image may exhibit a barrel distortion if the image is captured by the camera <NUM> when the camera <NUM> is too close to the intravenous set <NUM>. The relative locations of the graphical features <NUM> in the image may be used to correct for the barrel distortion and other types of distortions that may be present in the image.

<FIG> depicts a flowchart illustrating an example of a process <NUM> for machine learning enabled misload detection. Referring to <FIG> and <FIG>, the process <NUM> may be performed by the pump controller <NUM> to detect and respond to a variety of anomalous conditions at the pump <NUM>.

At <NUM>, the pump controller <NUM> may receive one or more images of a pump loaded with an intravenous set. In some example embodiments, the pump controller <NUM> may receive, from the camera <NUM>, one or more images of the pump <NUM> loaded with the intravenous set <NUM>. The camera <NUM> may be activated by the door <NUM> of the pump <NUM> being moved to an open position. The camera <NUM> may be further configured to capture images, for example, periodically, while the door <NUM> of the pump <NUM> remains in open position. As noted, the camera <NUM> may be configured to have a minimum field of view (FOV) such as, for example, <NUM>° or the like. Moreover, the camera <NUM> may be mounted to in a location such that the field of view (FOV) of the camera <NUM> includes one or more areas of interest such as, for example, at least a portion of the platen <NUM> and the intravenous set <NUM> loaded in the pump <NUM> (e.g., the portion of the intravenous set <NUM> between an upper fitment <NUM> and a lower fitment <NUM>), when the door <NUM> of the pump <NUM> is in an open position. In some cases, the field of view of the camera <NUM> may be limited to exclude certain areas including, for example, areas where surveillance is unsuitable, prohibited, or unnecessary.

At <NUM>, the pump controller <NUM> may apply a machine learning model trained to detect one or more nonconformities present in the one or more images. In some example embodiments, the machine learning engine <NUM> at the pump controller <NUM> may apply, to the images of the pump <NUM> loaded with the intravenous set <NUM>, one or more machine learning models trained to perform a variety of machine vision tasks including, for example, image enhancement, morphological filtering, blob detection, feature extraction, image segmentation, edge detection, object classification, optical character recognition (OCR), and/or the like. For example, the machine learning engine <NUM> may enhance the raw images captured by the camera <NUM> by performing edge detection (e.g., Laplace-Gaussian transformation, Canny edge detection, and/or the like) to determine the relative positions of the various components of the pump <NUM> and the intravenous set <NUM> depicted therein. The machine learning engine <NUM> may compare the enhanced images of the pump <NUM> loaded with the intravenous set <NUM> with one or more reference images depicting correctly loaded intravenous sets to determine whether the images of the pump <NUM> exhibits one or more nonconformities.

The machine learning engine <NUM> may be configured to detect a variety of nonconformities present in the images of the pump <NUM> loaded with the intravenous set <NUM>. One example of a nonconformity is a misload of the intravenous set <NUM> in which one or more portions of the intravenous set <NUM>, such as the upper fitment <NUM>, the pumping segment <NUM>, and the lower fitment <NUM>, are placed incorrectly in the pump <NUM>. Another example of a nonconformity may be the intravenous set <NUM> being of an incorrect type, a counterfeit product, or an expired set. Other nonconformities may include the intravenous set <NUM> not being primed properly of air, a missing component in the pump <NUM> and/or the intravenous set <NUM>, a damaged component in the pump <NUM> and/or the intravenous set <NUM>, and the presence of contaminants in the pump <NUM> and/or the intravenous set <NUM>.

At <NUM>, the pump controller <NUM> may perform a corrective action in response to an output of the machine learning model indicating a presence of one or more nonconformities in the one or more images of the pump loaded with the intravenous set. In some example embodiments, the pump controller <NUM> may perform one or more corrective actions when the output of the machine learning engine <NUM> indicates the presence of a nonconformity in the images of the pump <NUM> loaded with the intravenous set <NUM>. For example, the pump controller <NUM> may prevent the pump <NUM> from performing an infusion while a nonconformity is determined to be present at the pump <NUM>. Alternatively and/or additionally, the pump controller <NUM> may generate a message indicating, for example, a misload of the intravenous set <NUM> at the pump <NUM>. In some cases, the message may provide instructions for correcting the misload including by identifying the type and location of the misload at the pump <NUM>. The message may be presented via a user interface associated with the pump <NUM>.

Performing a corrective action a device may include transmitting one or more messages to adjust an operational state or functional element of the device. The message may include specific instructions to be executed by a processor of the device to manifest the change. The corrective action may include storing a value in a location of a storage device for subsequent retrieval by the device to be controlled, transmitting a value directly to the device to be controlled via at least one wired or wireless communication medium, transmitting or storing a reference to a value, and the like. For example, a control message may include a value to adjust a level of power from a power source of the controlled device. As another example, a control message may activate or deactivate a structural element of the controlled device such as a light, audio playback, a motor, a lock, a pump, a power supply, a display, or other component of a device described herein. A corrective action may include indirect control of the device by adjusting a configuration value used by the controlled device. For example, the control message may include a threshold value for a device characteristic (e.g., temperature, rate, frequency, etc.). The threshold value may be stored in a memory location and referred to by the controlled device during operation.

<FIG> depicts a block diagram illustrating a computing system <NUM> consistent with implementations of the current subject matter. Referring to <FIG> and <FIG>, the computing system <NUM> can be used to implement the pump controller <NUM> and/or any components therein.

As shown in <FIG>, the computing system <NUM> can include a processor <NUM>, a memory <NUM>, a storage device <NUM>, and input/output device <NUM>. The processor <NUM>, the memory <NUM>, the storage device <NUM>, and the input/output device <NUM> can be interconnected via a system bus <NUM>. The processor <NUM> is capable of processing instructions for execution within the computing system <NUM>. Such executed instructions can implement one or more components of, for example, the pump controller <NUM>. In some example embodiments, the processor <NUM> can be a single-threaded processor. Alternatively, the processor <NUM> can be a multi-threaded processor. The processor <NUM> is capable of processing instructions stored in the memory <NUM> and/or on the storage device <NUM> to display graphical information for a user interface provided via the input/output device <NUM>.

The memory <NUM> is a computer readable medium such as volatile or nonvolatile that stores information within the computing system <NUM>. The memory <NUM> can store data structures representing configuration object databases, for example. The storage device <NUM> is capable of providing persistent storage for the computing system <NUM>. The storage device <NUM> can be a floppy disk device, a hard disk device, an optical disk device, a tape device, a solid-state device, and/or any other suitable persistent storage means. The input/output device <NUM> provides input/output operations for the computing system <NUM>. In some example embodiments, the input/output device <NUM> includes a keyboard and/or pointing device. In various implementations, the input/output device <NUM> includes a display unit for displaying graphical user interfaces.

According to some example embodiments, the input/output device <NUM> can provide input/output operations for a network device. For example, the input/output device <NUM> can include Ethernet ports or other networking ports to communicate with one or more wired and/or wireless networks (e.g., a local area network (LAN), a wide area network (WAN), the Internet).

In some example embodiments, the computing system <NUM> can be used to execute various interactive computer software applications that can be used for organization, analysis and/or storage of data in various formats. Alternatively, the computing system <NUM> can be used to execute any type of software applications. These applications can be used to perform various functionalities, e.g., planning functionalities (e.g., generating, managing, editing of spreadsheet documents, word processing documents, and/or any other objects, etc.), computing functionalities, communications functionalities, etc. The applications can include various add-in functionalities or can be standalone computing products and/or functionalities. Upon activation within the applications, the functionalities can be used to generate the user interface provided via the input/output device <NUM>. The user interface can be generated and presented to a user by the computing system <NUM> (e.g., on a computer screen monitor, etc.).

One or more aspects or features of the subject matter described herein can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs, field programmable gate arrays (FPGAs) computer hardware, firmware, software, and/or combinations thereof.

These computer programs, which can also be referred to as programs, software, software applications, applications, components, or code, include machine instructions for a programmable processor, and can be implemented in a high-level procedural and/or object-oriented programming language, and/or in assembly/machine language. The machine-readable medium can alternatively or additionally store such machine instructions in a transient manner, such as for example, as would a processor cache or other random access memory associated with one or more physical processor cores.

To provide for interaction with a user, one or more aspects or features of the subject matter described herein can be implemented on a computer having a display device, such as for example a cathode ray tube (CRT) or a liquid crystal display (LCD) or a light emitting diode (LED) monitor for displaying information to the user and a keyboard and a pointing device, such as for example a mouse or a trackball, by which the user may provide input to the computer. For example, feedback provided to the user can be any form of sensory feedback, such as for example visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, speech, or tactile input. Other possible input devices include touch screens or other touch-sensitive devices such as single or multi-point resistive or capacitive track pads, voice recognition hardware and software, optical scanners, optical pointers, digital image capture devices and associated interpretation software, and the like.

As used herein a "user interface" (also referred to as an interactive user interface, a graphical user interface or a UI) may refer to a network based interface including data fields and/or other control elements for receiving input signals or providing electronic information and/or for providing information to the user in response to any received input signals. Control elements may include dials, buttons, icons, selectable areas, or other perceivable indicia presented via the UI that, when interacted with (e.g., clicked, touched, selected, etc.), initiates an exchange of data for the device presenting the UI. A UI may be implemented in whole or in part using technologies such as hyper-text mark-up language (HTML), FLASH™, JAVA™,. NET™, C, C++, web services, or rich site summary (RSS). In some embodiments, a UI may be included in a stand-alone client (for example, thick client, fat client) configured to communicate (e.g., send or receive data) in accordance with one or more of the aspects described. The communication may be to or from a medical device or server in communication therewith.

Claim 1:
A system (<NUM>), comprising:
at least one data processor (<NUM>); and
at least one memory (<NUM>) storing instructions which, when executed by the at least one data processor (<NUM>), result in operations comprising:
receiving, from a camera (<NUM>) at an infusion pump (<NUM>), one or more images of an area of interest of the infusion pump (<NUM>) loaded with an infusion set (<NUM>);
identifying, within the one or more images, a first component of the infusion pump (<NUM>) and a second component of the infusion set (<NUM>) in the area of interest, wherein the first component comprises a bezel, a membrane seal, a door (<NUM>), a platen (<NUM>), a locator feature (202a, 202b), or an air-in-line detector (<NUM>), and wherein the second component comprises an upper fitment (<NUM>), a lower fitment (<NUM>), or a pumping segment (<NUM>) of a tubing;
applying a machine learning model trained to detect one or more nonconformities present in the one or more images of the infusion pump (<NUM>) loaded with the infusion set (<NUM>) based at least on a relative position of the first component and the second component and by at least comparing, to one or more images of a correctly loaded infusion set (<NUM>), the one or more images of the infusion pump (<NUM>) loaded with the infusion set (<NUM>); and
in response to an output of the machine learning model indicating a presence of a nonconformity in the one or more images of the infusion pump (<NUM>) loaded with the infusion set (<NUM>), performing a corrective action.