Touchless interface for a medical treatment system

A dialysis machine comprising: one or more processing units configured to transmit control data; a pump configured to pump medical fluid to and from a patient based at least in part on control data received from the processing unit; an electronic panel comprising: a display surface, and at least one panel control unit configured to cause the electronic panel to display at least one user interface element that can be invoked by a user; at least one projector; and at least one camera; wherein the one or more processing units are configured to: process input received by the camera, determine a location of a physical object in a field of view of the camera based on the processed input, determine, based on processed input received on at least one occasion, that the location of the physical object represents an invocation of the at least one user interface element displayed on the electronic panel, and determine the control data based on the processed input.

TECHNICAL FIELD

This disclosure relates to an input device (e.g., a display) for a medical treatment system.

BACKGROUND

Dialysis is a treatment used to support a patient with insufficient renal function. Dialysis machines typically include input devices that can be used by nurses or doctors to input information related to treatment into the dialysis machine.

SUMMARY

In one aspect, a dialysis machine includes one or more processing units configured to transmit control data. The dialysis machine also includes a pump configured to pump medical fluid to and from a patient based at least in part on control data received from the processing unit. The dialysis machine also includes an electronic panel. The electronic panel includes a display surface and at least one panel control unit configured to cause the electronic panel to display at least one user interface element that can be invoked by a user. The dialysis machine also includes at least one projector and at least one camera. The one or more processing units are configured to process input received by the camera. The one or more processing units are also configured to determine a location of a physical object in a field of view of the camera based on the processed input. The one or more processing units are also configured to determine, based on processed input received on at least one occasion, that the location of the physical object represents an invocation of the at least one user interface element displayed on the electronic panel. The one or more processing units are also configured to determine the control data based on the processed input.

In some implementations, the projector includes a device that emits light.

In some implementations, the input received by the camera includes an image of pixels. Each pixel is defined by at least a u-coordinate value representing a horizontal position and a v-coordinate value representing a vertical position.

In some implementations, the position of the physical object is determined based on the u-coordinate value and the v-coordinate value of a pixel of the image.

In some implementations, the position of the physical object is determined by calculating an x-coordinate value, a y-coordinate value, and a z-coordinate value. The x, y, and z-coordinate values are each determined based on one or more of the following: the u-coordinate value, the v-coordinate value, a focal length of the camera in pixels, and a distance between the projector and the camera.

In some implementations, the at least one processor is configured to determine that the physical object is a physical object of interest.

In some implementations, the physical object is determined to be a physical object of interest based at least in part on a width of the physical object.

In some implementations, the physical object of interest is a finger of a human hand.

In some implementations, the projector emits a line. The length of the line depends on a distance between a point in space and the projector.

In some implementations, the dialysis machine includes four projectors and four cameras. A first projector is positioned above the electronic panel, a second projector is positioned below the electronic panel, a third projector is positioned to a left side of the electronic panel, and a fourth projector is positioned to a right side of the electronic panel.

In another aspect, a method performed by one or more processors of a dialysis machine includes processing visual input. The method also includes determining a location of a physical object based on the processed visual input. The method also includes determining, based on processed visual input received on at least one occasion, that the location of the physical object represents an invocation of at least one invokable user interface element displayed by an electronic panel of the dialysis machine.

In some implementations, the visual input includes information related to a light that is projected onto the physical object.

In some implementations, the light includes infrared light.

In some implementations, the visual input includes an image of pixels. Each pixel is defined by at least a u-coordinate value representing a horizontal position and a v-coordinate value representing a vertical position.

In some implementations, the position of the physical object is determined based on the u-coordinate value and the v-coordinate value of a pixel of the image.

In some implementations, the position of the physical object is determined by calculating an x-coordinate value, a y-coordinate value, and a z-coordinate value. The x, y, and z-coordinate values are each determined based on one or more of the following: the u-coordinate value, the v-coordinate value, a focal length, in pixels, of a camera that processes the visual input, and a distance between the camera and a projector that emits a light that is projected onto the physical object.

In some implementations, the method also includes determining that the physical object is a physical object of interest.

In some implementations, the physical object is determined to be a physical object of interest based at least in part on a width of the physical object.

In some implementations, the physical object of interest is a finger of a human hand.

In another aspect, a non-transitory computer-readable medium stores software that, when executed by one or more processors, performs a method including processing visual input. The method also includes determining a location of a physical object based on the processed visual input. The method also includes determining, based on processed visual input received on at least one occasion, that the location of the physical object represents an invocation of at least one invokable user interface element displayed by an electronic panel of a dialysis machine.

Implementations can include one or more of the following advantages.

In some implementations, the devices and techniques described herein can promote cleanliness and sterilization in a dialysis environment, thereby reducing the risk of facilitating the spread of infection and eliminating the need for the user to wear gloves. Cleanliness and sterilization can be especially important in a medical environment due to the fragile health of the patients. The touchless nature of the devices and techniques allow for a user to interact with the dialysis machine without touching the machine.

In some implementations, the electronic panel is configured to receive multiple inputs from the user. The multiple inputs may be received at different times (e.g., a gesture), or may be concurrent (e.g., multi-gesture input). The capability to receive multiple inputs from the user increases the number of distinct interactions that the user can have with the display118, thereby increasing the level of control that the user has over the dialysis machine.

Other aspects, features, and advantages of the invention will be apparent from the description and drawings, and from the claims.

DETAILED DESCRIPTION

At various points before, during, or after a medical fluid treatment such as a dialysis treatment, medical personnel may need to input information into a dialysis machine. For example, before treatment, a nurse may input patient parameters, such as a Patient ID. The nurse may also input medical treatment information, such as information related to the patient's treatment prescription.

The medical fluid treatment systems (e.g., dialysis systems) described herein can include an input device with a non-contact (e.g., touchless) interface for receiving input from a user. In some examples, the dialysis machine includes a display, a camera, a projector, and a processor. The display is configured to display user interface elements, such as user interface buttons, that can be invoked by a user without the user making contact with the display. Based on input received by the camera, the processor can determine a location of a finger of the user's hand in a field of view of the camera. The processor can also determine that the location of the finger represents an invocation of a particular user interface element that is being displayed.

In some examples, the user's finger may be positioned in a space in front of the display. The projector is configured to emit an infrared light in a plane that runs through the space in front of the display. When the user's finger intersects the plane of infrared light, an infrared line segment is projected onto the user's finger. The camera detects the infrared line segment that is projected onto the user's finger and constructs an image that includes a representation of the infrared line segment in pixels. The image may be two-dimensional and have a coordinate system wherein a u-coordinate represents a horizontal position of a pixel and a v-coordinate represents a vertical position of a pixel. In this way, a pixel can be characterized by a value (e.g., a numerical value) representing the u-coordinate and a value representing the v-coordinate.

The processor receives and processes the information received from the camera to determine whether the location of the finger represents an invocation of a particular user interface element. For example, the processor receives information related to the image constructed by the camera and determines that a physical object is positioned in a particular space in front of the display.

As a preliminary step, the processor may determine whether the physical object (e.g., the finger) is a physical object of interest. Such a determination may be based on a width of the finger, which may be determined based on the length of the infrared line segment projected on the finger.

If the finger is determined to be a physical object of interest, the processor determines the location of the finger in reference to the display (e.g., in terms of x, y, z-coordinates). The location of the finger is determined based at least in part on the image constructed by the camera, the u and v-coordinates of the infrared line segment, the position of the camera, the position of the projector, the projection angle of the projector relative to the display, and the focal length of the camera. The focal length of the camera is a measure of how strongly the system converges or diverges light, and corresponds to dimensions of objects that appear in the image constructed by the camera. An object within the camera's field of view tends to be in focus if the object is located at a distance from the lens that is close to the focal length. A longer focal length tends to correspond to a narrower angle of view, while a shorter focal length tends to correspond to a wider angle of view.

The location of the finger can be determined according to the following equations:

Once the x, y, z-coordinates of the finger are determined, the processor determines whether the location of the finger represents an invocation of a particular user element. The processor identifies a position on the display that is normal to the x, y, z-coordinates of the finger. If the identified position corresponds to a user interface element, the processor determines that the particular user interface element is being invoked. The dialysis machine may perform one or more actions based on the user interface element being invoked.

Use of the non-contact input device can promote cleanliness and sterilization in a dialysis environment, thereby reducing the risk of facilitating the spread of infection and eliminating the need for the user to wear gloves. Cleanliness and sterilization can be especially important in a medical environment due to the fragile health of the patients. The touchless nature of the input device allows for the user to interact with the dialysis machine without making physical contact with the machine.

Referring toFIG. 1, a hemodialysis system100includes a hemodialysis machine102to which a disposable blood component set104that forms a blood circuit is connected. As described below, the hemodialysis system100includes an input device such as an electronic panel (e.g., a display118).

In general, during hemodialysis, arterial and venous patient lines106,108of the blood component set104are connected to a patient and blood is circulated through various blood lines and components, including a dialyzer110, of the blood component set104. At the same time, dialysate is circulated through a dialysate circuit formed by the dialyzer110and various other dialysate components and dialysate lines connected to the hemodialysis machine102. Many of these dialysate components and dialysate lines are located inside the housing103of the hemodialysis machine102, and are thus not visible inFIG. 1. The dialysate passes through the dialyzer110along with the blood. The blood and dialysate passing through the dialyzer110are separated from one another by a semi-permeable structure (e.g., a semi-permeable membrane and/or semi-permeable microtubes) of the dialyzer110. As a result of this arrangement, toxins are removed from the patient's blood and collected in the dialysate. The filtered blood exiting the dialyzer110is returned to the patient. The dialysate that exits the dialyzer110includes toxins removed from the blood and is commonly referred to as “spent dialysate.” The spent dialysate is routed from the dialyzer110to a drain.

One of the components of the blood component set104is an air release device112. The air release device112includes a self-sealing vent assembly that allows air to pass therethrough while inhibiting (e.g., preventing) liquid from passing therethrough. As a result, if blood passing through the blood circuit during treatment contains air, the air will be vented to atmosphere as the blood passes through the air release device112.

As shown inFIG. 1, a dialysate container124is connected to the hemodialysis machine102via a dialysate supply line126. A drain line128and an ultrafiltration line129also extend from the hemodialysis machine102. The dialysate supply line126, the drain line128, and the ultrafiltration line129are fluidly connected to the various dialysate components and dialysate lines inside the housing103of the hemodialysis machine102that form part of the dialysate circuit. During hemodialysis, the dialysate supply line126carries fresh dialysate from the dialysate container124to the portion of the dialysate circuit located inside the hemodialysis machine102. As noted above, the fresh dialysate is circulated through various dialysate lines and dialysate components, including the dialyzer110, that form the dialysate circuit. As the dialysate passes through the dialyzer110, it collects toxins from the patient's blood. The resulting spent dialysate is carried from the dialysate circuit to a drain via the drain line128. When ultrafiltration is performed during treatment, a combination of the spent dialysate and excess fluid drawn from the patient is carried to the drain via the ultrafiltration line129.

The blood component set104is secured to a module130attached to the front of the hemodialysis machine102. The module130includes a blood pump132capable of driving blood through the blood circuit. The module130also includes various other instruments capable of monitoring the blood flowing through the blood circuit. The module130includes a door that when closed, as shown inFIG. 1, cooperates with the front face of the module130to form a compartment sized and shaped to receive the blood component set104. In the closed position, the door presses certain blood components of the blood component set104against corresponding instruments exposed on the front face of the module130. As will be described in greater detail below, this arrangement facilitates control of the flow of blood through the blood circuit and monitoring of the blood flowing through the blood circuit.

The blood pump132can be controlled by a blood pump module134. The blood pump module134includes a display window, a start/stop key, an up key, a down key, a level adjust key, and an arterial pressure port. The display window displays the blood flow rate setting during blood pump operation. The start/stop key starts and stops the blood pump132. The up and down keys increase and decrease the speed of the blood pump132. The level adjust key raises a level of fluid in an arterial drip chamber.

A drug pump192also extends from the front of the hemodialysis machine102. The drug pump192is a syringe pump that includes a clamping mechanism configured to retain a syringe178of the blood component set104. The drug pump192also includes a stepper motor configured to move the plunger of the syringe178along the axis of the syringe178. A shaft of the stepper motor is secured to the plunger in a manner such that when the stepper motor is operated in a first direction, the shaft forces the plunger into the syringe, and when operated in a second direction, the shaft pulls the plunger out of the syringe178. The drug pump192can thus be used to inject a liquid drug (e.g., heparin) from the syringe178into the blood circuit via a drug delivery line174during use, or to draw liquid from the blood circuit into the syringe178via the drug delivery line174during use.

Still referring toFIG. 1, the hemodialysis machine102includes a display118and a control panel120. The display118has a non-contact (e.g., touchless) interface for receiving input from a user. The display118and the control panel120allow the operator to input data, e.g., various different treatment parameters, to the hemodialysis machine102and to control the hemodialysis machine102. In addition, the display118conveys information to the operator of the hemodialysis system100.

The hemodialysis machine102includes one or more processing units configured to transmit control data (e.g., data that causes the dialysis machine102to perform one or more dialysis functions). In this example, the hemodialysis machine102includes a control unit105(e.g., a processor such as a microprocessor or microcontroller) that resides inside the machine and which is configured to transmit control data, communicate with the display118and the control panel120, and cause the hemodialysis machine102to carry out dialysis functions (e.g., starting or stopping a pump of the dialysis machine102). The control unit105is configured to receive data from the display118and the control panel120and control the hemodialysis machine102based on the received data, as described in more detail below. The hemodialysis machine102also includes a panel control unit109(e.g., a processor such as a microprocessor or microcontroller) that is configured to cause the display118to display one or more user interface elements that can be invoked by a user without the user making contact with the display118.

The hemodialysis machine102includes a projector101and a camera107that are affixed to the display118. The projector101is configured to emit a light (e.g., an infrared light) in a plane that runs through a space in front of the display118. If an object is present in the plane, the light is projected on the object. The camera107is configured to detect the light that is projected on the object and construct an image that includes a representation of the infrared line segment, as described in more detail below.

AlthoughFIG. 1is described in connection with a hemodialysis machine, it is specifically noted that the system and techniques described herein may be used with other types of dialysis and machines therefor, including peritoneal dialysis (PD).

FIG. 2ashows an example of the electronic panel (e.g., the display118). The display118includes a display surface202and a panel control unit (109ofFIG. 1). The display118presents one or more user interface elements204a-c. In this example, the user interface elements204a-care buttons that can be invoked by the user.

A three-dimensional coordinate system (210ofFIG. 2b) is associated with the display118. The coordinate system includes an x-axis, a y-axis, and a z-axis. The z-axis runs out of the paper/screen, and thus is not shown inFIG. 2a. The coordinate system has an origin at the bottom-left corner of the display118. The camera107is positioned along the y-axis at approximately x=0. In this example, the x-axis is represented such that negative values of x appear to the right, and positive values of x appear to the left. The projector101is positioned at x=−b and at approximately the same y value as the camera107.

FIG. 2bshows a perspective view of the display118ofFIG. 2ain reference to a three-dimensional coordinate system210. The projector101is positioned such that the projector101emits the infrared light in a particular plane206that runs through a space208(e.g., a three-dimensional area, sometimes referred to as a volume) in front of the display118. In this example, the projector101is tilted downwards. As such, the light is emitted away from the display in a downwards diagonal manner, starting at the top of the display118and running through the space208. The space208in front of the display118is represented by the dash-lined rectangular prism shown inFIG. 2b. When an object (e.g., the user's finger) intersects the plane206, the infrared light is projected onto the object.

FIG. 3ashows a perspective view of the display118ofFIGS. 2aand 2bin which a user's finger302is not intersecting the plane206. Although the user's finger302is positioned within the space208in front of the display118, the user's finger302is not in a position at which the projector101emits light. As described in more detail below, the hemodialysis system100can include additional projectors that emit light in additional planes that run through the space208in front of the display118. The inclusion of additional projectors can increase the number of positions within the space208in front of the display118at which the user's finger302can be detected (e.g., by intersecting one of the additional planes of light).

FIG. 3bshows an example of an image304that is constructed by the camera107based on the position of the user's finger302inFIG. 3a. The image304is associated with a coordinate system in which a u-coordinate value represents a horizontal position of a pixel and a v-coordinate value represents a vertical position of a pixel. An origin306of the image304corresponds to the x, y position of the camera107. More specifically, the origin306corresponds to a center of a lens of the camera107. In this example, because the camera107is positioned at the top-left corner of the display118, one quadrant of the image304has u, v-coordinate values that correspond to x, y-coordinate values that are within the space208in front of the display118. The quadrant is the top-right quadrant308of the image304. The top-right quadrant308of the image304is the quadrant that has u, v-coordinate values that correspond to x, y-coordinate values that are within the space208due to the mirror nature of camera images. In other words, from the camera's107front-facing perspective, the space208is located to the bottom-left of the camera107, but in the image304that is constructed by the camera107, the top-right quadrant308corresponds to the coordinates of the space208. If an object were to intersect the plane (206ofFIG. 3a), the camera107would detect the projected infrared light, and the representation of the infrared line segment would appear in the top-right quadrant308of the image304. However, in this example, the user's finger302is not intersecting the plane206. Therefore, the image300is substantially blank (e.g., it does not include a representation of the infrared line segment).

FIG. 3cshows a perspective view of the display118ofFIGS. 2aand 2bin which the user's finger302is intersecting the plane206of emitted infrared light. The position of the user's finger302causes a portion310of the infrared light emitted by the projector101to be projected onto the user's finger302.

As described above, the camera107detects the infrared light that is projected onto the user's finger302and constructs an image that includes a representation of the infrared light.FIG. 3dshows an example of an image312that is constructed by the camera107based on the position of the user's finger302inFIG. 3c. The representation of the infrared light appears as a segment314of pixels. The segment314is curved because the surface of the user's finger302is rounded.

In some implementations, the control unit (105ofFIG. 1) may determine whether the object that is represented by the segment314is a physical object of interest before proceeding with further processing. Such a determination may be based on a width of the314segment. For example, if the width of a segment is greater than a threshold (e.g., a predetermined threshold), the control unit105may determine that the object that caused the segment to be generated was likely not a finger of a user, and thus may identify the segment as an unintended or inappropriate input and choose to ignore the segment.

The threshold is predetermined if it is based on data that is available to the hemodialysis system100when a user starts using the non-contact interface at a particular time. In some implementations, the threshold may be based on data stored on and/or by the hemodialysis system100, such as configuration data that is stored at the time of manufacture. In some implementations, the data may be based on one or more calibrations of the hemodialysis system100. For example, the threshold may be determined by a calibration involving the user's finger, and the threshold may be subsequently modified based on additional calibrations.

The control unit105may determine an object location image point316based on the segment314. In the examples shown in the figures, the control unit105is a separate unit from the panel control unit109. As described above, the panel control unit109is configured to cause the display118to display the one or more user interface elements204a-c. The control unit105is responsible for, among other things, causing the hemodialysis machine102to carry out dialysis functions. Because the control unit105and the panel control unit109can be separate, isolated processors, a user interface malfunction involving the panel control unit109will not affect the dialysis functions carried out by the control unit105, thereby reducing the risk of the patient encountering an unsafe condition.

In some implementations, the control unit105may average the coordinate values of each pixel in the segment314to determine an average (e.g., mean) coordinate value of the segment314. The control unit105may then assign the average coordinate value to the object location image point316. Representing the segment314as a single object location image point316can simplify subsequent processing of information related to the image312. In this example, the object location image point316has coordinate values u1, v1in the image312.

Once the u, v-coordinate values of the object location image point316are known, the control unit105can determine the x, y, z-coordinate values of a point that corresponds to the location of the user's finger302within the space208in front of the display118.FIG. 4shows an example of a coordinate system400that illustrates now the x, y, z-coordinate values of the point are determined. The coordinate system400includes the three-dimensional coordinate system210ofFIGS. 2a, 2b, 3a, and 3c, as well as the u, v-coordinate system described with reference toFIGS. 3band 3d. The origin402of the coordinate system400corresponds to the location of the camera107. The origin402is represented as point c.

The x and u values are represented on one axis. Briefly referring back toFIGS. 2aand 2b, negative values of x represent positions that may be located directly in front of the display118, and positive values of x represent positions that are not located directly in front of the display118. This is due to the x-axis of the coordinate system210being defined as it is, e.g., such that positive x values correspond to locations that are not in front of the display118. Briefly referring back toFIGS. 3band 3d, positive values of u represent positions in the image that may correspond to locations that are directly in front of the display118, and negative values of u represent positions in the image that do not correspond to locations that are directly in front of the display118.

The y and v values are represented on one axis that runs out of the paper/screen. Briefly referring back toFIGS. 2aand 2b, positive values of y represent positions that may be located directly in front of the display118, and negative values of y represent positions that are not located directly in front of the display118. Briefly referring back toFIGS. 3band 3d, positive values of v represent positions in the image that may correspond to locations that are directly in front of the display118, and negative values of v represent positions in the image that do not correspond to locations that are directly in front of the display118.

For example, a point that has a negative x value and a positive y value is located directly in front of the display118provided the x value and the y value do not exceed the dimensions of the display. Similarly, a point that has a positive u value and a positive v value represents a position in the image that corresponds to a location that is directly in front of the display118provided the u value and the v value do not represent values that exceed the dimensions of the display118.

The z values are represented on one axis. The z-axis represents locations with reference to the surface of the display118. Briefly referring back toFIGS. 2aand 2b, positive values of z represent positions that are located in front of (but not necessarily directly in front of) the display118. Negative values of z represent positions that are located behind the display118. A focal length, f, of the camera107is represented as a negative value positioned on the negative portion of the z-axis. The dimensions of the image constructed by the camera107correspond to the focal length of the camera107.

The position of the projector101is represented as point p404. Briefly referring back toFIG. 2a, the distance between the projector101and the camera107is represented by length b. A projection angle of the light emitted from the projector101relative to the display118is represented as the angle θ. The object location image point316ofFIG. 3dis represented as point i406, and has coordinate values u1, v1. The z-coordinate of point i406the focal length, f, of the camera107.

The x, y, z-coordinate values of an object location point408, represented as point o408, which corresponds to the location of the user's finger302, can be determined based on one or more of the following: the u-coordinate value of the object location image point316(e.g., u1), the v-coordinate value of the object location image point316(e.g., v1), the focal length of the camera107in pixels, and the distance between the projector101and the camera107. In some example, the x, y, z-coordinate values of the object location point408can be determined according to the following equations:

Once the x, y, z-coordinate values of the object location point408are determined, the control unit105determines whether the object location point408(which corresponds to the position of the user's finger302intersecting the plane206) represents an invocation of a particular user interface element (e.g., a “press” of a button). In some examples, the control unit105compares the x, y, z-coordinate values of the object location point408to the x, y, z-coordinate values of user interface elements on the display118that are capable of being invoked. If the coordinate values of the object location point408fall within an area of a particular user interface element, the control unit105determines that the user is invoking the particular user interface element.

FIG. 5shows a perspective view of the display118in which a particular user interface element204b(e.g., a “stop blood pump” button) is invoked by the position of the user's finger302. The control unit105determines the coordinate values that define the areas of the user interface elements204a-c. The control unit105compares the x, y, z-coordinate value of the object location point (not shown) to the coordinate values that define the areas of the user interface elements204a-cto determine whether the object location point is located within the area of one of the user interface elements204a-c. In this example, the user's finger302is positioned at or close to the coordinate values of the object location point. The coordinate values of the object location point lie within the “stop blood pump” user interface element204b, as represented by a line502that is perpendicular to the display118and aligned with the user's finger302. The control unit105determines that the position of the user's finger302represents an invocation of the “stop blood pump” user interface element204b. The control unit105may determine control data based on the processed input (e.g., the input received by the camera107). In this example, the control unit105can determine control data that causes the hemodialysis machine102to stop the blood pump132.

FIGS. 6aand 6bshow a front-facing view and a perspective view, respectively, of the display118in which the hemodialysis system100includes four projectors601a-dand four cameras607a-d. The projectors601a-dare positioned at or near the middle of each edge of the display118, and the cameras607a-dare positioned at or near the corners of the display118.

Each projector601a-demits infrared light in a respective plane606a-dthat runs through the space208in front of the display118. As in the examples shown in the preceding figures, the first projector601aemits light in a downwards diagonal manner in a particular plane606athat runs from the top of the display118to the bottom of the space208in front of the display118. The second projector601bemits light in a sideways diagonal manner in a particular plane606bthat runs from the right side of the display118to the left side of the space208in front of the display118. The third projector601cemits light in an upwards diagonal manner in a particular plane606cthat runs from the bottom of the display118to the top of the space208in front of the display118. The fourth projector601demits light in a sideways diagonal manner in a particular plane606dthat runs from the left side of the display118to the right side of the space208in front of the display118. By having multiple projectors601a-d, there is more area within the space208in front of the display118for the infrared light to be projected onto the object. Thus, it can be said that a hemodialysis system100with additional projectors has fewer “blind spots” (e.g., locations within the space208in front of the display118that do not intersect a plane of light emitted by a projector). Similarly, by having multiple cameras607a-d, there is greater coverage of the space208in front of the display118for detecting the projected infrared light, further limiting potential blind spots.

As described above, each of the cameras607a-dis configured to detect the infrared light that is projected onto the object and construct an image that include a representation of the infrared light. The representation of the infrared light appears as a segment of pixels in the constructed images. The control unit105(shown inFIG. 1) is configured to determine whether the object that is represented by the segment of pixels is an object of interest (e.g., a finger of the user302), determine the u, v-coordinates of an object location image point in the constructed images based on the segment, determine the x, y, z-coordinate values of a point that corresponds to the location of the user's finger302within the space in front of the display118(e.g., the object location point), and determine whether the object location point represents an invocation of a particular user interface element (e.g., a “press” of a button).

As shown inFIG. 6b, the user's finger302intersects two planes—the plane610cof infrared light emitted by the third projector601c, and the plane610dof infrared light emitted by the fourth projector601d. The control unit105determines the coordinate values that define the areas of the user interface elements204a-c, and compares the x, y, z-coordinate value of the object location point (not shown) to the coordinate values that define the areas of the user interface elements204a-cto determine whether the object location point is located within the area of one of the user interface elements204a-c. In this example, the object location point lies within the “stop blood pump” user interface element204b, as represented by a line602that is perpendicular to the display118and aligned with the user's finger302. The control unit105determines that the position of the user's finger302represents an invocation of the “stop blood pump” user interface element204b.

In addition to reducing the number of blind spots within the space208in front of the display118, the use of four projectors601a-dand four cameras607a-dmay improve the accuracy of the system. In this example, because the user's finger302intersected two of the planes610c,610dof emitted infrared light, one or more of the images constructed by the cameras607a-dmay include a representation of the infrared light projected onto the user's finger302that has a different appearance than that of the segment314shown inFIG. 3d. For example, the image may include two segments, each segment corresponding to one of the planes610c,610d. Similarly, the u, v-coordinates of the object location image point may be determined based on the two segments (e.g., by averaging the coordinate values of each pixel of the two segments), thereby improving the accuracy of the determination.

In this example, one or more of the cameras607a-dmay have constructed an image that included a representation of the infrared light, and zero or more of the cameras607a-dmay have constructed an image that included no representation of the infrared light. In other words, zero or more of the cameras may have been unable to detect the infrared light projected onto the user's finger302. In cases in which multiple cameras607a-dconstruct an image that includes a representation of the infrared light, each of the corresponding images may be considered in determining the u, v-coordinates of the object location image point. For example, the u, v-coordinates of the object location image point may be determined by conflating information related to the segment(s) of the various images. In some implementations, the u, v-coordinates of each pixel of the segment(s) in a first image are averaged with the u, v-coordinates of each pixel of the segment(s) in a second image to determine the u, v-coordinates of the object location image point. In some implementations, coordinates related to one or more of the images undergo a conversion prior to the averaging to account for locational differences between the particular cameras607a-dused to construct the images.

FIG. 7is a flowchart700illustrating a technique for determining, by a processor of a dialysis machine, an invocation of a user interface element displayed by the dialysis machine. Visual input is processed (702). The visual input may be received by a camera. A location of a physical object is determined based on the processed visual input (704). The visual input can include information related to a light that is projected onto the physical object. The light can be emitted by a projector. The light may be in the infrared range. The visual input may include an image of pixels. Each pixel can be defined by at least a u-coordinate value representing a horizontal position and a v-coordinate value representing a vertical position, and the position of the physical object can be determined based on the u-coordinate value and the v-coordinate value of a pixel of the image. The position of the physical object can be determined by calculating an x-coordinate value, a y-coordinate value, and a z-coordinate value, wherein the x, y, and z-coordinate values are each determined based on one or more of the following: the u-coordinate value, the v-coordinate value, the focal length, in pixels, of the camera, and the distance between the camera and the projector. Whether the physical object is a physical object of interest can be determined. The physical object may be determined to be a physical object of interested based at least in part on a width of the physical object. The physical object of interest may be a finger of a human hand. The light that is projected onto the physical object can be a line, and the length of the line can depend on the distance between a point in space and an origin of the projected light (e.g., the projector). The location of the physical object can be determined as representing an invocation of at least one invokable user interface element (706). The user interface element may be displayed by an electronic panel of the dialysis machine. The determination can be based on the processed visual input received on at least one occasion.

In some implementations, the display118is configured to receive multiple inputs from the user. The multiple inputs may be received at different times (e.g., a gesture), or may be concurrent (e.g., multi-gesture input). The capability to receive multiple inputs from the user increases the number of distinct interactions that the user can have with the display118.

In some implementations, the user performs a gesture by moving an object (e.g., the user's finger) through the space208in front of the display118. A first input may be received when the user's finger intersects the plane206at a first position, and a second input may be received when the user's finger moves to a second position that intersects the plane206. Characteristics of the movement from the first position to the second position can determine the particular gesture that is being invoked. For example, the user may swipe his finger from the left to the right to cause the display118to present a previously-displayed screen (e.g., a “back” gesture), or the user may swipe his finger from the right to the left to cause the display118to present a next screen (e.g., a “next” gesture). Similarly, the user may swipe his finger from a top position to a bottom position to cause the display118to scroll down, or the user may swipe his finger from a bottom position to a top position to cause the display118to scroll up. In some implementations, the display118is configured to provide an indication when a gesture is detected. The visual indication may indicate the particular gesture that is detected.

In some implementations, the user performs a multi-gesture input by concurrently putting two object (e.g., a first and second finger of the user) in the space. In some implementations, one or both of the first finger and the second finger are moved through the space in a similar manner as described above. Characteristics of the positions and/or the movements of the fingers can determine the particular multi-gesture input that is being invoked. For example, the user may position his fingers in the space208in front of the display118such that each finger intersects the plane206, and subsequently move his two fingers closer together (e.g., a pinch). The pinch may cause the display118to zoom out. Similarly, the user may position his fingers in the space208in front of the display118and subsequently move his two fingers further apart (e.g., a spread). The spread may cause the display118to zoom in. In some implementations, the display118is configured to provide an indication when a multi-gesture input is detected. The visual indication may indicate the particular multi-gesture input that is detected.

While certain implementations have been described, other implementations are possible.

While the hemodialysis system has been described as including a display with a non-contact interface, in some implementations, the hemodialysis system may include one or more additional input devices. In some implementations, the hemodialysis system includes a keyboard (e.g., a traditional push-button QWERTY keyboard). The one or more additional input devices may be used in place of the display (e.g., as an alternative input device for users who do not wish to use the non-contact interface of the display), or may be used in addition to the display (e.g., to input data in a manner that is not easily input using the non-contact interface of the display). In some implementations, in addition to having the non-contact interface, the display may also be a touchscreen that is capable of receiving touch inputs.

While the hemodialysis system has been described as including both a control unit and a panel control unit (e.g., two separate processors), in some implementations, the hemodialysis system includes a single control unit that is configured to perform the functions of both the control unit and the panel control unit. For example, the hemodialysis system can include a single processor that is configured to transmit control data for the hemodialysis machine, process input received by the camera, determine the location of the physical object in the field of view of the camera, and determine that the location of the physical objects represents an invocation of a particular user interface element.

In cases in which the hemodialysis system includes multiple cameras, if one of the cameras is obstructed, the hemodialysis system can use a different camera to carry out the functions described herein. Further, the use of multiple unobstructed cameras may provide additional locational data related to the physical object. The additional locational data may allow the hemodialysis system to determine the location of the physical object without knowing one or more other pieces of information described herein. For example, a hemodialysis system that includes multiple cameras may be able to determine the x, y, z-coordinate values of the object without knowing one or both of the focal length of the camera and the distance between the projector and the camera.

In cases in which the hemodialysis system includes multiple projectors, if one of the projectors is obstructed, the hemodialysis system can use a different projector to carry out the functions described herein. Further, as described above, the use of multiple unobstructed projectors may enable the non-contact interface of the display to detect physical objects more positions within the space in front of the display. For example, multiple unobstructed projectors can emit light in additional planes that run through the space in front of the display, thereby resulting in fewer blind spots, and in some examples, eliminating blind spots completely.

While the hemodialysis system has been described as including i) one projector and one camera and ii) four projectors and four cameras, the hemodialysis system can include other numbers of projectors and/or cameras. For example, in some implementations, the hemodialysis system includes two projectors and one camera. In some implementations, the hemodialysis system includes two projectors and two cameras. In some implementations, the hemodialysis system includes two projectors and four cameras. In some implementations, the hemodialysis system includes four projectors and one camera. In some implementations, the hemodialysis system includes four projectors and two cameras.

In some implementations, one of more of the cameras of the hemodialysis system may be oriented in ways other than described above. For example, one or more of the cameras may be oriented such that the camera is tilted toward the display. Such orientation may improve the ability for the camera to detect infrared light that is projected onto an object, thereby eliminating blind spots. In some implementations, adjusting the orientation of the camera may require different equations for determining the x, y, z-coordinate values of the object location point than those described above. For example, the particular angle of tilt of the camera may cause the constructed image to be warped (e.g., stretched), and such warping may call for different equations that factor in the angle of tilt to correct for the warping. In some implementations, a warped image constructed by a tilted camera may undergo preprocessing to correct the warping, thereby eliminating the need for different equations.

While the light emitted by the projector has been described as being an infrared light, the projector may emit other types of light. In some implementation, the projector emits another type of light that is not visible to the human eye. In some implementations, the projector emits a type of light that is visible to the human eye.

While the control unit has been described as determining an object location image point by averaging the coordinate values of each pixel in the segment of the image, other techniques may be employed. In some implementations, the control unit computes an average of the coordinate values of the endpoint pixels of the segment. In some implementations, the control unit computes the midpoint of the segment. In some implementations, a single object location image point is not determined, and instead, the coordinate values of the pixels of the segment are themselves used to represent the location of the object. In some implementations, the coordinate values of the pixels of the segment are processed in some other way to represent the location of the object.

While the invocation of a user interface element has been described as causing the dialysis machine to stop the blood pump, the techniques described herein can be used to cause the dialysis machine to perform one or more other actions in response to a user interface element being invoked. Such functions may or may not be related to dialysis.

While some specific examples of gestures and multi-gesture inputs have been described that can be received by the display, these examples are not exhaustive. In some implementations, the display is configured to receive one or more other gestures and multi-gesture inputs.

While the non-contact input device has been principally described as being part of a hemodialysis machine, the input device could alternatively be included in other types of medical treatment systems. Examples of other medical treatment systems in which the input device can be used include hemofiltration systems, hemodiafiltration systems, apheresis systems, cardiopulmonary bypass systems, and peritoneal dialysis systems.

FIG. 8is a block diagram of an example computer system800. For example, referring toFIG. 1, the control unit105, the panel control unit109, or both (as separate units or as a single unit) could be examples of the system800described herein. In this example, the system800includes a processor810, a memory820, a storage device830, and an input/output device840. Each of the components810,820,830, and840can be interconnected, for example, using a system bus850. In some implementations, the control unit105, the panel control unit109, or both (as separate units or as a single unit) could be examples of the processor810(e.g., as oppose to the control unit105and/or the panel control unit109being examples of the entire system800. The processor810is capable of processing instructions for execution within the system800, as described in detail above. The processor810can be a single-threaded processor, a multi-threaded processor, or a quantum computer. The processor810can be capable of processing instructions stored in the memory820, on the storage device830, or both. The processor810may execute operations such as those described in detail above with respect to the control unit105and the panel control unit109.

The memory820stores information within the system800. In some implementations, the memory820is a computer-readable medium. The memory820can, for example, be a volatile memory unit or a non-volatile memory unit.

The storage device830is capable of providing mass storage for the system800. In some implementations, the storage device830is a non-transitory computer-readable medium. The storage device830can include, for example, a hard disk device, an optical disk device, a solid-date drive, a flash drive, magnetic tape, or some other large capacity storage device. The storage device830may alternatively be a cloud storage device, e.g., a logical storage device including multiple physical storage devices distributed on a network and accessed using a network.

The input/output device840provides input/output operations for the system800. In some implementations, the input/output device840includes one or more of network interface devices (e.g., an Ethernet card), a serial communication device (e.g., an RS-232 10 port), and/or a wireless interface device (e.g., an 802.11 card, a 3G wireless modem, or a 4G wireless modem). A network interface device allows the system800to communicate (e.g., transmit and receive data) with other devices. In some implementations, the input/output device840includes driver devices configured to receive input data and send output data to other input/output devices (e.g., the display118, the control panel120, a keyboard, and/or a printer, among others). In some implementations, mobile computing devices, mobile communication devices, and other devices are used. The system described herein may be used with any one or more of, including combinations of, appropriate wireless communication technologies, such as cellular or mobile network technologies, WiFi technologies, and/or other short distance wireless communication technologies, including Bluetooth and/or near field communication (NFC). The wireless communications may involve appropriate security and encryption protocols or standards, and may be used in conjunction with appropriate wireless hardware and software components that support such wireless communication technologies.

While an example computer system800has been described with reference toFIG. 8, implementations of the subject matter and the functional operations described above can be implemented in other types of digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Implementations of the subject matter described in this specification, such as software for determining an invocation of a user interface element displayed by a dialysis machine (e.g., as described with reference toFIG. 6), can be implemented as one or more computer program products, (e.g., one or more modules of computer program instructions encoded on a tangible program carrier), such as a computer-readable medium, for execution by, or to control the operation of, a processing system. The computer readable medium can be a machine readable storage device, a machine readable storage substrate, a memory device, a composition of matter effecting a machine readable propagated signal, or a combination of one or more of them.

Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile or volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks or magnetic tapes; magneto optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), e.g., the Internet.