HIGH DENSITY DISTANCE SENSOR ARRAY ALTERNATIVE TO SURFACE ELECTROMYOGRAPHY FOR THE CONTROL OF POWERED UPPER LIMB PROSTHESES

Systems and methods for a wearable sensor system including a compressible material, a two-dimensional array of distance sensors, a support structure, and a controller. The compressible material is positionable relative to a tissue surface and the two-dimensional array of distance sensors is configured relative to the compressible material to detect compressive deformations of the compressible material. The support structure is configured to hold the compressible material in place relative to the tissue surface such that muscle movements at the tissue surface cause the compressive deformations of the compressible material and is also configured to restrict movement of the two-dimensional array during the muscle movements. The controller is configured to receive a signal from the two-dimensional array indicative of the compressive deformation of the compressive material at a location of each distance sensor

BACKGROUND

The present disclosure relates to methods and system for detecting and monitoring movements of a hand. More particularly, in some implementations, the present disclosure relates methods and systems for detecting muscle movements for controlling a powered prosthesis.

SUMMARY

Despite the technological achievements of modern-day prostheses, the average person with upper limb amputation (ULA) is unable to gain a significant level of prosthetic embodiment. In general, a level of embodiment can be described as how well a person projects and attaches their sense of self to their body, other individuals, objects, and concepts. Prosthetic embodiment in particular has to do with the extent that a person identifies a prosthetic device as part of their self-identity and body. Usually, in literature, the amount of time an amputee wears a prosthesis is used as a correlate for a level of their prosthetic embodiment. However, for most people with ULA, there is very little prosthetic embodiment.

In some implementations, the present disclosure provides methods and systems for a wearable two-dimensional, high-density array of distance sensors for use in controlling a prosthesis or an animatronic device based on sensed muscle movements along a tissue surface. For example, in some implementations, the operation of a powered prosthetic hand is controlled based on changes in the shape of the forearm due to movements of the forearm muscles. As a user contracts the muscles in his/her forearm, there is a change in distance between the muscles and one or more of the sensors of the array of distance sensors positioned on the forearm surface. The distance sensors detect the change in distance and detect where on the arm the change occurred. A control signal based on the detected sensor information is transmitted to the prosthetic or animatronic hand, which then accomplishes the desired motion. This control device and methodology will enable a patient to perform real-time, direct, robust, and simultaneous control of multiple degrees of freedom.

In other implementations, the two-dimensional array of distance sensors is configured for placement on a different muscle surface. For example, in some implementations, the two-dimensional array of distance sensors is configured for placement on a leg surface and outputs signals indicative of changes in the surface of the leg due to movements of the leg muscles. Those output signals are then used, in some implementations, to operate the actuators of a powered leg and/or foot prosthesis. Similarly, in other implementations, the two-dimensional array of distance sensors is configured for placement on a chest surface and outputs signals indicative of changes in the surface of the chest due to movements of the pectoral muscles. Those output signals are then used, in some implementations, to operate the actuators of a powered arm prosthesis.

In still other implementations, the two-dimensional array of distance sensors is configured to monitor muscle movements in order to control other non-prosthetic system. For example, in some implementations, forearm muscle movements are monitored by the two-dimensional array of distances sensors in order to determine movements and/or placement of a user's hand for controlling a virtual reality (VR) or augmented reality (AR) systems.

In some embodiments, the invention provides a wearable sensor system including a compressible material, a two-dimensional array of distance sensors, a support structure, and a controller. The compressible material is positionable relative to a tissue surface and the two-dimensional array of distance sensors is configured relative to the compressible material to detect compressive deformations of the compressible material. The support structure is configured to hold the compressible material in place relative to the tissue surface such that muscle movements at the tissue surface cause the compressive deformations of the compressible material and is also configured to restrict movement of the two-dimensional array during the muscle movements.

The controller is configured to receive a signal from the two-dimensional array indicative of the compressive deformation of the compressive material at a location of each distance sensor and to determine a gesture operation based on the signal.

DETAILED DESCRIPTION

As noted above, the average person with ULA is unable to gain a significant level of prosthetic embodiment. However, the present disclosure enables great strides with osseointegration and neural prostheses, which can restore sensation by utilizing slanted electrode arrays. For the 96% of amputees who are not supported by Veterans Affairs, their upper-limb prosthesis may cost around $35,000-$75,000 with very little insurance coverage. The most effective prostheses in this price range use surface electromyography (sEMG) sensors and have less than a third of the degrees of freedom (DoF), or unique motions, of their natural counterpart. This is because every unique motion adds cost, bulk, and complexity to a prosthesis system. Beyond the problem of low resolution, sEMG requires filtering and excessive calibration and cannot differentiate between changes in muscle length, size, or speed of contraction. Also, these sEMG systems require excessive maintenance. Amputees must take time off from work to go to occupational therapists, physical therapists, and prosthetists or orthotists to keep their artificial limb working properly for the rest of their life. Due to these negative issues with upper-limb prostheses, 44-73% of people with ULA (based on level of amputation) do not use any prosthesis and they often feel disillusioned with the unintuitive expensive prostheses available to them.

The present disclosure provides cost efficient components for an electronically controllable prostheses at about 1/10th of the cost and circuit complexity of sEMG systems. Also, geometric anatomical measurements are made for the control of prosthesis hand geometry. Unlike EMG technology measurements, the geometric measurements enabled by the present disclosure can be directly related to muscle force, length, and velocity as measured using the high-density array of distance sensors. This provides for control of several degrees of freedom simultaneously where prosthesis motor actuation can vary in speed and position for unique motions of the prosthetic hand. For example, a person using a prostheses that is controlled based on the present disclosure could control the prosthesis intuitively and could play the piano, which is not presently available to a person with a ULA. The present disclosure provides simultaneous control of position, velocity, and force of prosthesis movement and gestures as intended and controlled by the user of the prosthesis. These improvements in controlling a prosthesis further increase a user's prosthetic embodiment and reduction in phantom pain. Furthermore, the present disclosure provides intuitive and easy to follow calibration of the prosthetic controller.

People with below-elbow amputations would benefit from the ability to consistently use their existing anatomy to restore hand dexterity with higher functionality as provided by the present disclosure. They would experience higher functionality than provided by earlier prosthetic control systems. For example, the present disclosure enables real-time, direct, and robust control. The natural, intuitive, and comfortable interface provides simultaneous control of multiple DoFs. In some embodiments a simple calibration method and system requires relatively limited training. The present distance sensor array prosthesis control device is designed for daily use, for use over long periods of time, and for a variety of indoor and outdoor environments. It is non-invasive and can be worn without adhesives. Users may include below elbow upper limb amputees that benefit from control of a below elbow prosthesis.

FIG. 1Ais a block diagram of a prosthesis control system including an array of distance sensors.FIG. 1Aincludes a prosthesis control system100comprising a compressive material108, a flexible band110, distance sensor array112, a multiplexer114, a signal processing circuitry116, a prosthesis controller118, a prosthesis device120, and a calibration computing system122. The prosthesis controller118controls the prosthesis device120based on output from the distance sensor array112. In some embodiments, the compressive material108, the flexible band110, the distance sensor array112, the multiplexer114, the signal processing circuitry116, and the prosthesis controller118are components of an integrated wearable prosthesis control assembly for interaction between a person wearing the integrated assembly and a prosthesis including a hand that is also worn by the user such as described in further detail in the examples below.

In some implementations, the distance sensor array112includes a two-dimensional array of light intensity sensors disposed on, attached to, or otherwise held in place relative to a user's body part (e.g., the user's forearm) by the flexible band110. The flexible band110may be a wearable band and may be wrapped or positioned around a user's forearm over the muscles that will be activated by the user and sensed by the distance sensor array112when the user controls the prosthesis device120. In other implementations, the flexible band110is provided as a part of a wearable control and socket system that is further configured to selectively attach the prosthesis device120to a user's limb. In some embodiments, the flexible band110may be fastened, for example, by a double D ring loop strap, Velcro, or another suitable fastener. In some embodiments, the flexible band110may be made of a polymer material and may be injection molded, 2D printed, 3D printed, laser-cut, or die-pressed, for example. In some embodiments, the distance sensor array112, the multiplexer114, the signal processing unit116, and prosthesis controller118may be integrated into one wearable band prosthesis control assembly.

In the example ofFIG. 1A, the sensors of the distance sensor array112and the flexible band110are selectively positioned over the muscles of the forearm such that a 2-dimensional or 3-dimensional image or mapping of positions of the muscles may be obtained from the distance sensor measurements. The compressive material108may be disposed between the distance sensor array112and the user's forearm. The compressive material108helps to prevent motion of the distance sensor array112due to underlying muscle movement or changes in muscle thickness so that relative distance to the distance sensors112caused by the muscle thickness changes can be detected.

In some embodiments, the 2D array of distance sensors includes a plurality of light intensity sensors or LIDAR sensors. In some such implementations, each sensor includes a light source (e.g., a light emitting diode (LED)) and a light sensor (e.g., a phototransistor or photodiode). Light is emitted from the light source into the compressive material layer and reflected back towards the sensors where it is sensed by the phototransistor. The output signal of the phototransistor is indicative of a distance (or a change in distance) between the sensor and the tissue surface. In some implementations (as described in further detail below), the compressive material layer includes a reflective surface or a reflective layer is positioned on the opposite side of the compressive material layer from the sensor array to reflect light emitted by the sensors back towards the sensors of the distance sensor array112. In some implementations, the reflective material may be used to prevent skin color or moisture on the skin surface from affecting light intensity measurements by the distance sensor array112. When the flexible band and compressive material are fixed to a user's forearm and a muscle or multiple muscles of the forearm are used to control intended hand movements, the distance sensor array112detects a change in distance between the muscles and the sensors of the array.

As noted above, the distance sensor array112may comprise a high-density array of light intensity sensors. The density of the sensors used in the array112may affect the resolution of muscle movement detection and the level of control or fineness of articulation in the movements of prosthesis parts in the prosthesis device120. The number of sensors in the array and the placement density of the sensors can vary in different implementations. In some examples, the sensor array may include 5 sensors, 250 sensors, or 1000 or more sensors. For example, the flexible band110may comprise a distance sensor array112comprising twenty-five sensors. The arrangement of the sensors of the array may vary depending the density of the array, the muscle group(s) and/or tissue surfaces on which the array with be positioned, and/or the positions of muscle motions that are used to map to movements or gestures of the prosthetic device120. For example, in some embodiments, the sensors may be arranged in a triangular, rectangular, or radial two-dimensional (2D) grid. However, the disclosure is not limited to any specific number or arrangement of sensor elements in the distance sensor array112.

In some embodiments, the distance sensor array112outputs analog signals. In the example ofFIG. 1A, electrical power from a power source113is applied to all of the sensors in the array112. The multiplexer114receives an output signal from each sensor in the sensor array112and indexes the sensors of the array one at a time based on a control input from the prosthesis controller118. Each sensor's signal is serially output from the multiplexer114and may be conditioned by the signal processing electronics116to provide a well-resolved digital input to the prosthesis controller118and/or to the calibration computing system122. For example, in some embodiments, the signal processing electronics include a voltage follower, a voltage subtracter, and a voltage amplifier to rectify the distance sensor signals and/or to limit the distance sensor signals to between about 0V and 5V. In some embodiments, the voltage subtracter and voltage amplifier may each have digital potentiometers or rheostat that allows the prosthesis control system to automatically rectify the signal (e.g., between 0V and 5V). In other examples, the circuit may be customized to the tolerances of the design or manually adjusted analogue potentiometers could be utilized. In some embodiments, the prosthesis controller118and/or the calibration computing system may include a microcontroller/computer/control system that is coupled to the signal processing electronics via a USB C, HDMI, or other suitable wired communication mechanism. However, the prosthesis controller118is not limited with regard to any specific types of communication interfaces.

One example of the prosthesis controller118is shown in further detail inFIG. 1Cand includes, among other things, an electronic processor820, a computer-readable non-transitory memory830, and a communication interface850that may be communicatively coupled via a bus870. The communication interface850may be communicatively coupled to the multiplexer114, the signal processing electronics116, and the calibration computing system122. The memory830stores instructions that are executed by the electronic processor820to control the multiplexer114, read distance sensor signals from the distance sensor array112, control the actuators/motors of the prosthesis device120, and/or calibrate the prosthesis control system100according to the disclosure described herein. For example, the electronic processor820may move the fingers and/or thumb of the prosthesis device120based on information of the distance sensor signals. In some embodiments, the prosthesis controller118may be operable to control wrist movement of the prosthetic device120. In some embodiments, the prosthesis controller118may include a user interface860, a display device840and/or a graphical user interface810to receive input from a user and/or provide information to a user.

The prosthesis controller118may be communicatively coupled to the calibration computing system122for calibration of the prosthesis control system100. The calibration computing system122may include among other things, an electronic processor, a memory, and a communication interface communicatively coupled to the prosthesis controller118. The calibration computing system122may also include a user interface, a display device, and a graphical user interface for interaction with a user during calibration of prosthesis control system100(see description below). In some embodiments, the calibration computing device122may be a portable device, such as a laptop, a smart phone, or a dedicated device. In some embodiments the calibration computing device122and the prosthesis controller118may be integrated as one wearable device, for example, as integrated in or attached to the flexible band110.

The prosthesis controller118determines which distance sensor locations and distance sensor signal values that are received from the array112correspond to which fingers and finger movements or gestures of the prosthesis device120through a calibration sequence. As described in further detail below, the calibration sequence facilitated by the calibration computing device122may instruct the user to flex the muscles that they believe correspond to specific hand motions that will be made by their prosthesis or as if the movements are made by their missing hand. In some embodiments, the calibration sequence consists of 32 (25) gestures and may start with all fingers open then all fingers closed. From there the sequence proceeds through the binary (full close and full open) combinations of each finger (including thumb). However, the disclosure is not limited to any specific calibration gestures or sequence of calibration gestures and the calibration method may be based on the level of control implemented for the prosthesis device120. In some embodiments, gestures may be repeated. For example, a calibration sequence may run through every gesture five times and average the distance sensor signals. Another step may include having the user use their muscles to intend to open and close their missing hand along with control of the prosthesis device120to get a reading of the relation of a sensed muscle diameter change to prosthesis finger position. This is because the relationship of diameter change to muscle length is not linear and not necessarily predictable.

FIG. 1Bis another block diagram of the prosthesis control system including the array of distance sensors112. Like the example ofFIG. 1A, the system ofFIG. 1Bincludes a high-density distance sensor array112configured to provide a signal output for each sensor to the multiplexer114. The multiplexer114is controlled by the microcontroller118to produce a serialized signal output which is then provided from the multiplexer114to additional adaptive signal processing circuitry116. The microcontroller118then operates the actuators and motors of the prosthesis120based on the processed distance signals. However, in addition to a computer system and application122for calibration, the example ofFIG. 1Balso includes a portable calibrator125and is further configured to show the relation of a user123wearing the wearable band prosthesis control assembly to other components of the system.

In the example ofFIG. 1A, electrical power from the power source113is applied to all sensors in the sensor array112during operation of the system. However, due to the operation of the multiplexer114, the signal from only a single sensor of the array112is included in the output signal of the multiplexer114at a given time.FIG. 2Aillustrates an example of a control system200in which electrical power is further conserved by using a de-multiplexer (DEMUX)203to apply electrical power to only one sensor (or a subset of sensors) at a given time. A multiplexer (such as multiplexer114in the example ofFIG. 1A) receives multiple input signals and provides one of those input signals as the output of the multiplexer based on the status of the control signal provided to the multiplexer. Conversely, a “demultiplexer” (such as DEMUX203in the example ofFIG. 2A) receives a single input signal and applies that input signal to one or a plurality of output signal lines based on the status of the control signal input provided to the demultiplexer.

In the example ofFIG. 2A, a power source201is coupled to the DEMUX203as the “input signal” for the DEMUX203. Each output signal line of the DEMUX203is coupled to a different sensor in the 2D distance sensor array205. Accordingly, depending on the control signal input provided to the DEMUX203(for example, by the prosthesis controller209), the DEMUX203operates to connect the electrical power from the power source201to only one sensor of the sensor array205and, thereby, only one distance sensor of the array205is energized & operational at a given time. The outputs of all of the distance sensors in the array205are coupled to the same shared output line of the array205and, because only one distance sensor receives electrical power at a time, the shared output line produces a signal indicative of the distance sensed by the presently energized sensor.

In other words, in the example ofFIG. 1A, a shared power source is coupled to all of the sensors in the array and the serialized output is produced by using the multiplexer114to selectively control which sensor output signal is connected to the output signal. In contrast, in the example ofFIG. 2A, the serialized output signal is generated by using the DEMUX203to selectively control which individual sensor of the array205receives operating power.

Despite this different configuration, the output signal from the 2D sensor array205in the example ofFIG. 2Ais functionally equivalent to the output signal from the multiplexer114in the example ofFIG. 1A. Accordingly, the system200ofFIG. 2Asimilarly operates by providing the serialized output signal to signal processing electronics207and ultimately to the prosthesis controller209, which, in turn, operates the motors and actuators of the prosthesis device211based on the serialized distance signal. The system200may also be coupled to a calibration computing system213as discussed above in reference to the examples ofFIGS. 1A, 1B, and 1C.

As illustrated in the example ofFIG. 2B, the signal processing electronics207may include digital potentiometers (e.g., rheostats) (221,223), instrumentation amplifier(s)225, and one or more passive components (e.g., wires, resistors, and capacitors). Each sensor227in the sensor array205are configured to operate as “proximity sensors” and are arranged in a 2-dimensional array indexed one position at a time by the DEMUX203. In some implementations of the multiplexer-based control system ofFIG. 1Aand the DEMUX-based control system ofFIG. 2A, each “distance sensor”227in the respective array includes an infrared light emitting diode (IR LED) that projects infrared light. This projected light reflects off an object and is then sensed by a phototransistor in the sensor that allows a current to flow proportional in magnitude to the intensity of the reflected light received by the sensor. The intensity of the light received is proportional in magnitude to the distance and reflectivity of the object. The voltage output is then measured by an analog-to-digital converter (ADC) (e.g., a component of the signal processing electronics207or the prothesis controller209) and the distance measured by the sensor can be calculated based on the measured voltage.

In some implementations, each sensor227of the array includes two resistors—one to set the voltage and wattage of the IR LED and the other to set the voltage and wattage of the phototransistor output. In the example ofFIGS. 2A and 2B, only one sensor is powered at a time and, therefore, in some such implementations, the LED drains and transistor outputs of every sensor in the array205can be connected to the same two resistors. This decreases component and assembly costs. Additionally, in some implementations using a DEMUX203to selectively apply electrical power to individual sensors (or groups of sensors), only one instrumentation amplifier (INAMP)225is needed per DMUX sensor array as the voltages are large enough to preserve their integrity without a voltage follower circuit. The negative input of the INAMP is set to equal the minimum possible voltage of the sensor being measured. This makes the instrumentation amplifier output at the minimum sensor value close to 0v. This also removes much of the electrical noise from the output as this noise is on the sensor and the negative INAMP input. This negative INAMP input is set with a voltage divider that has a digital potentiometer (rheostat)221as one of its legs. The gain of the INAMP225is set so that the maximum INAMP output is close to a maximum ADC input (as determined, for example, during a calibration procedure). High-frequency noise is removed from the output of the potentiometers and the INAMP225with external resistors and/or capacitors.

Similarly,FIG. 1Dillustrates an example of a filtering circuit for an implementation that uses a multiplexer114to selectively and successively couple the output signal from each individual sensor227in the sensor array to the output signal. Like the example ofFIG. 2B, the filtering circuit in the example ofFIG. 1Dalso includes an instrumentation amplifier (INAMP)135and a pair of digital potentiometers131,133.

The graph ofFIG. 2Cillustrates the operation of the signal filtering provided by the circuit in the example ofFIG. 2Bby graphing the unfiltered output signal of the two-dimensional array (signal231), the noise component233, and the filtered output signal with the noise component removed (signal235). As shown in the graph ofFIG. 2C, the operation of the filter circuitry removes the noise component and adjusts the amplitude of the output signal to fit a defined maximum amplitude and a define minimum amplitude (e.g., 5v and 0v, respectively, in the example ofFIG. 2C).

In some implementations, control system, such as those illustrated in the examples ofFIG. 1A, 1B, or2A, may be adapted to operate selectively and interchangeably with one or more different commercially available prosthetic hands.FIG. 3illustrates three examples of commercially available prosthetic hands that may be used with the control systems described in the examples above. The examples illustrated inFIG. 3include Ottobock prosthetic hands and cosmetics, including from left to right, small system inner hand, small MyoHand VariPlus Speed, and medium Michelangelo hand. Although these specific examples are shown inFIG. 3, many other prosthetic devices may be used with and controlled by the control systems described herein. Furthermore, although examples presented in this disclosure may be described in reference to controlling a prosthetic hand, these examples can be adapted in other implementations to provide for muscle-movement-based control of other types of prosthetic devices (e.g., powered prosthetic arms or legs) and/or other non-prosthetic systems and actuators.

As described above, the control system operates by using a support structure to hold the two-dimensional sensors in place and positioning a compressible layer between the array and a tissue surface.FIGS. 4A and 4Billustrate two examples of such configurations. In the example ofFIG. 4A, a plurality of distance sensors403are arranged in a two-dimensional array on a flexible, but incompressible, printed circuit board (PCB) layer401. The array of sensors403is positioned adjacent to a compressible layer405formed of an opaque material (e.g., a compressible foam). A series of holes407are formed through the compressible layer405aligned with the position of each sensor403so that the light from each distance sensor403is able to pass through the compressible layer405. A layer of reflective material409is positioned adjacent to the compressible layer405opposite the distance sensor403. In this configuration, light is emitted from a sensor403into its respective hole407in the compressible layer405, is reflected by the reflective layer409, and the reflected light is detected by the sensor403. In some implementations, the use of opaque material for the compressive layer ensures the distance measured by each individual sensor403is not affected by ambient light or light emitted by other sensors403in the array. However, in other implementations, partially or entirely translucent materials may be used for the compressible layer and the signal processing system may be adapted to detect muscle shape and/or movement based—not only on light reflected back to the sensor—but also based on how light from other sensors403in the array may be observed/sensed by a given sensor403. In some implementations of the example ofFIG. 4A, the sensors403may be affixed to the PCB layer401. However, in other implementations, each individual sensor403may be embedded (or otherwise placed) in its respective hole407in the compressible layer405as shown in the example ofFIG. 4B, which shows the sensors403and the compressive layer405without the support structure (e.g., PCB layer401) in place.

FIG. 5illustrates one example of the two-dimensional array of distance sensors and the compressible layer integrated into a single wearable control device and also illustrates an example of how that wearable control device interfaces with the tissue surface to detect muscle movements. The example ofFIG. 5illustrates two separate layered structures—the layers of the wearable control device501and the layers corresponding to the user503. In the example ofFIG. 5, the wearable control device501is positioned on an external skin surface507of a user to detect movements (e.g., changes in shape) of the muscles505below the skin surface. In some implementations, a user may also wear a “sock” garment between the skin surface507and the wearable control device501.

The wearable control device501includes a fabric layer511that is placed in contact with the skin surface507(or the sock509) when the device501is worn and an aesthetic covering521enclosing the functional layers and components of the wearable control device501. The wearable controller device501includes a two-dimensional array of distance sensors mounted to a flexible printed circuit board (i.e., PCB layer517). The PCB layer517is positioned adjacent to a compressible layer515opposite a reflector layer513. A flexible, incompressible support structure519is positioned between the PCB layer519and the aesthetic covering521. In the configuration illustrated inFIG. 5, the layers of the wearable control device501are secured to the user503such that movements of the muscles505cause corresponding movements of the reflector layer513by compressing the compressible layer513while the distance sensors of the PCB layer517are held in place by the support structure519. Accordingly, movements of the muscle505cause changes in the distance between the reflector layer513and each individual distance sensor.

It is noted thatFIG. 5presents just one example of a wearable control device501including a two-dimensional array of distance sensors and a compressible layer to monitor muscle movements/position. Other configurations are possible. For example, as discussed above, in some implementations, the wearable control device501does not include a reflector layer513. Instead, the wearable control device501in other implementations may include an array of individual reflectors embedded into the compressible layer513or, alternatively, may expose the compressible layer515directly (or through one or more translucent materials) to the skin surface507so that light from each sensor is reflected directly by the skin surface507.

Furthermore, although the example ofFIG. 5operates by moving the reflector layer513in response to muscle movements while the distance sensors remain stationary, in some other implementations, the position of the 2D sensor array and the reflector layer513may be reversed such that movement of the muscle505causes corresponding relative movement of each sensor in the 2D array by compressing the compressible layer515while the reflector layer513is held stationary by the support structure519.

FIGS. 6A through 6Dillustrate a specific example of a wearable control system including the layered arrangement illustrated inFIG. 5. In this example, the wearable control system is configured as a socket that selectively couples a prosthetic hand to an amputated arm. An example of the support structure519is shown inFIG. 6A. The support structure519is formed of a flexible material that is incompressible or significantly less compressible than the material of the compressible layer515. In the example ofFIG. 6A, the support structure519is laser cut as a two-dimensional form including a tongue601extending from a distal end of a fingerless glove section603. The tongue601is configured to fold over the user's hand/arm to protect the skin surface from pinching or abrasion when the wearable control device is secured to the user's arm by straps (as described below). The fingerless glove section603includes holes for a thumb and each of four fingers and is sized to wrap around a hand (either a prosthetic hand or a user's actual hand) when the wearable control device is secured to the user's arm. A proximal end of the fingerless glove section603is coupled to a sensor array support section605that is configured to wrap around a forearm of the user and to hold the sensor array in place when the wearable control system is secured to the user's arm. The support structure519further extends to an upper arm support section607at its proximal end. As illustrated further below, the upper arm support section607in this example is used to secure the wearable control device to a user's upper arm for anchoring and includes a cut-away section that is to be positioned over the inner elbow of the user's arm to prevent pinching and irritation when the user's elbow joint bends. The support structure519includes additional holes cut into it to fix the distance sensors and the printed circuit board in place while allowing the distance sensors to see changes in muscle conformation. In some implementations, the grooves may be cut into the support structure519to facilitate bending.

FIG. 6Billustrates an example of the printed circuit board layer517of the wearable control device503. In this example, 60 different distance sensors611are mounted to the surface of a flexible printed circuit board (PCB)517in a two-dimensional array pattern and are communicatively coupled by printed circuit traces613. However, as noted above, the exact number of sensors and the arrangement of the sensors in the two-dimensional array may be adjusted/altered in other implementations. As also described in detail above, the sensors are either all turned on at the same time and read one-at-a-time using a multiplexer (as illustrated inFIG. 1A) or share a common output channel and are selectively powered on one-at-a-time using a demultiplexer (as illustrated inFIG. 2A).

The assembled wearable control system is illustrated inFIG. 6C. As discussed above in reference toFIG. 5, in the assembled wearable control system503, the sensor array PCB layer517and the compressible layer515are housed within the aesthetic covering521. In the example ofFIG. 6C, both the aesthetic covering521and the fabric layer511on the underside are cut in the same shape as the support structure519such that the assembled device also resembles the shape of the support structure519include a tongue601, a fingerless glove section603, a sensor array support section605, and an upper arm support section607. In some implementations, the fabric layer511is affixed to the aesthetic covering521(e.g., by adhesive or by sewing) to encase the support structure519, the compressible layer515, the sensor array PCB layer517, and the reflector layer513. However, in other implementations, the aesthetic covering521and/or the fabric layer511are selectively removable for washing.

As shown in the example ofFIG. 6C, the prosthetic hand device621is positionable in the fingerless glove section603of the wearable control system503by extending the fingers and thumb of the prosthetic hand device621through the applicable holes in the fingerless glove section603. A prosthesis controller623is communicatively coupled to both the prosthetic hand621and the internal sensor array of the wearable control system503.

In the example ofFIG. 6C, the wearable control system also includes multiple straps625arranged along the outer edges of the fingerless glove section603, the sensor array support section605, and the upper arm support section607. To secure the wearable control system to the arm503of a user, the tongue601is placed along the user's arm503(extending from the prosthetic hand621towards the user's elbow), the body of the wearable control system is wrapped around the arm (over the tongue601), and the straps are used to secure the wearable control system in place around the user's arm503. In the example ofFIG. 6C, the straps625are provided as “hook-and-loop” straps. However, in other implementations, the wearable control system may be secured around the user's arm by other fasteners including, for example, lace, rivets, snaps, or toggles.

Furthermore, in some implementations, a series of holes are formed at the position of each strap625through the fabric layer511, the support structure519, and the aesthetic covering521. Accordingly, in some such implementations, the layers of the wearable control system can be assembled by extending and securing the straps625through each of these holes in the different layers and can be disassembled (e.g., for washing of the fabric layer511) by removing the straps625.

FIG. 6Dillustrates the wearable control system secured to the prosthetic hand621and the user's arm503. As shown inFIG. 6D, when the wearable control system is secured to the user's arm503, the sensor array section605is positioned along the palm-side of the user's forearm and extends by wrapping around the outer side of the forearm.FIG. 6Dalso shows the placement of the opening627adjacent to the inner elbow of the user's arm to allow for movement of the elbow joint without obstruction or causing irritation.

FIG. 7illustrates an example of a method for controlling an actuator (such as, for example, the prosthetic hand621in the example ofFIG. 6D) using a two-dimensional array of distance sensors with a shared output channel and a demultiplexer to selectively apply power to only one sensor in the array at a time (e.g., the control system ofFIG. 2A). Electrical power from a power source is applied to the input line of the demultiplexer (step701). The control input to the demultiplexer is adjusted by a controller (e.g., the prosthesis controller209) (step703) to cause the demultiplexer to selectively apply operating power from the power source to only one of the sensors in the array. The controller (e.g., the prosthesis controller209) then reads the shared signal output channel of the array (step705) and, after the expiration of a defined period of time (step707), adjusts the control input provide to the demultiplexer (step703) to cause the demultiplexer to apply operating power to a different sensor in the array. By controlling the demultiplexer in this way to sequentially applying operating power to each sensor in the array—one sensor at a time, a serialized signal is generated on the shared output channel of the array indicative of the muscle shape and movement. This serialized output signal is received by the controller (e.g., the prosthesis controller209) and used to control the operation of an actuator (e.g., an actuator/motor of the prosthetic hand211).

FIG. 8illustrates an example of a method for controlling an actuator (such as, for example, the prosthetic hand621in the example ofFIG. 6D) using a two-dimensional array of distance sensors with power continuously applied to all of the sensors and using a multiplexer to selectively adjust which sensor output is coupled to the output channel of the multiplexer. First, electrical power from a power source is applied to all of the sensors in the array (step801). Because all of the sensors in the array are receiving operating power, every sensor in the array will also provide an output signal indicative of its measured distance to the multiplexer. A serialized output signal is then generated by periodically adjusting the control input to a multiplexer (steps803and807). A controller (e.g., the prosthesis controller118) (step803) reads the serialized output signal at the output channel of the multiplexer (step805) and controls the operation of an actuator (e.g., an actuator/motor of the prosthetic hand120) based on the serialized output signal.

As discussed above in referenceFIGS. 1A through 2B, a calibration computing system122,213can be used to calibrate the control system for particular movements and to optimize the output of each sensor in the array based on observed signal outputs for each respective sensor.FIG. 9illustrates an example of one such calibration process. First, the calibration computing system122,213outputs a first movement instruction to a user (step901). In some implementations, this instruction may be for the user to fully extend all fingers of a hand or to form a tight fist. The serialized signal output is monitored (step903) and the controller analyzes the signal output to identify specific sensors that are positioned adjacent to the muscles (or muscle groups, areas, etc.) that change shape during the instructed “movement” (step904). The specific sensor that is identified as corresponding to a muscle movement location for a particular gesture movement is referred to herein as the “indicative sensor” for that particular movement. This process for identifying the “indicative sensor(s)” is repeated for a series of additional movement instructions (step907). When the signal output has been measured and the “indicative sensor(s)” have been identified for each prescribed movement in the calibration procedure (step905), the collected signal corresponding to each individual sensor is analyzed (step909) to determined a sensed maximum and minimum output value for the sensor (step911). The calibration system then defines a gain and/or baseline voltage value for the sensor that causes the minimum voltage output value for the sensor to be close to the minimum analog input value for the analog-to-digital converter and to cause the maximum voltage output value for the sensor to be close to the maximum analog input value for the analog-to-digital converter (step913). This is repeated for each sensor in the array (step917). When all of the sensors in the array have been calibration (step915), the calibration process is complete (step919) and the control system is operated according to the gain settings and baseline voltage values defined for each sensor.

The methods and systems described in the examples above provide a control system positionable on a tissue surface of a user that uses a two-dimensional array of distance sensors and a compressible layer to monitor movements of muscles below the tissue surface. Systems and/or actuators are then controlled based on these sensed muscle movements. In several of the examples discussed above and as briefly illustrated again inFIG. 10, the system is adapted to position the sensor array1001to detect movements of muscles in a user's forearm corresponding to intended movements of fingers and the sensed output from the two-dimensional array of distance sensors is used to control the actuators of a powered hand prosthesis1003. However, these systems can be adapted in other implementations to monitor different muscle groups and to control different systems.

For example,FIG. 11illustrates an example in which the 2D sensor array (operated by a multiplexer or demultiplexer)1101is positioned on a chest of a user to monitor movements of pectoral muscles. The output of the sensor array system1101is then used to provide control signals for the operation of an arm prosthesis1103.FIG. 12presents another alternative example in which the 2D sensor array1201is positionable on a leg surface to monitor movements of leg muscles and the output of the sensor array system1201is then used to provide control signals for the operation of a leg or foot prosthesis1203. For example, the sensor array might be positioned on a surface of the upper leg to monitor movements of the thigh muscle and to then control operation of a prosthetic knee based on the thigh muscle movements. Alternatively, the sensor array1201might be positionable on a lower leg surface to monitor movements of a calf muscle and to then control operation of a prosthetic ankle based on the calf muscle movements.

However, implementations of the 2D sensor array systems (such as those described above) are not necessarily limited to control of prosthetic devices in cases of amputation. Instead, the output signal of the 2D sensor array may be used to control other actuators or as a user interface input to other systems. For example, in a virtual reality system, a 2D sensor array1301may be positionable on a forearm of a user in order to track movement of a user's hand. This movement can then be used by the VR system controller1303as a user control input to the virtual reality system and images/interfaces displayed to the user on a VR display1305may be adjusted based on this user control input.

This movement tracking/detection functionality is also applicable to augmented reality (AR) systems in which movements of a user's hand, for example, are detected based on the output of the 2D sensor array1301and analyzed by an AR system controller1303. In addition to using hand position as a user control input for adjusting the images and/or interfaces displayed to the user on an AR display1305, the AR system controller1303may be further configured to control the operation of an actuator1307based on the determined hand position/movements.

Although the examples describe above focus specifically on measuring muscle movements (i.e., changes in muscle shape and thickness), in some implementations, additional functionality may be incorporated into the same system. For example, in some of the examples described above, the two-dimensional array of distance sensors includes a plurality of light-intensity distance sensors. In addition to measuring distances, some light-intensity distance sensors can also be operated to provide plethysmography and pulse oximetry functionality. Accordingly, in some implementations, the systems described herein can be further adapted to provide additional functions such as, for example, plethysmography and pulse oximetry with additional programming of the controller and, in some cases, without any additional components or circuitry. Similarly, in other implementations, the systems and structures described herein (including, for example, the layered configuration) can be configured to provide systems for plethysmography and/or pulse oximetry without including any muscle movement measurement of actuator/prosthesis control functionality.

Thus, the invention provides, among other things, systems and method for identifying bodily movements based on muscle movements measured at a tissue surface by a two-dimensional array of distance/proximity sensors configured to monitor the muscle movements based on compression of a compressible layer. Other features and advantages of this invention are set forth in the following claims.