Patent ID: 12229348

DETAILED DESCRIPTION

Specific embodiments of the invention will now be described in detail with reference to the accompanying figures. Like elements in the various figures are denoted by like reference numerals for consistency.

In the following detailed description of embodiments of the invention, numerous specific details are set forth in order to provide a more thorough understanding of the invention. However, it will be apparent to one of ordinary skill in the art that the invention may be practiced without these specific details. In other instances, well-known features have not been described in detail to avoid unnecessarily complicating the description.

Throughout the application, ordinal numbers (e.g., first, second, third) may be used as an adjective for an element (i.e., any noun in the application). The use of ordinal numbers is not to imply or create a particular ordering of the elements nor to limit any element to being only a single element unless expressly disclosed, such as by the use of the terms “before.” “after,” “single.” and other such terminology. Rather the use of ordinal numbers is to distinguish between the elements. By way of an example, a first element is distinct from a second element, and the first element may encompass more than one element and succeed (or precede) the second element in an ordering of elements.

In general, embodiments of the invention provide a system, a method, and a non-transitory computer readable medium (CRM) for tracking fingers of a hand. More specifically, one or more embodiments of the invention are directed to a wearable sensor patch and processor that determine finger pose information by illuminating the backside of a user's hand and tracking the metacarpal bones based on the backscattered light from the skin surface of the backside of the hand. Because the metacarpal bones are connected the fingers by a complex array of musculature and located closely to the skin of the backside of the hand, it is possible to extrapolate finger pose information for the fingers based on the relative movement and deflection of the metacarpals. The finger pose information may then be transmitted from the finger tracking system to another device as input information (e.g., for a virtual model of the user's hands). In one or more embodiments, finger pose information may include a position and/or orientation of one or more fingers of the hand. In one or more embodiments, finger pose information may include a position and/or orientation for individual bones of the hand. In one or more embodiments, finger pose information may include a position and/or orientation estimate for the entire hand (e.g., a gesture or activity classification). It will be appreciated that any combination of hand, finger, bone and/or classification information may be included in the finger pose information, and the disclosure is not particularly limited to the above configurations.

FIGS.1A-1Bshow a hand10demonstrating a finger pose and a configuration of bones in the hand10.

InFIG.1A, the hand10changes from a first finger pose (left) to a second finger pose (right) that grasps an object20between a thumb10aand an index finger10b. Specifically, in the first finger pose, the thumb10a, the index finger10b, a middle finger10c, a ring finger10d, and a pinky finger10eare extended and straight. In the second finger pose, the object20is disposed the tips of the thumb10aand the index finger10bwhich have rotated at their respective knuckle joints to secure the object20.

InFIG.1B, the various finger bones12,14,16,18of the hand10are shown to illustrate how the finger pose is related to the relative position and orientation of the bones. As shown in the first finger pose (left), each digit of the hand10includes a metacarpal12, proximal phalanx14connected to the metacarpal12, and a distal phalanx18disposed at the end of each digit. With the exception of the thumb10a, each finger10b,c,d,ealso has an intermediate phalanx16that connects the proximal phalanx14and the distal phalanx18. In the second finger pose (right), the metacarpals12a,b, the proximal phalanges14a,b, the intermediate phalanx16b, and the distal phalanges18a,bmoved in order to secure the object20at the tips of the thumb10aand index finger10b. As discussed above, based on detecting the relative movement and deflection of the metacarpals12, it is possible to extrapolate finger pose information due to the musculature connecting metacarpals12to the various phalanges.

FIG.2A-2Bshow an example of a finger tracking system200in accordance with one or more embodiments of the invention.

As shown inFIG.2A, the system200includes a sensor patch210that attaches to the back of the hand10and communicates, via a connector patch220, with a support band230that is a wearable wristband device equipped with a processor (e.g., a smart watch, a fitness tracker). Each of the components of the system200is described in further detail below. In one or more embodiments, the sensor patch210, the connector patch220, and/or the support band230may be combined into a single wearable patch device that tracks the fingers of the hand10.

FIG.2Bshows the sensor patch210attached to the backside of the hand10in a region corresponding to the metacarpals12a,b,c,d,e(i.e., the metacarpals12) during the transition between the first finger pose and the second finger pose shown inFIG.1B. To extrapolate finger pose information based on the metacarpals12, the sensor patch210remains in conformal contact with the hand10and emits optical radiation into the hand10to track the movements of the metacarpals12. The sensor patch210collects backscattered light intensities from the skin surface of the backside of the hand10and relays the data to the support band230that analyzes the data to determine finger pose information. The sensor patch210can be attached to the backside of the hand10for continuous monitoring (e.g., 24 hours/day, 7 days/week) of finger pose information or removed and replaced and the user's discretion (e.g., many on/off cycles) to provide finger pose information on demand.

FIG.3shows a schematic of the finger tracking system200in accordance with one or more embodiments of the invention. The system200has multiple components, and may include, for example, the sensor patch210, the connector patch220, and the support band230.

The sensor patch210includes the optical components required to emit and collect the optical radiation used to detect movement of the metacarpals12. The sensor patch210is described in further detail below with respect toFIG.4.

The connector patch220includes connectors at each end of conductive wires to connect to the sensor patch210and the support band230. The connectors may be an array of electrical contacts or an adapter (e.g., a plug or socket that can be attached/detached) to facilitate easy and fast connections to the sensor patch210and the support band230. The conductive wires may be inkjet-printed conductive metallic lines or fine grade metal wires lines attached to an adhesive tape. The connector patch220may be flexible and configured to conform and/or adhere to the hand10to improved user comfort.

The support band230has multiple components, and may include, for example, a memory232, a processor234, a battery236, and a transceiver238. The memory232may be random access memory (RAM), cache memory, flash memory, or a storage drive that stores information for the sensor patch210and the support band230. The processor234may be an integrated circuit (e.g., one or more cores, or micro-cores) for processing instructions for the sensor patch210and the support band230. The battery236may be a rechargeable battery (e.g., lithium-ion or any other appropriate medium, 3V, 5V, or any appropriate low voltage level for operating the other components) for powering the sensor patch210and the support band230. The transceiver238may be a wired or wireless communications circuit (e.g., data port, antenna(s) array, communications bus) that allows the system200to communicate with an external device, such as a user device or a network240. Although the support band230inFIG.3is shown as having four components (232,234,236, and238), in other embodiments of the invention, the support band230may have more or fewer components. Furthermore, the functionality of each component described above may be shared among multiple components or performed by a different component. For example, each component (232,234,236, and238) may be utilized multiple times in serial or parallel to carry out repeated, iterative, or parallel operations.

The system200may also include one or more input device(s) (not shown), such as a button, touchscreen, camera, microphone, or any other type of input device for the user to provide information directly to the system200rather than through the transceiver238. Further, the system200may include one or more output device(s) (not shown), such as a screen (e.g., a liquid crystal display (LCD), light emitting diode (LED) display, organic light emitting diode (OLED) display, or any other display device to provide information directly to the user rather than through the transceiver238. One or more of the output device(s) may be the same or different from the input device(s). The system200may connect to a network240(e.g., a local area network (LAN), a wide area network (WAN) such as the Internet, mobile network, or any other type of network) via the transceiver238to exchange information between the system200and any external device.

Further, one or more processing elements of the support band230may be located at a remote location and may be connected to the other elements over the network240. For example, one or more embodiments of the invention may be implemented by spreading the information processing across a distributed system having a plurality of nodes that include distinct computing and storage devices (i.e., cloud computing). Each node may correspond to a computer processor with associated physical memory. Each node may alternatively correspond to a computer processor or micro-core of a computer processor with shared memory and/or resources.

Software instructions executed by the support band230may be in the form of computer readable program code to perform embodiments of the invention and may be stored, in whole or in part, temporarily or permanently, on a non-transitory computer readable medium such as a CD, DVD, storage device, a diskette, a tape, flash memory, physical memory, or any other computer readable storage medium. Specifically, the software instructions may correspond to computer readable program code that when executed by a processor(s), is configured to perform embodiments of the invention.

FIG.4shows a section of the sensor patch210in accordance with one or more embodiments of the invention. The drawings shown is not to scale and the dimensions of the component shown may be smaller or larger in any dimension.

The sensor patch210has multiple components, and may include, for example, a substrate212, a light emitting layer214, a photodiode216, and a sectioning mask218. Although the sensor patch210inFIG.4is shown as having four components (212,214,216, and218), in other embodiments of the invention, the sensor patch210may have more or fewer components. Furthermore, the functionality of each component described below may be shared among multiple components or performed by a different component. For example, the sectioning mask218may be directly integrated into the structure of the photodiode216. In addition, each component (212,214,216, and218) may be utilized multiple times in serial or parallel to carry out a repeated operation or an iterative operation. For example, the sensor patch210may include a plurality of photodiodes216to track each finger of the hand10. Each of the components of the sensor patch210is described in further detail below.

The substrate212is a flexible backer sheet that supports the other components of the sensor patch210. In one or more embodiments, the substrate212may be a layer of single-sided adhesive tape (e.g., flexible medical-grade adhesive tape) that is configured to attach the sensor patch210to the backside of the hand10and conform the light emitting layer214to the backside of the hand10. The characteristics of the flexible medical-grade adhesive tapes are: to be capable for the extended use with the maximizes comfort and designed specifically for use in electronic devices. The adhesive tape used in the substrate212may be any appropriate flexible medical-grade adhesive tape that permits the sensor patch210to be removed and reapplied to the hand10multiple times. In one or more embodiments, the adhesive portion of the substrate212may be replaceable. Alternatively, the entire sensor patch210and/or connector patch220may be disposable portions of the system200. In one or more embodiments, the substrate212may include an elastic band or a flexible clamp that aids in keeping the entire sensor patch210applied to a surface of the hand10.

The light emitting layer214is a flexible planar light source that controllably emits one or more wavelengths of light. For example, the light emitting layer214may be a flexible OLED sheet that can be used to emit one or more wavelengths in the visible (e.g., blue, green, red) and infrared (e.g., near-infrared, mid-infrared) range based on the absorption spectrum of the flesh on the hand10. The OLED sheet may be a multilayer and/or multiwavelength LED design created by inkjet printing or any appropriate thin film fabrication process.

By using a planar light source (e.g., instead of point LED sources), a more uniform illumination profile is achieved and light detection errors in the photodiode caused by size differences of the hand and inhomogeneous locations of the metacarpal bones can be reduced or minimized. In one or more embodiments, the light emitting layer214may include several independent OLED sections to more precisely control the distribution of light emitted in the hand10. Furthermore, the OLED sheet may include a shadow mask214athat prevents scattered light from the backside of the photodiode216from reaching an adjacent photodiode216or other components of the sensor patch210. As shown inFIG.4, the light rays400emitted from the light emitting layer214do not impinge on the photodiode216due to the shadow mask214a.

The light emitting layer214is controlled by the support band230. For example, the light emitting layer214may be continuously powered or operated with a duty cycle of 50% or less (e.g., powered on-off in a cyclic manner to reduce overall power consumption, improve background measurement rate, filter out ambient contributions or baseline measurement shifts).

The photodiode216may be a single pixel detector configured to detect backscattered light intensities from the skin surface of the backside of the hand10. Alternatively, the photodiode216may be a more complex photodetector (e.g., an multipixel array in one or more dimensions, a charge coupled device (CCD), may include one or more wavelength filters corresponding to the wavelength(s) of the light emitting layer214). The photodiode216is pressed against the surface (any appropriate surface) of the hand10to detect changes in the optical properties of the surface of the hand10(e.g., backscattering and/or absorption of the OLED light) due to the movement one or more metacarpals12caused by the movement of the fingers10a,b,c,d,e.

The photodiode216is connected to the support band230. For example, the photodiode216may transmit signals to the support band230continuously, periodically, asynchronously, or on demand for processing and analysis. In one or more embodiments with a plurality of photodiodes216(e.g., a single photodiode216for each metacarpal12of the hand10), the support band230may coordinate data from each photodiode216to determine finger pose information based on movements of multiple metacarpals12.

The sectioning mask218is an opaque mask that blocks the transmission of light and prevents detection of stray light by the photodiode216. The sectioning mask218is disposed between the photodiode216and the surface of the sensor patch210that contacts the hand10. The sectioning mask218may form the surface of the sensor patch210that contacts the hand10. The sectioning mask218may be a sheet with apertures corresponding to each photodiode216or one or more opaque masks disposed on a transparent film. In embodiments with multiple photodiodes216and a light emitting layer214that extends across the width of the hand10, the sectioning mask218may prevent each photodiode216from detecting backscattered light from a region of the hand that corresponds to an adjacent photodiode. As shown inFIG.4, the backscattered light rays410afrom the region directly in front of the photodiode216(e.g., from the corresponding metacarpal) are allowed to impinge on the photodiode216while the backscatter light rays410bfrom the adjacent regions of the hand10are blocked.

FIGS.5A-5Bshow examples of a finger tracking system200in accordance with one or more embodiments of the invention.

InFIG.5A, the sensor patch210is equipped with five photodiodes216a,b,c,d,ethat correspond to the five metacarpals12a,b,c,d,eof the hand10. The sensor patch210is equipped with a single continuous OLED sheet as the light emitting layer214. The sensor patch210is equipped with a sectioning mask218that divides the emission regions of the OLED sheet into five regions that correspond to the five metacarpals12a,b,c,d,e, of the hand10. The sensor patch210is attached to the hand by a single-sided adhesive tape in the substrate212. The size of the sensor patch210may be 10-20 cm long and 0.5-1.0 cm wide, and less than 1 mm thick. However, the dimensions of the sensor patch210are not limited to these ranges and any appropriate dimensions may be used to adequately position the light emitting layer214and the photodiodes216at positions to monitor bones within the hand10.

As shown more clearly inFIG.5B, the photodiodes216a,b,c,d,eand are aligned with the five metacarpals12a,b,c,d,e, of the hand10. Accordingly, output signals from the five photodiodes216a,b,c,d,eare processed by a processor (e.g., in the support band230) and analyzed simultaneously. For example, the signals from the five photodiodes216a,b,c,d,emay be analyzed by a pretrained pattern recognition machine learning model that determines the finger pose information (e.g., classification model based on predetermined finger actions or finger poses). The machine learning model is discussed in further detail with respect toFIG.8.

FIGS.6A-6Cshow examples of finger tracking systems in accordance with one or more embodiments of the invention. InFIG.6A, the substrate212of the sensor patch210includes a flexible clamp212a1and a conforming elastic layer212a2that presses the sensor patch210against the backside (top surface) of the hand10. InFIG.6B, the substrate212of the sensor patch210includes a rigid base plate212b1that is attached to an elastic band212b2that pulls the rigid base212b1and sensor patch210against the backside of the hand10. InFIG.6C, the substrate212of the sensor patch210is an elastic band212cthat pulls the sensor patch210against the backside of the hand10. Increasing the contact pressure of the sensor patch210on the hand10may improve optical coupling (e.g., better transmission into the hand10and detection by the photodiode216).

FIG.7shows a flowchart in accordance with one or more embodiments of the invention. One or more of the individual processes inFIG.7may be performed by the system200ofFIG.3, as described above. One or more of the individual processes shown inFIG.7may be omitted, repeated, and/or performed in a different order than the order shown inFIG.7. Accordingly, the scope of the invention should not be limited by the specific arrangement as depicted inFIG.7.

At710, the sensor patch210is attached to a skin surface of a backside of a hand10. The relatively thin layer of flesh between the skin and metacarpals12on the backside of the hand10allows for a direct correlation between the backscattered light intensities from the skin surface of the backside of the hand10and the movement of the metacarpal bones12. Alternatively, the sensor patch210may be long enough or wide enough to extend around other portions of the hand10or to any other appropriate region of the hand10to obtain backscattered light intensities from any bones of the hand10. The substrate212of the sensor patch210may be flexible to allow the light emitting layer214to conform to the backside of the hand10.

In one or more embodiments, the sensor patch210includes a plurality of photodiodes216and the sensor patch210is disposed such that the plurality of photodiodes216are aligned at positions that correspond to the metacarpals12of the hand10.

At720, the light emitting layer214emits a first wavelength of light into the hand10. The first wavelength of light may be any wavelength that penetrates the skin and flesh of the hand10. In one or more embodiments, the first wavelength of light may be in the visible regime (e.g., 400-700 nm) or in the infrared regime (e.g., >700 nm).

In one or more embodiments, at725, the light emitting layer214emits a second wavelength of light into the hand10. The second wavelength of light may be any wavelength that penetrates the skin and flesh of the hand10. In one or more embodiments, the second wavelength of light may be in the visible regime (e.g., 400-700 nm) or in the infrared regime (e.g., >700 nm). The first wavelength of light and the second wavelength of light are different wavelength to allow for more complex analysis. The first wavelength of light and the second wavelength of light may be emitted at different times. In one or more embodiments, each of the wavelengths of light may be emitted with a duty cycle of 50% or less.

In one or more embodiments, the light emitting layer214may emit any number of wavelengths of light to preform multiple different measurements (e.g., bone tracking, vital sensing) sequentially or simultaneously. For example, oxygenated blood and deoxygenated blood have absorption peaks at different wavelengths (e.g., 555 nm for deoxygenated hemoglobin, and 579 nm for oxygenated hemoglobin) and the relative difference in backscattered light intensities from the two wavelengths can be used to monitor the user's vital health signals (e.g., pulse, blood oxygen level, breathing rate).

In one or more embodiments, at720and/or725, the first and/or second wavelength of light emitted by the light emitting layer214is prevented from backscattering off of the photodiode216by a shadow mask214a.

At730, the photodiode216detects backscattered light intensities from the skin surface of the backside of the hand10. Detecting of the backscattered light intensities from the skin surface of the hand10may include spatially filtering the backscattered light a sectioning mask218. The sectioning mask218prevents detection of stray light by the photodiode216. For example, the sectioning mask218may be shaped and disposed to surround each of a plurality of photodiodes216. In the example shown inFIGS.5A-5B, light backscattered from the skin surface above the metacarpal12c, which is primarily monitored by photodiode216c, is prevented from impinging on photodiodes12b,dby the sectioning mask218.

At740, a processor determines finger pose information by analyzing the detected backscattered light intensities from the photodiode216. In one or more embodiments, the detected backscattered light intensities are communicated from the photodiode216to a support band230(e.g., a wearable wristband device such a smart watch or a fitness tracker) that includes the processor234. The output signals from the one or more photodiodes216may be signal processed (e.g., filtering, amplifying, smoothing, etc.) and then analyzed simultaneously by a pretrained pattern recognition machine learning model, which is discussed in further detail below with respect toFIG.8. The machine learning model may identify finger orientations or classify the output signals into various finger actions as the finger pose information. Some examples of the classification may include picking up a soft object; picking up a hard object; typing on a keyboard, typing on a personal device (e.g., smart phone); holding a handle (e.g., a racket/rod/club/bike handle for a sport activity); holding a tool (e.g., a tool handle/grip). The accuracy of the finger pose information may be increased due to the simultaneously monitoring and associating the output signals that are caused by the movements of multiple metacarpals12.

At750, the finger pose information is transmitted from the system200to an external device (e.g., a smart phone, a computer, a network, etc.). In one or more embodiments, the finger pose information is transmitted by a transceiver238in the support band230. The transceiver238may use wired or wireless communication protocols to transmit the finger pose information.

FIG.8shows implementation example in accordance with one or more embodiments of the invention. In one or more embodiments, the system200uses a machine learning model that includes a deep learning neural network to determine the finger pose information. For example, one or more machine learning algorithms are used to train a machine learning model800to accept scattered light intensity820and output finger pose information830. In some embodiments, real, synthetic, and/or augmented (e.g., curated or supplemented data) scattered light intensity data may be combined to produce a large amount of interpreted data for training the machine learning model800.

The neural network810may be a deep learning neural network and may include one or more hidden layers812a,b,c,d, where each hidden layer includes one or more modelling nodes (i.e., neurons). The neural network810may be pretrained by a labelled dataset and may be semi-supervised or unsupervised (e.g., such one short learning). Furthermore, the hidden layers812a,b,c,dmay include various hidden layer types (e.g., convolutional, pooling, filtering, down-sampling, up-sampling, layering, regression, dropout, etc.). In some embodiments, the number of hidden layers may be greater than or less than the four layers shown inFIG.8. The hidden layers812a,b,c,dcan be arranged in any order.

Each neuron in the neural network810may combine one or more data points (e.g., from multiple photodiodes216from the input backscattered light intensities820) and associate the data with a set of coefficients (i.e., weighted values within the neural network810). Using the coefficients, the neurons amplify or reduce the value of the input data to assign an amount of significance of the input data. Through the training of the neural network810, the neurons are trained to determine which data inputs should receive greater priority in determining one or more specified outputs (e.g., specific finger poses or finger action categories). The weighted inputs and outputs are communicated through the neurons and a neuron's activation function may pass connect neurons between one or more of the hidden layers812a,b,c,dwithin the neural network810. In other words, the activation function of each neuron may determine how the output of that neuron progresses to other neurons and hidden layers812before a final output is determined.

For example, the input data820may be convolved with pre-learned filters that are designed to highlight specific characteristics. In one or more embodiments, training data is directly obtained by emitting the first wavelength of light into the hand and detecting the backscattered light intensities during known finger poses (i.e., an initial calibration by the user). In one or more embodiments, the user's training data is supplemented by previously obtained real, synthetic, and/or augmented data. Using the available data, a machine learning algorithm trains the machine learning model800and deep learning neural network810to accept the input backscattered light intensities820and output the finger pose information830.

The above example is for explanatory purposes only and not intended to limit the scope of how the backscattered light intensities820are analyzed to produce finger pose information830. WhileFIG.8shows an example configuration, other machine learning configurations may be used without departing from the scope of the disclosure. For example, one or more of the individual components shown inFIG.8may be omitted, repeated, replaced with an appropriate alternative model different from what is shown inFIG.8. Accordingly, the scope of the invention should not be limited by the specific configuration depicted inFIG.8.

Embodiments of the invention may have one or more of the following advantages: reducing the size, bulkiness, heaviness, complexity, and intrusiveness of finger tracking systems compared to systems that utilize actuated gloves; improving the structural flexibility, comfort while wearing, comfort over usage duration, and adaptability of the form factor (e.g., flexible design and esthetics) of finger tracking systems; reducing manufacturing cost (low cost components), and manufacturing complexity (easy scale up, roll to roll, pick and place manufacturing); reducing hardware and computational resource requirements (i.e., less processing power, lower memory requirements, lower power requirements, lower communication bandwidth requirements) compared to computationally expensive machine vision systems; improving the ability to visualize and simulate a user's presence in a virtual environment.

Although the disclosure has been described with respect to a limited number of embodiments, those skilled in the art, having benefit of this disclosure, will appreciate that various other embodiments may be devised without departing from the scope of the present invention. Accordingly, the scope of the invention should be limited only by the attached claims.