Patent ID: 12249185

DETAILED DESCRIPTION

Overview

The present disclosure provides for using multiple IMUs to recognize particular user activity, such as particular types of exercises and repetitions of such exercises. The IMUs may be located in consumer products, such as smartwatches and earbuds. Each IMU may include an accelerometer and a gyroscope, each with three axes of measurement, for a total of 12 raw measurement streams. In some examples, additional IMUs may also provide data, thus resulting in additional measurement streams. Further, each set of three axes can be combined into a spatial norm. Thus, in the example of two IMUs producing 12 data streams, adding the spatial norms would provide for a final total of 16 data capture tiles per training image.

The system and method described herein provides an ability to count repetitions of a single exercise without having to resort to using complicated window detection methodologies. A simple fixed overlapping window is used, and the number of repetitions may be determined using auto-correlation techniques, instantiation type convolutional neural networks (CNNs), or a combination of these or other techniques. The method may be deployed using efficient models, which may be retrainable in a final layer by the users themselves. For example, a user could retrain the model on their own device to recognize customizable exercise types, and their own unique motion profile during exercise.

Just a few examples of the types of exercises that may be detected include bicep curls, barbell press ups, push ups, sit ups, squats, chin-ups, burpees, jumping jacks, etc. It should be understood that any of a variety of additional types of exercises may be detected as well. The system may be trained by a user to detect a particular exercise selected or created by the user. In training the machine learning model, non-exercise may be included as well. For example, this may help to identify and distinguish other types of movements of the user, and thereby reduce false positive detection of exercises. Such non-exercise may include, by way of example only, walking, climbing stairs, opening doors, lifting various objects, sitting in chairs, etc.

Transfer learning may be implemented to retrain the top few layers of an efficient image recognition model, such as a MobileNet image recognition model. Images of IMU raw data subplots may be used as training examples, allowing for high accuracy with a small number of training examples.

Example Systems

FIG.1is a pictorial diagram of an example system in use. User102is wearing wireless computing devices180,190. In this example, the wireless computing devices include earbuds180worn on the user's head and a smartwatch190worn on the user's wrist. The earbuds180and smartwatch190may be in wireless communication with a host computing device170, such as a mobile phone. The host computing device170may be carried by the user, such as in the user's hand or pocket, or may be placed anywhere near the user. In some examples, the host computing device170may not be needed at all.

The wireless computing devices180,190worn by the user may detect particular types of exercises and repetitions of such exercises performed by the user102. For example, as shown the user102is doing jumping jacks. The smartwatch190, which is typically fixed to the user's arm, will detect relatively large, quick, sweeping movements. The earbuds190, which are typically fixed in the user's ears, will detect bouncing up and down. The wireless computing devices180,190may communicate such detections to each other or to the host device170. Based on the combination of detected movements, one or more of the devices170-190may detect that the user102is doing jumping jacks.

While in the example shown the wireless computing devices180,190include earbuds and a smartwatch, it should be understood that in other examples any of a number of different types of wireless devices may be used. For example, the wireless devices may include a headset, a head-mounted display, smart glasses, a pendant, an ankle-strapped device, a waist belt, etc. Moreover, while two wireless devices are shown as being used to detect the exercises inFIG.1, additional wireless devices may also be used. Further, while two earbuds180are shown, the readings detected by each earbud may be redundant, and therefore detection of the user's movements may be performed using only one earbud in combination with the smartwatch190or another device worn by the user102.

FIG.2further illustrates the wireless computing devices180,190, in communication with the host computing device170, and features and components thereof.

As shown, each of the wearable wireless devices180,190includes various components, such as processors281,291, memory282,292, transceiver285,295, and other components typically present in wearable wireless computing devices. The wearable devices180,190may have all of the components normally used in connection with a wearable computing device such as a processor, memory (e.g., RAM and internal hard drives) storing data and instructions, user input, and output.

Each of the wireless devices180,190may also be equipped with short range wireless pairing technology, such as a Bluetooth transceiver, allowing for wireless coupling with each other and other devices. For example, transceivers285,295may each include an antenna, transmitter, and receiver that allows for wireless coupling with another device. The wireless coupling may be established using any of a variety of techniques, such as Bluetooth, Bluetooth low energy (BLE), etc.

Each of the wireless devices180,190may further be equipped with one or more sensors286,296capable of detecting the user's movements. The sensors may include, for example, IMU sensors287,297, such as an accelerometer, gyroscope, etc. For example, the gyroscopes may detect inertial positions of the wearable devices180,190, while the accelerometers detect linear movements of the wearable devices180,190. Such sensors may detect direction, speed, and/or other parameters of the movements. The sensors may additionally or alternatively include any other type of sensors capable of detecting changes in received data, where such changes may be correlated with user movements. For example, the sensors may include a barometer, motion sensor, temperature sensor, a magnetometer, a pedometer, a global positioning system (GPS), camera, microphone, etc. The one or more sensors of each device may operate independently or in concert.

The host computing device170may be, for example, a mobile phone, tablet, laptop, gaming system, or any other type of mobile computing device. In some examples, the mobile computing device170may be coupled to a network, such as a cellular network, wireless Internet network, etc.

The host device170may also include one or more processors271in communication with memory272including instructions273and data274. The host device170may further include elements typically found in computing devices, such as output275, input276, communication interfaces, etc.

The input276and output275may be used to receive information from a user and provide information to the user. The input may include, for example, one or more touch sensitive inputs, a microphone, a camera, sensors, etc. Moreover, the input276may include an interface for receiving data from the wearable wireless devices180,190. The output275may include, for example, a speaker, display, haptic feedback, the interface with the wearable wireless devices for providing data to such devices, etc.

The one or more processor271may be any conventional processors, such as commercially available microprocessors. Alternatively, the one or more processors may be a dedicated device such as an application specific integrated circuit (ASIC) or other hardware-based processor. AlthoughFIG.2functionally illustrates the processor, memory, and other elements of host170as being within the same block, it will be understood by those of ordinary skill in the art that the processor, computing device, or memory may actually include multiple processors, computing devices, or memories that may or may not be stored within the same physical housing. Similarly, the memory may be a hard drive or other storage media located in a housing different from that of host170. Accordingly, references to a processor or computing device will be understood to include references to a collection of processors or computing devices or memories that may or may not operate in parallel.

Memory272may store information that is accessible by the processors271, including instructions273that may be executed by the processors271, and data274. The memory272may be of a type of memory operative to store information accessible by the processors271, including a non-transitory computer-readable medium, or other medium that stores data that may be read with the aid of an electronic device, such as a hard-drive, memory card, read-only memory (“ROM”), random access memory (“RAM”), optical disks, as well as other write-capable and read-only memories. The subject matter disclosed herein may include different combinations of the foregoing, whereby different portions of the instructions273and data274are stored on different types of media.

Data274may be retrieved, stored or modified by processors271in accordance with the instructions273. For instance, although the present disclosure is not limited by a particular data structure, the data274may be stored in computer registers, in a relational database as a table having a plurality of different fields and records, XML documents, or flat files. The data274may also be formatted in a computer-readable format such as, but not limited to, binary values, ASCII or Unicode. By further way of example only, the data274may be stored as bitmaps comprised of pixels that are stored in compressed or uncompressed, or various image formats (e.g., JPEG), vector-based formats (e.g., SVG) or computer instructions for drawing graphics. Moreover, the data274may comprise information sufficient to identify the relevant information, such as numbers, descriptive text, proprietary codes, pointers, references to data stored in other memories (including other network locations) or information that is used by a function to calculate the relevant data.

The instructions273may be executed to detect a type of exercise performed by the user based on raw data received from the sensors286,296of the wireless wearable devices180,190. For example, the processor271may execute a machine learning algorithm whereby it compares images of the received raw data with stored image corresponding to particular exercises, and detects the exercise performed based on the comparison. Moreover, the instructions273may be executed to detect a number of repetitions of the exercise, such as by using a window.

In other examples, the analysis of the sensor data may be performed by either or both of the wearable wireless devices180,190. For example, each of the devices180,190includes a processor281,291and memory282,292, similar to those described above in connection with host device170. These processors281,291and memories282,292may receive data and execute the machine learning algorithm to detect the type of exercise performed.

FIG.3illustrates the wireless wearable devices180,190in communication with each other and the host device170. The wireless connections among the devices may be, for example, short range pairing connections, such as Bluetooth. Other types of wireless connections are also possible. In this example, the devices170-190are further in communication with server310and database315through network150. For example, the wireless wearable devices180,190may be indirectly connected to the network150through the host device170. In other examples, one or both of the wireless wearable devices180,190may be directed connected to the network150, regardless of a presence of the host device170.

These network150may be, for example, a LAN, WAN, the Internet, etc. The connections between devices and the network may be wired or wireless.

The server computing device310may actually include a plurality of processing devices in communication with one another. According to some examples, the server310may execute the machine learning model for determining a particular type of exercise being performed based on input from the IMUs of multiple wearable devices. For example, the wearable devices180,190may transmit raw data detected from their IMUs to the server310. The server310may perform computations using the received raw data as input, determine the type of exercise performed, and send the result back to the host170or one or both of the wearable devices180,190.

Databases315may be accessible by the server310and computing devices170-190. The databases315may include, for example, a collection of data from various sources corresponding to particular types of exercises. For example, the data may include images of raw data streams from IMUs or other sensors in wearable devices, the raw data streams corresponding to particular types of exercise. Such data may be used in the machine learning model executed by the server310or by any of the host device170or the wearable devices180,190.

Regardless of whether the detection of exercise type is performed at the server310, at the host170, at one or both of the wearable devices180,190, or some combination thereof, any of several different types of computations may be performed. These different types include at least (1) peak detector window segmentation using input from multiple IMUs positioned on different locations of the user's body, or (2) fixed window segmentation from the multiple IMUs. Moreover, any of the devices170-190or310may be capable of counting the number of repetitions of each exercise.

Peak detector window segmentation using input from multiple IMUs positioned on different locations of the user's body detects “peaks” in signal data, such as raw accelerometer and gyroscope data.FIG.4Aillustrates an example of raw accelerometer data from an earbud, andFIG.4Billustrates an example of raw accelerometer data from a smartwatch. In this particular example, the data was obtained while doing squats, but it should be understood that the analysis can be applied to data for any of a variety of exercises. Each figure includes three waveforms: one for each of the x, y, and z directions. For example, inFIG.4A, wave402corresponds to the y direction, such as a vertical direction; wave406corresponds to an x direction, such as lateral or side-to-side movement with respect to the user; and wave404corresponds to a z direction, such as forward/backward movement relative to the user. Similarly, inFIG.4B, wave412corresponds to the y direction, wave416corresponds to the x direction, and wave414corresponds to the z direction.

The data from a first IMU may be timescaled to match time from a second IMU. For example, measurements from the IMU in an earbud may be timescaled to match measured time from an accelerometer in a smartwatch. This may include resampling all data to a length of the smartwatch accelerometer data. A low pass filter may be applied to the raw data. By way of example only, the low pass filter may be a Butterworth filter or any of a variety of other types of filter.

A “window” captures each repetition using custom peak detection and thresholding. A window, in peak detection segmentation, may refer to capturing one complete repetition of an exercise. The window may begin anywhere during the exercise repetition. For example, it could start in the middle of one pushup and end in the middle of a second pushup. The window may begin/end at a zero crossing, or some peak or trough, or any other feature that can be extracted from the data. For a fixed window, the window may be defined by a time length. For example, the time length may be 4 seconds, with windows overlapping every 2 seconds (50% overlap). This window may capture a partial repetition or multiple repetitions, depending on how fast the person exercises. Another option would be to have an autoscaling window, that autoscales the time length of the window based on the most recent repetition frequency estimate from the autocorrelation calculation. In that case the window is still determined by time, and may not exactly capture a full repetition. For example, the repetition length may be over or underestimated, but the time window would in general be close to the repetition length of the exercise.

Accordingly, for example, for a waveform of raw data received from an IMU during the exercise, a peak of the waveform is detected. Analysis of the waveform may further identify other characteristics to indicate the beginning and end of a repetition. If two IMUs are used, each having an x, y, and z axis, 12 raw data streams are included in each window. Adding 4 norms of each set of x, y, and z brings this to 16 data streams total. The norm may be computed as, for example, square_root(x{circumflex over ( )}2+y{circumflex over ( )}2+z{circumflex over ( )}2). Other mathematical manipulations could also be possible that may provide additional beneficial information to the model. Accordingly, 16 subplots are provided for each image, such as in a 4×4 grid.

FIGS.5A-Eillustrate example images created using the peak detection window segmentation technique described above with two IMUs for various types of exercises. For example,FIG.5Aillustrate an example image for a bicep curl,FIG.5Billustrates an example image for a weight press,FIG.5Cillustrates an example image for a pushup,FIG.5Dillustrates an example image for a situp, andFIG.5Eillustrates an example image for a squat. These images may be used, for example, for training the machine learning model to recognize exercises by movements detected by the IMUs in the wearable devices.

FIG.6illustrates an example image for a non-exercise. Examples of non-exercises may include, without limitation, walking, climbing stairs, working at a desk, opening doors, picking up objects, etc. Non-exercise images may be included in the training data to help distinguish between particular exercise types and the non-exercises. Recognizing such distinctions may help to prevent false positive detection of exercises. While one example image in shown inFIG.6, it should be understood that any number of example images corresponding to various non-exercises may be included in the training data.

According to another type of computation, the fixed window segmentation from the multiple IMUs, the data processing flow described above is modified by using a fixed window of a predetermined width. By way of example only, the fixed window may be 4 seconds in width, with 50% overlap. Because it is fixed, it is not synchronous with the exercise repetition, and may even have multiple or partial repetitions within a single windowed training or test example image. Because the window start is random with respect to the exercise frequency, it is possible to generate more training examples from the same set of data by varying the initial window start time. By stepping the window start time by 0.5 s, the total number of training examples per exercise is multiplied by 8.

According to yet another example, the fixed window may be autoscaled according to the repetition frequency estimate. This option does not require peak detection or thresholding.

FIGS.7A-Billustrate example images of exercises using fixed window segmentation. In these examples,FIG.7Arepresents a first example of a bicep curl, andFIG.7Brepresents a second example of a bicep curl. For example, the first example ofFIG.7Awas generated using a first start time for the window, while the second example ofFIG.7Bwas generated using a second start time different than the first start time. For example, if using a 4 s window, the second start time may be anywhere between 0.5 s-3.5 s later than the first start time. However, it should be understood that the width of the window may be varied, such as by using 2 s, 6 s, 10 s, etc. Moreover, the start time increments for the window may also be varied from 0.5 s to any other value. The machine learning model thus learns patterns within the image, without relying on specific start or stop points.

In addition to being able to classify exercise type, the number of repetitions of a given exercise type may be counted in real time. With peak detector windowing, each test image, when classified, becomes a single repetition. With fixed windowing, counting repetitions may be performed using a different method, such as autocorrelation or model instantiation.

In using auto-correlation, a sliding auto-correlation window of a fixed duration is used during exercise. The repetition frequency can therefore be extracted.FIG.8illustrates an example of autocorrelation. Using a 4 s sliding window during 10 s of pushups, an indicator of repetition frequency can be extracted. For example, the indicator may be the location of the highest peak, excluding the peak at time t=0. Once the repetition frequency is known, the number of repetitions in a given exercise sequence can be calculated as the exercise duration in seconds multiplied by the repetition frequency. As long as the devices are detecting a specific exercise type, a timer could run to track total duration. Using this method, the user could receive an update once every few seconds on a repetition counter.

In using instantiation, the model could learn to count the repetitions itself by using instantiation. In this method, the model is trained not only with labeled examples, but also with segments, or “instances” of a repetition. For example, in a training example image that contains 10 repetitions, each repetition may be labeled in the training example. In this way, the model learns to estimate how many repetitions are in a given image.

Example Methods

In addition to the operations described above and illustrated in the figures, various operations will now be described. It should be understood that the following operations do not have to be performed in the precise order described below. Rather, various steps can be handled in a different order or simultaneously, and steps may also be added or omitted.

FIG.9illustrates an example method900of detecting a type of exercise being performed by a user. The method may be performed by one or more processors in a first or second wearable device or in a host device in communication with the wearable devices.

In block910, first sensor data is received from one or more first sensors of the first wearable device. The first wearable device may be, for example, an earbud. The one or more first sensors may include an IMU, such as including a gyroscope and accelerometer. Each of the gyroscope and accelerometer may produce data streams for measurements in the x, y, and z directions.

In block920, second sensor data is received from one or more second sensors of the second wearable device. The second wearable device may be, for example, a smartwatch. Similar to the first sensors, the one or more second sensors may also include an IMU producing data streams for measurements in multiple spatial directions.

In block930, an image is generated based on the first sensor data and the second sensor data. The image may include a plurality of tiles or subplots, wherein each tile or subplot depicts a separate data stream from the first and second sensors. For example, a first tile depicts a data stream from the accelerometer of the first sensor in the x direction, a second tile depicts a data stream from the accelerometer of the first sensor in the y direction, and so on. Using two IMUs, each having two sensors measuring in three directions, twelve tiles may be generated. Additionally, each set of three (x, y, z) axes can be combined into a spatial norm, for a final total of 16 data capture tiles per image.

In block940, a type of exercise performed by the user during receipt of the first and second sensor data is determined based on the type of exercise. For example, a machine learning model is applied to the generated image. The machine learning model may be trained using image data, as described in further detail below in connection withFIG.10. The machine learning model may be, for example, an image based deep neural network, such as a convolutional neural network (CNN).

In block950, a number of repetitions of the exercise may be determined. For example, the machine learning model may be trained to recognize repetitions if it is trained with images wherein the repetitions are labeled. According to other examples, the repetitions may be detected by using autocorrelation to determine a repetition frequency, and then multiplying that repetition frequency by a duration of the exercise type.

FIG.10illustrates an example method1000of training a machine learning model to detect exercise types. The method may be performed, for example, by one or more processors in a wearable device, a coupled host device, and/or on a remote server connected to the wearable devices or host through a network.

In blocks1010-1020, data may be received as described above in connection with blocks910-920ofFIG.9. However, the data may be received over an extended period of time as the user performs several types of exercises and non-exercises.

In block1030, the received data may be used to generate one or more first training images. For example, similar to generation of the image described above in connection with block930orFIG.9, the first training image may include a plurality of image tiles, each depicting a different data stream for data collected while the user was performing a first type of exercise. Similarly, in block1040, the received data may be used to generate a second training image depicting data streams collected while the user was performing a second type of exercise different than the first type. This may be repeated for any number of different types of exercise.

In block1050, a third training image is generated using techniques similar to those described in block1030. However, the third training image may correspond to non-exercise. For example, the data represented in the tiles of the third image may be collected while the user is performing activities other than specific exercises. Such activities may include any of a variety of non-exercises, such as teeth-brushing, cooking, sitting, etc.

In block1060, the first, second, and third images are input to the machine learning model as training data. In this regard, the machine learning model learns to recognize various types of exercises, as well as to distinguish such exercises from non-exercises. Such distinction can help to reduce false positive detections.

The foregoing system and methods are advantageous in that they provide for highly accurate detection of exercises using IMUs in products available to and commonly used by users.

Unless otherwise stated, the foregoing alternative examples are not mutually exclusive, but may be implemented in various combinations to achieve unique advantages. As these and other variations and combinations of the features discussed above can be utilized without departing from the subject matter defined by the claims, the foregoing description of the embodiments should be taken by way of illustration rather than by way of limitation of the subject matter defined by the claims. In addition, the provision of the examples described herein, as well as clauses phrased as “such as,” “including” and the like, should not be interpreted as limiting the subject matter of the claims to the specific examples; rather, the examples are intended to illustrate only one of many possible embodiments. Further, the same reference numbers in different drawings can identify the same or similar elements.