Patent Description:
There has been some initial investigation into using an image based neural network to accomplish Inertial Measurement Unit (IMU) activity recognition. Image based models tend to be much more mature and generalized. In this sense, the "feature" based models appear similar to "feature" based extraction in earlier days of image recognition, which had limited success. Existing image based systems use a peak and threshold detector to determine where to window a single repetition of an exercise type. However, the threshold will vary person to person, and can be easily spoofed in the presence of noise. Also, there is no universal guidance for this method across all exercise types, requiring a custom analytic window detection methodology for each possible exercise type, while also trying to make the methodology robust across multiple users. <NPL>, discloses determining human activity based on knee-mounted sensors.

One aspect of the disclosure provides a method for detecting exercise. The method includes receiving, by one or more processors, first sensor data from one or more first sensors of a first wearable device, receiving, by the one or more processors, second sensor data from one or more second sensors of a second wearable device, generating, by the one or more processors, an image based on the first sensor data and the second sensor data, the image comprising a plurality of subplots, wherein each subplot depicts a data stream, and determining, using the one or more processors, based on the image, a type of exercise performed by a user during receiving of the first sensor data and the second sensor data. Determining the type of exercise performed includes executing a machine learning model, such as an image based machine learning model. Further, the model is trained by generating one or more first training images based on data collected from the first and second wearable devices while the user performs a first type of exercise, and inputting the one or more first training images into the machine learning model as training data. Training may further include generating one or more second training images based on data collected from the first and second wearable devices while the user performs a second type of exercise, and inputting the one or more second training images into the machine learning model as training data, as well as generating one or more third training images based on data collected from the first and second wearable devices while the user performs one or more activities that are not classified as exercise, and inputting the one or more third training images into the machine learning model as examples of non-exercise. Generating the one or more first training images includes fixed window segmentation. Fixed window segmentation may include varying a start time of a window having a predetermined length of time, wherein each variation of the start time in the window generates a separate one of the one or more first training images. The method further includes autoscaling a time length of the window based on a most recent repetition frequency estimate.

According to some examples, the method may further include determining a number of repetitions of the determined type of exercise. For example, this may include determining a repetition frequency using a sliding autocorrelation window, and computing the number of repetitions based on the repetition frequency and a duration of the type of exercise. In another example, determining the number of repetitions may include counting a number of repetitions in a training image, labeling the repetitions in the training image, and inputting the training image into a machine learning model as training data.

Another aspect of the disclosure provides a system for detecting exercise, including one or more first sensors in a first wearable device, one or more second sensors in a second wearable device, and one or more processors in communication with the one or more first sensors and the one or more second sensors. The one or more processors are configured to receive first sensor data from the one or more first sensors of the first wearable device, receive second sensor data from the one or more second sensors of the second wearable device, generate an image based on the first sensor data and the second sensor data, the image comprising a plurality of subplots, wherein each subplot depicts a data stream, and determine, based on the image, a type of exercise performed by a user during receiving of the first sensor data and the second sensor data. According to some examples, the first wearable device may be an earbud and the second wearable device may be a smartwatch. The one or more processors may reside within at least one of the first wearable device or the second wearable device, and/or within a host device coupled to the first wearable device and the second wearable device.

Yet another aspect of the disclosure provides a non-transitory computer-readable medium storing instructions executable by one or more processors in for performing a method of detecting a type of exercise. Such method includes receiving first sensor data from one or more first sensors of a first wearable device, receiving second sensor data from one or more second sensors of a second wearable device generating an image based on the first sensor data and the second sensor data, the image comprising a plurality of subplots, wherein each subplot depicts a data stream, and determining based on the image, a type of exercise performed by a user during receiving of the first sensor data and the second sensor data.

The present disclosure provides for using multiple IMUs to recognize particular user activity, such as particular types of exercises and repetitions of such exercises. The IMUs may be located in consumer products, such as smartwatches and earbuds. Each IMU may include an accelerometer and a gyroscope, each with three axes of measurement, for a total of <NUM> raw measurement streams. In some examples, additional IMUs may also provide data, thus resulting in additional measurement streams. Further, each set of three axes can be combined into a spatial norm. Thus, in the example of two IMUs producing <NUM> data streams, adding the spatial norms would provide for a final total of <NUM> data capture tiles per training image.

The system and method described herein provides an ability to count repetitions of a single exercise without having to resort to using complicated window detection methodologies. A simple fixed overlapping window is used, and the number of repetitions may be determined using auto-correlation techniques, instantiation type convolutional neural networks (CNNs), or a combination of these or other techniques. The method may be deployed using efficient models, which may be retrainable in a final layer by the users themselves. For example, a user could retrain the model on their own device to recognize customizable exercise types, and their own unique motion profile during exercise.

Just a few examples of the types of exercises that may be detected include bicep curls, barbell press ups, push ups, sit ups, squats, chin-ups, burpees, jumping jacks, etc. It should be understood that any of a variety of additional types of exercises may be detected as well. The system may be trained by a user to detect a particular exercise selected or created by the user. In training the machine learning model, non-exercise may be included as well. For example, this may help to identify and distinguish other types of movements of the user, and thereby reduce false positive detection of exercises. Such non-exercise may include, by way of example only, walking, climbing stairs, opening doors, lifting various objects, sitting in chairs, etc..

Transfer learning may be implemented to retrain the top few layers of an efficient image recognition model, such as a MobileNet image recognition model. Images of IMU raw data subplots may be used as training examples, allowing for high accuracy with a small number of training examples.

<FIG> is a pictorial diagram of an example system in use. User <NUM> is wearing wireless computing devices <NUM>, <NUM>. In this example, the wireless computing devices include earbuds <NUM> worn on the user's head and a smartwatch <NUM> worn on the user's wrist. The earbuds <NUM> and smartwatch <NUM> may be in wireless communication with a host computing device <NUM>, such as a mobile phone. The host computing device <NUM> may be carried by the user, such as in the user's hand or pocket, or may be placed anywhere near the user. In some examples, the host computing device <NUM> may not be needed at all.

The wireless computing devices <NUM>, <NUM> worn by the user may detect particular types of exercises and repetitions of such exercises performed by the user <NUM>. For example, as shown the user <NUM> is doing jumping jacks. The smartwatch <NUM>, which is typically fixed to the user's arm, will detect relatively large, quick, sweeping movements. The earbuds <NUM>, which are typically fixed in the user's ears, will detect bouncing up and down. The wireless computing devices <NUM>, <NUM> may communicate such detections to each other or to the host device <NUM>. Based on the combination of detected movements, one or more of the devices <NUM>-<NUM> may detect that the user <NUM> is doing jumping jacks.

While in the example shown the wireless computing devices <NUM>, <NUM> include earbuds and a smartwatch, it should be understood that in other examples any of a number of different types of wireless devices may be used. For example, the wireless devices may include a headset, a head-mounted display, smart glasses, a pendant, an ankle-strapped device, a waist belt, etc. Moreover, while two wireless devices are shown as being used to detect the exercises in <FIG>, additional wireless devices may also be used. Further, while two earbuds180 are shown, the readings detected by each earbud may be redundant, and therefore detection of the user's movements may be performed using only one earbud in combination with the smartwatch <NUM> or another device worn by the user <NUM>.

<FIG> further illustrates the wireless computing devices <NUM>, <NUM>, in communication with the host computing device <NUM>, and features and components thereof.

As shown, each of the wearable wireless devices <NUM>, <NUM> includes various components, such as processors <NUM>, <NUM>, memory <NUM>, <NUM>, transceiver <NUM>, <NUM>, and other components typically present in wearable wireless computing devices. The wearable devices <NUM>, <NUM> may have all of the components normally used in connection with a wearable computing device such as a processor, memory (e.g., RAM and internal hard drives) storing data and instructions, user input, and output.

Each of the wireless devices <NUM>, <NUM> may also be equipped with short range wireless pairing technology, such as a Bluetooth transceiver, allowing for wireless coupling with each other and other devices. For example, transceivers <NUM>, <NUM> may each include an antenna, transmitter, and receiver that allows for wireless coupling with another device. The wireless coupling may be established using any of a variety of techniques, such as Bluetooth, Bluetooth low energy (BLE), etc..

Each of the wireless devices <NUM>, <NUM> may further be equipped with one or more sensors <NUM>, <NUM> capable of detecting the user's movements. The sensors may include, for example, IMU sensors <NUM>, <NUM>, such as an accelerometer, gyroscope, etc. For example, the gyroscopes may detect inertial positions of the wearable devices <NUM>, <NUM>, while the accelerometers detect linear movements of the wearable devices <NUM>, <NUM>. Such sensors may detect direction, speed, and/or other parameters of the movements. The sensors may additionally or alternatively include any other type of sensors capable of detecting changes in received data, where such changes may be correlated with user movements. For example, the sensors may include a barometer, motion sensor, temperature sensor, a magnetometer, a pedometer, a global positioning system (GPS), camera, microphone, etc. The one or more sensors of each device may operate independently or in concert.

The host computing device <NUM> may be, for example, a mobile phone, tablet, laptop, gaming system, or any other type of mobile computing device. In some examples, the mobile computing device <NUM> may be coupled to a network, such as a cellular network, wireless Internet network, etc..

The host device <NUM> may also include one or more processors <NUM> in communication with memory <NUM> including instructions <NUM> and data <NUM>. The host device <NUM> may further include elements typically found in computing devices, such as output <NUM>, input <NUM>, communication interfaces, etc..

The input <NUM> and output <NUM> may be used to receive information from a user and provide information to the user. The input may include, for example, one or more touch sensitive inputs, a microphone, a camera, sensors, etc. Moreover, the input <NUM> may include an interface for receiving data from the wearable wireless devices <NUM>, <NUM>. The output <NUM> may include, for example, a speaker, display, haptic feedback, the interface with the wearable wireless devices for providing data to such devices, etc..

The one or more processor <NUM> may be any conventional processors, such as commercially available microprocessors. Alternatively, the one or more processors may be a dedicated device such as an application specific integrated circuit (ASIC) or other hardware-based processor. Although <FIG> functionally illustrates the processor, memory, and other elements of host <NUM> as being within the same block, it will be understood by those of ordinary skill in the art that the processor, computing device, or memory may actually include multiple processors, computing devices, or memories that may or may not be stored within the same physical housing. Similarly, the memory may be a hard drive or other storage media located in a housing different from that of host <NUM>. Accordingly, references to a processor or computing device will be understood to include references to a collection of processors or computing devices or memories that may or may not operate in parallel.

Memory <NUM> may store information that is accessible by the processors <NUM>, including instructions <NUM> that may be executed by the processors <NUM>, and data <NUM>. The memory <NUM> may be of a type of memory operative to store information accessible by the processors <NUM>, including a non-transitory computer-readable medium, or other medium that stores data that may be read with the aid of an electronic device, such as a hard-drive, memory card, read-only memory ("ROM"), random access memory ("RAM"), optical disks, as well as other write-capable and read-only memories. The subject matter disclosed herein may include different combinations of the foregoing, whereby different portions of the instructions <NUM> and data <NUM> are stored on different types of media.

Data <NUM> may be retrieved, stored or modified by processors <NUM> in accordance with the instructions <NUM>. For instance, although the present disclosure is not limited by a particular data structure, the data <NUM> may be stored in computer registers, in a relational database as a table having a plurality of different fields and records, XML documents, or flat files. The data <NUM> may also be formatted in a computer-readable format such as, but not limited to, binary values, ASCII or Unicode. By further way of example only, the data <NUM> may be stored as bitmaps comprised of pixels that are stored in compressed or uncompressed, or various image formats (e.g., JPEG), vector-based formats (e.g., SVG) or computer instructions for drawing graphics. Moreover, the data <NUM> may comprise information sufficient to identify the relevant information, such as numbers, descriptive text, proprietary codes, pointers, references to data stored in other memories (including other network locations) or information that is used by a function to calculate the relevant data.

The instructions <NUM> may be executed to detect a type of exercise performed by the user based on raw data received from the sensors <NUM>, <NUM> of the wireless wearable devices <NUM>, <NUM>. For example, the processor <NUM> may execute a machine learning algorithm whereby it compares images of the received raw data with stored image corresponding to particular exercises, and detects the exercise performed based on the comparison. Moreover, the instructions <NUM> may be executed to detect a number of repetitions of the exercise, such as by using a window.

In other examples, the analysis of the sensor data may be performed by either or both of the wearable wireless devices <NUM>, <NUM>. For example, each of the devices <NUM>, <NUM> includes a processor <NUM>, <NUM> and memory <NUM>, <NUM>, similar to those described above in connection with host device <NUM>. These processors <NUM>, <NUM> and memories <NUM>, <NUM> may receive data and execute the machine learning algorithm to detect the type of exercise performed.

<FIG> illustrates the wireless wearable devices <NUM>, <NUM> in communication with each other and the host device <NUM>. The wireless connections among the devices may be, for example, short range pairing connections, such as Bluetooth. Other types of wireless connections are also possible. In this example, the devices <NUM>-<NUM> are further in communication with server <NUM> and database <NUM> through network <NUM>. For example, the wireless wearable devices <NUM>, <NUM> may be indirectly connected to the network <NUM> through the host device <NUM>. In other examples, one or both of the wireless wearable devices <NUM>, <NUM> may be directed connected to the network <NUM>, regardless of a presence of the host device <NUM>.

These network <NUM> may be, for example, a LAN, WAN, the Internet, etc. The connections between devices and the network may be wired or wireless.

The server computing device <NUM> may actually include a plurality of processing devices in communication with one another. According to some examples, the server <NUM> may execute the machine learning model for determining a particular type of exercise being performed based on input from the IMUs of multiple wearable devices. For example, the wearable devices <NUM>, <NUM> may transmit raw data detected from their IMUs to the server <NUM>. The server <NUM> may perform computations using the received raw data as input, determine the type of exercise performed, and send the result back to the host <NUM> or one or both of the wearable devices <NUM>, <NUM>.

Databases <NUM> may be accessible by the server <NUM> and computing devices <NUM>-<NUM>. The databases <NUM> may include, for example, a collection of data from various sources corresponding to particular types of exercises. For example, the data may include images of raw data streams from IMUs or other sensors in wearable devices, the raw data streams corresponding to particular types of exercise. Such data may be used in the machine learning model executed by the server <NUM> or by any of the host device <NUM> or the wearable devices <NUM>, <NUM>.

Regardless of whether the detection of exercise type is performed at the server <NUM>, at the host <NUM>, at one or both of the wearable devices <NUM>, <NUM>, or some combination thereof, any of several different types of computations may be performed. These different types include at least (<NUM>) peak detector window segmentation using input from multiple IMUs positioned on different locations of the user's body, or (<NUM>) fixed window segmentation from the multiple IMUs. Moreover, any of the devices <NUM>-<NUM> or <NUM> may be capable of counting the number of repetitions of each exercise.

Peak detector window segmentation using input from multiple IMUs positioned on different locations of the user's body detects "peaks" in signal data, such as raw accelerometer and gyroscope data. <FIG> illustrates an example of raw accelerometer data from an earbud, and <FIG> illustrates an example of raw accelerometer data from a smartwatch. In this particular example, the data was obtained while doing squats, but it should be understood that the analysis can be applied to data for any of a variety of exercises. Each figure includes three waveforms: one for each of the x, y, and z directions. For example, in <FIG>, wave <NUM> corresponds to the y direction, such as a vertical direction; wave <NUM> corresponds to an x direction, such as lateral or side-to-side movement with respect to the user; and wave <NUM> corresponds to a z direction, such as forward/backward movement relative to the user. Similarly, in <FIG>, wave <NUM> corresponds to the y direction, wave <NUM> corresponds to the x direction, and wave <NUM> corresponds to the z direction.

The data from a first IMU may be timescaled to match time from a second IMU. For example, measurements from the IMU in an earbud may be timescaled to match measured time from an accelerometer in a smartwatch. This may include resampling all data to a length of the smartwatch accelerometer data. A low pass filter may be applied to the raw data. By way of example only, the low pass filter may be a Butterworth filter or any of a variety of other types of filter.

A "window" captures each repetition using custom peak detection and thresholding. A window, in peak detection segmentation, may refer to capturing one complete repetition of an exercise. The window may begin anywhere during the exercise repetition. For example, it could start in the middle of one pushup and end in the middle of a second pushup. The window may begin/end at a zero crossing, or some peak or trough, or any other feature that can be extracted from the data. For a fixed window, the window may be defined by a time length. For example, the time length may be <NUM> seconds, with windows overlapping every <NUM> seconds (<NUM>% overlap). This window may capture a partial repetition or multiple repetitions, depending on how fast the person exercises. Another option would be to have an autoscaling window, that autoscales the time length of the window based on the most recent repetition frequency estimate from the autocorrelation calculation. In that case the window is still determined by time, and may not exactly capture a full repetition. For example, the repetition length may be over or underestimated, but the time window would in general be close to the repetition length of the exercise.

Accordingly, for example, for a waveform of raw data received from an IMU during the exercise, a peak of the waveform is detected. Analysis of the waveform may further identify other characteristics to indicate the beginning and end of a repetition. If two IMUs are used, each having an x, y, and z axis, <NUM> raw data streams are included in each window. Adding <NUM> norms of each set of x, y, and z brings this to <NUM> data streams total. The norm may be computed as, for example, square_root(x^<NUM> + y^<NUM> + z^<NUM>). Other mathematical manipulations could also be possible that may provide additional beneficial information to the model. Accordingly, <NUM> subplots are provided for each image, such as in a 4x4 grid.

<FIG> illustrate example images created using the peak detection window segmentation technique described above with two IMUs for various types of exercises. For example, <FIG> illustrate an example image for a bicep curl, <FIG> illustrates an example image for a weight press, <FIG> illustrates an example image for a pushup, <FIG> illustrates an example image for a situp, and <FIG> illustrates an example image for a squat. These images may be used, for example, for training the machine learning model to recognize exercises by movements detected by the IMUs in the wearable devices.

<FIG> illustrates an example image for a non-exercise. Examples of non-exercises may include, without limitation, walking, climbing stairs, working at a desk, opening doors, picking up objects, etc. Non-exercise images may be included in the training data to help distinguish between particular exercise types and the non-exercises. Recognizing such distinctions may help to prevent false positive detection of exercises. While one example image in shown in <FIG>, it should be understood that any number of example images corresponding to various non-exercises may be included in the training data.

According to another type of computation, the fixed window segmentation from the multiple IMUs, the data processing flow described above is modified by using a fixed window of a predetermined width. By way of example only, the fixed window may be <NUM> seconds in width, with <NUM>% overlap. Because it is fixed, it is not synchronous with the exercise repetition, and may even have multiple or partial repetitions within a single windowed training or test example image. Because the window start is random with respect to the exercise frequency, it is possible to generate more training examples from the same set of data by varying the initial window start time. By stepping the window start time by <NUM>, the total number of training examples per exercise is multiplied by <NUM>.

The fixed window is autoscaled according to the repetition frequency estimate. This option does not require peak detection or thresholding.

<FIG> illustrate example images of exercises using fixed window segmentation. In these examples, <FIG> represents a first example of a bicep curl, and <FIG> represents a second example of a bicep curl. For example, the first example of <FIG> was generated using a first start time for the window, while the second example of <FIG> was generated using a second start time different than the first start time. For example, if using a <NUM> window, the second start time may be anywhere between <NUM>-<NUM> later than the first start time. However, it should be understood that the width of the window may be varied, such as by using <NUM>, <NUM>, <NUM>, etc. Moreover, the start time increments for the window may also be varied from <NUM> to any other value. The machine learning model thus learns patterns within the image, without relying on specific start or stop points.

In addition to being able to classify exercise type, the number of repetitions of a given exercise type may be counted in real time. With peak detector windowing, each test image, when classified, becomes a single repetition. With fixed windowing, counting repetitions may be performed using a different method, such as autocorrelation or model instantiation.

In using auto-correlation, a sliding auto-correlation window of a fixed duration is used during exercise. The repetition frequency can therefore be extracted. <FIG> illustrates an example of autocorrelation. Using a <NUM> sliding window during <NUM> of pushups, an indicator of repetition frequency can be extracted. For example, the indicator may be the location of the highest peak, excluding the peak at time t=<NUM>. Once the repetition frequency is known, the number of repetitions in a given exercise sequence can be calculated as the exercise duration in seconds multiplied by the repetition frequency. As long as the devices are detecting a specific exercise type, a timer could run to track total duration. Using this method, the user could receive an update once every few seconds on a repetition counter.

In using instantiation, the model could learn to count the repetitions itself by using instantiation. In this method, the model is trained not only with labeled examples, but also with segments, or "instances" of a repetition. For example, in a training example image that contains <NUM> repetitions, each repetition may be labeled in the training example. In this way, the model learns to estimate how many repetitions are in a given image.

<FIG> illustrates an example method <NUM> of detecting a type of exercise being performed by a user. The method may be performed by one or more processors in a first or second wearable device or in a host device in communication with the wearable devices.

In block <NUM>, first sensor data is received from one or more first sensors of the first wearable device. The first wearable device may be, for example, an earbud. The one or more first sensors may include an IMU, such as including a gyroscope and accelerometer. Each of the gyroscope and accelerometer may produce data streams for measurements in the x, y, and z directions.

In block <NUM>, second sensor data is received from one or more second sensors of the second wearable device. The second wearable device may be, for example, a smartwatch. Similar to the first sensors, the one or more second sensors may also include an IMU producing data streams for measurements in multiple spatial directions.

In block <NUM>, an image is generated based on the first sensor data and the second sensor data. The image may include a plurality of tiles or subplots, wherein each tile or subplot depicts a separate data stream from the first and second sensors. For example, a first tile depicts a data stream from the accelerometer of the first sensor in the x direction, a second tile depicts a data stream from the accelerometer of the first sensor in the y direction, and so on. Using two IMUs, each having two sensors measuring in three directions, twelve tiles may be generated. Additionally, each set of three (x, y, z) axes can be combined into a spatial norm, for a final total of <NUM> data capture tiles per image.

In block <NUM>, a type of exercise performed by the user during receipt of the first and second sensor data is determined based on the type of exercise. For example, a machine learning model is applied to the generated image. The machine learning model may be trained using image data, as described in further detail below in connection with <FIG>. The machine learning model may be, for example, an image based deep neural network, such as a convolutional neural network (CNN).

In block <NUM>, a number of repetitions of the exercise may be determined. For example, the machine learning model may be trained to recognize repetitions if it is trained with images wherein the repetitions are labeled. According to other examples, the repetitions may be detected by using autocorrelation to determine a repetition frequency, and then multiplying that repetition frequency by a duration of the exercise type.

<FIG> illustrates an example method <NUM> of training a machine learning model to detect exercise types. The method may be performed, for example, by one or more processors in a wearable device, a coupled host device, and/or on a remote server connected to the wearable devices or host through a network.

In blocks <NUM>-<NUM>, data may be received as described above in connection with blocks <NUM>-<NUM> of <FIG>. However, the data may be received over an extended period of time as the user performs several types of exercises and non-exercises.

In block <NUM>, the received data may be used to generate one or more first training images. For example, similar to generation of the image described above in connection with block <NUM> or.

<FIG>, the first training image may include a plurality of image tiles, each depicting a different data stream for data collected while the user was performing a first type of exercise. Similarly, in block <NUM>, the received data may be used to generate a second training image depicting data streams collected while the user was performing a second type of exercise different than the first type. This may be repeated for any number of different types of exercise.

In block <NUM>, a third training image is generated using techniques similar to those described in block <NUM>. However, the third training image may correspond to non-exercise. For example, the data represented in the tiles of the third image may be collected while the user is performing activities other than specific exercises. Such activities may include any of a variety of non-exercises, such as teeth-brushing, cooking, sitting, etc..

In block <NUM>, the first, second, and third images are input to the machine learning model as training data. In this regard, the machine learning model learns to recognize various types of exercises, as well as to distinguish such exercises from non-exercises. Such distinction can help to reduce false positive detections.

Claim 1:
A method for detecting exercise, comprising:
receiving, by one or more processors, first sensor data (<NUM>, <NUM>, <NUM>) from one or more first sensors (<NUM>) of a first wearable device (<NUM>);
receiving, by the one or more processors, second sensor data (<NUM>, <NUM>, <NUM>) from one or more second sensors (<NUM>) of a second wearable device (<NUM>);
generating, by the one or more processors, an image based on the first sensor data (<NUM>, <NUM>, <NUM>) and the second sensor data (<NUM>, <NUM>, <NUM>), the image comprising a plurality of subplots, wherein each subplot depicts a data stream received from the one or more first sensors (<NUM>) and the one or more second sensors (<NUM>), respectively;
determining, using the one or more processors, based on the image and comprising executing a machine learning model, a type of exercise performed by a user during receiving of the first sensor data (<NUM>, <NUM>, <NUM>) and the second sensor data (<NUM>, <NUM>, <NUM>);
training the machine learning model, the training comprising:
generating one or more first training images based on sensor data collected from the one or more first and second sensors (<NUM>, <NUM>) of the first and second wearable devices (<NUM>, <NUM>) while the user performs a first type of exercise, wherein the one or more first training images includes a plurality of image tiles, each depicting a different data stream for the sensor data; and
inputting the one or more first training images into the machine learning model as training data, wherein
generating the one or more first training images comprises fixed window segmentation and autoscaling a time length of the window based on a most recent estimate of a repetition frequency of the first type of exercise.