Patent Description:
The overall health and well-being of infants is of paramount concern to parents and other caregivers. Moreover, for new parents, adapting a newborn to sleeping and feeding cycles can be difficult, often causing stress. Hence, determining and recording the activity of infants such as whether the infant is sleeping, sitting up, or feeding, can be useful.

Solutions exist for tracking the activity of adults. For example, fitness and sleep trackers can track exercise such as steps taken, sleep cycles, and more. But these solutions are not adapted to the unique requirements of infants. For example, adult fitness trackers are unfit to be worn by infants.

Existing infant monitoring solutions also suffer from deficiencies. For example, solutions that simply transmit audio from an infant's room to the parents cannot measure activity or sleep. Some products measure an infant's blood oxygenation, alerting the parents if the infant's blood oxygen level drops below a certain threshold. But these solutions require cumbersome sensors to be attached to the infant, which requires skin contact, and can be kicked off. Camera-based infant monitoring solutions exist also, but such camera-based systems simply indicate the presence of the infant. Some camera-based solutions use artificial intelligence but still cannot discern small movements from large movements. As such, these solutions lack the precision to determine the activity of the infant, such as whether the infant is in a light sleep or deep sleep.

<CIT> discloses mechanisms and processes for an infant data aggregations system.

<CIT> discloses a portable monitoring device for monitoring the risks of Sudden Infant Death Syndrome.

According to one aspect of invention there is a method for determining activity as set out in claim <NUM>.

Features, embodiments, and advantages of the present disclosure are better understood when the following Detailed Description is read with reference to the accompanying drawings.

Aspects described herein provide solutions for activity classifications systems, specifically activity classification systems designed to detect the movement of an infant and determine whether the infant is performing a specific activity such as sleeping, feeding, nursing, or sitting up.

In an example, an activity classification system includes a movement sensor and an external monitor. The sensor is attached to an infant's clothing or an absorbent article such as a diaper. The sensor, which can include an accelerometer or a gyroscope, provides measurements to the monitor via a wireless communication channel. The monitor is an electronic device that can receive the measurements from the sensor and execute a monitor application and a predictive model such as a machine learning model, state-flow-model, or algorithm to determine activities performed by the infant. In this example, the predictive model is trained to determine, based on the infant's movement, an activity that the infant is performing such as sleep or sitting up. The activity classification system can then indicate to an operator of the monitor the predicted activity of the infant, for example that the infant is in a deep sleep.

Aspects of the present disclosure provide certain advantages over existing solutions. For example, using precise movements gathered from the accelerometer or the gyroscope, the activity classification system can distinguish activities being performed by an infant. For example, the activity classification system can be configured (e.g. trained) to distinguish deep sleep from light sleep, whether the infant is on its stomach versus on its back, or whether the infant is nursing. Further, certain aspects described herein can use predictive models to further refine the system's ability to determine activity. For example, the classification system can distinguish light sleep from deep sleep or measure an infant's breathing rate.

<FIG> depicts a block diagram of an activity classification system, according to certain aspects of the present disclosure. Activity classification system <NUM> incudes components that measure the movement of an infant and one or more processors that analyze the measurements to determine the infant's activity. Movement can be determined by inertial measurements, which include acceleration or angular velocity measurements. The one or more processors use an activity function or a predictive model such as a classifier to determine the infant's activity.

Activity classification system <NUM> includes one or more sensors <NUM>, a monitor <NUM> connected wirelessly to the sensor <NUM> via wireless network <NUM>, an activity classification server <NUM>, or mobile device <NUM>.

In an example, monitor <NUM>, which can also be referred to as a hub, can be small enough to be positioned nearby an infant such as on a table or a dresser. Monitor <NUM> communicates wirelessly with sensor <NUM>, and optionally, with other sensors such as volatile organic compound sensors, urine wetness sensors, noise sensors, light sensors, electrical conductivity sensors, optical sensors, capacitance sensors, color change sensors, and the like. Monitor <NUM> can aggregate data from sensor <NUM> or the other sensors.

Monitor <NUM> can optionally include user interface <NUM>, on which alerts, status information, or other information can be displayed. User interface <NUM> can include buttons, dials, knobs, a touchscreen, etc. Monitor <NUM> can communicate over data network <NUM> to mobile device <NUM>, and cause mobile device <NUM> to sound alarms, alerts, or provide other information.

In an aspect, user interface <NUM> can also receive feedback from a user that can help the monitor application <NUM> better determine when an infant is sleeping. For example, monitor application <NUM> can ask a user whether an infant is sleeping, and user interface <NUM> can receive an indication from a user as to whether the infant is actually sleeping. Such information can be incorporated as training data into a predictive model and can help a predictive model become more accurate over time.

In an example, the predictive model is trained to determine, based on the infant's movement, an activity that the infant is performing such as sleep or sitting up. The predictive model can be trained by data gathered for an individual or group of individuals.

For example, the predictive model can receive training data that represents a particular individual. The training data can include the individual's movements as determined by sensor <NUM> and an associated ground truth (correct activity, as designated by a training label). Thus, upon receiving new data from sensor <NUM>, the trained predictive model predicts an activity. In another example, the predictive model can be trained based on training data gathered from a group of individuals, which need not include the individual whose activity is later predicted. Therefore, the predictive model can learn from the recorded behavior of a particular individual or a group of individuals and use the learned behavior to predict an activity of the individual or another individual.

In order to measure movement, sensor <NUM> is positioned on or affixed to a person such as an infant. Sensor <NUM> includes one or more sensor devices such as an accelerometer <NUM> or a gyroscope <NUM>. Sensor <NUM> also includes a wireless transceiver <NUM> and a processor <NUM>. Sensor <NUM> gathers measurements of movement and sends the measurements to an external device such as monitor <NUM>. Based on the measurements, monitor <NUM> can determine the activity being performed by the infant and can take action such as notifying an operator as to what activity the infant is performing.

Securing the sensor <NUM> can be accomplished in different manners. For example, sensor <NUM> can be attached to a diaper or an item of clothing. Alternatively, sensor <NUM> can be pinned or adhered to clothing or an absorbent article or diaper, or fixed in any other way to the infant. Sensor <NUM> can be attached, removed, and reattached, resulting in the sensor being reusable. Sensor <NUM> need not touch an infant's skin. Sensor <NUM> can be inserted into a pouch and removed as necessary. Sensor <NUM> can be placed anywhere on an infant's body. But as discussed further, placement of the sensor <NUM> near the infant's waist allows certain advantages such as the ability to determine respiration rate.

To determine movement, processor <NUM> receives measurements from one or more sensors such as accelerometer <NUM> and gyroscope <NUM>. Processor <NUM> transforms the measurements into a form suitable for transmission over wireless network <NUM>, such as by encoding the measurements into a network packet. Processor <NUM> provides the measurements to wireless transceiver <NUM>, which transmits the measurements to an external device such as monitor <NUM>.

Accelerometer <NUM> measures acceleration of the infant in one or more dimensions. For example, accelerometer <NUM> can be a three-dimensional accelerometer that measures acceleration in the x, y, and z directions. In this case, accelerometer <NUM> provides a triplet of numerical values corresponding to the x, y, and z directions.

Gyroscope <NUM> measures angular velocity. Gyroscope <NUM> can output a signal proportional to the angular velocity of the infant. Angular velocity changes in the direction of a torque applied to the gyroscope. Accordingly, when an infant wearing sensor <NUM> rolls over, gyroscope <NUM> detects an increase in angular velocity. When the infant stops rolling, the angular velocity returns to zero.

The direction component of the angular velocity can be used in various ways. For example, the direction of the velocity can help indicate which side, e.g., left or right, stomach or back, the infant is positioned. Processor <NUM> can sample gyroscope <NUM> at specific instances in time and obtain the angular velocity on a periodic basis.

The output from the accelerometer <NUM> or gyroscope <NUM> can be used by the activity function or predictive model to provide indications of an infant's activity, breathing rate, or orientation such as on which side an infant is nursing or bottle feeding.

Processor <NUM> periodically samples accelerometer <NUM> and/or gyroscope <NUM> to obtain measurements as needed. Processor <NUM> provides the sampled measurements to wireless transceiver <NUM> for transmission to an external device. Processor <NUM> can sample accelerometer <NUM> or gyroscope <NUM> at different rates. For example, processor <NUM> can use an adaptive sample rate. For example, when the detected level of movement based on the measurements from the accelerometer <NUM> and the gyroscope <NUM> are below a threshold, processor <NUM> can sample less frequently.

A reduction in sample rate can help reducing power consumption by disabling devices such as the wireless transceiver <NUM> or entering into and remain in a low power state. Processor <NUM> can also perform additional processing or analysis on the measurements before providing the measurements to wireless transceiver <NUM>. For example, processor <NUM> may only provide measurements via wireless transceiver <NUM> to an external device when the measurements indicate a threshold level of activity. In this manner, processor <NUM> can save power by not causing the wireless transceiver <NUM> to transmit data. Conversely, when the measurements indicate activity above a second threshold, processor <NUM> can sample the accelerometer <NUM> and gyroscope <NUM> more frequently. In this manner, processor <NUM> can obtain more detailed measurements during bursts of activity.

The processor <NUM> transmits the measurements to an external device such as monitor <NUM> using the wireless transceiver <NUM>. In some examples, the processor <NUM> can transmit measurements to the external device, e.g., the monitor <NUM>, as they are received from the accelerometer <NUM> and gyroscope <NUM>, or in some examples it can buffer the measurements and send a group of measurements to the monitor <NUM> all at once.

The wireless transceiver <NUM> is configured to send and receive radio communications to enable communications between the sensor <NUM> and one or more external devices, such as monitor <NUM>. The wireless transceiver <NUM> can provide wireless communications using any suitable wireless protocol such as Bluetooth or WiFi to transmit data or other information, such as a measurement of acceleration from accelerometer <NUM> or a measurement of angular velocity, to the monitor <NUM>. In some examples, the wireless transceiver <NUM> can also transmit other information, such as a status message that indicates whether the sensor <NUM> is operational, in a stand-by mode, or deactivated. In an aspect, wireless transceiver <NUM> can receive radio transmissions from an external device. For example, wireless transceiver can receive information and commands from monitor <NUM>. For example, monitor <NUM> can send a message to wireless transceiver <NUM> that causes the sensor <NUM> to turn on, enter a low power state, or turn off.

In this example, monitor <NUM> includes output device <NUM>, processor <NUM>, monitor application <NUM>, wireless transceiver <NUM>, and predictive model <NUM>. Monitor <NUM> operates in conjunction with sensor <NUM> and optionally activity classification server <NUM> to determine an activity being performed by the infant. To do so, the monitor <NUM> receives sensor measurements such as acceleration and angular velocity from sensor <NUM> and uses the monitor application <NUM> in conjunction with the predictive model <NUM> to determine an activity based on the received sensor measurements. In this example, the predictive model <NUM> is a state-machine or algorithm, but may be any suitable type of predictive model in different examples such as a machine learning model or a classification model.

Wireless transceiver <NUM> can send and receive messages via a wireless protocol such as Bluetooth, WiFi, or any other wireless protocol. To enable wireless communications in this example, the wireless transceiver <NUM> can be paired with a sensor device such as sensor <NUM>. Wireless transceiver <NUM> receives sensor data from wireless transceiver <NUM> obtained from accelerometer <NUM> or gyroscope <NUM>.

Monitor <NUM> uses the measurements to determine an activity being performed by the infant. For example, monitor <NUM> can apply the measurements to the predictive model <NUM>, which can determine whether an infant wearing sensor <NUM> is feeding on the left hand side, feeding on the right hand side, sleeping, awake and playing on its back, being held, sitting, or performing some other activity.

Based on the determined activity, monitor <NUM> can take action. For example, monitor <NUM> can display a message to an operator or sound an alert or it can use output device <NUM> to alert an operator of monitor <NUM> about an activity, such as whether an infant wearing the sensor <NUM> is awake or asleep. Output device <NUM> can be a display such as an LCD or LED display, touchscreen display, light, flashing light, speaker, or other output device.

In an aspect, predictive model <NUM> can determine the breathing rate of the infant. For example, if the sensor <NUM> is placed on the infant's waist or stomach, then the infant's breathing will cause the sensor to move back and forth in one direction causing the sensor to output one or more sensor signals indicating the movement. Monitor application <NUM> can determine the infant's frequency of breathing from the measurement of activity. Monitor application <NUM> can also use the activity levels to distinguish a small fall from a large fall, for example, a fall that requires intervention. Similarly, monitor application <NUM> can determine when the infant's heart is beating and thereby determine the heartrate of the infant. Heartrate and breathing rate can be used to help determine when the infant is in different stages of sleep such as light sleep, deep sleep, about to wake up, etc. Heart rate, or the differences between resting heart rate and active heart rate, can also indicate the infant's health such as whether the infant has a fever.

In a further aspect, predictive model <NUM> can determine future activities from past activities. For example, predictive model <NUM> can be trained to recognize trends such as a time series of sensor values that indicate deep sleep for a period of time followed by increasing levels of movement. Such a trend in the time series of acceleration data can indicate that an infant is waking up. Some examples are illustrated with respect to <FIG>.

In an aspect, the monitor <NUM> can use the wireless transceiver <NUM> to connect via data network <NUM> to an activity classification server <NUM>. Data network <NUM> can be a local network, the Internet, or both. For example, wireless transceiver <NUM> can wirelessly to activity classification server <NUM> or mobile device <NUM>.

Activity classification server <NUM> can be configured to perform additional analysis such as population-level analysis of the movements of multiple infants or training of one or more predictive models and providing predictive models to the monitor <NUM>.

Activity classification server <NUM> includes predictive model <NUM>, processor <NUM>, and network adaptor <NUM>. Network connectivity from the monitor <NUM> to an activity classification server <NUM> provides several advantages. For example, via data network <NUM>, monitor <NUM> can receive software updates from a vendor or transmit status information to the vendor. Software updates can include updates to predictive model <NUM>.

Predictive model <NUM> can be a machine learning model such as a decision tree classifier or a regression model. Predictive model <NUM> can be trained to perform similar functions as predictive model <NUM>. For example, predictive model <NUM> can be trained to determine whether a wearer of the sensor is feeding on the left hand side, feeding on the right hand side, sleeping, awake and playing on its back, being held, or sitting. In particular, movement detected by gyroscope <NUM> is useful to predict feeding.

Updates to the predictive models are possible. For example, predictive model <NUM> can also update predictive model <NUM>. For example, as explained further herein, predictive model <NUM> can be trained using data from multiple monitors 110a-n. In this manner, predictive model <NUM> can become more accurate because it has been trained on a larger data set than might be available from only one monitor. Updating predictive model <NUM> with the training of predictive model <NUM> provides each monitor 110a-n the ability to benefit from population-based training.

Network adaptor <NUM> connects to a data network such as data network <NUM>. In this manner, network adaptor <NUM> can communicate with monitor <NUM> via wireless transceiver <NUM>. Network adaptor <NUM> can therefore receive updates such as new training data gathered from monitor <NUM>. Network adaptor <NUM> can also provide updated software or trained predictive models to monitor <NUM>.

Processor <NUM> is processor that receives data from network adaptor <NUM>. over data network <NUM>. Processor can be a processor such as processor <NUM> described in <FIG>. Processor <NUM> can perform similar operations as processor <NUM>. For example, processor <NUM> can execute an application such as monitor application <NUM>. Such an application can train a predictive model such as predictive model <NUM>, receive measurement data over data network <NUM>, and use predictive model <NUM> to determine the activity of an infant.

Mobile device <NUM> can be a smart phone, tablet, laptop, or other device. Mobile device <NUM> can include a graphical user interface that outputs data such as how long an infant has slept, a predicted nursing time, etc., as further depicted in <FIG>. Mobile device <NUM> can send alerts to an operator such as an indication that the infant as woken up, fallen asleep, is crying, etc. Mobile device <NUM> can communicate with monitor <NUM> or activity classification server <NUM> over data network <NUM>. Mobile device <NUM> can include perform some or all of the processing handled by processors <NUM>, <NUM>, or <NUM>.

Sensor <NUM>, monitor <NUM>, and activity classification server <NUM> operate in conjunction with each other to determine an infant's activity. Processing of the measurement data and determination of the infant's activity can be performed on sensor <NUM>, e.g. via processor <NUM>, monitor <NUM>, e.g. via processor <NUM>, or activity classification server <NUM>, e.g., via processor <NUM>.

For example, processor <NUM> can receive the measurements such as acceleration and angular velocity and sensor <NUM> can provide the unmodified measurement data to monitor <NUM> across wireless network <NUM>. In turn, processor <NUM> receives the measurements from sensor <NUM> via wireless transceiver <NUM>. Processor <NUM>, for example via monitor application <NUM>, uses an activity function to determine the activity of the infant.

Alternatively, processor <NUM> can derive an activity function from the acceleration and angular velocity measurements. Processor <NUM> provides the activity function to the wireless transceiver <NUM>. Processor <NUM> receives the activity measurement function data and makes a prediction of the infant's activity.

In a further aspect, monitor <NUM> can also provide the measurements from sensor <NUM>, via data network <NUM>, to activity classification server <NUM>, which can determine the infant's activity. Activity classification server <NUM> may provide additional processing resources, resulting in a more accurate prediction of the infant's activity, or it may update its predictive model <NUM> based on the received measurements and/or additional training data that is annotated based with ground truth (e.g., an indication of the actual activity such as sleeping or awake).

In an aspect, activity classification server <NUM> connects to multiple activity monitors <NUM> and aggregates measurement data or determines activities for multiple infants. Activity classification server <NUM> can analyze the data for trends. Trends that are difficult to discern can be easier to discern using multiple sets of data. For example, in conjunction with demographic information such as the age of the infants being measured, activity classification server <NUM> can determine the average amount of sleep received by an infant of a certain age.

Activity classification server <NUM> can also aggregate training data from multiple infants and improve the training of predictive model <NUM>. Predictive model <NUM> can be deployed into the predictive model <NUM> of each monitor <NUM>.

<FIG> depicts examples of sensor outputs from an accelerometer, according to certain aspects of the current disclosure. <FIG> incudes graph <NUM>, which includes waveforms <NUM>, <NUM> and <NUM>. Waveform <NUM> represents a measurement of activity in the x dimension of sensor <NUM>, waveform <NUM> represents a measurement of activity in the y dimension of sensor <NUM>, and waveform <NUM> represents a measurement of activity in the z dimension of sensor <NUM>. Graph <NUM> includes a vertical axis that indicates acceleration (g), and a horizontal axis that indicates time (hours).

As can be seen in graph <NUM>, bursts of activity, shown as higher values, are present such as burst <NUM>, which correlates with movement detected by sensor <NUM>. A burst of activity may occurs when the infant moves in some way that causes one or more sensors to output sensor signals indicating, for example, acceleration. For example, when the infant rolls over, a burst of activity is detected because one or more accelerometers output sensor signals indicating accelerations that are more than a threshold value away from zero acceleration, or an angular velocity in the direction of the roll (not shown in <FIG>) that is more than a threshold value away from zero angular velocity. Similarly, when the infant is resting, the infant's breathing can appear as bursts of activity at the breathing rate. In contrast, during periods without bursts of activity, the infant is either not moving or moving very little. As explained further, monitor application <NUM> can distinguish between movements using an activity function or a predictive model <NUM>.

<FIG> is a flowchart of an exemplary method used to determine activity from movement sensors, according to certain aspects of the present disclosure. For example purposes, method <NUM> is illustrated with respect to monitor application <NUM>. But method <NUM> can be performed by software executing on a processor such as processor <NUM> of a remote device such as activity classification server <NUM>.

At block <NUM>, monitor application <NUM> receives, from a movement sensor, a time series of data including an inertial measurement for each of a set of time periods. Inertial measurements can include acceleration or angular velocity. For example, accelerometer <NUM> provides a triplet of numerical values corresponding to the x, y, and z directions to processor <NUM>, which provides the triplet to wireless transceiver <NUM>. Processor <NUM> periodically samples accelerometer <NUM> to create a time series of data. Processor <NUM> annotates each triplet with a timestamp, creating a pair that includes sensor measurement and timestamp. Processor <NUM> can also sample gyroscope <NUM> on a periodic basis. In conjunction with the measurement data from accelerometer <NUM>, processor <NUM> can provide a set of data to monitor <NUM>. The set of data can include a gyroscope measurement, e.g. angular velocity, an accelerometer measurement, e.g., a triplet of x-y-z values, and a timestamp.

In an aspect, activity classification system <NUM> analyzes measurement data in real-time. For example, processor <NUM> can cause wireless transceiver <NUM> to transmit each sampled triplet and timestamp pair separately to monitor <NUM> in real-time. In this manner, monitor application <NUM> can update an activity measurement function or predictive model <NUM> in real-time.

Alternatively, activity classification system <NUM> can analyze a block of samples at a time. For example, processor <NUM> can buffer the pairs until a threshold number of pairs have been received, then transmit a batch consisting of all the pairs gathered to monitor <NUM>. Processor <NUM> can also buffer the pairs until a threshold amount of time has passed, then provide the pairs to monitor <NUM>. In this manner, processor <NUM> can analyze movement over different windows of time.

Other types of movement sensors can be used such as vibration sensors, ultrasonic sensors, or passive infrared sensors. Such sensors can have single or multiple dimensions. Processor <NUM> can sample such sensors and create a set of data containing sensor values and timestamp to send to monitor <NUM>.

At block <NUM>, monitor application <NUM> calculates, from a subset of the time series of data, an activity function from statistical data derived from the inertial measurement. Statistical data can include data such as (i) a statistical variance of the inertial measurement or (ii) a root-mean-square of the inertial measurement. Monitor application <NUM> uses an activity measurement function in order to determine activity level. Different measurements of activity can be derived. For example, monitor application <NUM> can calculate the statistical variance, standard deviation, or the root mean square (RMS) of the signal. Monitor application <NUM> can use another customized metrics based on the accelerometer or gyroscope data. For example, a customized metric that quantifies the level of activity A can be calculated for a given number n of samples with the following function, where Sx, Sy, and Sz are the sum of the square differences from the respective means in the x, y, and z dimensions respectively:
<MAT>.

At block <NUM>, monitor application <NUM>, determines an activity indicated by the subset of time series data based on a measure from the activity function being greater than a first threshold and less than a second threshold. Monitor application <NUM> can determine an activity such as sleeping or awake based on a level of activity being with a range of values. For example, if the activity function measures a level of activity below a first threshold but above zero, then the monitor application <NUM> determines that the infant is in light sleep. If the activity function measures a level of movement below a second, lower, threshold, then the monitor application determines that the infant is in a deep sleep. As discussed further with respect to <FIG>, monitor application <NUM> can use a state machine to determine activity states.

At block <NUM>, monitor application <NUM> provides the activity to a user interface. For example, the monitor application <NUM> can output the activity such as "deep sleep" or "light sleep," for example, by providing an silent alert to output device <NUM> to indicate to a caregiver that the infant is asleep. The monitor application <NUM> can also log the activity and a timestamp in a memory and provide the logged information to activity classification server <NUM>.

Monitor application <NUM> can also output information derived from the activity of the infant. For example, output device <NUM> can provide an indication to the operator such as "the baby has been asleep for two hours. " Monitor application <NUM> can also cause a sound such as an alert from output device <NUM>, for example, when the infant has woken up after being asleep for a predetermined amount of time. Monitor application <NUM> can also transmit information to a remote device, such as activity classification server <NUM>. In this manner, the activity classification system <NUM> can be useful for caregivers of multiple infants, for example, at a hospital or a daycare.

As discussed, in an aspect, activity classification system <NUM> can use a predictive model such as predictive model <NUM> or predictive model <NUM> to determine the infant's activity in addition to or instead of algorithms or state machines. Monitor application <NUM> provides the accelerometer measurements, the gyroscope measurements, or the output of an activity measurement function to the predictive model <NUM>, or both.

Predictive models discussed herein can be machine learning models such as decision tree classifier or regression models. Other models are possible. Predictive model <NUM> is trained to determine whether a wearer of the sensor is feeding on the left hand side, feeding on the right hand side, sleeping, awake and playing on its back, being held, or sitting. Other detectable activities may include sitting, playing, crawling, walking, etc. Monitor application <NUM> provides data for one or more periods of time to predictive model <NUM>. In this manner, predictive model <NUM> may therefore determine an activity based on present or past activity level.

Predictive model <NUM> can be updated or upgraded, for example, via a data network <NUM>. As discussed further herein, predictive model <NUM> can also be trained locally on the monitor <NUM>. For example, an operator such as a parent can indicate to the monitor <NUM> that the wearer of the sensor was asleep or awake during a particular time period. The monitor application <NUM> can then update predictive model <NUM> based on the additional information provided. In this manner, predictive model <NUM> can be personalized to an individual or infant. Potential benefits include greater accuracy relative to using data from other individuals.

<FIG> is a flowchart of an exemplary method used to determine activity from a movement sensor by using a predictive model, according to certain aspects of the present disclosure. For example purposes, method <NUM> is illustrated with respect to monitor application <NUM> and predictive model <NUM>. But activity classification server <NUM> and predictive model <NUM> can also perform method <NUM>.

At block <NUM> of method <NUM>, monitor application <NUM> receives, from a an inertial measurement sensor, measurements for a time period. At block <NUM>, monitor application <NUM> receives the inertial measurements generally as described with respect to block <NUM>. The measurements can be obtained on one or more dimensions.

At block <NUM> of method <NUM>, monitor application <NUM> calculates, from the inertial measurements, an activity function or statistical data. Examples of activity functions include (i) a statistical variance of the of the inertial measurement and (ii) a root-mean-square of the inertial measurement. At block <NUM>, monitor application uses an activity measurement function generally as described with respect to block <NUM>.

At block <NUM> of method <NUM>, monitor application <NUM> provides the inertial measurements and/or the statistical data to a predictive model. More specifically, monitor application <NUM> provides sensor measurements and/or the output of the activity function to the predictive model <NUM>. The predictive model can be trained, for example, by using the process described with respect to <FIG>. When trained, the predictive model can identify, based on the training and the inertial measurements or statistical data, an activity from a list of activities.

Predictive model <NUM> is trained to determine activity from measurements that indicate movement. The predictive model <NUM> determines, based on its training, from a predefined set of classes, to which class the activity belongs. An exemplary list of activity classes includes feeding on the left side, feeding on the right side, sleeping, awake but playing on back, being held, and sitting.

Other training classes are possible. For example, predictive model <NUM> can be trained to distinguish deep sleep from light sleep, and activities such as crawling, rolling, sitting up, feeding, or nursing from each other. For example, monitor <NUM> may include a predictive model that is trained to distinguish between asleep, awake, stirring, or settled states, and another that is trained to distinguish between light sleep and deep sleep. Training is discussed further with respect to <FIG>. Stirring represents a state in which an infant is moving more than a first threshold amount and settled represents a state in which the infant has calmed down and is moving less than a second threshold amount.

At block <NUM> of method <NUM>, monitor application <NUM> receives, from the predictive model, a determined activity based on the inertial measurements. The determined activity can be from a list of predefined activities.

For example, predictive model <NUM> provides a prediction to monitor application <NUM> from one of the trained categories such as feeding on the left hand side, feeding on the right hand side, sleeping, awake and playing on its back, being held, or sitting. An illustration of activities determined by the predictive model is shown in <FIG>.

<FIG> depicts a graph <NUM> that shows activities determined by a predictive model based on movement detected from an accelerometer and a gyroscope, according to certain aspects of the present disclosure. Graph <NUM> includes waveform <NUM>, Wake state <NUM>, Feed state <NUM>, and Sleep state <NUM>. Graph <NUM> indicates activities that were determined based on the sensor information shown in graph <NUM> of <FIG>.

Waveform <NUM> represents an output from a predictive model such as predictive model <NUM> over time. In the aspect depicted by graph <NUM>, predictive model <NUM> is trained to determine one of three activities: wake, feed, or sleep. Graph <NUM> includes wake state <NUM> corresponding to waking up, feed state <NUM> corresponding to feeding or nursing, and sleep state <NUM> corresponding to sleeping. Accordingly, at a given point in time, the output represented by waveform <NUM> is in one of states <NUM>-<NUM>.

In graph <NUM>, the units of time are hours. Therefore, as can be seen, predictive model <NUM> has predicted that the infant has slept for various periods of time from a few hours to seven or more hours. The dots indicated by feed state <NUM> represent feeding, which indicates typical feeding times of <NUM>-<NUM> minutes measured by the predictive model.

States <NUM>-<NUM> can also represent annotated training information provided to the predictive model <NUM>. For example, data that indicates the infant's activity can be obtained via monitor <NUM> or another device and provided with the predicted states and a prompt for user input. For example, monitor <NUM> can prompt an operator such as a caregiver with a prompt such as "the monitor has determined the baby is asleep. True or false?".

Returning to <FIG>, at block <NUM> of method <NUM>, at block <NUM> of method <NUM>, monitor application <NUM> provides the determined activity to an output device, e.g., an external device or user interface. At block <NUM>, the monitor application <NUM> outputs the activity generally as described with respect to block <NUM>.

As discussed, predictive models such as predictive model <NUM> or predictive model <NUM> are trained before use. Predictive model <NUM> can be trained with supervised or unsupervised learning. With supervised learning, the predictive model is provided annotated or labeled training data that indicates the actual activity of the infant, such as whether the infant is asleep, awake, or feeding. The training data is provided to the predictive model, which creates a loss function and compares the loss function to the actual, annotated or labeled, value.

In an aspect, additional training can be performed locally by an operator of monitor application <NUM>. For example, training data can be augmented at runtime by the device such as a caregiver. For example, monitor <NUM> can prompt the operator "the activity monitor thinks your baby is sleeping, is that correct?" In this manner, the predictive model <NUM> can be continuously updated and improved over time. Predictive model <NUM> can be tailored to one specific infant's tendencies. For example, one infant may sleep lighter than another, causing a prediction for one infant to require adjustment for use with a second infant.

<FIG> depicts examples of sensor outputs from an accelerometer annotated with predicted states, according to certain aspects of the present disclosure. <FIG> includes graph <NUM>. Graph <NUM> shows accelerometer outputs on the x, y, and z axes for a sensor attached to an infant subject. Graph <NUM> is annotated with identified, or predicted activity states <NUM>, <NUM>, <NUM>, and <NUM>. Activity state <NUM> represents a state in which the infant is feeding. As can be seen, movement as detected by the accelerometer is relatively small compared to other states such as <NUM>, which represents a time at which the infant is lying on its back and playing with the baby gym. Activity state <NUM> represents a diaper change. State <NUM>, which as can be seen, shows relatively little to no movement, depicts when the infant is sleeping. Aspects described herein can be trained to predict activity states such as those depicted by <NUM>-<NUM>, and others such as lying on stomach, lying on side, etc..

<FIG> depicts examples of sensor outputs from an accelerometer that indicate sleep and waking up according to certain aspects of the present disclosure. <FIG> includes graph <NUM>, which depicts activity states <NUM> and <NUM>. Activity state <NUM> indicates data that represents an infant subject in the process of slowly waking up and building up to eventually crying. Activity state <NUM> represents that the infant is awake. Aspects described herein can be trained to distinguish between activity states <NUM> and <NUM>.

<FIG> depicts examples of sensor outputs that indicate a detected respiratory rate of an infant, according to certain aspects of the present disclosure. When an infant subject is asleep, the accelerometer can detect small movements in one or more directions that vary based on a frequency and identify those movements as respiration. <FIG> includes graphs <NUM> and <NUM>. Graph <NUM> depicts a waveform that correlates with breathing rate. Graph <NUM> depicts a waveform derived from the waveform in graph <NUM>, but filtered by a band-pass filter, e.g., that allows <NUM>-<NUM> frequencies to pass. As can be seen, the waveform <NUM> is easy to discern. Waveform <NUM> illustrates a breathing frequency of approximately <NUM> breaths per minute.

<FIG> is a flowchart of an exemplary method for training a predictive model to determine activity from an infant's movement, according to certain aspects of the present disclosure. Predictive model <NUM> can be trained by monitor application <NUM> executing on monitor <NUM>, or a computing system such as computing system <NUM> depicted in <FIG>. Similarly, an external device such as activity classification server <NUM> can train predictive model <NUM> using method <NUM>.

At block <NUM>, monitor application <NUM> receives training data including time series data indicating inertial measurements measured over a set of time periods, the training data annotated with training labels indicating, for each time period, a corresponding activity. In some cases, the training data labels are provided by a caregiver. For example, a caregiver can indicate what an infant was doing at a particular time, and monitor application <NUM> can combine the indication with the accelerometer or gyroscope data to provide training data to train one of the predictive models.

For example, the training data consists of multiple instances of training data. Each instance includes a measurement of movement. A measurement of movement includes acceleration, e.g., an x-y-z triple, or a measure of angular velocity from a gyroscope. Each instance therefore represents a snapshot in time of movement.

The training data is annotated with labels that correspond to real, measured activity such as whether an infant was asleep or awake. For example, each instance of training data is annotated with a label that indicates an actual determined activity of an infant, specifically, the activity that an infant was performing at the time that corresponds to that instance of training data, e.g., when the measurements were taken.

In an aspect, training data can include instances that represent a time period. For example, an instance can represent a period of one second. Accordingly, the instance may include multiple samples of movement data with one associated activity annotation.

At block <NUM>, monitor application <NUM> provides the annotated training data to the predictive model. Monitor application <NUM> provides one instance of training data to the predictive model <NUM> at a time.

At block <NUM>, monitor application <NUM> iteratively adjusts the predictive model based on a loss function that is based on a comparison of the prediction to the training label by determining, by the predictive model, a prediction of whether the data corresponding to a time period corresponds to one of the activities.

Specifically, the predictive model <NUM> outputs an activity from a predefined set of activities. A set of activities could be, for example, feeding on the left side, feeding on the right side, sleeping, awake but playing on back, being held, and sitting. The loss function is a comparison of the predicted activity, e.g., what activity determined by the predictive model to correspond to the time series data, with the activity as indicated by the training label. The predicted activities can be represented numerically or in another fashion.

At block <NUM>, monitor application <NUM>, reduces the value of a loss function based on a comparison of the determined activity from the predictive model and the labeled activity. The loss function computes the difference between the determined activity and the labeled activity, e.g., the activity that the infant who was the subject of the training data was actually performing at that time. Predictive model <NUM> attempts to minimize the loss function with each iteration and adjusts its internal parameters accordingly.

Monitor application <NUM> repeats steps <NUM>-<NUM> with subsequent instances of training data. Predictive model continues to update and reduce the value of the loss function over each iteration.

After a sufficient amount of training data, the predictive model learns to distinguish sleeping from not sleeping. For example, after a sufficient number of examples of deep sleep and light sleep, predictive model can learn to distinguish deep sleep from light sleep. Other training examples include distinguishing when an infant is in its belly versus on its back, and whether the infant is nursing and if so, on which breast. The orientation of the infant is useful for distinguishing breastfeeding, bottle feeding, and eating solid food. For example, certain orientations such as completely vertical or completely horizontal are less likely than other orientation.

At block <NUM>, a monitor application <NUM> tests the trained model with a set of test data that is separate from the training data and measures a level of accuracy of the prediction. The test data is separate from the training data that is used for training purposes as described with respect to blocks <NUM>-<NUM>. The test data is not provided to the predictive model <NUM> until testing. In this manner, the test data is new to the predictive model <NUM> and therefore cannot influence the model's prediction. The test data therefore can be used to rigorously test whether the training has worked. If the predictive model fails a threshold number of tests, then the model is further trained.

In an aspect, activity classification system <NUM> can be integrated with a system that can measure urine in a diaper, temperature, humidity, bowel movements of the infant, or other data. For example, an integrated sensor can include sensor <NUM>, e.g., accelerometer <NUM> and gyroscope <NUM>, and additional sensors such as temperature, humidity, and the like. Sensor <NUM>, specifically processor <NUM>, can transmit additional measurements such as temperature measurements and humidity measurements to monitor <NUM>. In turn, monitor application <NUM> can then determine when the infant's diaper needs to be changed and provide an alert, for example, via output device <NUM>. In a further aspect, monitor <NUM> can receive data transmitted from a sensor that measures the color of a color strip. The monitor <NUM> can detect, by the color of the color strip, a volume of urine present in the infant's diaper. Monitor <NUM> can output an alert to a caregiver via output device <NUM>.

In some cases, activity monitor <NUM> can use a state machine to determine an activity state of an infant. An exemplary state machine is shown in <FIG>. The state machine can be implemented by monitor <NUM>, activity classification server <NUM>, mobile device <NUM>, or another device. The predicted state can be output to a caregiver on a periodic basis.

<FIG> is a state diagram of an exemplary state machine to determine activity levels, according to aspects of the present disclosure. <FIG> depicts state machine <NUM>, which includes various states and transitions between the states.

More specifically, state machine <NUM> includes four states: stirring <NUM>, awake <NUM>, asleep <NUM>, and settled <NUM>. Other states are possible. State machine <NUM> also includes examples of transitions between states. For example, transition <NUM> is a transition between stirring <NUM> and awake <NUM>. Transition <NUM> is a transition between stirring <NUM> and asleep <NUM>. Transition <NUM> is a transition between awake <NUM> and asleep <NUM>. Transition <NUM> is a transition between awake <NUM> and settled <NUM>. Finally, transition <NUM> is a transition between asleep <NUM> and settled <NUM>.

Transitions <NUM>-<NUM> can be trigged by detection of infant activity, e.g., by using methods <NUM> or <NUM>.

In some cases, measurements from the accelerometer or gyroscope can cause short transitions in state. One example is when an infant is asleep but stirs during sleep for a short time, e.g., five minutes. Such a short periods can be erroneously labeled awake, causing a care giver to be concerned or frustrated with the monitor. In another example, an "asleep" state is preceded by a certain amount of settled time, which is really the infant going to sleep, e.g., calming down. Further, during the day, time periods of low activity can be erroneously measured as asleep.

In some cases, a user of the activity monitor may not expect or know how to act on such activity state detections. Accordingly, in some cases, activity monitor can use a smoothing filter to smooth out short or seemingly erratic transitions. A smoothing filter helps reduce confusion caused by transitions that are later superseded by another transition. For example, if the activity monitor determines that an infant is asleep, then subsequently stirs for a short period of time and falls back asleep, then the infant is not in the process of waking up. Rather, the activity monitor should not report the stirring period.

An example of one such smoothing filter is depicted in <FIG> and is described further with respect to <FIG>.

<FIG> is a flowchart of an exemplary method <NUM> performed by a smoothing filter, according to certain aspects of the present disclosure. For discussion purposes, <FIG> is explained with respect to <FIG>.

<FIG> are diagrams illustrating exemplary operations performed by a smoothing filter, according to aspects of the present disclosure.

At block <NUM>, a smoothing filter changes all segments of activity designated as "settled" to a subsequent state. <FIG> depicts graphs <NUM> and <NUM>, which each depict state ("awake," "stirring," "settled," and "asleep") over time. More specifically, graph <NUM> depicts prediction <NUM>, which represents an output of the activity monitor at block <NUM>. Output <NUM> represents the output after block <NUM> has completed.

As can be seen, graph <NUM> includes areas <NUM>, <NUM>, and <NUM>, which indicate transitions to the settled state. As depicted in area <NUM>, periods in which the predicted state transitioned from settled to asleep were re-classified as asleep. As depicted in output <NUM> subsequent to area <NUM>, the transitions from settled to awake identified in areas <NUM> and <NUM> are converted to awake.

At block <NUM>, the smoothing filter changes all segments of activity designated as "stirring" to "sleep. " <FIG> depicts graphs <NUM> and <NUM>, which each depict a state over time. More specifically, graph <NUM> depicts prediction <NUM>, which represents an input to the smoothing filter at block <NUM>. Graph <NUM> represents the output after block <NUM> has completed.

As can be seen, areas <NUM>, <NUM>, and <NUM>, each of which represents a brief time in which the state is "stirring" before and after periods of "asleep. " Areas <NUM>-<NUM> are removed or "smoothed" by the smoothing filter. As can be seen, graph <NUM> depicts a constant "asleep" state.

At block <NUM>, the smoothing filter changes all segments that are (i) designated as "asleep," are (ii) below a threshold in length, and are (iii) during the day to "awake. " <FIG> depicts graphs <NUM> and <NUM>, which each depict a state over time. More specifically, graph <NUM> depicts prediction <NUM>, which represents an input to the smoothing filter at block <NUM>. Graph <NUM> represents the output after block <NUM> has completed.

As can be seen, areas <NUM>-<NUM> are areas in which the predicted state changes from awake to asleep to awake, remaining in the asleep state for only a short period of time. Such periods of activity may have been erroneously identified as asleep, because they are too short to really be asleep. Accordingly, the smoothing filter changes these states to awake, resulting in graph <NUM>. Examples of threshold in length (or time) include <NUM>, <NUM>, or <NUM> minutes.

At block <NUM>, the smoothing filter changes all segments that are (i) designated as "awake," are (ii) below a threshold in length, and are (iii) during the night to "asleep. " <FIG> depicts graphs <NUM> and <NUM>, which each depict a state over time. More specifically, graph <NUM> depicts prediction <NUM>, which represents an input to the smoothing filter at block <NUM>. Graph <NUM> represents the output after block <NUM> has completed. The output depicted in graph <NUM> can be output to a user interface.

As can be seen, areas <NUM>-<NUM> represent areas predicted as "awake" within areas of "asleep. " Such periods of activity may have been erroneously identified as awake, because they are too short to really be awake, given that the infant has been asleep for a relatively long period of time and continues to be asleep afterwards. Accordingly, the smoothing filter changes these states to asleep, resulting in graph <NUM>.

Alternatively, or in addition, monitor application <NUM> can display historical sleep information for the infant in graphical format or on a calendar display, to enable a caregiver to visual sleep trends or routines.

<FIG> and <FIG> depict exemplary user interfaces for viewing sleep patterns of an infant, according to certain aspects of the present disclosure. <FIG> includes user interface <NUM> and graph <NUM>. User interface <NUM> includes a sleep routine display that indicates, for multiple days of the week and for each time period during the day, whether an infant subject was predicted to be asleep or awake. <FIG>, which includes user interface <NUM>, shows the total sleep per day for an infant in graph <NUM>. Graph <NUM> can be filtered based on a weekly, monthly, or lifetime basis.

Monitor application <NUM> can also output feeding information derived from the activity of the infant. For example, output device <NUM> can provide information about when the last feeding occurred, how long a feeding event last, and whether a nursing event took place on the left or the right breast. Monitor application <NUM> may also enable caregivers to input information from bottle feeding events. Monitor application <NUM> can display historical feeding information for the infant in graphical format or on a calendar display, to enable a caregiver to visualize feeding trends or routines.

<FIG> depicts exemplary user interfaces for viewing feeding patterns of an infant, according to certain aspects of the present disclosure. <FIG> includes user interface <NUM>, sleeping or feeding key <NUM>, and graph <NUM>. Graph <NUM>, which is read in conjunction with feeding key <NUM>, shows time periods of each day for multiple days, whether an infant subject was sleeping, feeding, or neither.

<FIG> includes user interface <NUM>, which depicts graph <NUM>, and textual information <NUM>. User interface <NUM> allows filtering by week, month, or lifetime. Graph <NUM> shows the hours spent feeding, or nursing, each day. Textual information <NUM> provides specific feedback to an operator such as averages.

<FIG> includes user interface <NUM>, which depicts textual information box <NUM> and recent information <NUM>. Textual information box <NUM> provides an operator of the device detailed information such as number of feedings for the current day, total time nursed for the day, and observations. Recent information <NUM> shows the last feeding time and duration.

<FIG> depicts a block diagram of a processor configured to determine activity of a wearer of a sensor, according to certain aspects of the present disclosure. Some or all of the components of the computing system <NUM> can belong to the processor <NUM> or processor <NUM> of <FIG>. For example, monitor application <NUM> and predictive model <NUM> may operate on the computing system <NUM>. The computing system <NUM> includes one or more processors <NUM> communicatively coupled to one or more memory devices <NUM>. The processor <NUM> executes computer-executable program code, which can be in the form of non-transitory computer-executable instructions, stored in the memory device <NUM>, accesses information stored in the memory device <NUM>, or both. Examples of the processor <NUM> include a microprocessor, an application-specific integrated circuit ("ASIC"), a field-programmable gate array ("FPGA"), or any other suitable processing device. The processor <NUM> can include any number of processing devices, including one.

The memory device <NUM> includes program code <NUM> and program data <NUM>. Program code <NUM> can include code executed by monitor application <NUM>, predictive model <NUM>, predictive model <NUM>, a mobile application executing on mobile device <NUM>, or any other application described herein. Program data <NUM> can include training data for predictive models, acceleration or gyroscope measurements, or program data for any application described herein such as monitor application <NUM>, predictive model <NUM>, or predictive model <NUM>.

The memory device <NUM> includes any suitable computer-readable medium such as electronic, optical, magnetic, or other storage device capable of providing a processor with computer-readable instructions or other program code. Non-limiting examples of a computer-readable medium include a magnetic disk, a memory chip, a ROM, a RAM, an ASIC, optical storage, magnetic tape or other magnetic storage, or any other medium from which a processing device can read instructions. The instructions may include processor-specific instructions generated by a compiler or an interpreter from code written in any suitable computer-programming language, including, for example, C, C++, C#, Visual Basic, Java, Python, Perl, JavaScript, and ActionScript.

The computing system <NUM> may also include a number of external or internal devices such as input or output devices. For example, the computing system <NUM> is shown with an input/output ("I/O") interface <NUM> that can receive input from input devices or provide output to output devices. A bus <NUM> can also be included in the computing system <NUM>. The bus <NUM> can communicatively couple one or more components of the computing system <NUM> and allow for communication between such components.

The computing system <NUM> executes program code that configures the processor <NUM> to perform one or more of the operations described above with respect to <FIG> or <FIG>. The program code of the monitor application <NUM>, which can be in the form of non-transitory computer-executable instructions, can be resident in the memory device <NUM> or any suitable computer-readable medium and can be executed by the processor <NUM> or any other one or more suitable processor. Execution of such program code configures or causes the processor(s) to perform the operations described herein with respect to the processor <NUM>. In additional or alternative aspects, the program code described above can be stored in one or more memory devices accessible by the computing system <NUM> from a remote storage device via a data network. The processor <NUM> and any processes can use the memory device <NUM>. The memory device <NUM> can store, for example, additional programs, or data used by the applications executing on the processor <NUM> such as the monitor application <NUM>.

The computing system <NUM> can also include at least one network interface <NUM>. The network interface <NUM> includes any device or group of devices suitable for establishing a wired or wireless data connection to one or more data networks. Non-limiting examples of the network interface <NUM> include an Ethernet network adapter, WiFi network, a modem, and/or the like. The computing system <NUM> is able to communicate with one or more other computing devices or computer-readable data sources via a data network using the network interface <NUM>.

Unless specifically stated otherwise, it is appreciated that throughout this specification discussions utilizing terms such as "processing," "computing," "calculating," "determining," and "identifying" or the like refer to actions or processes of a computing device, such as one or more computers or a similar electronic computing device or devices, that manipulate or transform data represented as physical electronic or magnetic quantities within memories, registers, or other information storage devices, transmission devices, or display devices of the computing platform.

The system or systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device can include any suitable arrangement of components that provide a result conditioned on one or more inputs. Suitable computing devices include multi-purpose microprocessor-based computer systems accessing stored software that programs or configures the computing system from a general purpose computing apparatus to a specialized computing apparatus implementing one or more aspects of the present subject matter. Any suitable programming, scripting, or other type of language or combinations of languages may be used to implement the teachings contained herein in software to be used in programming or configuring a computing device.

Claim 1:
A computer-implemented method for determining activity, the method comprising:
receiving, from an inertial measurement sensor, a plurality of inertial measurements in three dimensions for a time period;
calculating statistical data derived from the inertial measurements;
providing the plurality of inertial measurements and the statistical data to a predictive model;
receiving, from the predictive model, a determined activity from a plurality of predefined activities based on the inertial measurements, wherein the plurality of predefined activities comprises: (i) asleep, (ii) stirring, and (iii) settled, wherein stirring represents movement more than a first threshold amount, and settled represents movement less than a second threshold amount;
determining, based on the predictive model, a plurality of activities, each corresponding to a different one of a plurality of time periods;
adjusting an activity from the plurality of activities by:
when the activity is stirring in a predetermined time period between time periods whose determined activity is asleep, changing the activity to asleep, and/or
when the activity is settled in a predetermined time period before a time period whose determined activity is asleep, changing the activity to asleep; and
providing the determined activity to an output device.