Methods and devices for detection of eating behavior

Systems, devices, and methods are provided for detecting and correcting eating behavior. A device may receive audio data, determine that the audio data is indicative of consumption of a product by a user. The device may determine, based on the product, a measureable attribute associated with the user. The device may receive first data associated with the measureable attribute. The device may determine that the first data exceeds a threshold. The device may generate a message for presentation.

BACKGROUND

To address health issues related to the consumption of food, beverage, or other products, medical professionals may recommend consuming more or less of certain types of products. In addition, devices may monitor biomedical characteristics of patients, allowing patients and medical professionals to address health-related issues. Some methods of monitoring the effects of consumable products on patients may be inconvenient and ineffective.

Certain implementations will now be described more fully below with reference to the accompanying drawings, in which various implementations and/or aspects are shown. However, various aspects may be implemented in many different forms and should not be construed as limited to the implementations set forth herein; rather, these implementations are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art. Like numbers in the figures refer to like elements throughout. Hence, if a feature is used across several drawings, the number used to identify the feature in the drawing where the feature first appeared will be used in later drawings.

DETAILED DESCRIPTION

Overview

Example embodiments described herein provide certain systems, methods, and devices for detection and correction of eating behavior.

Consumable products, such as foods, beverages, nutritional products, vitamins, pharmaceuticals, biologics, and others, may cause effects on people who consume them. For example, because some carbohydrates may be broken down into sugar, such as glucose, some foods high in carbohydrates may cause an increase in a person's blood sugar. Alcohol content may cause an increase in a person's blood alcohol levels or heart rate. Foods or beverages with high fat, salt, or caloric content may cause higher blood pressure. Caffeine may increase a person's heart rate.

Consumable products also may exhibit characteristics associated with their consumption. For example, the opening of a can or bottle with a carbonated beverage may exhibit a distinct sound. The chewing of crunchy food such as potato chips may sound different than the chewing of an apple.

Patients and medical professionals may benefit from customizing the monitoring of biomedical data based on when and what a patient may be consuming. For example, the sampling rate for a person's heartrate may increase when a person eats or drinks to better capture the effects of the consumable product on the person, and the movement or activity of a person may be monitored differently when a user is consuming a product. Because an audio sampling may be captured at a low frequency at small intervals, the audio sampling may result in the capture of audio data used to identify when a person is eating or drinking, and to trigger a higher frequency sampling or higher sampling rate for more detailed data collection.

Therefore, by detecting audio associated with consumption of a product and/or opening of a package, the process for determining that someone is consuming a consumable product and identifying that consumable product may be automated. By identifying biomedical data which may be affected by the consumption of a particular type of consumable product, additional data including the biomedical data may be monitored in customizable ways for the effects that the consumable product has on the person who consumed the product. In this manner, systems, devices, and methods may detect eating and drinking behavior, and may correct the behavior by encouraging a person to avoid consuming certain products or to substitute healthier alternative products.

In one or more embodiments, a wearable device such as a watch, a ring, glasses, headbands, or medical device may record audio (with a user's consent) using one or more audio sensors (e.g., microphones, electromyography sensors). Captured audio data may be analyzed by the device or sent to another device for analysis. The audio data may be converted to a sound profile. For example, a sound profile may include a frequency distribution of captured audio signals over time. A device may compare the sound profile to known sound profiles of consumable products. For example, the crunch of potato chips may match a known sound profile for potato chips. The crisp sound of a user biting into an apple may have a distinct sound profile, as may the sound of swallowing a liquid, opening a carbonated beverage or bag, opening and closing a refrigerator, an active microwave, and the like. Audio profiles of consumable products may be differentiated from audio profiles of other types of noises or sounds, such as talking (e.g., voice) or certain types of background noise (e.g., sounds of musical instruments, automobiles, computer devices, etc.). Machine learning using neural networks or other types of machines may be used to identify sounds and words to identify when a user is consuming a product, about to consume a product, and has recently consumed a product. Using sound profiles, a device may determine a specific product or type of product that a person may be consuming.

In one or more embodiments, a device may determine characteristics of a product once the product has been identified. For example, a cheeseburger may have high cholesterol and may trigger a higher blood pressure for a person, as may potato chips or other foods known to be salty. Candy may include sugar which may cause an increase in a person's blood glucose levels. Spicy or acidic products may cause indigestion or acid reflux. A caffeinated product may increase a person's heart rate. When a device determines the product or type of product that a person may be consuming, the device may determine corresponding characteristics of the product, and may determine data which may be associated with the effects of the characteristics. For example, if a characteristic of a sugary food or drink is to increase blood glucose levels, a device may determine that blood glucose data may indicate the effects of consuming the sugary food or drink. If caffeine products are known to increase heartrate, a device may determine that monitoring a user's heartrate may provide an indication of the effects of consuming caffeine.

In one or more embodiments, a device may determine that another device or an application is responsible for detecting or otherwise collecting data associated with a characteristic of a consumable product. For example, a blood glucose monitor may measure blood glucose levels. A heartrate monitor may capture heartrate data. A hydration sensor may measure a user's dehydration. A pacemaker may recognize a person's electrocardiogram. A thermometer may measure a person's temperature. An accelerometer, magnetometer, wireless signals (e.g., Bluetooth or Wi-Fi signals), global navigation satellite system signals may be used (with a user's consent) to determine a device's motion or location, and the motion or location data may confirm if the user is at a location (e.g., a restaurant) or moving (e.g., motioning an arm or hand toward the face) in a manner which indicates a likely consumption of a product (e.g., and may be used to supplement audio data for the purpose of determining when a user is consuming a product). A hydrogen sensor may measure a user's indigestion. When a device determines a characteristic of a consumable product and an associated type of data which may measure the effects of the characteristic on a person consuming the consumable product, the device may identify another device or an application responsible for capturing the associated type of data, and may request the associated data. The request for the data may include specification of a sampling rate or frequency. For example, one device may request that another device provide data captured at a particular rate or frequency (e.g., a higher sampling rate or frequency than normal). Such may allow devices to conserve power and resources (e.g., by not sampling at higher rates or frequencies unless a user is consuming something).

In one or more embodiments, with a user's consent, a device may help a user regulate their intake of consumable products and may provide recommendations for products, when to consume or not consume, locations where consumable products are available, nutritional information, warnings/alerts, alarms to medical professionals or other parties or devices, and the like. For example, when a device detects that a user is eating food late at night (e.g., outside of a normal window of time associated with eating meals), the device may present alarms or messages encouraging the user to eat something healthier or to wait until the next meal, or to indicate the effects that consuming a product may have on the person. The device may provide recommendations of healthier products to substitute, such as substituting fruit and vegetables for a less healthy product.

In one or more embodiments, devices may connect to one another using a variety of connection methods such as Wi-Fi, Bluetooth, and ultrasound, and may use direct or other peer-to-peer connections (e.g., Wi-Fi Direct, neighbor awareness networking, etc.) to communicate with one another. For example, a smart phone, tablet, or other mobile device may execute applications which may collect data from other devices, such as heartrate monitors, blood glucose monitors, pacemakers, thermometers, hydrogen sensors, exercise monitors, step monitors, and the like. A device may capture audio, and when the audio indicates a user's consumption of a product, the device may request that a biomedical sensor or another mobile device in communication with the biomedical sensor provide other data, such as additional audio data, biomedical data, user profile or preference data, and the like. A device may collect such data, with user consent, and may analyze the data to determine that a user is consuming a product, what the product is, characteristics of the product, data indicative of or effected by the characteristics, and the effects of the product on a person. When audio data is strongly indicative of consumption (e.g., a confidence level associated with the sound profile of a consumable product exceeds a threshold confidence), the data collected by a mobile device may be analyzed for the effects that a consumable product has on a person. When the confidence level associated with the sound profile of a consumable product does not exceed a threshold confidence, data may be collected from or sent to a remote service (e.g., a cloud server) for analysis or collection.

In one or more embodiments, time and duration of consumption may be indicative of an amount of a product consumed and/or when the product is consumed. If audio data indicates that a user is consuming products outside of normal meal times or that a user is consuming products for a long time (e.g., longer than a time threshold), a device may determine that a user may be consuming too much, too little, and/or consuming products and unhealthy times of day (e.g., right before sleeping). Such may trigger the generation of messages or alarms, and/or the capturing of relevant biomedical data to monitor.

Illustrative Processes and Use Cases

FIG.1illustrates an example system100for detection and correction of eating behavior, in accordance with one or more example embodiments of the present disclosure.

Referring toFIG.1, the system100may include a user102wearing a wearable device104shown as a watch. The user102may have a user device106, which may be in communication with one or more biomedical devices (e.g., device108, device110). The user device106may collect data using one or more applications (e.g., application112). For example, the application112may collect audio data captured by the wearable device104, time data (e.g., a time of day, a duration associated with captured data), heart data (e.g., heartrate data or electrocardiogram data captured by the device108), blood glucose data (e.g., captured by the device108or the device110), movement data (e.g., captured by the user device106), and location data (e.g., as captured by the user device106). The data captured by the wearable device104, the user device106, the device108, and/or the device110may be collected with user consent (e.g., the user may be prompted to confirm whether to allow data to be collected and/or tracked).

Still referring toFIG.1, the wearable device104may capture (e.g., record) audio data of the user102(with user consent) consuming one or more consumable products according to a variety of scenarios. In scenario114, a user may be drinking a liquid with one hand, and wearing the wearable device104with the other hand/arm. In scenario116, a user may be eating a product with one hand, and wearing the wearable device104with the other hand/arm. In scenario118, a user may be drinking a liquid with the same hand/arm wearing the wearable device104. In scenario120, a user may be eating a product with the same hand/arm wearing the wearable device104. In any scenario, the wearable device104may capture audio such as chewing, swallowing, opening a package or container (e.g., a bottle, can, bag, box, jar, etc.), opening or closing a refrigerator or microwave, or audio of a person talking (e.g., audio including keywords regarding the consumption of a product or location where consumable products may be sold).

Still referring toFIG.1, the user device106may communicate (e.g., using one or more communication networks130) with one or more servers140(e.g., cloud-based servers), and with one or more devices150(e.g., computer device152, treadmill154, refrigerator156) using the one or more communication networks130or using a direct connection (e.g., Wi-Fi, Bluetooth, ultrasound). The one or more servers140may receive data captured by the wearable device104, the user device106, the device108, the device110, and/or the one or more devices150and may analyze the data, or any combination of the one or more servers140, the user device106, and the wearable device104may analyze the captured data. With user consent, the one or more servers140may provide user data, such as health data, data regarding the user's product consumption habits and history, exercise and other activity data, and the like. The one or more devices150may provide data indicating when a user exercised or bought consumable products (e.g., using browsing or other search history from the computer device152, or medical data such as medical history or prescription product history from the computer device152). Such data from the one or more devices150may indicate activity options (e.g., exercising options available to a user) and for analysis regarding whether a user is exercising after consuming certain types of products.

In one or more embodiments, the user device106may execute one or more applications, such as the application112, which may collect biomedical and/or other data from the device108and/or the device110. For example, the device108and/or the device110may include a blood glucose monitor, a heartrate monitor, electrodes, a pacemaker, a thermometer, a hydration monitor, a hydrogen sensor, or other sensors or devices capable of detecting user data with a user's consent. One or more applications executable by the user device106may collect and analyze data from the device108and/or the device110, and may send the data to the one or more servers140for analysis, or may analyze the data locally on the user device106. The analysis of the data may be supplemented by data from the one or more devices150(e.g., to determine user purchasing and/or exercising habits).

In one or more embodiments, the wearable device104may include a wearable wireless device (e.g., bracelet, watch, glasses, ring, etc.) capable of capturing audio with one or more sensors (e.g., one or more microphones and/or electromyography sensors, not shown). The one or more sensors may be arranged to detect sounds at different levels and/or in different ranges or directions from the wearable device104. The use of multiple sensors may allow for noise cancelation (e.g., background noise suppression) while preserving sounds relevant to consumption of a product. The audio data may be analyzed by the wearable device104, the user device106, and/or the one or more servers140to determine whether the audio data indicates that the user102is consuming a product. For example, captured audio data may be analyzed by the wearable device104or sent to another device (e.g., the user device106or the one or more servers140) for analysis. The audio data may be converted to a sound profile. For example, a sound profile may include a frequency distribution of captured audio signals over time. The wearable device104, the user device106, or the one or more servers140may compare the sound profile to known sound profiles of consumable products. For example, the crunch of potato chips may match a known sound profile for potato chips. The crisp sound of a user biting into an apple may have a distinct sound profile, as may the sound of swallowing a liquid, opening a carbonated beverage or bag, opening and closing a refrigerator, an active microwave, and the like. Audio profiles of consumable products may be differentiated from audio profiles of other types of noises or sounds, such as talking (e.g., voice) or certain types of background noise (e.g., sounds of musical instruments, automobiles, computer devices, etc.). Machine learning modules142may use neural networks or other types of machines (e.g., implemented by the one or more servers140) may be used to identify sounds and words to identify when a user is consuming a product, about to consume a product, and has recently consumed a product. Using sound profiles, the wearable device104, the user device106, or the one or more servers140may determine a specific product or type of product that a person may be consuming.

In one or more embodiments, when captured audio by the wearable device104or another device matches an audio profile of a consumable product, the wearable device104, the user device106, or the one or more servers140may determine one or more characteristics associated with the consumable product. For example, a cheeseburger may have high cholesterol and may trigger a higher blood pressure for a person, as may potato chips or other foods known to be salty. Candy may include sugar which may cause an increase in a person's blood glucose levels. Spicy or acidic products may cause indigestion or acid reflux. A caffeinated product may increase a person's heart rate. When the wearable device104, the user device106, or the one or more servers140determines the product or type of product that a person may be consuming, the wearable device104, the user device106, or the one or more servers140may determine corresponding characteristics of the product, and may determine data which may be associated with the effects of the characteristics. For example, if a characteristic of a sugary food or drink is to increase blood glucose levels, the wearable device104, the user device106, or the one or more servers140may determine that blood glucose data may indicate the effects of consuming the sugary food or drink. If caffeine products are known to increase heartrate, the wearable device104, the user device106, or the one or more servers140may determine that monitoring a user's heartrate may provide an indication of the effects of consuming caffeine.

In one or more embodiments, when the wearable device104, the user device106, or the one or more servers140determines one or more characteristics associated with a consumable product, the wearable device104, the user device106, or the one or more servers140may determine a measurable attribute to capture, an application (e.g., the application112) that may capture data indicative of the measurable attribute, and/or a device (e.g., the device108, the device110, the one or more devices150) associated with capturing and/or providing the data indicative of the measurable characteristic. The measurable attribute may include blood glucose levels or other blood sugar levels, heartrate, electrocardiogram data, hydrogen data, breathing data, perspiration data, movement or activity data, biomedical cell data, skin data, tissue data, circulatory data, blood content, blood alcohol data, and the like. With user consent, the wearable device104, the user device106, or the one or more servers140may determine which device and/or application may provide such data for the measurable attribute.

In one or more embodiments, when the wearable device104, the user device106, or the one or more servers140determines a device and/or application which may capture and provide data for the measurable attribute associated with a characteristic of a consumable product, the wearable device104, the user device106, or the one or more servers140may request the data for the measurable attribute at a particular sampling rate or frequency. For example, a device may sample the data for the measurable characteristic at one sampling rate or frequency, and the wearable device104, the user device106, or the one or more servers140may request that the sampling rate or frequency be increased at least for a time period (e.g., until it is determined that the user is no longer consuming the product or a threshold time after consumption of the product). The wearable device104, the user device106, or the one or more servers140may receive the data captured at an increased frequency or sampling rate, allowing the capturing device to conserve resources by sampling at a lower frequency or sampling rate outside of requests from the wearable device104, the user device106, or the one or more servers140.

In one or more embodiments, the wearable device104, the user device106, or the one or more servers140may receive captured data associated with a measurable attribute, may analyze the data, and may determine an association between the data of the measurable attribute and the product, along with similar products. The wearable device104, the user device106, or the one or more servers140may store data associated with a user and with a product that indicates user reactions associated with the measurable attribute (e.g., changes in heartrate, blood sugar, hydration levels, breathing levels, heart waves, blood pressure, neurological data, and the like).

In one or more embodiments, the wearable device104, the user device106, or the one or more servers140may generate one or more messages, alarms, or alerts based on the data. For example, if the wearable device104, the user device106, or the one or more servers140determines from prior association data that a product or similar product causes a negative biomedical effect of a user, the wearable device104, the user device106, or the one or more servers140may generate a message or alarm intended to discourage the user from consuming the product, may recommend substitute products known to cause less of the biomedical effect (e.g., not associated with the consumable product's characteristics), and/or may send alerts to other devices to let other users know that the person is consuming a product known to cause a negative biomedical effect. As shown inFIG.1, the wearable device104may display nutritional information, including the content of sodium and fat, along with a recommendation to try a healthier product (e.g., to substitute an apple for salty potato chips). Messages and alerts may provide any combination of information related to a user's health (e.g., heartrate, blood glucose, blood pressure, etc.), information about what the user is consuming (e.g., nutritional information, health-related effects, etc.), messages to encourage or discourage consumption of certain products, offers for similar or substitute products, notifications of locations where products may be purchased, notifications regarding a user's exercise habits and/or exercise options and locations (e.g., that the treadmill154or other exercise equipment like an exercise bicycle are available and have not been used since a given time), and the like.

In one or more embodiments, with user consent, recording audio or other data by the wearable device104may activate or change based on a time, location, position, or movement of the wearable device104. For example, when a time of day is within a selected or known meal time (e.g., a range of time in the morning for breakfast, a range of time in the afternoon for lunch, a range of time in the evening for dinner), the wearable device104may activate recording or increase sampling frequency. In this manner, activation may include initiating or powering on one or more components of the wearable device104(e.g., such as microphones or other audio sensors), and may include adjusting a sampling rate or frequency. When the wearable device104is in a position or orientation associated with consumption of a product, the wearable device104may activate recording or increase sampling frequency. For example, using accelerometer, magnetometer, or other device data, the wearable device104may determine that a user's arm or hand is at an angle (e.g., within an angular range) with respect to one or more additional sensors on other devices and/or with respect to gravity known to be associated with bringing a consumable product to a user's face for consumption.

In one or more embodiments, with user consent, when the wearable device104is in such a position or orientation, the wearable device104may activate a timer to determine the duration that the wearable device104is in the position or orientation. The timer may capture time indicating a duration (e.g., how long a user is consuming a product), which may be correlated with an amount of product consumption (e.g., the longer the duration, the more product is consumed). The wearable device104may deactivate recording or decrease sampling frequency when the wearable device104determines that it is no longer in a consumption position or orientation. The wearable device104may determine the time at which a user may be consuming a product, and may generate messages based on the time (e.g., to not eat in between meals). The wearable device104may use global navigation satellite system data, Wi-Fi data, Bluetooth data, ultrasound data, accelerometer data, magnetometer data, or other data to identify its location. The wearable device104may determine (e.g., using a map or other type of application executable on the wearable device104or by the user device106) whether the device's current location is at or near (e.g., within a distance threshold) of a restaurant or other provider of consumable products, and may generate offers, incentives, alternative options, or messages discouraging the consumption of certain products.

In one or more embodiments, to identify products based on what the user is determined to be consuming, the wearable device104, the user device106, and/or the one or more servers140may use product identifiers. For example, when audio data matches a sound profile associated with consumption of a product, the product may have a product identifier. The wearable device104, the user device106, and/or the one or more servers140may store and/or access data including related or different products. For example, given a product identifier, the wearable device104, the user device106, and/or the one or more servers140may identify other products having similar characteristics (e.g., health characteristics, nutritional content, types of products, a same brand, same effects on a person's health, such as decreased heartrate or blood pressure, etc.) or substitute products (e.g., healthier products not known to cause the same level of effects such as heartrate or blood pressure changes, products with less content of certain ingredients such as sugar or fat, etc.).

In one or more embodiments, the one or more communications networks130may include, but not limited to, any one of a combination of different types of suitable communications networks such as, for example, broadcasting networks, cable networks, public networks (e.g., the Internet), private networks, wireless networks, cellular networks, or any other suitable private and/or public networks. Further, any of the one or more communications networks130may have any suitable communication range associated therewith and may include, for example, global networks (e.g., the Internet), metropolitan area networks (MANs), wide area networks (WANs), local area networks (LANs), or personal area networks (PANs). In addition, any of the one or more communications networks may include any type of medium over which network traffic may be carried including, but not limited to, coaxial cable, twisted-pair wire, optical fiber, a hybrid fiber coaxial (HFC) medium, microwave terrestrial transceivers, radio frequency communication mediums, white space communication mediums, ultra-high frequency communication mediums, satellite communication mediums, or any combination thereof.

FIG.2illustrates an example process200for detection and correction of eating behavior, in accordance with one or more example embodiments of the present disclosure.

Referring toFIG.2, a person202wearing a wearable device204(e.g., having functionality as described with regard to the wearable device104ofFIG.1) may consume a product206(e.g., a liquid or type of beverage). With user consent, the wearable device204may capture audio208of the person202consuming the product (e.g., a swallowing sound). At block212of the process200, the wearable device204may detect the audio208(e.g., using one or more microphones or other audio sensors). The audio208may include chewing, swallowing, opening the product206, words spoken by the person202, sounds made by the product206(e.g., carbonated beverage sounds), or other audio.

Still referring toFIG.2, the wearable device204optionally may consider its position, orientation, and/or movement. For example, at block214, the wearable device204may detect its position, movement, or orientation (e.g., angles, rotation, movement directions, etc.), and at block216may determine that the wearable device204is in a position within a threshold range (e.g., angular range indicative of a tilt, height within a threshold distance of a user's face, angular range with respect to gravity or one or more other devices, etc.). Based on the audio208and optionally the position, movement, or orientation data, the wearable device at block218may determine that the person202is consuming the product206. For example, the wearable device204may determine that the audio208matches one of more sound profiles for various products or types of products (e.g., food, beverage, etc.). The sound profiles may be ranked (e.g., with respective scores indicating the likelihood that the sound profile matches the audio), and the sound profile with the highest score may be selected. The product or product type corresponding to the selected sound profile may be identified by the wearable device204as the product206.

Still referring toFIG.2, at block220, the wearable device204may determine that additional data associated with a measurable attribute (e.g., heartrate, blood pressure, blood sugar, etc.) is to be captured. For example, when the product206is known (e.g., based on a product profile stored and accessed based on the corresponding product identifier) to have high cholesterol content, the wearable device204may determine corresponding characteristics of the product (e.g., increased blood pressure), and may determine data (e.g., blood pressure data) which may be associated with the effects of the characteristic. When a characteristic of a sugary food or drink is to increase blood glucose levels, the wearable device204may determine that blood glucose data may indicate the effects of consuming the sugary food or drink. If caffeine products are known to increase heartrate, the wearable device204may determine that monitoring a heartrate of the person202may provide an indication of the effects of consuming caffeine. The wearable device204may determine that another device (e.g., the device108or the device110ofFIG.1) or an application (e.g., the application112ofFIG.1) is responsible for detecting or otherwise collecting data associated with a characteristic of the product206. At block222, when the wearable device204determines a characteristic of the product206and an associated type of data (e.g., a measurable attribute) which may measure the effects of the characteristic on the person202consuming the product206, the wearable device204may identify another device (e.g., the device108or the device110ofFIG.1) or an application (e.g., the application112ofFIG.1) responsible for capturing the associated type of data (e.g., as additional data), and may request and receive the associated data. The additional data may include additional audio data, existing audio data, biomedical data, data from one or more other devices, and/or other types of data. The request for the data may include specification of a sampling rate or frequency with which to capture or otherwise detect the additional data. At block224, the wearable device may receive the additional data.

In one or more embodiments, the request for additional data (e.g., biomedical data) sent by the wearable device204may be sent to another device (e.g., the user device106ofFIG.1), which may execute one or more applications (e.g., the application112ofFIG.1), which may collect biomedical and/or other data from other devices (e.g., the device108and/or the device110ofFIG.1). The other devices may include a blood glucose monitor, a heartrate monitor, electrodes, a hydration monitor, a hydrogen sensor, or other sensors or devices capable of detecting user data with a user's consent.

In one or more embodiments, the wearable device204may include a wearable wireless device (e.g., bracelet, watch, glasses, ring, etc.) capable of capturing audio with one or more sensors (e.g., a microphone, not shown). The audio data may be analyzed by the wearable device204or another device to determine whether the audio208indicates that the person202is consuming the product206and what the product206is.

In one or more embodiments, the wearable device204or another device may analyze the additional data, and may determine an association between the additional data of a measurable attribute and the product206, along with similar products (e.g., identified using the product identifier for the product206). The wearable device204or another device may store data associated with the person202and with the product206that indicates user reactions associated with the measurable attribute (e.g., changes in heartrate, blood sugar, hydration levels, breathing levels, heart waves, blood pressure, neurological data, and the like).

In one or more embodiments, the wearable device204may generate one or more messages, alarms, or alerts based on the audio208and/or the additional data. For example, if the wearable device204determines from prior association data that the product206or a similar product causes a negative biomedical effect of the person202or another person, the wearable device204may generate a message or alarm intended to discourage the person202from consuming the product206, may recommend substitute products known to cause less of the biomedical effect (e.g., not associated with the consumable product's characteristics), and/or may send alerts to other devices to let other users know that the person202is consuming the product206known to cause a negative biomedical effect. The wearable device204may display nutritional information, including the content of sodium and fat, along with a recommendation to try a healthier product (e.g., to substitute water for a sugary drink). Messages and alerts may provide any combination of information related to the person's health (e.g., heartrate, blood glucose, blood pressure, etc.), information about what the person202is consuming (e.g., nutritional information, health-related effects, etc.), messages to encourage or discourage consumption of certain products, offers for similar or substitute products, notifications of locations where products may be purchased, notifications regarding a user's exercise habits and/or exercise options and locations, and the like.

In one or more embodiments, with user consent, recording the audio208or other data by the wearable device204may activate or change based on a time, location, position, or movement of the wearable device204. For example, when a time of day is within a selected or known meal time (e.g., a range of time in the morning for breakfast, a range of time in the afternoon for lunch, a range of time in the evening for dinner), the wearable device204may activate recording or increase sampling frequency. In this manner, activation may include initiating or powering on one or more components of the wearable device204(e.g., such as microphones or other audio sensors), and may include adjusting a sampling rate or frequency. When the wearable device204is in a position or orientation associated with consumption of a product, the wearable device204may activate recording or increase sampling frequency. For example, using accelerometer, magnetometer, or other device data, the wearable device204may determine that a user's arm or hand is at an angle (e.g., within an angular range) with respect to one or more additional sensors on other devices and/or with respect to gravity known to be associated with bringing a consumable product to a user's face for consumption.

In one or more embodiments, with user consent, when the wearable device204is in such a position or orientation, the wearable device204may activate a timer to determine the duration that the wearable device204is in the position or orientation. The timer may capture time indicating a duration (e.g., how long a user is consuming a product), which may be correlated with an amount of product consumption (e.g., the longer the duration, the more product is consumed). The wearable device204may deactivate recording or decrease sampling frequency when the wearable device204determines that it is no longer in a consumption position or orientation. The wearable device204may determine the time at which a user may be consuming a product, and may generate messages based on the time (e.g., to not eat in between meals). The wearable device204may use global navigation satellite system data, Wi-Fi data, Bluetooth data, ultrasound data, accelerometer data, magnetometer data, or other data to identify its location. The wearable device204may determine (e.g., using a map or other type of application executable on the wearable device204) whether the device's current location is at or near (e.g., within a distance threshold) of a restaurant or other provider of consumable products, and may generate offers, incentives, alternative options, or messages discouraging the consumption of certain products.

WhileFIG.2shows the user202consuming the product206with the same hand/arm that is wearing the wearable device204, as discussed below with regard toFIG.3, the wearable device204may be used in a similar manner to determine that the person202is consuming the product206even when the wearable device204is worn by the opposite arm/hand (or is at or near another part of the person's body).

FIG.3illustrates an example process300for detection and correction of eating behavior, in accordance with one or more example embodiments of the present disclosure.

Referring toFIG.3, a person302wearing a wearable device304(e.g., having functionality as described with regard to the wearable device104ofFIG.1) may consume a product306(e.g., a food, nutritional product, or other type of consumable product) from a package307. With user consent, the wearable device304may capture audio308of the person302consuming the product (e.g., a chewing and/or swallowing sound) and/or from the package307(e.g., a crinkling sound of a bag, a sound of a top being popped or unscrewed, a sound of a box being ripped open, etc.). At block312of the process300, the wearable device304may detect the audio308(e.g., using one or more microphones or other audio sensors). The audio308may include chewing, swallowing, opening the package307, words spoken by the person302, sounds made by the product306(e.g., crunchy sounds, chewy sounds, etc.), or other audio.

Still referring toFIG.3, the wearable device304optionally may consider its position, orientation, and/or movement. For example, at block314, the wearable device304may detect its position, movement, or orientation (e.g., angles, rotation, movement directions, etc.), and may determine that the wearable device304is in a position within a threshold range. Based on the audio308and optionally the position, movement, or orientation data, the wearable device at block316may determine that the person302is consuming the product306(e.g., eating). For example, the wearable device304may determine that the audio308matches one of more sound profiles for various products or types of products (e.g., food, beverage, etc.). The sound profiles may be ranked (e.g., with respective scores indicating the likelihood that the sound profile matches the audio), and the sound profile with the highest score may be selected. The product or product type corresponding to the selected sound profile may be identified by the wearable device304as the product306.

Still referring toFIG.3, at block318, the wearable device304may determine that additional data associated with a measurable attribute (e.g., heartrate, blood pressure, etc.) is to be captured. For example, when the product306is known (e.g., based on a product profile stored and accessed based on the corresponding product identifier) to be salty (e.g., if the product306is a potato chip), the wearable device304may determine corresponding characteristics of the product (e.g., increased blood pressure), and may determine data (e.g., blood pressure data) which may be associated with the effects of the characteristic. When a characteristic of a product having high sugar content is to increase blood glucose levels, the wearable device304may determine that blood glucose data may indicate the effects of consuming the sugary product. The wearable device304may determine that another device (e.g., the device108or the device110ofFIG.1) or an application (e.g., the application112ofFIG.1) is responsible for detecting or otherwise collecting data associated with a characteristic of the product306. At block320, when the wearable device304determines a characteristic of the product306and an associated type of data (e.g., additional data) which may measure the effects of the characteristic on the person302consuming the product306, the wearable device304may identify another device (e.g., the device108or the device110ofFIG.1) or an application (e.g., the application112ofFIG.1) responsible for capturing the associated type of data, and may request and receive the associated data. The additional data may include additional audio data, existing audio data, biomedical data, data from one or more other devices, and/or other types of data. The request for the data may include specification of a sampling rate or frequency with which to capture or otherwise detect the additional data. At block322, the wearable device may receive the additional data.

In one or more embodiments, the request for additional data (e.g., biomedical data) sent by the wearable device304may be sent to another device (e.g., the user device106ofFIG.1), which may execute one or more applications (e.g., the application112ofFIG.1), which may collect biomedical and/or other data from other devices (e.g., the device108and/or the device110ofFIG.1). The other devices may include a blood glucose monitor, a heartrate monitor, electrodes, a hydration monitor, a hydrogen sensor, or other sensors or devices capable of detecting user data with a user's consent.

In one or more embodiments, the wearable device304may include a wearable wireless device (e.g., bracelet, watch, glasses, ring, etc.) capable of capturing audio with one or more sensors (e.g., a microphone, not shown). The audio data may be analyzed by the wearable device304or another device to determine whether the audio308indicates that the person302is consuming the product306and what the product306is (e.g., the specific product as identified by type and brand, or a type or category of the product306such as food, beverage, medicine, nutritional product, fruit, vegetable, snack, candy, burger, chips, water, cola/soda, sugary drink, vitamin, etc.).

In one or more embodiments, the wearable device304or another device may analyze the additional data, and may determine an association between the additional data of a measurable attribute and the product306, along with similar products (e.g., identified using the product identifier for the product306). The wearable device304or another device may store data associated with the person302and with the product306that indicates user reactions associated with the measurable attribute (e.g., changes in heartrate, blood sugar, hydration levels, breathing levels, heart waves, blood pressure, neurological data, and the like).

In one or more embodiments, the wearable device304may generate one or more messages, alarms, or alerts based on the audio308and/or the additional data. For example, when the wearable device304determines from prior association data that the product306or a similar product causes a negative biomedical effect of the person202or another person, the wearable device304may generate a message or alarm intended to discourage the person232from consuming the product306, may recommend substitute products known to cause less of the biomedical effect (e.g., not associated with the consumable product's characteristics), and/or may send alerts to other devices to let other users know that the person302is consuming the product306known to cause a negative biomedical effect. The wearable device304may display nutritional information, including the content of sodium and fat, along with a recommendation to try a healthier product (e.g., to substitute water for a sugary drink). Messages and alerts may provide any combination of information related to the person's health (e.g., heartrate, blood glucose, blood pressure, etc.), information about what the person302is consuming (e.g., nutritional information, health-related effects, etc.), messages to encourage or discourage consumption of certain products, offers for similar or substitute products, notifications of locations where products may be purchased, notifications regarding a user's exercise habits and/or exercise options and locations, and the like.

In one or more embodiments, with user consent, recording the audio308or other data by the wearable device304may activate or change based on a time, location, position, or movement of the wearable device304. For example, when a time of day is within a selected or known meal time (e.g., a range of time in the morning for breakfast, a range of time in the afternoon for lunch, a range of time in the evening for dinner), the wearable device304may activate recording or increase sampling frequency. In this manner, activation may include initiating or powering on one or more components of the wearable device304(e.g., such as microphones or other audio sensors), and may include adjusting a sampling rate or frequency. When the wearable device304is in a position or orientation associated with consumption of the product306, the wearable device304may activate recording or increase sampling frequency. For example, using accelerometer, magnetometer, or other device data, the wearable device304may determine that a user's arm or hand is at an angle (e.g., within an angular range) with respect to one or more additional sensors on other devices and/or with respect to gravity known to be associated with bringing a consumable product to a user's face for consumption.

In one or more embodiments, with user consent, when the wearable device304is in such a position or orientation, the wearable device304may activate a timer to determine the duration that the wearable device304is in the position or orientation. The timer may capture time indicating a duration (e.g., how long a user is consuming a product), which may be correlated with an amount of product consumption (e.g., the longer the duration, the more product is consumed). The wearable device304may deactivate recording or decrease sampling frequency when the wearable device304determines that the person302is no longer consuming the product (e.g., the audio308stops and/or no additional sound is identified from the package307). The wearable device304may determine the time at which a user may be consuming a product, and may generate messages based on the time (e.g., to not eat in between meals). The wearable device304may use global navigation satellite system data, Wi-Fi data, Bluetooth data, ultrasound data, accelerometer data, magnetometer data, or other data to identify its location. The wearable device304may determine (e.g., using a map or other type of application executable on the wearable device304) whether the device's current location is at or near (e.g., within a distance threshold) of a restaurant or other provider of consumable products, and may generate offers, incentives, alternative options, or messages discouraging the consumption of certain products.

FIG.4Aillustrates a flow diagram for a process400for detection and correction of eating behavior, in accordance with one or more example embodiments of the present disclosure.

At block402, a device (e.g., the wearable device104ofFIG.1), with user consent, may receive audio data associated with the consumption of a consumable product (e.g., the product206ofFIG.2, the product306and/or the package307ofFIG.3). For example, with user consent, the device may record audio with one or more audio sensors (e.g., microphones). The recording may be constant, periodic, based on user input, based on times of day, based on movement, position, or orientation of the device, based on the location of the device (e.g., global positioning coordinates), or the like. In one example, a user may be drinking a liquid with one hand, and wearing the device with the other hand/arm. In another example, a user may be eating a product with one hand, and wearing the device with the other hand/arm. In another example, a user may be drinking a liquid with the same hand/arm wearing the device. In another example, a user may be eating a product with the same hand/arm wearing the device. In any scenario, the device may capture audio such as chewing, swallowing, opening a package or container (e.g., a bottle, can, bag, box, jar, etc.), opening or closing a refrigerator or microwave, or audio of a person talking (e.g., audio including keywords regarding the consumption of a product or location where consumable products may be sold). The device may be worn on a hand, arm, leg, ankle, around the head or neck, or at another location of a body. The product may include any combination of food, beverage, medical products, nutritional products, or any other consumable product.

At block404, the device may determine that the audio data matches an audio profile for a consumable product or multiple consumable products. For example, the sound of one consumable product may be different when combined with another consumable product. The audio data may be converted to a sound profile. For example, a sound profile may include a frequency distribution of captured audio signals over time. A device may compare the sound profile to known sound profiles of consumable products. For example, the crunch of potato chips may match a known sound profile for potato chips. The crisp sound of a user biting into an apple may have a distinct sound profile, as may the sound of swallowing a liquid, opening a carbonated beverage or bag, opening and closing a refrigerator, an active microwave, and the like. Audio profiles of consumable products may be differentiated from audio profiles of other types of noises or sounds, such as talking (e.g., voice) or certain types of background noise (e.g., sounds of musical instruments, automobiles, computer devices, etc.). Machine learning using neural networks (e.g., the one or more machine learning modules142ofFIG.1) or other types of machines may be used to identify sounds and words to identify when a user is consuming a product, about to consume a product, and has recently consumed a product. Using sound profiles, the device may determine a specific product or type of product that a person may be consuming (or combination of products).

At block406, the device may determine a characteristic of the consumable product or products. For example, when the device determines the product that is being consumed, the device may identify the product and an associated product identifier (or multiple identifiers for a combination of products). The product identifier may be stored on the device or elsewhere (e.g., the user device106or the one or more servers140ofFIG.1) in addition to characteristics of the product, such as nutrition content, ingredients, product categories, health effects (e.g., changes to blood sugar, blood pressure, cholesterol, heartrate, perspiration, breathing rate, acid reflux, indigestion, fatigue, blood flow, blood alcohol level, medical side effects, etc.).

At block408, the device may determine, based on the characteristic, one or more measurable attributes of a user to assess. For example, when a characteristic of a product is associated with impacting heartrate, the measurable attribute may be a user's heartrate. When a characteristic is a change in blood pressure, the measurable attribute may be a user's blood pressure. When a characteristic is blood sugar, the measurable attribute may be a user's blood sugar. When the characteristic is blood alcohol, the measurable attribute may be a user's blood alcohol level. When the characteristic is a hydration level, the measurable attribute may be a user's hydration or perspiration. When the characteristic is indigestion or heartburn, the measurable attribute may be a user's hydrogen level. When the characteristic is a known medical side effect, the measurable attribute may be any combination of readings which may indicate whether the side effect is occurring (e.g., breathing, allergic reactions, arrhythmias, etc.). The device may determine another device (e.g., the device108or the device110ofFIG.1) and/or application (e.g., the application112ofFIG.1) which may detect and/or collect data measuring the measurable attribute. For example, when the measurable attribute is a user's heartrate, the device may identify a heartrate monitor and/or an application which may collect heartrate data of a user. When the measurable attribute is blood sugar, the device may identify a glucose monitor and/or application which collects blood glucose data.

At block410, with user consent, the device may obtain data based on the measurable attribute. In particular, the device may send a request for the data associated with the measurable attribute. For example, when the measurable attribute is heartrate, the request may indicate that heartrate data is requested. When the measurable attribute is blood sugar, the request may indicate that blood sugar data is requested. The request may provide parameters, such as time/duration of the recorded data, and at which sampling rate or frequency. For example, the device which detects or captures the data for the measurable attribute (e.g., a heartrate monitor which captures heartrate data) may capture or detect the data at a sampling rate or frequency. Because of the determination that a user may be consuming a product, the requesting device may request sampling at higher sampling rates or frequencies to collect more data for analysis. The request may indicate multiple sampling rates and frequencies based on different times (e.g., a first sampling rate or frequency at one time, and a second sampling rate or frequency at another time). The measurable attribute may be additional audio data at the same frequency or sampling rate as the previously received audio data, or may be additional audio data at a different frequency or sampling rate. The device may receive the data from the capturing device, or from another device which collects the data (e.g., the user device106or the one or more servers140ofFIG.1). The device may specify the time for the data to be delivered and in what format. The device may receive the data according to the requested time and/or format. The request may indicate a device or application associated with capturing and/or providing the requested data.

At block412, with user consent, the device may analyze the data. For example, the device may determine that the data associated with the measurable attribute is further associated with the consumable product. For example, the device may analyze the effects that the consumable product has on a user, and may provide an indication of the effects to be stored with the product identifier for future use. When the effects are a change in heartrate, blood pressure, blood sugar, breathing rate, acid reflux, blood alcohol, perspiration, or the like, the device may associate the effects with the product that was consumed. In this manner, when the device determines a characteristic of a product (e.g., at block406), the device may consider the effects as characteristics of the product or similar products. The device may determine whether the data confirms the characteristic of the product and/or whether the product was correctly identified. receive the data associated with the measurable attribute. The device may determine that the data exceeds a threshold (e.g., a heartrate threshold, a blood pressure threshold, a blood sugar threshold, an electrocardiography threshold, other biomedical thresholds, a threshold time associated with consumption, etc.). Based on the data exceeding the threshold, the device may determine that consumption of a product has caused the consumer's biomedical data to change, that the user is consuming a product outside of a time range, that the user may be consuming too much or too little of the product, that the user is swallowing or chewing too much or too little, and the like. The messages may be generated to indicate such findings, to recommend adjustments to consumption habits, and the like.

At block414, with user consent, the device may perform one or more actions based on the data. The device may generate, based on the data, one or more messages for presentation. For example, the messages may display nutrition or other health information related to the product, a representation of the data for the measureable attribute (e.g., an indication that the data shows the user's blood sugar, heartrate, etc. are effected by consumption of the product). The one or more messages may provide product recommendations for similar or different products, may recommend consumption adjustments, may notify other devices that the person is consuming the product at a particular time, may notify the user of exercise options, and more. The device may request additional information for analysis at the same or different frequency or sampling rates. For example, the device may request additional audio data, time data, device data (e.g., orientation data, movement data, location data, etc.). The device may send instructions to other devices (e.g., user devices, smart home devices, exercise equipment, microwaves, refrigerators, etc.), such as instructions to change or stop detection of data, instructions to display messages (e.g., requesting that a person adjust behavior or consumption habits), instructions to log that a user is consuming the product at a given time, etc.

FIG.4Billustrates a flow diagram for a process450for detection and correction of eating behavior, in accordance with one or more example embodiments of the present disclosure.

At block452, a device (e.g., the wearable device104ofFIG.1) may determine identifiers associated with a consumable product (e.g., the product206ofFIG.2, the product306ofFIG.3), packaging (e.g., the package307ofFIG.3), and/or environmental sounds (e.g., background noises, sounds of devices such as refrigerators, microwaves, cooking appliances, product storage, etc.). For example, when audio data matches a sound profile associated with consumption of a product, the product may have a product identifier. Packaging, such as bottles, cans, bags, boxes, containers, etc. may have product identifiers. Environmental sounds may have identifiers. Any combination of identifiers may be assessed by the device to determine that someone is consuming a product or type of product. For example, the combination of a chewing or swallowing sound and a package sound may indicate a product or type of product. The device may identify certain background noises and cancel the background noises, allowing for analysis on other sounds that may be more relevant to product consumption. The device or another device (e.g., the user device106and/or the one or more servers140ofFIG.1) may store and/or access data including related or different products. When a product has been identified (e.g., based on an audio profile match from received audio data of a person consuming the product or discussing the consumption of the product), the device may identify the product identifier of the product. For example, the matching sound profile for the product identified as being consumed may be stored on the device or the other device with data such as the product identifier, characteristics of the product, measurable attributes of the product, effects that the product has had on one or more users, and the like.

At block454, the device may determine, based on the product identifier, one or more similar products (e.g., having one or more of the same characteristics or effects on a user as the identified product, products having one or more of the same ingredients, products having nutritional content within a range of the product, etc.) and/or one or more different products (e.g., products not having one or more of the same characteristics or user effects as the product, products known to cause the opposite effects of a user, products having nutritional content outside of a range of the product, etc.). For example, given a product identifier, the device or the other device may identify other products having similar characteristics (e.g., health characteristics, nutritional content, types of products, a same brand, same effects on a person's health, such as decreased heartrate or blood pressure, etc.) or substitute products (e.g., healthier products not known to cause the same level of effects such as heartrate or blood pressure changes, products with less content of certain ingredients such as sugar or fat, etc.).

At block456, with user consent, the device may generate one or more messages for presentation (e.g., using the device or another device) based on one or more similar products or one or more different products. For example, as shown inFIG.1, when the identified product is potato chips, the device may display a recommendation for a substitute product such as an apple. In such an example, the apple may be one of multiple products identified as having different characteristics, effects, ingredients, nutritional content, etc. from potato chips. The device may identify the potato chip product identifier (e.g., a specific potato chip product or a categorical product identifier for potato chips in general), and based on characteristics, measurable attributes, content, and/or effects stored in association with the product identifier, may find a corresponding product identifier stored with similar or different characteristics, measurable attributes, content, and/or effects. The messages may select one or more products or product categories (e.g., eat fruit instead of a less healthy product) and may include a recommendation, offer, incentive, or nutritional or health information for the similar or different product. For example, the messages may indicate similar products, similar products that are healthier, products that may be purchased nearby (e.g., within a distance threshold), substitute products, nutritional information for other products (e.g., content of nutritional ingredients such as sodium, fat, sugar, etc.), and/or health information (e.g., effects that a product may have on biomedical data such as heart rate, blood pressure, blood sugar, body temperature, etc.). The messages may be displayed using one or more methods, including text, graphs, audio, vibrations, and the like.

Referring toFIG.4B, the process450may refer to one or more steps associated with block414ofFIG.4A.

FIG.4Cillustrates a flow diagram for a process470for detection and correction of eating behavior, in accordance with one or more example embodiments of the present disclosure.

At block472, with user consent, a device (e.g., the wearable device104ofFIG.1) may determine a time and/or duration of a consumption of a consumable product (e.g., the product206ofFIG.2, the product306ofFIG.3). The time may refer to the time of day. For example, the device may determine the time when any portion of captured audio (e.g., the audio208ofFIG.2, the audio308ofFIG.3) occurred. The duration may refer to an entire time of a captured clip of audio, or to a portion of audio beginning when the device determines that the user is consuming a product and ending when the device determines that the user is no longer consuming a product (e.g., when the audio no longer matches a sound profile associated with consumption of a product).

At block474, with user consent, the device may determine whether the time is within a time range. For example, certain time ranges may be associated with times when a person is expected to consume a product (e.g., meal times, times to take medicine or nutritional products, times to eat or drink based on a user's health, times to eat or drink based a user's schedule, times to eat or drink based on a user's exercise habits, times selected by a user, etc.). When the device determines that the consumption is within a time range (e.g., normal meal hours, times to take medicine, etc.), the device may continue to block476. When the user is consuming a product outside of normal times (e.g., a late-night snack, taking medicine too soon, eating or drinking when the user's health or medical treatment does not allow eating or drinking), the device may proceed to block478.

At block476, with user consent, the device may generate one or more messages with information about one or more products. For example, if the product being consumed is associated with positive effects on the user, the messages may include offers or incentives to consume and/or purchase more of the product or similar products. The messages may encourage a user to continue to consume such products within the time range. The messages may include information or offers regarding substitute (e.g., healthier) products. The device may indicate that a user is chewing too loudly or talking while eating. At block478, with user consent, the device may generate one or more messages discouraging consumption. For example, the messages may indicate that a user should not consume during this time, may suggest substitute products to consume, may sound alarms, and/or may notify other devices that the user is consuming a product outside of the approved time range.

At block480, the device may determine whether the duration exceeds a threshold duration. For example, when a consumption time is limited to a duration (e.g., thirty minutes to complete a meal), and the duration exceeds the duration, the device may determine that the user is consuming too much. A drink of a liquid may be associated with the duration, which may correspond to an amount of liquid. For example, when a user is supposed to drink a certain amount of water, the duration may indicate whether the user consumed that amount, consumed too much, or consumed too little. The duration may be set based on user preferences and/or schedules, based on known meal times, or based on quantities of products to consume at a given time. When the duration is within the threshold duration, the device may proceed to block482. When the duration exceeds the threshold duration, the device may continue to block484.

At block482, with user consent, the device may generate one or more messages encouraging a user to maintain the consumption level associated with the duration and/or to increase consumption. For example, when a user is expected to drink a certain volume of water associated with the duration, and the user's consumption time for the liquid is below the threshold duration, the device may generate messages encouraging the user to drink more water. When the user is determined to be eating for a time within the threshold duration, the device may determine that the user has not eaten too much or too long, and may generate messages encouraging the use to continue to eat within the threshold duration of time.

At block484, with user consent, the device may generate one or more messages encouraging a user to reduce consumption. For example, if the user is determined to be eating a product for a duration longer than the threshold duration, the device may generate messages reminding a user of the duration, indicating the nutritional content of the product, or indicating the effects that the product may have on one or more measurable attributes.

Referring toFIG.4C, the process470may refer to one or more steps associated with block414ofFIG.4A.

FIG.4Dillustrates a flow diagram for a process490for detection and correction of eating behavior, in accordance with one or more example embodiments of the present disclosure.

At block492, with user consent, a device (e.g., the wearable device104ofFIG.1) may identify one or more devices (e.g., the user device106, the one or more devices150ofFIG.1). For example, block492represent actions corresponding to block414ofFIG.4A. When the device has identified that a user is consuming a product, has obtained data for a measurable attribute of the product, and has analyzed the data, the device may identify one or more devices to which to send instructions or messages.

At block494, the device may generate one or more instructions based on the data for the product. For example, when the device determines that a user is consuming a product, the device may generate a notification for a smart refrigerator or other smart device (e.g., a smart microwave) to log the consumption, to display or otherwise output messages requesting that the person adjust consumption behavior (e.g., to prevent dispensing a product, to block access to a product, to change or stop operation, etc.). The instructions may indicate to a device to stop recording or otherwise detecting data (e.g., stop recording audio, stop or change collection of biomedical data, etc.). At block496, the device may send the one or more instructions. The instructions may be sent directly to a device, or may be sent through another device (e.g., the user device106, the one or more servers140ofFIG.1).

FIG.5illustrates a flow diagram for a process500for detection and correction of eating behavior, in accordance with one or more example embodiments of the present disclosure.

At block502, with user consent, a device (e.g., the wearable device104ofFIG.1) may determine a time of day when a user is to consume a product (e.g., the product206ofFIG.2, the product306ofFIG.3). The time of day may correspond to meal times, times when a user is to take a medicinal or other health product, times set based on a user's schedule, times when a user is known to perform activities (e.g., exercising), or other times. The times may be preset, selected by users, or based on data (e.g., calendar data from an application executing on the device or another device).

At block504, with user consent, the device may activate one or more sensors on the device. Activating may include powering on a sensor or changing the operating state of a sensors, such as modifying a sampling rate or frequency. For example, when the sensors are microphones, the device may activate a microphone by powering on the microphone and/or by setting a sampling rate or frequency with which to capture audio. The sensors may capture audio at one sampling rate or frequency, and based on the time of day, the device may change (e.g., increase) the sampling rate or frequency for more data to analyze over a time period.

At block506, with user consent, the device may receive captured data by the one or more sensors. Captured audio data may be analyzed by the device or sent to another device for analysis. The audio data may be converted to a sound profile. For example, a sound profile may include a frequency distribution of captured audio signals over time. A device may compare the sound profile to known sound profiles of consumable products. For example, the crunch of potato chips may match a known sound profile for potato chips. The crisp sound of a user biting into an apple may have a distinct sound profile, as may the sound of swallowing a liquid, opening a carbonated beverage or bag, opening and closing a refrigerator, an active microwave, and the like. Audio profiles of consumable products may be differentiated from audio profiles of other types of noises or sounds, such as talking (e.g., voice) or certain types of background noise (e.g., sounds of musical instruments, automobiles, computer devices, etc.). Machine learning using neural networks or other types of machines may be used to identify sounds and words to identify when a user is consuming a product, about to consume a product, and has recently consumed a product. Using sound profiles, a device may determine a specific product or type of product that a person may be consuming.

At block508, the device may determine that the user is no longer consuming the product. The device may determine, based on captured audio, that the use is no longer consuming a product based on whether the captured audio matches audio associated with a known product and when the audio no longer matches audio associated with a known product.

At block510, the device may deactivate the one or more sensors. For example, the device may deactivate (e.g., lower power, sampling rate, or frequency) the one or more sensors when the time of day (or time period of day) has passed.

FIG.6illustrates a block diagram of an example of a machine600(e.g., implemented in whole or in part by the wearable device104ofFIG.1, the user device106ofFIG.1, the one or more servers140ofFIG.1, the device108ofFIG.1, the device110ofFIG.1, the one or more devices150ofFIG.1, the wearable device204ofFIG.2, the wearable device304ofFIG.3) or system upon which any one or more of the techniques (e.g., methodologies) discussed herein may be performed. In other embodiments, the machine600may operate as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine600may operate in the capacity of a server machine, a client machine, or both in server-client network environments. In an example, the machine600may act as a peer machine in Wi-Fi direct, peer-to-peer (P2P) (or other distributed) network environments. The machine600may be a wearable device or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term “machine” shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein, such as cloud computing, software as a service (SaaS), or other computer cluster configurations.

The machine (e.g., computer system)600may include any combination of the illustrated components. For example, the machine600may include a hardware processor602(e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof), a main memory604and a static memory606, some or all of which may communicate with each other via an interlink (e.g., bus)608. The machine600may further include a power management device632, a graphics display device610, an alphanumeric input device612(e.g., a keyboard), and a user interface (UI) navigation device614(e.g., a mouse). In an example, the graphics display device610, alphanumeric input device612, and UI navigation device614may be a touch screen display. The machine600may additionally include a storage device (i.e., drive unit)616, a signal generation device618(e.g., a biomedical data signal or other data signal), a consumption regulation device619, a network interface device/transceiver620coupled to antenna(s)630, and one or more sensors628, such as a sound detecting sensor (e.g., a microphone), one or more electromyography sensors (e.g., to detect swallowing), accelerometers, magnetometers, location sensors, and the like. When using multiple sensors628, the sensors may be arranged to detect sounds in different directions and at different distances. The machine600may include an output controller634, such as a serial (e.g., universal serial bus (USB), parallel, or other wired or wireless (e.g., infrared (IR), near field communication (NFC), etc.) connection to communicate with or control one or more peripheral devices (e.g., a printer, a card reader, other sensors, etc.)).

The consumption regulation device619may carry out or perform any of the operations and processes (e.g., process400ofFIG.4A, process450ofFIG.4B, process470ofFIG.4C, process490ofFIG.4D, process500ofFIG.5) described and shown above.

In one or more embodiments, the consumption regulation device619may be implemented a wearable device (e.g., the wearable device104ofFIG.1) or as a medical device (e.g., the device108or the device110ofFIG.1). The consumption regulation device619may record audio (with a user's consent) using one or more audio sensors (e.g., the one or more sensors628). Captured audio data may be analyzed by the consumption regulation device619or sent to another device (e.g., the user device106or the one or more servers140ofFIG.1) for analysis.

In one or more embodiments, the consumption regulation device619may be implemented in a user device (e.g., the user device106) or a server device (e.g., the one or more servers150ofFIG.1). The consumption regulation device619may convert audio data to a sound profile. For example, a sound profile may include a frequency distribution of captured audio signals over time. The consumption regulation device619may compare the sound profile to known sound profiles of consumable products. The consumption regulation device619may differentiate audio profiles of consumable products from audio profiles of other types of noises or sounds, such as talking (e.g., voice) or certain types of background noise (e.g., sounds of musical instruments, automobiles, computer devices, etc.). The consumption regulation device619may be used to identify sounds and words to identify when a user is consuming a product, about to consume a product, and has recently consumed a product. Using sound profiles, the consumption regulation device619may determine a specific product or type of product that a person may be consuming.

In one or more embodiments, the consumption regulation device619may determine characteristics of a product once the product has been identified. For example, a cheeseburger may have high cholesterol and may trigger a higher blood pressure for a person, as may potato chips or other foods known to be salty. Candy may include sugar which may cause an increase in a person's blood glucose levels. Spicy or acidic products may cause indigestion or acid reflux. A caffeinated product may increase a person's heart rate. When the consumption regulation device619determines the product or type of product that a person may be consuming, the device may determine corresponding characteristics of the product, and may determine data which may be associated with the effects of the characteristics. For example, if a characteristic of a sugary food or drink is to increase blood glucose levels, the consumption regulation device619may determine that blood glucose data may indicate the effects of consuming the sugary food or drink. When caffeine products are known to increase heartrate, the consumption regulation device619may determine that monitoring a user's heartrate may provide an indication of the effects of consuming caffeine.

In one or more embodiments, the consumption regulation device619(e.g., when implemented on the wearable device104ofFIG.1) may determine that another device or an application is responsible for detecting or otherwise collecting data associated with a characteristic of a consumable product. For example, a blood glucose monitor may measure blood glucose levels. A heartrate monitor may capture heartrate data. A hydration sensor may measure a user's dehydration. An accelerometer, magnetometer, wireless signals (e.g., Bluetooth or Wi-Fi signals), global navigation satellite system signals may be used (with a user's consent) to determine a device's motion or location, and the motion or location data may confirm if the user is at a location (e.g., a restaurant) or moving (e.g., motioning an arm or hand toward the face) in a manner which indicates a likely consumption of a product (e.g., and may be used to supplement audio data for the purpose of determining when a user is consuming a product). A hydrogen sensor may measure a user's indigestion. When the consumption regulation device619determines a characteristic of a consumable product and an associated type of data which may measure the effects of the characteristic on a person consuming the consumable product, the consumption regulation device619may identify another device or an application responsible for capturing the associated type of data, and may request the associated data. The request for the data may include specification of a sampling rate or frequency. For example, consumption regulation device619may request that another device provide data captured at a particular rate or frequency (e.g., a higher sampling rate or frequency than normal). Such may allow devices to conserve power and resources (e.g., by not sampling at higher rates or frequencies unless a user is consuming something).

In one or more embodiments, with a user's consent, the consumption regulation device619may help a user regulate their intake of consumable products and may provide recommendations for products, when to consume or not consume, locations where consumable products are available, nutritional information, warnings/alerts, alarms to medical professionals or other parties or devices, and the like. For example, when the consumption regulation device619detects that a user is eating food late at night (e.g., outside of a normal window of time associated with eating meals), the consumption regulation device619may present alarms or messages encouraging the user to eat something healthier or to wait until the next meal, or to indicate the effects that consuming a product may have on the person. The de consumption regulation device619vice may provide recommendations of healthier products to substitute, such as substituting fruit and vegetables for a less healthy product.

In one or more embodiments, the consumption regulation device619may be implemented at the one or more servers140ofFIG.1to receive data from the wearable device104or the user device106ofFIG.1. For example, the consumption regulation device619may receive captured audio data and determine that the user is consuming one or more products. The consumption regulation device619, implemented at the one or more servers140, the wearable device104, and/or the user device106ofFIG.1may determine characteristics and measurable attributes of a product. The consumption regulation device619may request additional data for the measurable attributes, including by specifying a frequency, sampling rate, time, and/or format of the data.

In one or more embodiments, the consumption regulation device619may be implemented at the one or more devices150ofFIG.1. For example, the consumption regulation device619may record data of a user such as biomedical data, exercise data, consumption data, product inventory (e.g., a smart refrigerator or freezer), and may send and receive data associated with consumption recommendations, exercise recommendations, etc.

It is understood that the above are only a subset of what the consumption regulation device619may be configured to perform and that other functions included throughout this disclosure may also be performed by the consumption regulation device619.