Recommending activity sensor usage by image processing

Methods, systems, and computer program products for recommending activity sensor usage by image processing are provided herein. A computer-implemented method includes identifying, based on (i) sensor data from one or more sensors during a user activity and (ii) video data associated with the user performing the user activity, positioning of the one or more sensors with respect to the user; identifying the user activity being performed based on the video data; assessing data quality for the sensor data based on (i) the identified positioning of the one or more sensors and (ii) the identified user activity; and generating a recommendation for re-positioning at least one of the one or more sensors based on (i) the assessing and (ii) historical data pertaining to sensor data associated with the identified user activity.

FIELD

The present application generally relates to information technology, and, more particularly, to sensor management.

BACKGROUND

A growing number of individuals use activity sensors, for example, to obtain information pertaining to their daily physical activity and performance. Activity sensors commonly capture data from user movements, provided by components such as accelerometers, gyroscopes, etc. Additionally, models can be used to correlate the captured data to a predefined set of movements. However, challenges existing using current approaches due to a lack of training examples and improper positioning of such sensors, which limits the ability to capture meaningful information about the user movements.

SUMMARY

In one embodiment of the present invention, techniques for recommending activity sensor usage by image processing are provided. An exemplary computer-implemented method can include steps of identifying, based on (i) sensor data from one or more sensors during a user activity and (ii) video data associated with the user performing the user activity, positioning of the one or more sensors with respect to the user, wherein the one or more sensors are at least one of (a) positioned within a given proximity of a user and (b) worn by the user; identifying the user activity being performed based on the video data; assessing data quality for the sensor data based on (i) the identified positioning of the one or more sensors and (ii) the identified user activity; and generating a recommendation for re-positioning at least one of the one or more sensors based on (i) the assessing and (ii) historical data pertaining to sensor data associated with the identified user activity.

Another embodiment of the invention or elements thereof can be implemented in the form of a computer program product tangibly embodying computer readable instructions which, when implemented, cause a computer to carry out a plurality of method steps, as described herein. Furthermore, another embodiment of the invention or elements thereof can be implemented in the form of a system including a memory and at least one processor that is coupled to the memory and configured to perform noted method steps. Yet further, another embodiment of the invention or elements thereof can be implemented in the form of means for carrying out the method steps described herein, or elements thereof; the means can include hardware module(s) or a combination of hardware and software modules, wherein the software modules are stored in a tangible computer-readable storage medium (or multiple such media).

DETAILED DESCRIPTION

As described herein, an embodiment of the present invention includes recommending activity sensor usage by image processing. At least one embodiment of the invention includes utilizing sensor information and camera information during a physical activity to increase the accuracy in recognizing the movements being made/performed by the user (wearing the sensor(s)) during the activity, and generating a recommendation for the best use and/or placement of the sensor(s).

One or more embodiments, as further detailed herein, includes increasing sensor data capturing accuracy during a physical (user) activity by using video data obtained from an external camera. The video information is processed automatically using one or more computer vision algorithms capable of informing the system (server) what activity is being performed (by the user), and enabling such information to be added and/or incorporated into the machine learning algorithm of the sensors responsible for estimating motion-related information. The one or more computer vision algorithms are additionally capable of analyzing the execution of the movement of the user during the activity in question, to suggest one or more (improved) positioning of the sensor(s), so as to extract (additional) information during the data capturing phase of the sensor(s) execution. Also, in at least one embodiment of the invention, the one or more computer vision algorithms are capable of estimating, using the visual information derived from the camera, the movements being performed by the user, so as to increase the confidence of the sensor(s) when detecting the same movement in the future. Additionally, the sensors can be used as visual markers to assist in one or more video processing algorithms. In such an embodiment, the video information can enhance sensor analysis as well as computer vision algorithms.

As also detailed herein, at least one embodiment of the invention can include implementing computer vision techniques to improve and/or enhance identification of physical activities by allowing inclusion of new/additional recognized movements to a knowledge base, and suggesting specific positioning of sensors for improved recognition of activities.

FIG. 1is a flow diagram illustrating techniques for recommending activity sensor usage, according to an exemplary embodiment of the invention. In step102, the user places one or more sensors (for example, a smart watch) on his or her body and additionally positions a camera (for example, a smart phone) at an appropriate location for capturing video data of the user. In step104, once the sensor(s) and camera are in place, the user starts an activity (for example, exercise, functional training, etc.). In step106, the data captured includes the data generated by the sensor(s) and captured by the camera. The recorded video serves as input to one or more computer vision algorithms. Algorithms for motion estimation can include, but are not limited to, optical flow, template matching, and feature matching via random sample consensus (RANSAC). Algorithms for action recognition can include, but are not limited to, support vector machine- (SVM-) based classification of hard-coded features (such as histogram of oriented gradient (HOG) and histogram of optical flow (HOF)) and deep learning using convolutional neural networks (CNNs) and long short-term memories (LSTMs). In step108, the camera and the sensor(s) send captured data to a client module that, in turn, sends the data to a server module120(that is, a back-end of the system). The server module120includes an activity log database that contains the data sent by the client modules, which includes sensor(s) data and video recorded. Additionally, in step110, the system uses one or more computer vision algorithms to identify the one or more sensors in use. The visual appearance of the sensor itself serves as a point of reference; if the visual appearance is not discriminant enough, a fiducial marker may be added to the sensor.

In step112, the system uses one or more computer vision algorithms to identify the one or more regions of the user's body that are equipped with a sensor, using the visual input from the sensor itself. Additionally, steps110and112can be performed on a client module itself in cases wherein there is computation power available at the client module. In step114, the system classifies the current activity (of the user) based on information from an activity knowledge base122used to train one or more classification algorithms, and identifies the activity being performed (for example, running, practicing yoga, sleeping, etc.). The camera can use the sensor visual movements to assist in the identification of the activity.

In step116, the identification output is sent to the client module, which notifies the user. If the sensor is located in a body region that is not optimal for measuring the activity being practiced, the user is informed, via the client module, and can re-position the sensor to a notified position. Additionally, a best practice can be suggested that involves ways to improve the sensor's data quality, aiming at richer analytics (for example, when practicing yoga, the important information to be taken into account can include the orientation of the gyroscopes, for positions in which a person stands still). Further, in step118, the system updates the activity knowledge base122. If the tuple (sensor(s), activity, and body region(s)) pertaining to the configuration the user was using prior to the recommendation is not in the database122, the system creates a new class for the tuple. For existing tuples, the knowledge base122is enriched with the new data.

The activity knowledge base122includes a database containing all video image and sensor data used by the machine learning algorithms. Additionally, the activity knowledge base122can be created, for example, by applying computer vision algorithms to public videos of professionals teaching how to perform exercises, which might also contain textual information describing the activities or even sensors used in the videos. Such textual information can be used to label the activities, regions, and/or sensors used in the activities. The activity knowledge base122can also include information extracted from literature (for example, textbooks, research papers, etc.).

FIG. 2is a diagram illustrating system architecture, according to an embodiment of the invention. By way of illustration,FIG. 2depicts a camera202, one or more sensors204, a client module206, a server module208, and a computer vision module210. The client module is responsible for communicating with the user (for example, informing user regarding the re-positioning of the one or more sensors204). The server module208is responsible for hosting the computer vision module210, which is a software component capable of running computer vision algorithms (that may, for example, have high computation cost at the server) for performing one or more steps detailed inFIG. 1(such as, for example, step114).

As described herein, one or more embodiments of the invention include techniques for increasing user-worn sensor204accuracy during a physical user activity by processing video input (from external camera202) via the computer vision module210. In at least one embodiment of the invention, the computer vision module210can determine and inform the server module208the activity being performed by the user. Additionally, the computer vision module210can analyze the execution of the user movement(s) to suggest a beneficial positioning of the sensor(s)204. Further, the computer vision module210can estimate, using visual information derived from the camera202, the one or more movements being performed by the user, so as to increase the confidence of the sensor204when detecting the same movement.

By way merely of illustrating one or more embodiments of the invention, consider the following use cases. In a first example use case involving home exercises, at least one embodiment of the invention can include allowing the activity sensor to discover successful characteristics of “new” movements to help identify such movements, and personalize the sensor activity to the user's needs and routine. In a second example use case involving physiotherapy, using enriched data obtained by combining sensor data and camera data, at least one embodiment of the invention can include accurately identifying the exercise being performed, providing one or more recommendations of how to optimize the set of sensors in use, given that physiotherapy movements require precise movements, and allowing such specialized support to scale. The recommendations offered by one or more embodiments of the invention can consider an activity knowledge base (such as component122inFIG. 1), trained with examples of correct movements (often supervised by specialists). Hence, such capabilities make it possible to scale-up recommendations involving activities requiring precise movements.

FIG. 3is a flow diagram illustrating techniques according to an embodiment of the present invention. Step302includes identifying, based on (i) sensor data from one or more sensors during a user activity and (ii) video data associated with the user performing the user activity, positioning of the one or more sensors with respect to the user, wherein the one or more sensors are at least one of (a) positioned within a given proximity of a user and (b) worn by the user. Identifying positioning of the one or more sensors can include detecting a shape associated with the one or more sensors, a color associated with the one or more sensors, and/or a texture associated with the one or more sensors. Additionally, identifying positioning of the one or more sensors can include implementing a fiducial marker in connection with the one or more sensors.

Step304includes identifying the user activity being performed based on the video data. Identifying the user activity can include utilizing one or more human body models, statistically decomposing a human body into a hierarchical structure, and/or computing displacement information from an optical flow field. Additionally, identifying the user activity can include implementing an image classifier, generated based on training data, to determine one or more classes of user activities relevant to the sensor data and the video data. One or more embodiments of the invention can include using classifiers such as support vector machines (SVMs), random trees, (deep) convolutional neural networks, etc., and/or we face recognition-based approaches using local binary patterns (LBP), Haar cascades, etc.

Step306includes assessing data quality for the sensor data based on (i) the identified positioning of the one or more sensors and (ii) the identified user activity. Assessing can include determining a noise-to-signal ratio for the sensor data, and/or utilizing a look-up a table containing pre-defined tuples associated with user activities.

Step308includes generating a recommendation for re-positioning at least one of the one or more sensors based on (i) said assessing and (ii) historical data pertaining to sensor data associated with the identified user activity. The recommendation can include combining two or more of the sensors, and/or a recommendation of one or more additional user activities.

The techniques depicted inFIG. 3can also include outputting, to the user, a notification containing the recommendation for re-positioning at least one of the one or more sensors. Further, the techniques depicted inFIG. 3can also include updating, based on the recommendation, one or more machine learning algorithms of the one or more sensors, wherein the one or more machine learning algorithms are responsible for estimating motion-related information. Additionally, at least one embodiment of the invention can include storing the sensor data, the video data, the identified positioning of the one or more sensors, the identified user activity, and the recommendation in a knowledge base. Further, one or more embodiments of the invention can include determining positioning of at least one camera associated with capturing video data, wherein determining the positioning of at least one camera can include inferring a camera view with respect to the user, and utilizing a look-up table containing pre-defined tuples associated with camera views.

A data processing system suitable for storing and/or executing program code will include at least one processor402coupled directly or indirectly to memory elements404through a system bus410. The memory elements can include local memory employed during actual implementation of the program code, bulk storage, and cache memories which provide temporary storage of at least some program code in order to reduce the number of times code must be retrieved from bulk storage during implementation.

Input/output or I/O devices (including, but not limited to, keyboards408, displays406, pointing devices, and the like) can be coupled to the system either directly (such as via bus410) or through intervening I/O controllers (omitted for clarity).

Additionally, it is understood in advance that implementation of the teachings recited herein are not limited to a particular computing environment. Rather, embodiments of the present invention are capable of being implemented in conjunction with any type of computing environment now known or later developed.

Characteristics are as follows:

Service Models are as follows:

Deployment Models are as follows:

Virtualization layer70provides an abstraction layer from which the following examples of virtual entities may be provided: virtual servers71; virtual storage72; virtual networks73, including virtual private networks; virtual applications and operating systems74; and virtual clients75. In one example, management layer80may provide the functions described below. Resource provisioning81provides dynamic procurement of computing resources and other resources that are utilized to perform tasks within the cloud computing environment. Metering and Pricing82provide cost tracking as resources are utilized within the cloud computing environment, and billing or invoicing for consumption of these resources.

In one example, these resources may include application software licenses. Security provides identity verification for cloud consumers and tasks, as well as protection for data and other resources. User portal83provides access to the cloud computing environment for consumers and system administrators. Service level management84provides cloud computing resource allocation and management such that required service levels are met. Service Level Agreement (SLA) planning and fulfillment85provide pre-arrangement for, and procurement of, cloud computing resources for which a future requirement is anticipated in accordance with an SLA.

Workloads layer90provides examples of functionality for which the cloud computing environment may be utilized. Examples of workloads and functions which may be provided from this layer include: mapping and navigation91; software development and lifecycle management92; virtual classroom education delivery93; data analytics processing94; transaction processing95; and activity sensor recommendation generation96, in accordance with the one or more embodiments of the present invention.

At least one embodiment of the present invention may provide a beneficial effect such as, for example, using image data to enrich sensor data, and recommending particular sensor(s) usage for a detected activity.