Patent Description:
The healthcare of an infant is the most important task for a parent. Advances in technology allow helping parents and caregivers with the task of tracking the development of the babies and quickly detecting certain anomalies. When it comes to the nutrition of the infant, it is important to detect whether there are anomalies in the digestive system, or whether the infant's body is absorbing all the nutrients it needs to.

For assessing the performance of the digestive system, analyzing the stool pattern is known to provide good insights. Scales to compare stools with a set of stool analysis scale scores help to classify the type of stool and to retrieve conclusions from there. Examples of such scales are the Bristol Stool Form Scale (BSS) and the Amsterdam Stool Scale. The BSS, which consists of seven images of different stool consistencies, allows assessment of stool consistency (scale <NUM> for hard lumps to scale <NUM> for watery stools), in an objective manner in adults. The BSS may also be used to characterize the stool of infants and young children.

When parents visit health care professionals (HCPs) because they are concerned about the wellbeing of their infants, HCPs usually ask questions about the consistency of the stool of the infants, and these questions are difficult to answer for most parents. When parents are asked to keep a log of the stool consistency of their infants, it is difficult for them to identify the stool consistency and the associated stool analysis scale score that fits a stool of their kids.

It would be desirable to have a system where parents and caregivers could keep track of the stool pattern, i.e. stool consistency, frequency and colour, of their babies in real time. It would further be desirable, that irrespective of which caregiver (parent, grandparent, nanny or day care) is changing the diaper, or helping the child to use a potty or toilet chair, the stool pattern is tracked in an objective and consistent manner. It would also be desirable to have a system that, based on the observed stool patterns, provides an indication, either that everything is normal, which would provide ease of mind to the parents and caregivers, or that the infant's stool pattern is not behaving as expected and that is advisable to visit an HCP.

Nowadays, with the habit of people carrying their smartphones, tablet computers or other portable devices with them continuously, there is an effort to provide mobile applications (apps) to be executed by these portable devices that can make normal life tasks easier for the users. Furthermore, the procedure of taking photos of the infant in order to see its development is a common practice for parents, by means of which they can approximately see how the infant is growing.

This can also be applied to keeping track of stool patterns. Programs or apps are known which allow to introduce or to capture images of stool and manually select a score of a stool analysis scale that better suits the stool on the image, in order to keep a log of the digestive system performance throughout time. Also programs or apps are known which allow the use of colour recognition techniques to automatically detect the colour of stool.

Although these apps may help parents and caregivers analyse some characteristics of the stool of their babies, it is sometimes difficult to manually make a classification, and it is tedious to manually keep a log, and thus there is a need for a method to more accurately analyse characteristics of stool of an infant, in a manner that is easy for the parents or caregivers, and that yet provides an accurate and fast classification.

Documents <CIT> and <NPL>", represent the closest prior art. <CIT> discloses using random forest algorithms to classify colors in stool and Vandenplas et Al disclose a study on the BITSS scale.

The invention provides a method and a system for analysing the consistency of stool according to the appended claims.

The invention further provides a computer program product for analysing the consistency of stool, comprising a computer readable medium comprising codes executable by at least one processor to execute a method according the invention.

<FIG> depicts an overview of an embodiment of the present invention. Although embodiments of the present invention comprise stool being placed in a diaper, a potty, a bed pan, a toilet chair, a toilet with plateaus, or other similar locations, in the embodiment of <FIG>, an example is depicted in which the stool is placed in a diaper. A diaper <NUM> comprising infant stool <NUM> is placed in an open position on a surface <NUM>, and a portable device <NUM> is used by a user, to capture an image of the open diaper <NUM> with the stool <NUM>. The captured image can be used by an app running on the portable device, and the app may input the captured image to a model comprising a CNN which will perform a series of operations in order to obtain a classification vector with probabilities of the possible scores in a stool analysis scale, and in order to obtain a predicted score based on the classification vector (i.e., the score with the highest probability). In order to obtain the best results from the classification process, it is desirable that the captured image has good characteristics, and therefore, that certain predetermined conditions are met. As an example of the conditions to be met, the stool <NUM> should be as recent as possible, so that its properties (colour, consistency) have not yet changed due to for instance that a part of the stool is absorbed by the diaper. The image should hence be captured shortly after the stool has been deposited, and in the embodiment of <FIG>, shortly after the diaper <NUM> has been filled with stool, and a suitable period to capture the image is up to ten minutes from the time the diaper has been filled. The colour of the stool in the captured image may be also used, by the CNN or by a different algorithm, to provide additional information. The app of the portable device may automatically detect the colour of the stool from the captured image, and analyse it in order to provide information. This allows to also provide information related to whether the colour is normal or not, which together with the consistency, can allow for a better determination of possible anomalies.

Another example of a condition which should be met is that the image is captured with sufficient light to clearly distinguish the features in the image, but not with too much light as this may modify the real colour and appearance of the image features. An example of how this can be achieved is by using natural light (daylight) or light from a ceiling lamp. Using the flash of the camera is however less preferred, as it could change the appearance of the image.

Another example of a condition is that the background of the image should preferably be regular, such as following a pattern, or uniform. If the stool is placed in a potty, a bed bad a toilet chair, or a toilet with plateaus, it is desirable that the background is uniform. If the stool is placed in a diaper, as represented in the embodiment of <FIG>, it is preferable that the diaper <NUM> is in the foreground of the image. A surface <NUM> of a table which is uniform is an example of a suitable background.

Another example of a condition that should be met is that no other object should be present in the image. If the stool is placed in a potty, a bed bad a toilet chair, or a toilet with plateaus, it is desirable that only the stool and the background are present in the image, no other objects or parts of a body. If the stool is placed on a diaper, it is desirable that no other object other than the diaper <NUM> containing the stool <NUM> is present. Parts of the baby's body, or other objects, should not be present in the image. If, however, the captured image has undesirable objects surrounding the stool, it may be possible to cut the captured image to eliminate the undesirable objects before the image is input to the CNN. Other pre-processing steps may be performed such as modifying the resolution of the captured image, changing the format of the image, or other steps to remove noise in the image.

It should be noted that embodiments of the invention may use only one of the predetermined conditions, or any combination of them. It should also be noted that other predetermined conditions can be used, as long as they help the user obtain an image which is suitable for being input to the CNN. Which predetermined conditions are to be used may be determined and modified in the settings of the app, or may be predefined, and a controller of the portable device may control which predetermined conditions are used, and how they are determined.

Other examples of conditions that should be met can be seen in <FIG>, which show different dispositions of a diaper comprising a stool according to the present invention. Again, the example of a stool in a diaper is represented, but the skilled person will understand that the features may similarly be applied to other embodiments in which the stool is provided in a potty, a bed bad, a toilet chair, a toilet with plateaus, or the like. In <FIG>, an open diaper <NUM> comprising stool <NUM> can be seen, with a uniform background <NUM>, which in this case corresponds to the top of a uniform-surface table. In <FIG>, no other objects other than the diaper are present in the image. An image like the one in <FIG> could be considered suitable for the method of the present invention.

According to an embodiment, the portable device <NUM>, before capturing the image, for example before pointing the camera to the stool, or when the user is pointing the camera to the stool, may give indications to the user to remind him/her of some conditions that should be met ( the light should be enough, and the like). These conditions may be some or all of the predetermined conditions described above, or different conditions. The portable device may additionally guide the user in order to bring the portable device <NUM> closer or further to the stool, in <FIG>, in the diaper <NUM>, to change the angle or the light source in the room in order to achieve better light in the image, and so on, and provide an indication of when the image could be suitable. This may allow the user to capture an image which has good characteristics and will improve the success rate of the classification. According to an embodiment, the portable device <NUM> may automatically capture the image when it detects that the portable device <NUM> is located at a suitable distance from the stool, or the diaper <NUM>, the light conditions are suitable, or when any other desirable condition is met. The condition or conditions to be met in order for the portable device to decide to automatically capture the image may be previously defined.

In <FIG>, the diaper <NUM> is not in a suitable position for capturing the image, as the diaper is not completely open and hence the stool <NUM> cannot be sufficiently identified. According to an embodiment of the invention, by seeing the guidance displayed in the portable device the user can be made aware that this is an unsuitable position and can open the diaper. According to another embodiment, the app will provide the guidance before pointing the camera to the stool, and the user will be aware that for example the diaper needs to be open, so that an image like the one represented in <FIG> is not captured.

<FIG> shows a flow chart illustrating a method according to the present invention. In step <NUM>, stool <NUM> is provided. As seen above, it can be provided in a diaper, which then will be open in order to capture the image, or in a potty, a bed bad, a toilet chair, or a toilet with plateaus. In step <NUM>, an image of the stool <NUM> is captured by a camera comprised in the portable device <NUM>. The capturing step may be performed by an app being run in the portable device <NUM>, which the user may initialize.

In step <NUM>, the captured image is provided to the input layer of a convolutional neural network (CNN). A CNN is a neural network suitable for classifying images, and unlike regular neural networks, the layers of a CNN have neurons arranged in three dimensions: width, height and depth. For the purposes of this disclosure, a CNN is understood to be a neural network, for example a feed-forward neural network, with multiple layers, comprising layers that transform an input 3D volume to an output 3D volume. In an embodiment, a CNN comprises an input and an output layer, with multiple hidden layers in between. Each of the hidden layers can be one of a convolutional layer, a pooling layer, a fully connected layer and a normalization layer. Such neural networks can for example be implemented using the TensorFlow library (Abadi et al "TensorFlow: A System for Large-Scale Machine Learning", 12th USENIX Symposium on Operating Systems Design and Implementation, <NUM>).

Throughout the layers of the CNN, the full input image, colour or grayscale, is reduced to a single vector of class scores, or classification vector. In the present invention, step <NUM> comprises processing the captured image to obtain a classification vector and a predicted score according to the stool analysis scale. The image captured with the portable device is the input image to the CNN, either in Red, Green, Blue colour or as a grayscale image. The output single vector is a classification vector comprising the probability values of each possible score. The predicted is score can be set to the score with the highest probability among the scores in the classification vector.

Several stool analysis scales can be used, for example the Bristol Stool Form Scale (BSS), the Amsterdam Stool Scale, or any other suitable scale, such as the Brussels Infant and Toddler Stool Scale (BITSS), currently under development (VandenPlas et al. , Development of the Brussels Infant and Toddler Stool Scale ('BITSS"): Protocol of the study', BMJ Open <NUM>; <NUM>:e014620.

If for example the BSS is used, the vector is a 1x1x7 vector comprising probabilities for each of the seven possible scores which form the scale.

The step <NUM> may be performed by a controller of the portable device <NUM>, or may be performed by a server in communication with the portable device <NUM>. In the second scenario, the controller of the portable device will instruct a transceiver of the portable device <NUM> to transmit the captured image to a server, and the processing of the captured image, that is, the computing of the CNN to provide a classification vector, and/or a predicted score, is performed by the server. This is suitable when the portable device lacks sufficient computational and graphical power to perform the operations required by the CNN. After the classification vector has been obtained by either the server or the portable device <NUM>, step <NUM> consists of storing information about the classification vector or about the score. This information may be stored in the portable device <NUM> or in the server, or in both. If information about the classification vector is stored, the portable device may then make a decision of the most likely score by selecting the score with the highest probability value among the scores in the classification vector. If the score is stored, the step of making a decision may be omitted. This score is then displayed by the portable device <NUM> executing the application. In another embodiment, the server may also perform the selection of the score based on the probability values of all the scores, and send the score to the portable device. The information about the predicted score comprises at least one of the score itself, the captured image, information about a date and/or time at which the image was captured, the classification vector obtained with the CNN, or some additional information which may be useful for the parents and caregivers and for the app.

<FIG> illustrates a diagram of the training of images according to the present invention. When the app for providing information about a stool is being executed in the portable device <NUM>, the captured image is input to a model comprising a CNN which will provide as a result a classification vector comprising probability values for the possible scores within a stool analysis scale, or a predicted score from said classification vector. In order to provide the classification vector, the CNN needs to be previously pre-trained, that is, it needs to be provided with enough information to be able to classify an input image. This may be performed with a set of labeled images (labeled with a score of a stool analysis scale), so that from the set of labeled images, the CNN can learn to automatically predict a specific label. Transfer learning may also be used in order to answer a specific image recognition task (like predicting a score of a BSS or BITSS from picture of a stool). This methodology allows building a model that will answer a specific question without collecting thousands of pictures and training a model from scratch. In this approach, a pre-trained model may be used and only the last layers are customized in order to predict the score. The CNN to be used may be based on known CNNs (for example based on Tensorflow libraries), but at least the last layer is customized, so that the output of the CNN provides a classification vector containing each of the scores of a stool analyisis scale. The training of the at least last layer of the CNN is performed as follows. In step <NUM>, an initial set of images is provided. Examples of a suitable set size are at least <NUM> images per score, at least <NUM> images in total (comprising the images for all the scores of the scale), more preferably at least <NUM>, and more preferably at least <NUM> images. These images are manually labeled with one score of the stool analysis scale to be used, such as the BSS. Among the initial set of images, a subset is selected in step <NUM> as test images, and a subset is selected in step <NUM> as training images. With the subset of training images, in step <NUM> the train model is implemented, and during the train model, the values given to certain elements in the operations being performed are updated by comparing the operation results with the labels that the images initially have. In that way, the CNN "learns" how to better predict the right score for an image. After that, the subset of test images is used to evaluate the model in step <NUM>. Once this iteration is done, step <NUM> may be optionally performed, wherein the size of the training set may be increased (and therefore the size of the test set is decreased) and the iteration process is repeated from step <NUM>. This fine tuning step <NUM> may increase the accuracy of the model. Once the CNN is trained, the values it uses for the elements in the different operations are known, and with them the CNN is able to predict the classification of an input image, such that an image of a stool in a diaper captured by a portable device. The trained model comprising the CNN may then be run in the portable device, or in the server, to provide a score for a captured image, as already illustrated above. According to embodiments of the invention, additionally to the fine tuning step <NUM>, the app may allow to collect additional images to be used for retraining the model, if the initial data set is not adequate enough to reach an acceptable accuracy.

During the training step, and in order to improve the accuracy and reliability of the classification, a step can be performed in which, for the manual labelling of the images, assessment is given by at least one, preferably two, parents or caregivers (for example mothers), wherein these parents or caregivers preferably have no relation with the infants from which the initial set of images was obtained. In a preferred embodiment, where the assessment of two parents or caregivers (or a combination) is used, disagreement in their assessment is solved by the assessment of a healthcare professional in order to obtain the final score, that is, label.

In an example according to an embodiment of the present invention, a total of <NUM> images is used as the initial set of images. Among these images, a subset of <NUM> images is selected as training images, and a subset of <NUM> images is selected as test images. The table <NUM> below shows an example of an evaluation performed with the <NUM> test images:
<IMG>.

As seen in table <NUM>, from the <NUM> test images, <NUM> images were identified as pertaining to score <NUM> by both a human and a machine learning classification. Similarly, <NUM> images were identified as pertaining to score <NUM> by both the human and the machine learning classification, and the like, as seen in the cells in the diagonal of table <NUM>. This represents a degree of agreement of <NUM>%. This level of agreement is even higher if an error of ±<NUM> score is considered, as shown in the cells adjacent to the diagonal elements in table <NUM>, from which it can be seen that the agreement reaches a <NUM>%.

According to embodiments of the present invention, the classification of the stool consistency that is provided by the app being run in the portable device or on a server follows the seven possible scores of the BSS.

In an embodiment according the invention, the portable device's operating system is iOS and/or Android. The app run in the portable device <NUM> which provides the classification of the stool may be as follows:.

The entries may be modified and deleted. The entries may also be shared with a health database (server) shared with the infant's healthcare provider (HCP) so that if there is an indication given for need for special care, the portable device can warn the user to contact the HCP, who will then have quick access to the relevant information to allow faster decision making on the next steps.

It may be possible for the HCP to compare the predicted score with the actual image and to amend the score if needed. In an embodiment, such a corrected sample is entered automatically into a set of further training data, so that the model can be retrained to improve the accuracy.

All the images captured with the app may be stored in the portable device and/or on a remote server with which the mobile device may communicate, which will allow HCP to have access to the data and to assess the data of their patients if granted access rights.

<FIG> schematically shows a portable device <NUM> according to an embodiment of the present invention. The portable device <NUM> has a display unit <NUM>, which may be touch screen suitable for displaying information and handling user input. The device <NUM> further has a camera <NUM> for recording images and video clips, a processor <NUM> for processing recorded images, a memory <NUM> for storing images, program data, the CNN, and the like, and a communication unit <NUM> for communication with other devices over wired or wireless connections. In an embodiment the processor <NUM> is programmed to process recorded images using the CNN and to generally implement the processes as described in this application.

<FIG> schematically shows a portable device <NUM> and a server <NUM> according to an embodiment of the present invention. The portable device <NUM> and server <NUM> can communicate over a wired or wireless link. In an embodiment, the portable device <NUM> sends recorded images to the server. The server has a processor, a memory, and a communication unit. The server <NUM> can be programmed to process received images using the CNN and to send back the results to the portable device <NUM>. In addition, the server may store the obtained results and/or the received images and/or any intermediate calculation results. The server may be further arranged to implement the training method described in reference to <FIG>.

In the foregoing description of the figures, the invention has been described with reference to specific embodiments thereof. It will, however, be evident that various modifications and changes may be made thereto without departing from the scope of the invention as summarized in the attached claims.

Claim 1:
A computer-implemented method of analysing the consistency of stool, comprising the steps of:
- providing stool of an infant,
- capturing, with a portable device comprising a camera, an image of the stool,
- providing the captured image to an input layer of a pre-trained convolutional neural network, CNN;
- processing, by the portable device, the captured image using the CNN to obtain, from a final layer of the CNN, a classification vector and to obtain information about a predicted score from the classification vector;
wherein the CNN has been obtained by transfer learning, by customizing at least the final layer of the CNN so that each element of the classification vector corresponds to a respective score of a stool consistency analysis scale, and
- storing information about the predicted score,
- wherein the image is a colour image, and wherein the method further comprises automatically detecting, by the portable device, the colour of the stool and analysing the colour to provide additional information.