Patent Description:
It is typical for skin treatment devices to make use of software applications (apps) which can be downloaded on user electronic devices such as smartphones or tablets. These electronic devices are coupled to the skin treatment device, by means of connections such as Wireless Fidelity (WiFi) or Bluetooth. The apps can be used for controlling an operation of the skin treatment device, displaying messages to the user and the like.

For control actions relating to skin treatment, the app may prompt the user to input a dimension such as length of a body part via a suitable user interface (e.g. touchscreen) of the electronic device. Conventionally, the user manually measures the length of the body part to be treated by the skin treatment device, e. using a standard measuring tape. The input length is used by the app to calculate treatment parameters which are customized to the user. For example, in a skin treatment such as an Intense Pulsed Light (IPL) device, the treatment parameters include a number of light flashes, the intensity of light, etc. required to treat the body part.

<NPL>, discloses an automatic skin ulcer region assessment framework to accurately segment a wound in a patient's skin ulcer image and automatically estimate its area and size. The assessment framework uses an encoder/decoder deep neural network to perform segmentation of the wound in the image. The assessment framework also detects a measurements ruler/tape present in the image and automatically estimates the pixel density of the image. This allows an accurate measurement of the size of the wound.

<NPL> discloses an automatic method for determining anthropometric dimensions of the human body using two-dimensional images. The method uses an efficient algorithm for automatically determining the anthropometric dimensions using fiducial points that are detected from frontal and lateral views of body silhouettes. The accuracy of the detected fiducial points is improved by segmenting the human body silhouette in both views. A large number of primary anthropometric dimensions are obtained by calculating the difference between two relevant fiducial points. A large number of secondary dimensional ratios are obtained directly from the primary dimensions, and circumferential dimensions are estimated using an ellipoid model.

It is an object of the present invention to provide an improved solution which can automate measurement of a dimension of the body part, making it more convenient to the user.

The user is also prone to making errors in measurement, resulting in an inaccurate calculation of treatment parameters. It is another object of the invention to provide a solution which can improve the accuracy of the measured body part.

In the field of computer vision, human body key point detection relates to detecting human body key points or fiducial points in an image. However, the method requires complex image processing routines, thus making it slow for real time applications.

It is yet another object of the present invention to provide an automated solution addressing the challenges associated with image processing methods such as key point detection. This is achieved by use of a hybrid method based on neural network(s) and key point detection.

Some neural networks are deep neural networks which include one or more hidden layers in addition to an output layer.

It is yet another object of the present invention to provide a technique for training neural networks, which networks can be used to output a measurement of a dimension of a body part in an input image on an electronic device.

For example, the specification provides methods, systems, and apparatus, including computer programs encoded on computer storage media, for carrying out any of the techniques disclosed herein. In some embodiments, a computer program comprises instructions that when executed by computing apparatus causes it to perform any of the techniques disclosed herein.

In one aspect, a method performed by one or more computers and a system thereof for outputting a measurement of a dimension of a body part in an input image is provided. The method includes receiving the input image captured using an image capturing unit, wherein the input image includes the body part and a reference object of a predefined dimension, predicting by a segmentation deep neural network, S-DNN, a segmentation mask of the input image, the segmentation mask differentiating image pixels of the body part from image pixels of a background region in the input image, detecting the reference object in the input image and processing the detected reference object to predict a pixel dimension of the reference object by an object detection deep neural network, OD-DNN, obtaining pixel coordinates of a plurality of key points of the body part using the segmentation mask, modifying the obtained pixel coordinates based on the predicted pixel dimension of the reference object, and measuring the dimension of the body part based on the modified pixel coordinates and outputting the measured dimension of the body part.

The method and system provide an improved measurement solution which can automate measurement of a dimension of the body part of the user, making a device incorporating the solution more user-friendly. As a result of the features used therein, the solution also improves the accuracy of the measured body part. The method further reduces the complexity of image processing routines, and makes them more suited to real time applications, and therefore can be implemented on devices with relatively less processing capability.

In one aspect, a method of training a plurality of neural networks and a system thereof used to control an electronic device to output a measurement of a dimension of a body part in an input image is provided. The input image is captured by an image capturing unit of the electronic device. The plurality of neural networks includes an object detection deep neural network, OD-DNN, and a segmentation deep neural network, S-DNN. The method includes the steps of initializing neural network parameter values of the S-DNN and OD-DNN, obtaining a training data set including a plurality of images, each image including the body part and a reference object of a predefined dimension, annotating the reference object and the body part in the images of the training data set to generate an annotated training data set, dividing a part of the annotated training data set into a validation data set, inputting the annotated training data set to the OD-DNN such that the OD-DNN learns to detect the reference object in the input image and proces the detected reference object to predict a pixel dimension of the reference object, inputting the annotated training data set to the S-DNN such that the S-DNN learns to predict a segmentation mask of the input image, wherein the segmentation mask differentiates image pixels of the body part from image pixels of a background region in the input image, validating, using the validation data set, a pixel dimension of the reference object learnt by the OD-DNN and a segmentation mask learnt by the S-DNN, and updating the neural network parameter values of the S-DNN and OD-DNN based on the validation.

The method and system provide a training solution for real world applications including controlling an electronic device to output a measurement of a body part. The output may be displayed to a user of the electronic device. In some implementations, the training solution continuously interacts with measurement input which is captured by one or more physical image capturing units.

In one aspect, one or more computer program products which include instructions which, when the program is executed by one or more computers, cause the one or more computers to carry out steps of the above-mentioned methods, are provided.

As mentioned, other features, aspects, and advantages of the subject matter will become apparent from the description, the figures, and the claims.

<FIG> is a flow chart of an example method <NUM> performed by one or more computers in a system for outputting a measurement of a dimension of a body part in an input image. An example of a dimension is length, but the present invention is not limited to measuring length. Other dimensions include but are not limited to breadth or height of the body part. For convenience, the method <NUM> will be described as being performed by a system of one or more computers located in one or more locations. For example, an image processing system with distributed architecture, appropriately programmed in accordance with this specification, can perform the method <NUM>. Method <NUM> may also be performed by a single computer, for example, by a processor of a user electronic device/computer. The order of the steps of method <NUM> is merely exemplary, for example, the skilled person is aware that it is possible to interchange orders of at least some of the steps of the method. The term "user" is any user who uses the method described herein and/or the system and/or a computer program based on the method and need not be the same person as the subject on which the treatment is carried out. The subject may be a human or an animal.

In step <NUM>, method <NUM> comprises receiving an input image. The input image comprises the body part which is intended to be treated, for example, a leg, an arm, or any other body part. It may be a two-dimensional (2D) image (e.g., represented as a 2D array of pixels) and may be acquired by an image scanner of any modality, for example, an image capturing unit of a user electronic device (smartphone camera).

The input image further comprises a reference object of a predefined dimension. It is difficult to obtain an accurate length of a region of interest (ROI) or desired object in a captured image. This is because, based on factors such as selected optical/digital zoom of the image capturing unit, image resolution, the angle of image capture, etc., the measurements of the ROI in an image may deviate from its actual values. Since all objects of an image share similar characteristics, such deviations may be compensated by referencing the ROI to another object in the same image, the other object having a known length.

Skin treatment devices are often equipped with at least one attachment with a standard length. In a particular example, a treatment aperture of this attachment can be used as the reference object. An example of a reference object is shown in <FIG>. In an IPL device (e.g. Philips Lumea), this could be the treatment aperture of a device attachment which is supplied with the IPL device. It has a known dimension of around <NUM>, specifically, <NUM>. It is understood that other dimensions are equally possible for the treatment aperture. In an example, the user is prompted by the app to capture an input image of the body part keeping the device attachment in the field of view of the camera.

In step <NUM>, method <NUM> comprises predicting at least one segmentation mask of the input image. A segmentation deep neural network (S-DNN) is used to process the input image. In an example, a ResNet model is used. The S-DNN is configured to receive the input image and process the input image in accordance with current (optimised) values of S-DNN parameters to predict the segmentation mask. Step <NUM> may be performed at a single location by a single computer (for example, at the user electronic device) or at a different location by another computer or processor. In the latter example, the S-DNN may be located in a distributed server such as a cloud. Hence, the computing is performed at the server location. The user electronic device can be configured to transmit the acquired input image to the server by means of wireless communication. In this example, the user electronic device and the server form a plurality of computers in a wireless communication system. Since the image processing is carried out apart from the user (client) electronic device, less resources are needed by the device at the user location.

The S-DNN is a semantic classifier, which classifies pixels in the input image into a plurality of classes or types. In an example, a predicted segmentation mask comprises the classes ROI and background. The image pixels of the ROI in the segmentation mask relate to a first class and the image pixels of the background region in the segmentation mask relate to a second class. The second class relating to the background comprises those image pixels outside a boundary of the ROI or the body part, and can optionally be filtered from the segmentation mask (shown darkened in <FIG>). In another example, a plurality of segmentation masks are generated. A first segmentation mask relates to the first class, and a second segmentation mask relates to the second class. In other words, the segmentation mask differentiates image pixels of the ROI, the body part, from image pixels of a background region in the user image. An example of a segmentation mask showing the first/ROI class is shown in <FIG>. The remaining steps of processing the image are carried on this predicted segmentation mask - a subset of the input image, making the process faster than while processing raw input image data.

In some implementations, the segmentation mask has a score, which represents a likelihood that the predicted mask is the most appropriate classification for the user. This score, amongst other parameters, may be used by the S-DNN for future predictions, i.e., for computing the segmentation mask of another input image of the same or similar body part at a later point of time.

As mentioned, the input image comprises the reference object of a predefined dimension/length/size, in order to obtain an accurate measurement of the ROI in the input image. In step <NUM>, method <NUM> detects the reference object in the input image and processes the reference object in the input image to predict an image pixel dimension, e.g., a (total) pixel length of the reference object, in other words, the length of a row of pixels (number of pixels) spanning the length of the reference object in the input image. The prediction acts as a correction factor and is applied to pixel coordinates of the ROI (body part) determined via keypoint detection. This is detailed below.

An object detection deep neural network (OD-DNN) is used to detect and process the reference object in the input image. In an example, a You only look once (YOLO) object detection convolutional neural network (CNN) model is used. The OD-DNN is configured to receive and process the input image in accordance with current (optimised) values of OD-DNN parameters to predict the pixel length of the reference object. The OD-DNN may further be configured to process the input image in accordance with current (optimised) values of OD-DNN parameters to detect the ROI. Like step <NUM>, step <NUM> may be performed at a single location by a single computer (for example, at the user electronic device) or at a different location by a distributed computer. In the latter example, the OD-DNN may be located in a distributed server such as a cloud. Hence, the processing or computing is performed at the server location. Similar to step <NUM>, the user electronic device can be configured to transmit the acquired input image to the server by means of wireless communication, the electronic device and the server forming a wireless communication system. Since the image processing is carried out apart from the user (client) electronic device, less resources are needed by the device at the user location.

In some implementations, the OD-DNN is trained to output pixel coordinates of a bounding box which outlines the reference object. A bounding box shows the spatial location of an object and is presented by a set of coordinates that tends to bound the object by the smallest enclosing box. An example of a bounding box is shown in <FIG>. While training the OD-DNN, the bounding box technique can be used to annotate or label the reference object in each image of a training image data set, so that the OD-DNN learns to obtain a pixel length from the predicted pixel coordinates of the bounding box in the input image with a degree of accuracy.

Like in step <NUM>, the predicted pixel length of the reference object has a score, which represents a likelihood that the prediction is the most appropriate (linear) regression. This score, amongst other parameters, may be used by the OD-DNN for future predictions.

In some implementations, the OD-DNN calculates a single/per pixel length, denoted as a measurement per pixel (MPP), based on the predefined dimension and the predicted pixel dimension of the reference object. In the field of computer vision, MPP is understood to quantify the number of inches or centimeters of the environment shown in the iamge encoded or represented per pixel length or per pixel breadth of the image. An example is shown in <FIG>. The task performed by the OD-DNN is then to determine how much real-world spatial length is encoded per pixel length or per pixel breadth of the image.

In the above equation, the predefined dimension of the reference object is denoted by the term ground truth length, and the predicted pixel dimension is denoted as the pixel length of the object. This metric acts as a correction factor and is applied to pixel coordinates of the ROI (body part) determined via keypoint detection. Some other advantages of measuring (reference) objects in an image with the MPP are that the metric is independent of the distance between an image capturing unit (camera) and the object, that it can be employed for any object of known dimensions in the image, is independent of the pixel density of the image and of camera calibration parameters such as skew coefficient and optical center.

In step <NUM>, method <NUM> obtains pixel coordinates of a plurality of key points of the body part from the segmentation mask obtained in step <NUM>. This is shown in <FIG>. The key point detection algorithm used in the invention is especially chosen to improve the resource usage of the method and/or the user electronic device, in other words, its internal functioning. Built on a non-AI based model, it has the advantage that it does not require prior training to perform a task. In contrast, an AI model typically requires large amounts of training data to learn the underlying patterns or features in the data. As such, it is suited to be implemented on the user electronic device. As mentioned, the algorithm processes a sub-set of the image data, making it faster to implement on a user device. In some embodiments, step <NUM> is performed by a remote/distributed computer in the system.

In an example, the key points of body part "lower leg" are chosen as the ankle and knee joints. An example of how these key points are detected is as follows. In <FIG>, the segmentation mask shows details of an image of the lower leg at a certain orientation. The height of the mask is h pixels, and the width is w pixels. A set of coordinate axes x (horizontal) and y (vertical) is defined with respect to the segmentation mask, with an arbitrary origin defined as (<NUM>,<NUM>). In <FIG>, the origin is located at the top-left corner. It is preferred that the lower leg is aligned at a non-zero angle α with respect to the horizontal axis of the mask. The recommended lower leg orientation allows to determine points of the desired key points using elementary calculus in a straightforward manner. The ankle and knee joints are identified as follows. Given the orientation of the leg, the leg outline pixel on the mask associated with a y-local minimum is assigned as the knee joint K. This is as shown in the right half portion of <FIG>. In other words, the pixel with the minimum y-coordinate is treated as the knee joint coordinate. To determine the ankle joint A, the method determines a set of four points T, H, Q and J in the lower leg, and the ankle joint co-ordinate A is determined by the point of intersection between the line segments TJ and HQ. A second local minimum y-coordinate point in the mask is determined as a toe pixel T. A local maximum y-coordinate point is determined as a heel pixel H. A local maximum point near an upper intersection of the foot and leg in the mask is selected as point Q, and point J is empirically defined as x(H) + <NUM> units where x(H) is the x-coordinate of the heel H.

An exemplary implementation to calculate the local minima/maxima is shown in <FIG>. A vertical slider of pixel width <NUM> can be scanned across the segmentation mask in a given direction. In <FIG>, for identifying the knee joint K, the vertical slider scans the mask in a given direction D, for example, left to right. The set of coordinate axes is defined, as mentioned above. The vertical slider identifies all ROI pixels on the mask outline (boundary) along the scanning direction and checks for the height of the pixel along said outline. The pixel with the lowest height with respect to the assigned y=<NUM> is chosen as the knee joint pixel y coordinate. The corresponding x coordinate is also identified. The implementation can be repeated for detection of each key point. It is clear to the skilled person that any identification means such as a cursor can be used instead of the vertical slider.

In some embodiments, the method calculates the distance between the obtained key points.

In step <NUM>, method <NUM> modifies the obtained pixel coordinates of the key points or the distance between the key points based on the predicted pixel dimension of the reference object. In some implementations, the method corrects the obtained coordinates using the pixel size derived from the measurement per pixel calculation in accordance with the equation in paragraph [<NUM>], hence, actual lower leg length = MPP * distance between the key points (pixel length of the ROI).

Such correction, based on reference to an object in the input image with known dimensions, accounts for ROI size variations due to factors such as selected optical/digital zoom of the image capturing unit, image resolution, the angle of image capture, etc..

In step <NUM>, method <NUM> outputs the measurement of the length of the body part based on the modified pixel coordinates. The measurement may be displayed by the user electronic device to the user by means of its display screen. The measured value is further transmitted to the skin treatment device for calculation of treatment parameters by the user electronic device. In some embodiments, the measured value can be directly transmitted to the skin treatment device, i.e., without further display to the user.

The method may use at least one of the input image, segmentation mask, the predicted pixel dimension, and the output measurement of the dimension of the body part for future predictions of the segmentation mask, the pixel dimension, and the output measurement of the dimension of the body part.

<FIG> illustrates an example image processing system <NUM> according to an exemplary embodiment of the present invention. It is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described herein are implemented.

The image processing system <NUM> is configured to receive and process an input image <NUM> depicting or including a body part to be treated. The body part is considered the ROI <NUM> in the input image. Input image <NUM> is acquired using an image capturing unit <NUM> which may form part of system <NUM>, or be external to it. As mentioned, the image capturing unit <NUM> may be part of a user electronic device, and the body part may be that of the user or another subject.

The system <NUM> processes the input image <NUM> to predict a segmentation mask of the input image. In doing so, it provides the input image as an input to an S-DNN <NUM> (as mentioned above). The S-DNN is configured to receive the input image <NUM> and process it in accordance with current values of S-DNN parameters to generate or predict a segmentation mask <NUM> (based on a hypothesis that approximates a function between an input to the DNN and an expected output). The segmentation mask characterizes a segmentation of the input image into at least one class, the class comprising image pixels of the ROI, the body part.

The system <NUM> may store the generated segmentation mask in a data store (e.g., a logical data storage area or a physical data storage device, not shown) wirelessly coupled to the system <NUM>. A user of the system <NUM> (e.g., a user operating a device at home) may subsequently retrieve the segmentation mask from the data store. In some cases, the system <NUM> may directly present the generated segmentation mask on a display device (e.g., a computer screen) visible to a user of the system <NUM>, e.g., using the display of the user electronic device.

The S-DNN is trained with differently initialized parameter values, on different training images, or both. An example process for training the S-DNN is described with reference to <FIG>. In some implementations, the S-DNN is pre-trained to predict segmentation masks of arbitrary images. The pre-trained models are further trained using customized training data images. The advantage of using pre-trained models are that the calculations converge faster, i.e., the optimized neural network parameters are obtained faster. In an example, the S-DNN is a semantic classifier. Some model architectures which can perform semantic segmentation include fully convolutional network (FCN), U-Net, Mask RCNN, etc. In an example, the S-DNN may be implemented using DeepLabv3 PyTorch model with an FCN-ResNet101 backbone.

To generate the output measurement <NUM> of the body part, the system <NUM> further provides the segmentation mask <NUM> to the key point detector <NUM>. The algorithm or model of the key point detector <NUM> may be embedded in a user device which is comprised internal or external to system <NUM>, or at a server location together with S-DNN and/or other neural networks (the latter shown in <FIG>). The implementation of the key point algorithm is as mentioned above. The key point detection algorithm can highly improve latency of predictions since the algorithm does not rely on resource-intensive deep neural network architectures for identifying the image pixels corresponding to the key points in the ROI. The output measurement of the body part comprises a dimension of the body part determined by locating key points among the ROI/body part image pixels in the segmentation mask. Optionally, system <NUM> outputs the measurement to a user device.

In some implementations, the system <NUM> processes the input image <NUM> further to modify the obtained dimension. In this case, input image <NUM> additionally includes a reference object <NUM>. The system <NUM> provides the input image <NUM> as an input to an OD-DNN <NUM> (specifics of OD-DNN as mentioned above). The OD-DNN processes the reference object <NUM> to predict a pixel dimension of the reference object <NUM>. In some implementations, the system <NUM> further calculates a pixel size based on the predicted pixel dimension of the reference object and a known dimension of the reference object <NUM>. System <NUM> corrects the obtained dimension of the body part <NUM> based on the predicted pixel dimension of the reference object or the derived pixel size/metric MPP. The key point detector <NUM> may apply this correction to the obtained pixel coordinates of the key points or the dimension of the body part calculated based on these pixel coordinates in its output <NUM>.

The OD-DNN is trained with differently initialized parameter values, on different training images, or both. An example process for training the OD-DNN is described with reference to <FIG>. In some implementations, the OD-DNN is pre-trained to predict pixel dimensions of arbitrary images. The pre-trained OD-DNN is further trained using customized training data images. In an example, the OD-DNN may be implemented using a Yolo AI model. Depending on the extent of accuracy desired in prediction of dimension of the body part, a heavyweight or lightweight (terms well-known in art) YOLO model may be used. Other object detection neural network architectures include fast R-CNN, single shot detector (SSD) etc..

Both S-DNN and OD-DNN may be trained using separate supervised learning procedures. The neural networks may also be trained by the same supervised training procedure, using the same training data set. The procedures may involve using manually labelled training images. An example of the latter approach is illustrated in <FIG>.

<FIG> is a flow chart of an exemplary method <NUM> for training an S-DNN and an OD-DNN according to an embodiment of the invention. The method <NUM> will be described as being performed by a system of one or more computers located in one or more locations. While it is possible to implement the method <NUM> using the system <NUM>, it is advantageous to train the neural networks using a separate procedure, prior to implementing them in the method of FIG. The trained OD-DNN and S-DNN can be used to predict the segmentation mask and the pixel dimension of the body part in the input image in method <NUM>. As mentioned, the input image may be captured by an image capturing unit of the electronic device. The electronic device may be one of the one or more computers in the system. Hence, the training procedure of <FIG> can be used to control an electronic device to obtain and/or output a measurement of a dimension of a body part in an input image.

In step <NUM>, the method comprises initializing the neural network parameter values of the S-DNN and OD-DNN. In some implementations, the method may initialize the S-DNN and OD-DNN neural network parameter values based on trained parameter values of another neural network with the same architecture as the S-DNN and OD-DNN, respectively, and which have been trained to perform similar predictions. For example, as mentioned above, S-DNN and OD-DNN may build on pre-trained models, i. , the OD-DNN and S-DNN are prior trained to detect objects in an arbitrary image and semantically segment an arbitrary image into at least one segment mask of a specific image class, respectively. In this case the initialization can be based on trained parameter values of the pre-trained S-DNN and OD-DNN.

In some implementations, the score of the predictions in method <NUM> by system <NUM> is considered at step <NUM> during initialization of the parameter values, so that the training of the neural networks can be adjusted every time method <NUM> is performed. As a result, over time, the system <NUM> learns to obtain and/or output a measurement of a dimension of a body part in an input image to a user of an electronic device with improved accuracy.

In step <NUM>, method <NUM> obtains a training examples or a training data set comprising a plurality of images, each image including the body part and a reference object of a predefined dimension. Each image in the training data set is further annotated by a human expert, in an automated or semi-automated manner. Annotation refers to a process by which a person, manually or using a computer, assigns a type to an object in the image for purposes of training a neural network.

In some implementations, method <NUM> obtains the training data set periodically from one or more computers in the system. For example, each time method <NUM> is performed by system <NUM>, the input image may be stored in the system for later training purposes. In this manner, the S-DNN and OD-DNN can be iteratively trained.

In step <NUM>, the method comprises annotating the reference object and the body part in the obtained training data set, to generate an annotated training data set.

In an example, in each image of the training data set, the image pixels of a particular class or particular classes in the training data set are annotated to generate an expected segmentation mask output. The annotated training data set is input to the S-DNN. Each example in the training data set input to the S-DNN has (i) a training image (input) and (ii) an annotated training segmentation mask showing image pixels of the body part class (expected output) in the training image. The S-DNN learns to map each input to each output using a possible hypothesis function.

In a training data set to the OD-DNN, the method comprises annotating the reference object in each image using annotation techniques such as by using the bounding box. As shown in <FIG>, the treatment aperture of a device attachment can be used as the reference object, therefore, the aperture can be annotated with a bounding box ABCD in each image of the training data set. Each example in the training data set to the OD-DNN has (i) a training image (input) and (ii) an annotated reference object (expected output) in the training image.

It is advantageous to modify the training data set to increase or augment the training set size, and hence the accuracy of model predictions. In some implementations, the method comprises generating different versions of each image in the training data set, using image augmentation techniques such as image translation and image flipping. Image augmentation also addresses the issue of overfitting of S-DNN and OD-DNN. With image augmentation, it is possible to obtain heterogeneous data that helps the models learn different patterns, hence avoiding overfitting.

In step <NUM>, the method comprises dividing or splitting a part of the annotated training data set into a validation data set. A part of the annotated training data set is chosen as the validation data set. It is thus ensured that the training and validation sets are obtained using the same distribution of data. This subset is used to evaluate the performance of the S-DNN and OD-DNN in their predictions.

In step <NUM>, the method comprises inputting the annotated training data set to the OD-DNN and the S-DNN such that the OD-DNN and the S-DNN learn to detect the reference object in the input image and process the detected reference object to predict a pixel dimension of the reference object, and to predict a segmentation mask of each image, respectively. The segmentation mask differentiates the image pixels of the ROI from the background in the image.

Step <NUM> comprises validating the segmentation mask and the pixel dimension of the reference object learnt by the trained S-DNN and OD-DNN, respectively, using the validation data set.

In an example, a trained YOLO OD-DNN achieves a mAP score of <NUM> on the validation data set. For each image in the validation set, the YOLO OD-DNN detects the reference object and predicts pixel co-ordinates A, B, C, D of a bounding box bounding the reference object. A trained DeepLab S-DNN achieves a mIOU score of <NUM> on the validation data set. For each image in the validation set, the DeepLab S-DNN predicts the segmentation mask showing image pixels of the ROI/body part.

Step <NUM> comprises updating the neural network parameter values of the S-DNN and OD-DNN based on the validation. The updated parameter values can further be used for testing the OD-DNN and S-DNN using a test data set comprising at least one image showing the body part and the reference object.

In some implementations, the pixel dimension of the reference object learnt by the OD-DNN is a single pixel size, preferably a measurement per pixel, calculated based on the predefined dimension and the predicted pixel dimension.

<FIG> shows an example of a system <NUM> for training neural networks <NUM> according to the method of <FIG>. The neural network training system <NUM> is an example of a system implemented on one or more computers in one or more locations, for example, as computer programs. System <NUM> may be implemented on system <NUM> (in other words, they be one and the same system), or as a separate system <NUM> wirelessly coupled to system <NUM>. There may be additional systems <NUM>, <NUM>,. coupled to system <NUM>. Systems <NUM>, <NUM> may be other training systems or image processing systems which may be configured to perform the method <NUM> or database management systems whose data may be used by system <NUM> to train neural networks <NUM>.

System <NUM> includes a training engine <NUM> which has access to a memory <NUM> shared by the one or more computers in the system for training neural networks S-DNN and OD-DNN. In some implementations, training engine <NUM> is a distributed processor.

Training engine <NUM> may be configured to initialize the neural network parameters of the S-DNN and the OD-DNN. It retrieves the parameters from memory <NUM>. Memory <NUM> may store trained parameter values of other neural networks with the same architecture as the S-DNN and OD-DNN, respectively, and which have been trained to perform similar predictions. If the S-DNN and OD-DNN build on pre-trained models, memory <NUM> may store trained parameter values of the pre-trained S-DNN and OD-DNN, respectively.

Training engine <NUM> may be further configured to obtain the training data set stored in memory <NUM>, including the annotated training set. In some implementations, the training set stored in memory <NUM> is periodically updated as mentioned above. The continually updated training set can be retrieved by the training engine <NUM>. Training engine <NUM> further obtains the pixel dimension of the reference object and the segmentation mask of each image in the training set predicted by the OD-DNN and S-DNN, respectively, validates the prediction (checks whether training criteria are satisfied, in some implementations, iteratively) and updates the neural network parameters of the S-DNN and the OD-DNN based thereupon, to reach optimized neural network parameters of the S-DNN and the OD-DNN. The trained neural networks S-DNN and the OD-DNN can be used to control an electronic device to obtain and/or output a measurement of a dimension of a body part in an input image. The optimized neural network parameters are used in a next initialization by training engine <NUM>.

Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

The terms "computer", "processor", "data processing apparatus", "electronic device" etc. refer to data processing hardware and encompasses all kinds of one or more apparatus, devices, and machines for processing data.

To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer or electronic device having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer.

Data processing apparatus or computers for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, i.e., inference, workloads.

Claim 1:
A method (<NUM>) performed by one or more computers for outputting a measurement (<NUM>) of a dimension of a body part (<NUM>) in an input image (<NUM>), the method comprising:
receiving (<NUM>) the input image captured using an image capturing unit (<NUM>), the input image comprising the body part and a reference object (<NUM>) of a predefined dimension;
predicting (<NUM>) by a segmentation deep neural network, S-DNN (<NUM>), a segmentation mask (<NUM>) of the input image, the segmentation mask differentiating image pixels of the body part from image pixels of a background region in the input image;
detecting the reference object in the input image and processing the detected reference object to predict (<NUM>) a pixel dimension (<NUM>) of the reference object by an object detection deep neural network, OD-DNN (<NUM>);
obtaining (<NUM>) pixel coordinates of a plurality of key points (A, H, J, K, Q, T) of the body part using the segmentation mask;
modifying (<NUM>) the obtained pixel coordinates based on the predicted pixel dimension of the reference object; and
measuring the dimension of the body part based on the modified pixel coordinates and outputting (<NUM>) the measured dimension of the body part.