Patent ID: 12190515

BEST MODE FOR CARRYING OUT THE INVENTION

Acne vulgaris (acne for short) consistently represents the top three most prevalent skin conditions in the general population in the world. Due to lack of knowledge or disregard for acne, or lack of means (time and money and doctors), many people with acne do not get a proper treatment. Acne vulgaris can lead to severe irritation, ugly long-lasting scars (about 10% of population have acne scars), and other issues if not diagnosed and treated correctly.

There are two common methods for acne severity assessment: counting and grading. Acne counting is a challenge jobs since normally acne is a small mark and has similar color to skin. Grading is the comparison between some descriptions or photos. Grading method can be easy to use but can be inconsistent among doctors since it is a subjective method.

As mentioned above, doctors can miss some acne during their annotation process. If an artificial intelligence (AI) model or machine learning (ML) model learn or are trained with those images, the images with the missed annotation will create a confusion and inconsistency (because of non-labeled acne, the AI/ML model will learn or be trained that the missed annotated images are not acne when those images should be labeled as acne).

Embodiments of the compute system with an acne diagnostic mechanism provide more consistent and accurate acne scoring and classifies different types of acne for diagnostics at least by recognizing multiple acne in a given image and assessing each while other often miss acne locations as well as limit the number of acne in a given image. As examples of embodiments, the compute system with an acne diagnostic mechanism can identify, segment, or a combination thereof all or multiple acne and acne-like on a given image. Continuing with the example, the identification and segmentation can be implemented by a segmentation model or module, which will be described more later. Embodiments of the compute system with an acne diagnostic mechanism do not need doctors to do this work and avoiding some of the challenges as noted above. Further, embodiments eliminate or reduce the probability of missing acne.

Continuing with examples of embodiments, the compute system with an acne diagnostic mechanism can perform segmentation, generating more precise results than bounding boxes. Further continuing the examples, the compute system with an acne diagnostic mechanism can classify the detected acne. The compute system with an acne diagnostic mechanism can utilize annotated data from doctors as input as labeled data to training the AI/ML models of the compute system with an acne diagnostic mechanism and can compensate for the missing acne annotation by the doctors leading to solving inconsistency problem.

The compute system with an acne diagnostic mechanism can utilize acne counting, acne scoring, and acne segmentation to compute area of each acne, without requiring training for the area calculation of each acne. The compute system with an acne diagnostic mechanism process through acne area calculation for each acne provide consistent results for diagnostics. Continuing the examples for embodiments, the compute system with an acne diagnostic mechanism generates a severity from 0 to 100, which is precise or objective value while also being sensitive in the changing of number of acne and each of its severity. The consistent and accurate scoring as well as the acne area computation from the compute system with an acne diagnostic mechanism can also be utilized to track the progress of treatment.

The following embodiments are described in sufficient detail to enable those skilled in the art to make and use the invention. It is to be understood that other embodiments would be evident based on the present disclosure, and that system, process, or mechanical changes may be made without departing from the scope of an embodiment of the present invention.

In the following description, numerous specific details are given to provide a thorough understanding of the invention. However, it will be apparent that the invention may be practiced without these specific details. In order to avoid obscuring an embodiment of the present invention, some well-known circuits, system configurations, and process steps are not disclosed in detail.

The drawings showing embodiments of the system are semi-diagrammatic, and not to scale and, particularly, some of the dimensions are for the clarity of presentation and are shown exaggerated in the drawing figures. Similarly, although the views in the drawings for ease of description generally show similar orientations, this depiction in the figures is arbitrary for the most part. Generally, the invention can be operated in any orientation. The embodiments of various components as a matter of descriptive convenience and are not intended to have any other significance or provide limitations for an embodiment of the present invention.

The term “module” or “unit” or “circuit” referred to herein can include or be implemented as or include software running on specialized hardware, hardware, or a combination thereof in the present invention in accordance with the context in which the term is used. For example, the software can be machine code, firmware, embedded code, and application software. The software can also include a function, a call to a function, a code block, or a combination thereof. The word “module” or “model” can be also be used interchangeable depending on the context it is described or used in the written description. The “model” can represent one or more artificial intelligence models, machine learning models, or a combination thereof. The term “acne pimples” referred to herein means any type of acne including white heads, black heads, pustules, cysts, nodules, papules, comedones, or discolorations, without limitation.

Also, for example, the hardware can be gates, circuitry, processor, computer, integrated circuit, integrated circuit cores, memory devices, a pressure sensor, an inertial sensor, a microelectromechanical system (MEMS), passive devices, physical non-transitory memory medium including instructions for performing the software function, a portion therein, or a combination thereof to control one or more of the hardware units or circuits. Further, if a “module” or “unit” or a “circuit” is written in the claims section below, the “unit” or the “circuit” is deemed to include hardware circuitry for the purposes and the scope of the claims.

The module, units, or circuits in the following description of the embodiments can be coupled or attached to one another as described or as shown. The coupling or attachment can be direct or indirect without or with intervening items between coupled or attached modules or units or circuits. The coupling or attachment can be by physical contact or by communication between modules or units or circuits, such as wireless communication.

The word “module” or “model” can be also be used interchangeable depending on the context it is described or used in the written description. The “model” can represent one or more artificial intelligence models, machine learning models, or a combination thereof. It is understood the models identified in the description can be operated concurrently, in sequence, or in alternative without changing the operation of the models.

It is also understood that the nouns or elements in the embodiments can be described as a singular instance. It is understood that the usage of singular is not limited to singular but the singular usage can be applicable to multiple instances for any particular noun or element in the application. The numerous instances can be the same or similar or can be different.

Referring now toFIG.1, therein is shown an example of a system architecture diagram of a compute system100with an acne diagnostic mechanism in an embodiment of the present invention. Embodiments of the compute system100provide standardized and objective acne scoring and area calculation for each of the acne, as described earlier.

The compute system100can include a first device102, such as a client or a server, connected to a second device106, such as a client or server. The first device102can communicate with the second device106through a network104, such as a wireless or wired network.

For example, the first device102can be of any of a variety of computing devices, such as a smart phone, a tablet, a cellular phone, personal digital assistant, a notebook computer, a wearable device, internet of things (IoT) device, or other multi-functional device. Also, for example, the first device102can be included in a device or a sub-system.

The first device102can couple, either directly or indirectly, to the network104to communicate with the second device106or can be a stand-alone device. The first device102can further be separate form or incorporated with a vehicle, such as a car, truck, bus, motorcycle, or a drone.

For illustrative purposes, the compute system100is described with the first device102as a mobile device, although it is understood that the first device102can be different types of devices. For example, the first device102can also be a non-mobile computing device, such as a server, a server farm, cloud computing, or a desktop computer.

The second device106can be any of a variety of centralized or decentralized computing devices. For example, the second device106can be a computer, grid computing resources, a virtualized computer resource, cloud computing resource, routers, switches, peer-to-peer distributed computing devices, or a combination thereof.

The second device106can be centralized in a single room, distributed across different rooms, distributed across different geographical locations, embedded within a telecommunications network. The second device106can couple with the network104to communicate with the first device102. The second device106can also be a client type device as described for the first device102.

For illustrative purposes, the compute system100is described with the second device106as a non-mobile computing device, although it is understood that the second device106can be different types of computing devices. For example, the second device106can also be a mobile computing device, such as notebook computer, another client device, a wearable device, or a different type of client device.

Also, for illustrative purposes, the compute system100is described with the second device106as a computing device, although it is understood that the second device106can be different types of devices. Also, for illustrative purposes, the compute system100is shown with the second device106and the first device102as endpoints of the network104, although it is understood that the compute system100can include a different partition between the first device102, the second device106, and the network104. For example, the first device102, the second device106, or a combination thereof can also function as part of the network104.

The network104can span and represent a variety of networks. For example, the network104can include wireless communication, wired communication, optical, ultrasonic, or the combination thereof. Satellite communication, cellular communication, Bluetooth, Infrared Data Association standard (IrDA), wireless fidelity (WiFi), and worldwide interoperability for microwave access (WiMAX) are examples of wireless communication that can be included in the communication path. Ethernet, digital subscriber line (DSL), fiber to the home (FTTH), and plain old telephone service (POTS) are examples of wired communication that can be included in the network104. Further, the network104can traverse a number of network topologies and distances. For example, the network104can include direct connection, personal area network (PAN), local area network (LAN), metropolitan area network (MAN), wide area network (WAN), or a combination thereof.

Returning to the description standardized and objective acne scoring of the embodiments of the compute system100, as an example, the compute system100provide functions to various users112, including patients and clinicians. The compute system100can provide functions to the users112in a number of ways.

For example, the compute system100can provide the functions for the users112with the first device102, the second device106, distributed between these two devices, or a combination thereof. Also as examples, the compute system100can provide a mobile applications for the patients, the clinicians, or a combination thereof. Further as an example, the compute system100can provide the functions via a web-browser based applications or a software to be executed on the first device102, the second device106, distributed between these two devices, or a combination thereof.

In one embodiment as an example, patient images114are taken and uploaded by the patient and reviewed by the clinician. In this embodiment, a patient launches the acne diagnostic mechanism via the mobile application and logs into the patient's account. The patient can be prompted to upload or take body images as the patient images114. The compute system100can guide a patient on photo guidelines for the patient images114and accepts or rejects the patient images114for retake based on a pre-specified criteria, e.g., distance, quality, blur, or a combination thereof. The compute system100can also provide guides for a patient on capturing videos as opposed to still photos. The patient images114can be selected from the video.

Once the patient images114, as required for analysis, are successfully uploaded, the compute system100can send or load the patient images114to an acne diagnostic module116for analysis including an acne artificial intelligence (AI)118. The acne diagnostic module116will be described later. For brevity and clarity and as an example, the acne diagnostic module116is shown inFIG.1as being executed in the second device106although it is understood that portions can operate on the first device102, such as the mobile app or the web-browser based application, can operate completely on the first device102, or a combination thereof. As a further example, the acne diagnostic module116can be implemented in software running on specialized hardware, full hardware, or a combination thereof.

The acne AI118can be software executed on a processor, core, ASIC, specialized GPU, or a combination thereof configured as a machine learning structure. The combination of hardware decision nodes including for example gates and switches combined with a machine learning software that can process the patient images114at a pixel level.

Based on analysis results, the compute system100can display information to the patient including a recommendation based on the patient images114, uploaded, for the patient to schedule a visit with your primary care physician or with a specialist based on an acne indication120, which may or may not be visible or displayed to the patient.

If the acne diagnostic module116provides the acne indication120below a pre-specified threshold122, the compute system100can display a message that based on the patient images114, uploaded, the patient may not need a visit with your primary care physician or with other specialists. The compute system100can provide a function allowing the patient to schedule a visit with the clinician.

Continuing the example, the compute system100can provide a function that allows the clinician to access the patient images114uploaded by the patient and an acne indication120, such as the acne score, the acne area, or a combination thereof through the web-based dashboard from the acne diagnostic mechanism. The compute system100allows the clinician to make edits to annotations determined by the acne diagnostic module116and the scores (if necessary) and saves the results. The clinician can utilize the acne indication120to make the diagnostic decision and provide necessary treatment steps (if applicable).

In a further embodiment as an example, the compute system100can allow a patient to schedule a visit with a primary care physician or with a specialist. A clinician can launch the acne diagnostic mechanism, such as a mobile application and logs in. The compute system100can be prompted to upload or take the patient images114of the patient's body or body parts to be analyzed by the acne diagnostic module116.

The compute system100can provide guidance to the clinician on the photo guidelines. The compute system100can accept or reject images for retake based on a pre-specified criteria, such as distance, quality, blur, or a combination thereof. Once the patient images114are successfully uploaded, the compute system100and send or load the patient images114to the acne diagnostic module116for analysis.

Continuing the example, the compute system100can similarly provide a function that allow the clinician to access the patient images114uploaded by the patient and the acne indication120, such as with the web-based dashboard from the acne diagnostic mechanism. The compute system100allows the clinician to make edits to annotations determined by the acne diagnostic module116and the scores (if necessary) and saves the results. The clinician can utilize the acne indication120to make the diagnostic decision and provide necessary treatment steps (if applicable).

Referring now toFIG.2, therein is shown an example of some of the processing stages201from patient images114, acne segmentation202, and acne classification204in an embodiment. The compute system100ofFIG.1, the acne diagnostic module116ofFIG.1, or a combination thereof can process the patient images114, such as the leftmost image depicted inFIG.2. The middle image depicted inFIG.2is an example of an acne segmentation202as an interim output of the acne diagnostic module116. The leftmost image represents the acne classification204based on the patient image114as an output that can be displayed for the user112ofFIG.1. The acne classification204is an example of an output of the acne diagnostic module116.

Referring now toFIG.3, therein is shown an example of a user interface display301of an embodiment. An example of the user interface display301in an embodiment can be displayed on the first device102, the second device106, or a combination thereof.FIG.3is as an example of the user interface display301of an output of the compute system100ofFIG.1, the acne diagnostic module116ofFIG.1, or a combination thereof to be displayed to the user112ofFIG.1.

In this example, the patient image114ofFIG.1is shown with the acne classification204ofFIG.2and non-acne areas302, such as the eyes being blacked out, can also serve an anonymization. In this example, the input is a facial image as the patient image114to the compute system100, the acne diagnostic module116, or a combination thereof. The output is shown in the user interface display301including an acne counting table304of each type of acne (to be described later), acne severity score306for the patient image114, acne severity score306for the patient image114, the acne indication120includes markers308indicating the center and radius of each of the type of acne in the acne counting table304, and comment area310.

The comment area310can provide communication between the clinician and the user112ofFIG.1. It is understood that the user interface display301can be transmitted to the user to provide analysis of the user's acne and feedback for treatment of the items listed in the acne counting table304. The user interface display301can be used to monitor changes in the acne counting table304over time.

Referring now toFIG.4, therein is shown an example of a block diagram of the compute system100with the acne diagnostic mechanism116in an embodiment. In the example depicted inFIG.4, the processing of the patient image114can be depicted at least as shown inFIG.2with acne segmentation202and acne classification204and can be performed by the compute system100ofFIG.1, the acne diagnostic module116ofFIG.1, or a combination thereof.

In this example, the patient image114is processed with Oops detection module402to detect whether if the patient image114is skin related or not. If yes, the flow can continue to Image quality check module404to verify if the quality of the patient image114is sufficiently detailed for detection with the acne diagnostic module116utilizing the Acne AI118ofFIG.1.

In this example, the block diagram can couple the oops detection module402to the image quality check module404. Continuing the example, the image quality check module404can function as a filter for preventing poor quality images from being used as input for an acne classification module412. Poor quality images refer to images that are too blurry or images that are of too bad luminosity (either too bright, too dark or too noisy). Eventually, the image quality check module404is a classification module whose possible output classes are acceptable, blurry, bad luminosity, or a combination thereof.

The image quality check module404can check the patient images114for a skin segmentation module406including a skin module406, a skin detection module406, or a combination thereof for processing through the acne diagnostic mechanism116. The image quality check module404can be implemented in a number of ways.

For example, the image quality check module404can check the patient images114that are input for certain quality criteria and meeting or exceeding the quality threshold122. As specific examples, the quality criteria can include a nominal metric416, a blurry metric418, a bad luminosity metric420, or a combination thereof. Further as a specific example, the quality criteria is a three dimensional vector with the nominal metric416, the blurry metric418, and the bad luminosity metric420.

The nominal metric416is used to measure the acceptability of the patient images114beyond the acceptability by the skin segmentation module406. The skin segmentation module406rejects or accepts each of the patient images114to include sufficient skin area or skin region as described earlier. The nominal metric416processes the patient images114further such that the image is of sufficient quality to determine a skin diagnosis. The nominal metric416represents an overall measure that can be used to determine the acceptability of the patient image114for skin diagnosis for further processing. The nominal metric416can include measures for clarity, lighting, non-intervening obstructions to the visibility of skin, resolution, or a combination thereof.

The blurry metric418is used to measure the how clear or blurry the patient image114being processed is. The value for the blurry metric418is set for the patient image114used for training the image quality check module404of what is considered clear and what is consider not clear or blurry. If the value of the blurry metric418indicates that the patient image114is not clear or blurry, then the acne diagnostic mechanism116or portions thereof cannot analyze the instance of the patient images114to compute the acne indication120. If the value of the blurry metric418indicates that the image is clear or not blurry, then the acne diagnostic mechanism116or portions thereof can analyze the instance of the patient images114to compute the acne indication120.

The bad luminosity metric420is used to measure the lighting or brightness or dimness of the patient image114being processed. The value for bad luminosity metric420is set for the patient image114used for training the image quality check module404of what is considered too dim and what is consider not dim. If the value of the bad luminosity metric420indicates that the image is dim, then the acne diagnostic mechanism116or portions thereof cannot analyze the instance of the patient images114to compute the acne indication120. If the value of the bad luminosity metric420indicates that the image is not too dim, then the acne diagnostic mechanism116or portions thereof can analyze the instance of the patient images114to compute the acne indication120.

The metrics of the quality criteria can be measured with quality threshold122collectively, as subsets, as equal priority, or of non-equal priority. The term equal priority refers to all the metrics are compared with equal weight and impact the meeting or exceeding the quality threshold122for the patient image114to be deemed acceptable and continue to be processed by the acne diagnostic mechanism116. The term non-equal priority refers to the varying weight of the metrics relative to each other where some can have more importance over the other metrics. As an example, one of the metrics of the quality criteria alone can be used to determine if the quality threshold122is met or not for the image to continue to be processed by the acne diagnostic mechanism116.

Returning to the quality threshold122, the quality threshold122can include a single value for all or some of the metrics of the quality criteria or can include a value for each of the metrics of the quality criteria. As an example, the quality threshold can include a nominal threshold422, a blurry threshold424, a bad luminosity threshold426, or a combination thereof.

As a specific example, if an instance of the patient images114is relevant or usable to compute the acne indication120by the acne diagnostic mechanism116, then the skin segmentation module406can determine that instance of the patient images114continues processing to the skin segmentation module406. In this example, the image quality check module404checks each of the instance of the patient images114and outputs a three dimensional vector for the scores for the nominal metric416, the blurry metric418, and the bad luminosity metric420. The value for the bad luminosity metric420can also represent the noise in the patient image114being processed.

Continuing with the specific example, the sum of output vector for the quality criteria does not need to be 1. There can be two high values at the same time: [0.0, 99.6, 98.0] for the nominal metric416, the blurry metric418, and the bad luminosity420, respectively, which means the input image can be blurry or not clear and in bad light quality.

The compute system100can set the nominal threshold422for the nominal metric416for example, if the value for the nominal metric416is greater than or equal to 0.6, the image quality check module404, the acne diagnostic mechanism116, or a combination thereof accept the input image or the instance of the patient images114being processed at the time. In other words, the nominal metric greater than or equal to the nominal threshold422alone can be the quality criteria and serve as the quality threshold, respectively, to determine the patient images114being processes as acceptable by the image quality check module404. In this example, the quality criteria can function as a priority encoder with the nominal metric416greater than or equal to the nominal threshold422to determine acceptability regardless of the values and comparison to the other two metrics and two thresholds.

When the nominal metric416is less than the nominal threshold422, the blurry metric418and the bad luminosity metric429are compared to the blurry threshold424and the bad luminosity threshold426, respectively. The greater or the maximum value between the blurry metric418and the bad luminosity metric420will determine whether the blurry threshold424or the bad luminosity threshold426, respectively, shall be used as the quality threshold.

In other words, as an example, if the value of the nominal metric416is lower than 0.6 but values for the blurry metric418and the bad luminosity metric420indicates bad light condition are lower than the blurry threshold424and the bad luminosity threshold426, respectively, then the image quality check module404, accepts the input image or the instance of the patient images114being processed at the time. Otherwise, the input image or the instance of the patient images114being processed at the time will be classified into blurry if the value for the blurry metric418is higher than the value for the bad luminosity metric420and vice versa. Continuing this example, the quality threshold122is shown to be 0.7. Further, the metric (blurry metric418or the bad luminosity metric420) with the larger value can be used to provide feedback to improve performance of the compute system100if the image quality check module404rejects the image.

Continuing the example, the patient images114that are processed and determined by the image quality check module404to continue processing by the compute system100, the acne diagnostic module116, or a combination thereof, the flow can process from the image quality check module404to the skin segmentation module406. The skin segmentation module406can segment skin region included eyes, nails, tattoo on skin, sick skin (for example psoriasis, skin tumors, etc.). The skin segmentation does not optionally segment scalp unless the scalp is visible (i.e., there are not too much hair covering it). The skin segmentation module406can ignore the object on skin such as clothes, bracelet, etc. However the skin segmentation module406can still segment the visible skin like skin under transparent objects, such as glasses.

Continuing with the example, the compute system100, the acne diagnostic module116, or a combination thereof can detect skin and blackout all non-skin region of the patient image114with the skin segmentation module406. The flow can continue to an acne segmentation module408to segment the acne and acne-like area of the segmented image407. As shown inFIG.2, the input image114is analyzed in small pixel regions to derive the segmented image202ofFIG.2. Based on the result of the acne segmentation module408, the compute system100, the acne diagnostic module116, or a combination thereof can continue to Acne separation algorithm (Algorithm A)410, described later, to get the center and radius information of each of an acne pimples409and acne-like areas. The compute system100, the acne diagnostic module116, or a combination thereof can crop the segmented image407into a target pixel array411with one of the acne pimples409or acne-like area in the center. The acne separation algorithm410can analyze the target pixel array411to separate a plurality of the acne pimples409that are adjacent in the segmented image407.

The compute system100, the acne diagnostic module116, or a combination thereof can score the cropped images using Acne classification module412. The acne AI118can include the acne segmentation module408, the acne separation algorithm410, and the acne classification module412for identifying and classifying the acne pimples409. The Acne classification module412can identify each of the acne pimples409into acne types413. The acne types413can be identified from among a white head409, a black head409, a pustule409, a cyst409, a nodule409, a papule409, a comedone409, and a discoloration409. The Acne classification module412generates a score from 0 to 5 for each cropped image. The higher score, the more severe it is. Using the Skin segmentation module406, Acne segmentation module408, and the Acne classification412, the compute system100, the acne diagnostic module116, or a combination thereof can compute (Algorithm B) in a presentation module414to generate the final score for each of the acne pimples409and acne-like area as well as their area.

The compute system100, the acne diagnostic module116, or a combination thereof can be trained each module individually. Once every single module performs well (that is the obtained metrics are greater or higher than some good threshold depending on each module. For example, the skin segmentation has to have similarity coefficient, such as a Jaccard score, higher than 0.8), the compute system100, the acne diagnostic module116, or a combination thereof can be trained, tested, or a combination thereof as the whole system together. In this training process, a test set is not part of the training set. The test set can include a variety of data for example different skin tone, different parts of the face or body, different resolution, etc. Every time, if any portion of the compute system100, the acne diagnostic module116, or a combination thereof can provide an update, which can be from one module, from one algorithm, the compute system100, the acne diagnostic module116, or a combination thereof can predict acne on those images by running through the acne diagnostic mechanism116. After the raw results, the compute system100, the acne diagnostic module116, or a combination thereof can run statistical tests and compare the analysis result with the one of an older version. If the result is better, the compute system100, the acne diagnostic module116, or a combination thereof can keep the update. Otherwise, we will not use it.

The compute system100, the acne diagnostic module116, or a combination thereof can compute acne severity score as follows:

S=200π⁢arc⁢tan⁡(c⁢∑i=0Nsi⁢aiA)(1.1)
where N is total number of acne, c is a fixed coefficient, siis the score of acne i, aiis the area of acne i and A is the total area of detected skin. The arctan function provides a linear increasing for the normal coverage of acne and a quite constant for the very severe acne face seeFIG.5. The acne severity score430has minimum value is 0 and maximum value is 100. The skin segmentation is described more later and to compute total skin area and the computation of acne area (described later).

Regarding the performance of the compute system100, the acne diagnostic module116, or a combination thereof, a number of metrics can be utilized. There are a number of possible metrics for loss functions and accuracy.

Regarding mean squared error and mean absolute error, if a vector of n predictions is generated from a sample of n data points on all variables, and Y is the vector of observed values of the variable being predicted, with Y being the predicted values, then the within-sample Mean squared error (MSE) of the predictor is computed as:

MSE=1n⁢∑i=1n(Yi-Y^i)2(2.1)

And the Mean absolute error (MAE) is computed as

MAE=1n⁢∑i=1n❘"\[LeftBracketingBar]"Yi-Y^i❘"\[RightBracketingBar]"(2.2)

These functions can quantify the errors made by the module: the lower their value is, the better it is for the module. They can also be considered as distances between true and predicted values, L1-distance for the MAE, and L2-distance for the MSE. They are mostly used in regression problems, where the output is a real number.

One benefit of the MSE is that it can be optimized due to its derivative. This means that the compute system100, the acne diagnostic module116, or a combination thereof can utilize this function as a loss function during the training of a regression problem. On the other side, MAE, which has not this easy optimization property due to the absolute value in its definition, is useful for measuring how far from the true values are predicted values. Indeed, this function gives values in the same scale and unit as the predicted values, which allow more interpretation and understanding for human eyes. Then MAE is often used as a metric during training, to ensure that the module performs well.

Regarding Binary Cross-Entropy loss, binary classification is basically a supervised learning algorithm which aims to classify inputs into one of two classes. The format of the output can be the probability to belong to each class, or directly the predicted class. Useful for binary classification problems with probabilities as output, the Binary Cross-Entropy loss (BCE) is defined as:

BCE=-1n⁢∑i=1n(Yi×log⁡(Y^i)+(1-Yi)×log⁡(1-Y^i))(2.3)

This function allows to compare each predicted probabilities Ŷito actual class output Yiwhich can be either 0 or 1. Then, it calculates the score that penalizes the probabilities based on the distance from the expected value. That means how close or far from the actual value. The BCE is also called the log loss because we apply the logarithm function. The reason behind using the logarithm function is that it penalizes less for small differences between predicted probabilities and true labels, but penalizes more when the difference is larger. Eventually, since the probabilities are between 0 and 1, the log values are negative, that is why the compute system100, the acne diagnostic module116, or a combination thereof can take the opposite of the average over the samples, to get a positive BCE value.

Regarding Jaccard score, the Jaccard index, also known as the Jaccard similarity coefficient, is a statistic used for gauging the similarity and diversity of sample sets. It measures the similarity between finite sample sets, it is defined as the ratio of the intersection over the union of the two sets.

J⁡(A,B)=❘"\[LeftBracketingBar]"A⋂B❘"\[RightBracketingBar]"❘"\[LeftBracketingBar]"A⋃B❘"\[RightBracketingBar]"(2.4)

Regarding intra-class correlation coefficient (ICC), the intra-class correlation coefficient (ICC) is a number, usually found to have a value between 0 and 1. It refers to correlations within a class of data (for example correlations within repeated measurements of weight), rather than to correlations between two different classes of data (for example the correlation between weight and length). Let the variable of observation is defined as:

Xij=μ+ai+eij(2.5)

Where a is the group effects and e is the residual effects which are independently normally distributed with mean 0 and

E⁡(Xij)=μ⁢and⁢Var⁡(ai)=σa2;Var⁡(eij)=σe2(2.6)

Then the ICC is defined as

ICC=σa2σa2+σe2(2.7)

The binary accuracy is defined as the ratio between number of correct predictions over the total number of predictions. For example, let the ground truth is [1, 1, 0, 1, 0, 0] and the prediction is [0, 1, 1, 0, 1, 0], then the number of correct predictions is 2 (the second position and the last position) and the total number of predictions is 6. So, the Binary accuracy in this case is 2/6=⅓. When the compute system100, the acne diagnostic module116, or a combination thereof compute Binary accuracy for one image, each pixel of that image is considered as an information valued either 0 or 1. The compute system100, the acne diagnostic module116, or a combination thereof can compare the predicted values with the ground truth pixel by pixel and get the Binary accuracy as the above formula.

For example, let the ground truth segmented image407(3×5 pixels) A and the prediction B be

A=[100000000000000];and⁢B=[010000000000000](2.8)

Binary accuracy of this example is 0.8 (there are 12 corrected prediction pixels over 15 pixels). However, Jaccard index in this case is 0 since there is only one annotated pixel which is not matched by the prediction.

The Oops detection module402can acts as a filter, preventing irrelevant images to be used as input for the image quality check module404. An irrelevant image refers to images that do not contain human skin or has more than 80% background. Eventually, the Oops detection module402is a regression module whose output gets a value between 0 and 1 in which a value close to 1 means probably irrelevant and vice versa. The compute system100, the acne diagnostic module116, or a combination thereof can include a threshold equal to 0.8 to separate the region of irrelevant and relevant, i.e., if output is greater than 0.8, the input is irrelevant, otherwise it is relevant.

As an example, the ImageNet dataset contains 310 K images in which there are 34 K irrelevant images and the remaining are relevant images. The dataset includes a variety of images of animals, plants, foods and people. It also includes images with very sick skin to normal skin in wide range of skin tone from white to darker skin.

The Oops detection module402not only filters irrelevant images, but also to eliminate an image taken from too far distance which makes the human skin part cover less than 20% area. To be able to detect the case, the compute system100includes a set of data in which there are images containing human skin covering less than 20% area. The dataset is supplemented with data augmentation creating a set of 94 K synthetic images in which a relevant image is merged with an irrelevant one to get a synthetic image in which the background of the relevant image is expanded.

The skin segmentation module406can be implemented in a number of ways. As an example, the implementation can be U-net architecture, which includes of an encoder-decoder scheme. The encoder reduces the spatial dimensions in every layer and increases the channels. The decoder increases the spatial dims while reducing the channels. As a specific example. EfficientNet architecture, which is a convolutional neural network architecture and scaling method that uniformly scales all dimensions of depth/width/resolution using a compound coefficient, can be an implementation for the classification and pre-trained on ImageNet dataset for the encoding.

Continuing with the example, the process can continue from the skin segmentation module406to the acne segmentation module408. As an example, the acne segmentation module408does not just segment the acne but it also segments the acne-like marks, for example acne scar or mole. The acne segmentation module408performs an object detection where compute system100, the acne diagnostic module116, or a combination thereof can detect every acne pimples409, such as irritated regions of the skin at or near the surface even of very small size. These are challenging problems because machine can get high accuracy without detecting anything. For example, if there are only 5% of skin area having acne, the machine will get 95% accuracy when it detects no acne and the acne pimples409do not often have well-defined borders which makes the acne segmentation task even more difficult. As a result, different doctors may segment the acne pimples409differently. Therefore, it is very hard to make the machine learn the correct acne segmentation. The compute system100, the acne diagnostic module116, or a combination thereof addresses these challenging problems at least through the selection of the architecture and training dataset.

Continuing with the example, the process can continue from the acne segmentation module408to the acne classification module412, the acne separation module410, or a combination thereof.

The acne classification module412can be trained in a number of ways with various datasets. For example, the acne classification module412can be trained with 128×128 pixels images. The training images are cut based on the acne segmentation module408. That is, after detecting the acne spots, the acne classification module412identifies the center point of each acne and let it be the center of 128×128 pixels image. In the input image, there can be many acne pimples409. The acne classification module412focuses on the acne at the center of the image and classify it, but do not classify the ones not in the center. For example, the acne classification module412utilizes region of interest (ROI) technique to focus on the chosen acne in the patient image114. That is the inputs of acne classification module412are one RGB image and one ROI image. As a specific example, the acne classification module can be based on an Xception structure, which is a deep convolutional neural network architecture, with a residual layer and multiplication step between original image and ROI.

In the example shown inFIG.4, the acne separation module410is shown between the acne segmentation module408and the acne classification module412, although the compute system100, the acne diagnostic module116, or a combination thereof can be implemented differently. For example, the acne separation module410can be implemented as part of the acne segmentation module408and the acne classification module412, or distributed between two.

The acne separation module410improves the performance of the acne classification module412because the acne segmentation module408can output the acne pimples409that is right next to another of the acne pimples409as one. The erroneous segmentation input to the acne classification module412results in an incorrect classification either because the two of the acne pimples409are of different types or the acne classification module412will see two of the acne pimples409as one hence the area or the size will be bigger. All of this sequence can produce an incorrect prediction. An example of the acne segmentation module408is further described inFIG.32toFIG.35. The acne separation module410can separate the acne pimples409that is right next to another of the acne pimples409by dividing the image into a target pixel array411each containing a single instance of the acne pimples409. The target pixel array411can be the smallest pixel array suitable for analysis of the acne pimples409. By way of an example, the target pixel array411can be the 32×32×1024 pixel array411for the image quality check module404, the 12×12×1632 pixel array411for the skin segmentation module406, and the 16×16×1632 pixel array411for the acne segmentation module408.

The flow can progress with one or more of the following leading to the presentation module414. As an example, the presentation module414can process inputs from the skin segmentation module406, the acne segmentation module408, the acne classification module412, or a combination thereof. The compute system100, the acne diagnostic module116, display module414, or a combination thereof can return the area of each type of acne as well as the area of each acne. To do that, the result from Acne classification module412, Acne segmentation module408, and Skin segmentation module406can be overlayed to form the acne indication120.

By way of an example, the Skin segmentation module406can provide the segmented image407, the Acne segmentation module408can provide the region of interest (ROI) having evidence of acne in the segmented image407, and the Acne classification module412can provide an area of each acne428and an acne severity score430. The compilation of the outputs of the Acne classification module412, Acne segmentation module408, and Skin segmentation module406can form the acne indication120.

The functional units or circuits in the first device102can work individually and independently of the other functional units or circuits. The first device102can work individually and independently from the network104, the second device106, other devices or vehicles, or a combination thereof.

The functional units or circuits described above can be implemented in hardware. For example, one or more of the functional units or circuits can be implemented using a gate, circuitry, a processor, a computer, integrated circuit, integrated circuit cores, a pressure sensor, an inertial sensor, a microelectromechanical system (MEMS), a passive device, a physical non-transitory memory medium containing instructions for performing the software function, a portion therein, or a combination thereof.

Referring now toFIG.5, therein is shown an example of a graphical representation for acne severity function501in an embodiment.FIG.5depicts the acne severity score502versus acne area504versus overall severity506. Acne severity function501has linear increasing for the normal coverage of acne and a quite constant for the very severe acne face.

Referring now toFIG.6, therein is shown an example of synthetic images601from input synthesis in an embodiment. The examples depicted inFIG.6, a skin area602and a larger skin area604is placed on top of an image of landscape606or trees608. The ImageNet dataset contains 94 K synthetic images discussed inFIG.4including 81 K images that human skin cover from 2% to 18% (called synthetic type 1) and 13 K images that human skin cover more than 20% (called synthetic type 2). The dataset labeled synthetic images601type 1 as 0.5 and synthetic images type 2 as 0.0. Examples of the synthetic images601are used to train the acne detection mechanism116ofFIG.4.

The training set includes data augmentation with random cropping, making boarder constant, resizing, random rotating, random flipping such that compute system100, the acne diagnostic module116, or a combination thereof produce output of size 384×384. For validation set, the compute system100, the acne diagnostic module116, or a combination thereof simply resize images to 384×384 which is the size of the machine learning algorithm's input.

Referring now toFIG.7, therein is shown an example of a block diagram for an inception module701of oops detection module402in an embodiment. The compute system100, the acne diagnostic module116, or a combination thereof can be implemented in an number of ways for the oops detection module402. For example the oops detection module402can be implemented with an Inception-ResNet-v2, which is a convolutional neural network that is trained on more than a million images from the ImageNet database, for the encoding part. The Inception-ResNet-v2 is a convolutional neural architecture that builds on the Inception family of architectures but incorporates residual connections (replacing the filter concatenation stage of the Inception architecture).

The convolutional neural network used in the Oops detection module402replaces the filter concatenation stage of the Inception architecture with the circuit ofFIG.7to incorporate residual connections that would otherwise be lost. A relu activation module702receives the patient image114for processing. The image is dispersed to a series of a detail level converter704, such as a 1×1 converter in a 32 segment array. The detail level converters704can be coupled to a broad area converter706, such as a 3×3 converter in a 32 segment array. The patient image114is submitted to three channels for analysis. The first channel consists of only one of the detail level converter704. The second channel consists of one of the detail level converter704coupled to one of the broad area converter706. The third channel consists of one of the detail level converter704coupled to one of the broad area converter706coupled to a second one of the broad area converter706. A detailed area converter708, such as a 1×1 converter in a 256 bit linear configuration receives input from the first channel, the second channel, and the third channel in order to provide detailed analysis over the entire input of the patient image114. A summing node710receives the input image of the patient image114and the output of the detailed area converter708in order to identify skin in the patient image114.

Referring now toFIG.8, therein is shown an example of a block diagram for another portion801of the oops detection module402in an embodiment. Given the patient image114, the oops detection module402can be implemented with a convolutional neural network (CNN) architecture that can return one output as illustrated inFIG.8. Compared with another encoding. Inception-ResNet-v2802gives us robust performance. The Oops detection module402contains 783 layers with about 54.3 million parameters.

A Conv_7B Layer module804receives the patient image114processed by the Inception-ResNet-v2802and identifies non-skin areas in the patient image114. The Conv_7B Layer module804is coupled to a dropout module806. The dropout module806can mask out the non-skin areas of the patient image114. The dropout module806is coupled to a dense+activation relu module808that can identify acne within the skin area602ofFIG.6of the patient image114that has not been marked out. The dense+activation relu module808is coupled to a loss module810and a minimum module812. The loss module810calculates the error interpretation of the analysis of the skin area602and identifies a mean absolute error for the calculations of the acne count, area, and size. The minimum module812applies the corrections of the loss module to the analysis of the skin area602from the dense+activation relu module808. The minimum module812provides the result814for incorporation in the user interface display301ofFIG.3.

The loss module810plays an important role in the training process. The loss module810punishes the module if it regresses wrong. Regarding the loss and metric, the metric to measure accuracy of the model is mean absolute error (MAE) that is a measure of errors between paired observations expressing the same phenomenon. MAE is calculated as the sum of absolute errors divided by the sample size:

MAE=1n⁢∑j=1n❘"\[LeftBracketingBar]"yj-y^j❘"\[RightBracketingBar]"(3.1)
where yjare the true labels and ŷJare the corresponding predictions.

It has been discovered that the oops detection module402can operate the Inception-ResNet-v2802as an artificial intelligence utilizing a convoluted neural network trained with the ImageNet data set to recognize all types of the acne pimples409that can be identified on the skin surface. The processing of the patient image114preserves the overall view while identifying and marking the acne pimples409on the skin surface. The loss module810can refine the model during execution in order to maintain a high degree of accuracy and improve the process of acne identification and analysis.

Referring now toFIG.9, therein is shown an example of the loss comparison chart901for the oops detection module402in an embodiment. The compute system100ofFIG.1, the acne diagnostic module116ofFIG.1, or a combination thereof can use the Shrinkage loss902to handle the data imbalance issue in learning deep regression networks. For this task, the compute system100, the acne diagnostic module116, or a combination thereof can use Shrinkage loss function, which is defined as below:

l⁡(y,y^)=m21+exp[a⁡(c-m)]
where m=|y−ŷ|, y is the true label and ŷ is the prediction, a and c are hyper-parameters controlling the Shrinkage speed and the localization respectively.

As shown in the example inFIG.9, the Shrinkage loss902only penalizes the importance of easy samples (when m<0.5) and keeps the loss of hard samples unchanged (when m>0.5) compared to the square loss L2904(with L2=m2). The focal loss L3906(with L3=m3) penalizes both the easy and hard samples.

The compute system100, the acne diagnostic module116, or a combination thereof set the value of a to be 10 to shrink the weight function quickly and the value of c to be 0.2 to suit for the distribution of m, which ranges from 0 to 1.

Referring now toFIG.10, therein is shown an example of the performance chart1001of the oops detection module402in an embodiment. The compute system100, the acne diagnostic module116, or a combination thereof can be trained for 95 epochs with nearly 40000 images in each epoch, about 80% data set is reserved for training set and the rest for validation. To measure the accuracy of the model, MAE as equation 3.1 can be utilized. The compute system100, the acne diagnostic module116, or a combination thereof can obtain 0.01 MAE score on validation.FIG.10illustrates for the performance of Oops detection module402.

A loss chart1002indicates the loss function of a training loss1006as compared to a verification loss1008based on the number of epochs used to train the acne diagnostic module116. As displayed in the loss chart1002, the verification loss1008tracks the training loss1006with 0.002 after 95 epochs of training.

An absolute error chart1004indicates the mean absolute error of a training error1010as compared to a verification error1012based on the number of epochs used to train the acne diagnostic module116. As displayed in the absolute error chart1004, the verification error1012tracks the training error1010with 0.1 mean absolute error between the predicted values and the absolute values after 95 epochs of training.

Referring now toFIG.11, therein are shown examples of patient images114classified as blurry images1101in an embodiment. The images depicted inFIG.11are examples used with the image quality check module404. The dataset includes 5297 images. 86% of them are of good quality, and 14% of them are of bad quality. The image quality check module404trained only with good quality images that were modified to transform them into bad quality images did not perform well with real bad quality images. The dataset to train the image quality check module404were supplemented to include real bad quality images. The dataset was also supplemented with images to include enough black skin images so that the model does not classify them as “too dark”.FIG.11are some examples of images of the training dataset.

A first blurry image1102is too far away to see details of the skin and a portion of the skin is obstructed by a bandage. A second blurry image1104is close enough, but is out of focus, which removes the required detail for analysis by the image quality check module404. A third blurry image1106is too far away and is out of focus making it a useful example of a blurry image for training purposes.

Referring now toFIG.12, therein are shown examples of patient images114classified as bad luminosity1201in an embodiment.FIG.12depict examples of different type of bad luminosity images used for training the image quality check module404.

A first bad luminosity image1202shows an extremely bright light washes out the details of the skin. A second bad luminosity image1204shows too little light to see details of the skin. A third bad luminosity image1206shows a colored light washes-out the detail of the skin making the image a good training item for bad luminosity.

Referring now toFIG.13, therein are shown examples of patient images114classified as acceptable1301in an embodiment.FIG.13depict examples of good images as opposed to bad images to those shown inFIG.11orFIG.12.

A first acceptable image1302provides sufficient detail of the face for analysis of the image quality check module404. A second acceptable image1304, again provides the correct detail, focus and luminosity to be analyzed correctly. A third acceptable image1306is also provided as a good training image for proper focus, luminosity, and detail of the skin for analysis.

Referring now toFIG.14, therein is shown an example of patient image114processing with data augmentation1401in an embodiment.FIG.14depicts examples of a patient image114being processed as part of the data augmentation1401to the data set for the compute system100ofFIG.1, the acne diagnostic module116ofFIG.1, or a combination thereof. As an example, the patient image114is considered a good image. The data augmentation process, in this example, transforms the good image into one or more bad images to generate balanced batch of inputs for the image quality check module404. The transformations change the good image's luminosity, contrast, add different levels of blur, or a combination thereof. Moreover, to make the model learn to focus on the skin only and not on the background, the compute system100, the acne diagnostic module116, or a combination thereof trains the model to focus on the skin and not the background, the skin segmentation module406to modify the background and keep the skin untouched, or to modify the skin and keep the background untouched.

As a training aide, a bad luminosity block1402can change the skin luminosity and the background luminosity in a block1404resulting in the high contrast image1406. The bad luminosity block1402can also change the whole image luminosity in a block1408resulting in the darkened image1410. A blurry image block1412can blur only the skin part while leaving the background untouched in a block1414resulting in the image1416. The blurry image block1412can also blur the entire image in a block1418resulting in the image1420. An acceptable image block1422can keep the skin area602untouched and blur the non-skin areas of the image in a block1424, resulting in a first acceptable image1426. The acceptable image block1422can keep the whole image untouched1428, resulting in a second acceptable image1430.

It has been discovered that the training of the image quality check module404can be trained by manipulating the patient image114to intentionally have less that acceptable conditions in a predictable manner by the bad luminosity block1402and the blurry image block1412for training purposes. This technique can provide balanced analysis of the patient image in normal operation of the image quality check module404.

Referring now toFIG.15, therein is shown an example of a patient image114processed with data augmentation1401and a classification1501in an embodiment.FIG.15depicts example outputs of the data augmentation flow ofFIG.14such that the compute system100ofFIG.1, the acne diagnostic module116ofFIG.1, or a combination thereof can also perform other data augmentation1401, such as rotation1502, transposition1504, RGB shift1506, Horizontal and Vertical flip and cutout. The leftmost image1502ofFIG.15is classified as acceptable. The middle image1504ofFIG.15is classified as blurry. The rightmost image1506ofFIG.15is considered bad luminosity.

Referring now toFIG.16andFIG.17, therein is shown an example of a block diagram1601of a portion of the image quality check module404in an embodiment and is an example of a more detailed block diagram of a portion of the image quality check module404in an embodiment. The image quality check module404was trained with MobileNet architecture, which is a light model. Indeed, our MobileNet model has only 2M trainable parameters. As the following figureFIG.16andFIG.17shows, the use of MobileNetSmall minimalistic pre-trained1701with ′imagenet′ weights, and had a few more layers afterwards included Global Average Pooling (GAP), Fully Connected Layers, and Dropout.

As an example, depicted inFIG.16, the image quality check module404with patient image114of size 1024×1024×3 is input in a block1602. The encoding of the patient image114can be reduced to 512×512×16 in a block1604, further reduced to 256×256×72 in a block1606. The reduction in size divides the pixels of the patient image114to expose details. The image is further reduced to 128×128×96 in a block1608, then reduced to 64×64×288 in a block1610, and finally reduced to a target pixel array411, such as 32×32×1024 pixel array411. The target pixel array411is processed by a GAP block1614. The GAP block1614averages the pixel values in each of the target pixel array411to produce 1024 averages representing the patient image114.

The output of the GAP block1614goes through a dropout process1616to disregard certain nodes in a layer at random during training. The dropout process1616is a regularization approach that prevents overfitting by ensuring that no units are codependent with one another. The remaining data is processed by a fully connected layer block1618, which is a neural network in which each neuron applies a linear transformation to the input vector through a weights matrix. As a result, all possible connections layer-to-layer are present, meaning every input of the input vector influences every output of the output vector. The output of the fully connected layer block1618provides an indication of normal, brightness, or blurriness to qualify the patient image114as acceptable or unacceptable.

Referring now toFIG.17, therein is shown an example of a more detailed block diagram1701of a portion of the image quality check module404in an embodiment. The more detailed block diagram1701depicts an exemplary block of the Image quality check module404contains Conv2D and DepthWise Conv2D as well as Batch normalization and Padding.

A batch normalization module1702provides normalization of the layers' inputs by re-centering and re-scaling. The batch normalization module1702provides a method used to make training of artificial neural networks faster and more stable. Batch normalization is a technique for training very deep neural networks that standardizes the inputs to a layer for each mini-batch. This has the effect of stabilizing the learning process and dramatically reducing the number of training epochs required to train deep networks. A Conv2d module1704provides a filtering of the input image. The Conv2d module1704can be scanned across the entirety of the patient image114as measured by height and width in pixels.

A Rectified Linear Unit (ReLU)1706is an activation function that introduces the property of non-linearity to a deep learning model and solves the vanishing gradients issue. The ReLU1706can help to prevent the exponential growth in the computation required to operate the neural network. A DepthWise conv2D module1708is a type of convolution that applies a single convolutional filter for each input channel in order to reduce the number of parameters being convolved. The DepthWise conv2D module1708can prevent a proliferation of parameters submitted to the convolutional neural network.

A conv2D module1710creates a convolution kernel that is convolved with the layer input to produce a tensor of outputs. Unlike the DepthWise conv2D module1708, the conv2D module1710applies the same kernel to all channels of the input image resulting in a single pixel per input frame. An adder module1712can combine the originally processed image with the highly filtered details added.

Referring now toFIG.18, therein are shown examples of performance graphs1801of the image quality check module404in an embodiment. The performance graphs1801include a loss function graph1802and an accuracy graph1804. Regarding the loss function graph1802and metrics for the image quality check module404, a Cross Entropy loss function is utilized and expressed as:

Loss=-∑i=1sizeoutputyi·log⁢y^i(4.1)

The loss function graph1802depicted inFIG.18is the evolution of the loss function declining as the number of epochs increases. The vertical scale of the loss function graph1802is a loss value as determined by equation 4.1. The horizontal scale of the loss function graph1802is the number of epochs used to train the image quality check module404. A single epoch can include 40,000 to 50,000 samples of images having different quality.

The accuracy graph1804depicted inFIG.18is the evolution of the accuracy metric increasing as the number of epochs increases. The vertical scale of the accuracy graph1804is accuracy of the predicted value of image quality as compared to the actual quality of the input image. The horizontal scale of the accuracy graph1804is the number of epochs used to train the image quality check module404.

It has been discovered that the convoluted neural network of the image quality check module404can maintain low loss and high degrees of accuracy of validation data after200epochs of training data has been processed. The stability of the validated data over the additional200epochs demonstrates the ability of the image quality check module404to properly determine the quality of the patient images114.

Referring now toFIG.19, therein are shown examples1901of patient images114for the skin segmentation module406in an embodiment. The example1901of the skin dataset depicted inFIG.19includes four samples of the 4516 images with smallest size of 117×99 pixels and maximum size of 5462×4096 pixels. The data set include variety in resolution images (both very close up shot to wide shot) as well as wide range of skin tone from white to darker skin. The examples1901included in the dataset of 4516 images are images of face and body with hair (3565), images of sick skin (667), images of nude body without hair (193), images of tattooed skin (92).

The data set was annotated by annotators using GNU Image Manipulation Program (GIMP). After annotation work, the data set was double checked to ensure that there is not error in the training set. In this example, the compute system100ofFIG.1, the acne diagnostic module116ofFIG.1, or a combination thereof performs resizing or random cropping each image to a size of 384×384 pixels and many other techniques in data augmentation to enrich the data set.

Regarding data augmentation1401, skin segmentation module406can provide an optional augmentation. For training set, firstly, the compute system100, the acne diagnostic module116, or a combination thereof can apply random flipping, random rotating, random changing brightness then the compute system100, the acne diagnostic module116, or a combination thereof can apply random re-scaling, random cropping, random padding, random shifting such that finally we get output of size 384*384 pixels. The images and the masks are applied with the same augmentations. For a validation set, the compute system100, the acne diagnostic module116, or a combination thereof, simply resize both images and their masks to 384*384 pixels which is the size of the model's input.

Referring now toFIG.20, therein is shown an example of a processing flow2001of the skin segmentation module406in an embodiment.FIG.20depicts an example of an implementation of the skin segmentation module406as a convoluted neural network with 508 layers with 25.7 million parameters.

Regarding the loss and metric to measure accuracy of the skin segmentation module406, as an example, the metric to measure accuracy/performance of the model is Jaccard score. The Jaccard score measures the similarity between two sets of data to see which members are shared and distinct.

Regarding accuracy, the compute system100ofFIG.1, the acne diagnostic module116ofFIG.1, or a combination thereof can be was trained for 200 epochs with 4516 images in each epoch, about 80% data set is reserved for training set and the rest for validation. To measure the accuracy of the model, the metric utilized is Jaccard score:

J⁡(A,B)=❘"\[LeftBracketingBar]"A⋂B❘"\[RightBracketingBar]"❘"\[LeftBracketingBar]"A⋃B❘"\[RightBracketingBar]"(5.1)

The Jaccard score is a value between 0 and 1. The value of 0 indicates the data sets are completely different and a value of 1 indicates the data sets are identical.

As an example, the loss function of the skin segmentation module406can be defined as:

l⁡(y,y^)=1-∑i,j⁢f1∑i,j⁢f2(5.2)f1=1-(1-y1)k,f2=(1-y2)h,y1=y⁢y^,y2=(1-y)⁢(1-y^)
where,h=2.5, k=2.5, y is the true label and ŷ is the prediction. The sum in the formula of the loss function l is taken over two indices: index i stands for index of data sample {yi}i, index j stands for the component of the vector label yi=(yij)j. This loss considers not only the mask (y and ŷ) but also the inverse mask (1−y and 1−ŷ). By varying parameters h, k we could have many different versions of this loss function for different purposes. For example, if we reduce k and increase h, we will increase the priority area of pixels valued 1 (standing for skin area602) compared to area of pixels valued 0 (standing for non-skin area). In this task, we take h=k=2.5 to somehow balance priority for the skin area602compared to non-skin area.

The skin segmentation module406receives an input image2002in a 384×384×3 format. The image is processed to reduce2004the input image2002to a 192×192×144 format to divide the image into smaller segments in order to expose details of the image. The differentiation of skin area2006and the non-skin area2008are the result of the frame reduction process. The frame reduction process continues down to the target pixel array411, such as the 12×12×1632 pixel array411, then is processed through a convoluted neural network to increase the number of channels, before converging the data from 12×12×488 back to a single frame of 384×384 to differentiate the skin area2006and the non-skin area2008.

Referring now toFIG.21, therein is shown an example of a block diagram for a portion2101of the skin segmentation module406in an embodiment.FIG.21depicts the architecture of the skin segmentation module406including one block contains 2D convolution, batch normalization, global average pooling and a skip connection.

The batch normalization module1702provides a method used to make training of artificial neural networks faster and more stable. A Fixed Dropout module2102is a mask that nullifies the contribution of some neurons towards the next layer and leaves unmodified all others. The adder module1712can combine the originally processed image with the highly filtered details added. An activation/swish module2104is a filter that decides whether a neuron should be activated or not to decide whether the neuron's input to the network is important or not in the process of prediction using simpler mathematical operations. A reshape module2106will return a new tensor which will have same values as tensor in the same order, except the shape will be different. A Conv2D/Swish module2108is a 2D convolution of a layer with only certain elements propagated. A Conv2D/Sigmoid module2110is a 2D convolution of a layer with a curve flattening at the high and low extremes. A multiply module2112can perform matrix multiplication between the two input images.

It has been discovered that the skin segmentation module406can process the input image to detect and separate the skin area2006and the non-skin area2008by applying neural network processes including the Fixed Dropout module2102, the activation/swish module2104, the reshape module2106, the Conv2D/Swish module2108, the Conv2D/Sigmoid module2110, and the multiply module2112.

Referring now toFIG.22, therein is shown an example of functional scaling2201representation for a portion of the skin segmentation module406in an embodiment.FIG.22depicts an example of an implement of the skin segmentation module406with EfficientNet. EfficientNet is a convolutional neural network architecture and scaling method that uniformly scales all dimensions of depth, width and resolution of network resources using a compound coefficient.

The EfficientNet scaling of depth of the network can capture richer and more complex features, width of the network can capture the fine-grained features and control the ease of training, and resolution of the network can capture more fine-grained features of the patient mage114. The EfficientNet can concurrently scale the depth, width, and resolution.

Referring now toFIG.23, therein is shown an example for the performance graphs2301of the skin segmentation module406in an embodiment. As described inFIG.21, the metric used as an example is the Jaccard score.FIG.23shows the performance graphs2301, as an example, up to 200 epochs of training with 0.901 Jaccard score and binary accuracy is 0.962 for the skin segmentation module406.FIG.23depicts the Jaccard score, binary accuracy and loss of the skin segmentation module406.

Referring now toFIG.24, therein is shown an example of process images2401in processing with the acne segmentation module408in an embodiment.FIG.24depicts an example of the acne segmentation dataset: original image2402, derived label2404and composite image2406of the acne segmentation module408on the original image2402.

Regarding the training of the acne segmentation module408, the training dataset includes 901 images including variety in acne severity, resolution that varies from smart-phone images to professional images, views that include close-up and wide shots, and skin tone including lighter skin to darker skin. These images are split into 5 folds of approximately 180 images each, where 1 fold is designated for validation and the other four for training. Since the compute system100ofFIG.1, the acne diagnostic module116ofFIG.1, or a combination thereof can segment acne and acne-like objects, the compute system100, the acne diagnostic module116, or a combination thereof does not required doctors to do the job. The dataset was annotated by trained annotators. The labeling strategy is as follows: For small acne like comedonal acne, papules and pustules, the annotation is a circle as the normal shape of an acne is a circle. However, for big acne like nodules and cysts, the annotation can be segmented to the exact shape of those acne.FIG.24shows one example in our dataset.

Regarding data augmentation1401for the acne segment module408, the acne segment module408can apply techniques of the data augmentation1401to both the patient images114and the composite image2406. Examples of the techniques include random rotation to an angle between −30 and 30 degrees, random flipping, random changing of brightness, random re-scaling between 0.7 to 1.3 times original size, and crop to the model size, or a combination thereof.

Referring now toFIG.25, therein is shown an example of a processing flow2501of the acne segmentation module408in an embodiment. The acne segmentation module408can be implemented in a number of ways. For example, the architecture of the acne segmentation module408can be U-Net architecture with ImageNet-pretrained EfficientNet B4 encoder. Model inputs is an RGB image2502of size 512×512×3. Model output is a mask2504of size 512×512.FIG.25depicts an example of the architecture as Unet with EfficientNet B4 encoder and input size 512×512×3. The RGB image2502can be reduced to the target pixel array411to create a 16×16×1632 pixel array411for analysis to identify the mask2504.

Regarding loss and metric, as an example, the compute system100ofFIG.1, the acne diagnostic module116ofFIG.1, the acne segmentation module408, or a combination thereof can utilize T-loss function for training and Jaccard score for validation. The T-loss function is defined by:

L⁡(y,y^)=1-1-(1-y⁢y^)k(1-(1-y)⁢(1-y^))h(6.1)

This loss considers not only the mask (y and ŷ) but also the inverse mask (1−y and 1−ŷ). By varying parameters h, k we could have many different versions of this loss function for different purposes. For example, if we reduce k and increase h, we will increase the priority area of pixels valued 1 compared to area of pixels valued 0. In this task, we take h=k=2.5 to balance priority for the 0 and 1 valued area.

Regarding the validation, the compute system100, the acne diagnostic module116, the acne segmentation module408, or a combination thereof can perform the validation with k-fold cross validation with k=5. The acne segmentation module408can be trained on 2 different folds. The final model is the average of the two trained ones, that is, the final prediction is the average of the predictions of the two trained models.

Referring now toFIG.26, therein are shown examples of input images2601for processing by the acne segmentation module408in an embodiment.FIG.26can depict patient images114, a portion of the patient images114, a cropped portion of the patient images114, or a combination thereof. The compute system100ofFIG.1, the acne diagnostic module116ofFIG.1, the acne segmentation module408, or a combination thereof can process the patient images114as shown inFIG.26that can have many acne in each of the patient images114. For example, the acne segmentation module408can focus on the acne at the center of the image. The input images2601are 128×128 pixels in size.

Referring now toFIG.27, therein are shown examples of regions of interest2701for processing by the acne classification module412in an embodiment.FIG.27depicts examples of corresponding images for the regions of interest2701to the RGB images shown inFIG.26.

Referring now toFIG.28, therein is shown an example of a distribution graph2801of the images processed in an embodiment. The distribution graph2801is an example of the images processed for training the acne classification module412. In this example, the acne classification module412can be trained on 12682 images. The distribution graph2801of types of acne and pus information is shown inFIG.28. The compute system100ofFIG.1, the acne diagnostic module116ofFIG.1, the acne classification module410, or a combination thereof take the average score as our ground truth. In the example of the data set shown inFIG.28, there is a wide range of severity of acne, from scar (non-acne) to very severe acne (severe nodule/cyst). Further in the example, distribution depicts the number of images in each class in our dataset. In the distribution classification, the images are defined pustule as a papule with pus. Similarly for cyst, it is nodule with pus.

Regarding data augmentation1401to improve the training of the acne classification module412, some techniques of the data augmentation1401to increase the number of data. Since the images and regions of interest come from previous acne segmentation module408, they are already square, so the step of cropping the image to make it square is skipped for this model. For the training images, we first randomly re-scale and rotate both image and regions of interest, before resizing the two inputs to the model size (128×128), then we apply one of these 3 augmentations: horizontal flip, vertical flip or combination of both with a probability p=0.75. Finally a Gaussian blur with a probability p=0.5 and brightness with a coefficient randomly chosen (between 0.8 and 1.1) is applied to the image. For the validation data, none of these augmentations were applied, we only resized to the model size the images and regions of interest.

Referring now toFIG.29, therein is shown an example of a block diagram2901for a portion of the acne classification module412in an embodiment. The example shown inFIG.29provides a block diagram2901of a model structure and layers or the acne classification module412. As described earlier, in order to train the acne classification module412to focus or identify on the centered lesion, the training dataset include both image2902and regions of interest2904as inputs of the acne classification module412. The training approach for this problem is as a regression not classification. The reason for this training approach is that doctors classify acne differently, especially for middle acne (severe comedonal or mild papule, severe papule or mild nodule). As the result, a scoring system for the compute system100ofFIG.1, the acne diagnostic module116ofFIG.1, the acne classification module412, or a combination thereof for calibrating the acne score430ofFIG.4from 0 to 5 based on scar and mole or non-acne marks rated as 0; micro-comedonal, comedonal rated from 1 to 2 depending on the size of the acne: papule, pustule rated from 2 to 4 depending on the size of the acne: nodule, cyst rated from 4 to 5 depending on the size of the acne. Compiling the acne score430includes accumulating the values from 0 to 5 based on the acne type413ofFIG.4severity grading of the acne pimples409.

As an example, the outputs are the severity score, the presence of pus, and the presence of acne scar. As a specific example, the first output has a reLU activation function and the corresponding values are between 0 for non-acne lesion and 5 for cyst. The two other outputs have a sigmoid activation function as the cases are binary: presence (1) or absence (0) of pus in the lesion, same logic applies to the acne scar output. The base architecture is similar to the Xception architecture. The specific example of the structure of the model is shown inFIG.29. It counts with more than 20M parameters and 162 layers.

The acne classification module412has an image entry2902, which receives the output from the acne segmentation module408, and a region of interest entry2904, which receives the regions of interest2701from the acne segmentation module408. Each of the entry points process in substantially the same channels including the Conv2D1710, a gaussian noise module2906, an activation SELU2908, a dropout module2910, the batch normalization1702, and the adder1712. The two channels are merged through the multiply module2112and added to an enhanced version of the image2902by the adder1712.

An Xception module2912is a convolutional neural network that is 71 layers deep. The Xception module2912can load a pretrained version of the network trained on more than a million images from the ImageNet database. The pretrained network, of the Xception module2912, can classify images into 1000 object categories. A dropout module2910can selectively eliminate specific nodes in a neural network. A dense sigmoid module2914is a layer activation function that can enable layers to the neural network. A dense ReLU module2916is a layer adding function for a neural network. A concatenate module2918can assemble multiple layers into a single layer. A concatenate_1 module2920can output a highlighted layer in the previously assembled layer.

Referring now toFIG.30, therein is shown an example of a block diagram3001for a portion of the acne classification module412in an embodiment.FIG.30depicts an example of a zoom in of the architecture of the acne classification module412. A Fixed Dropout module3002is a mask that nullifies the contribution of some neurons towards the next layer and leaves unmodified all others. As a specific example,FIG.30depicts one block contains 2D convolution1710, batch normalization1702, global average pooling1614and a depthwise conv2D1708. The depthwise conv2D1708is a type of convolution in which each input channel is convolved with a different kernel (called a depthwise kernel).

Referring now toFIG.31, therein is shown an example for the performance graph3101of the acne classification module412in an embodiment. Regarding loss and metric for the acne classification module412, the loss function can be implemented in a number of ways. As an example, the loss function is implemented to handle the different types of outputs of the model using a combination of three losses, defined as follows:

L⁡(Y,Y^)=MSE⁡(y1,y^1)+BCE⁡(y3,y^3)+F⁡(Y,Y^)(7.1)

Where Y=(y1, y2, y3) is the true label and Ŷ=(ŷ1, ŷ2, ŷ3) is the prediction, y1, y2, y3stand for each output, that is respectively score, pus, and scar. MSE is the mean square error, and BCE the binary cross-entropy. The function F, defined for our purpose is given by the following formula:

F⁡(Y,Y^)=-1n⁢∑i=1n(y2⁢i⁢log⁡(y2⁢i^)+(1-y2⁢i)⁢log⁡(1-y2⁢i^))×(2-min⁡(2,❘"\[LeftBracketingBar]"3-y1⁢i❘"\[RightBracketingBar]"))(7.2)

F corresponds to the reduced sum over all samples of the BCE for output pus, multiplied by a term that depend on the true score of the sample. This term allows the compute system100, the acne diagnostic module116, the acne classification model, or a combination thereof to only include this loss where the severity score of the lesion is high enough to consider the pus information.

Describing the different terms of this loss, the first term is the loss for the severity score of each lesion. In other words, the first term addresses regression problem, and a reasonable selection or option to select the MSE loss for the optimization of this output. The second term is the loss for scar classification, which addresses a binary classification problem. The third term is a modified BCE for pus classification. As for scar classification, a classical loss to use would be the BCE, but in the case of pus, this output is linked to the severity of the acne.

Continuing the example, if the acne lesion is too small and benign, as with micro-comedones and comedones (acne lesions with severity score≤1), the pus information is not relevant, the compute system100ofFIG.1, the acne diagnostic module116ofFIG.1, the acne classification module412, or a combination thereof should not be taken into account in the loss value. On the other hand, when the acne lesion is too severe, i.e. it is a cyst (severity score=5), by definition the lesion is filled with pus, so the pus information neither is relevant in that case. In this context, to avoid penalizing the acne classification module412when it is not necessary, the function F, that takes the value 0 when the severity score (y1) is not strictly between 1 and 5. That is possible using the term (2−min (2, |3−y1|)) as the graph of this function depicted inFIG.31. The selection of this term is not unique, only condition being to take the value 0 outside the interval (0, 5), but it allowed the acne classification module to reach a good performance of the model.

As an example, the compute system100, the acne diagnostic module116, the acne classification model412, or a combination thereof utilizes sum these three terms with no extra coefficient before each, meaning that same weight to each output. The training process also selected several coefficients without increasing the performance of the model. Another example is to add more importance to one specific output, regarding new data or new purposes. The metric for the Mean Average Error as in Equation 2.1. The MAE provides an indication how far from the true values the predictions are, either in term of severity score and pus and scar classification. On the validation set, the MAE is on average 0.125.

Regarding validation, in order to measure the performance of our model the compute system100, the acne diagnostic module116, the acne classification module412, or a combination thereof utilizes k-fold Cross-Validation to split our data into 5 sub-samples equally distributed. The validation data corresponds to one of the five sub-samples meanwhile the four others are used as training data. Two models have been trained with different folds for more than 1200 epochs and tested using this technique. For each epoch, 20 000 images were seen by the models. The final model selected as prediction is the average of the two trained models.

Referring now toFIG.32, therein is shown an example of a processing3201of the acne separation module410ofFIG.4in an embodiment.FIG.32depicts the processing by the acne separation module410with images where acne right next to the other be segmented as one. As an example, the acne separation module410can utilize geometrical methods and some mathematical analysis to separate the acne pimples409and classify them individually.

The segmented image407can be augmented with the markers308identifying the location and radius of each of the acne pimples409. The acne classification module412ofFIG.4can identify the acne type413ofFIG.4in order to support the assembly of the user interface display301ofFIG.3.

Referring now toFIG.33, therein is shown an example of a region of interest processing3301of different locations by the acne separation module410in an embodiment. In this example, the working premise can be that the shape of acne is circular. The compute system100ofFIG.1, the acne diagnostic module116ofFIG.1, or a combination thereof can annotate an acne by a circle that is a minimum circle that cover all of the acne.

Continuing with the example fromFIG.32andFIG.33, the acne segmentation module410can be trained with the dataset with the circle. The acne segmentation module410can be segment by a circle. In this example depicted dinFIG.32, the two acne shown are close to each other and returns2overlay circles as depicted inFIG.33.

As a specific example, in the case of two acne next to the other depicted inFIG.33, the acne segmentation module406can segment two overlapping circles. First example shown inFIG.33is two separated acne. The second example shown inFIG.33is two overlapping acne presented in different colors. The third example shown inFIG.33is how the prediction looks like in case of two overlapping acne.

Referring now toFIG.34, therein is shown an example of a stage3401of processing by the acne separation module410in an embodiment. The acne separation module410can be implemented in a number of ways.

For example, a multi-step process can be implemented. The acne separation module can determine whether the given region is made by one or more small circles. The output of the determination step is either nothing (in case of one circle) or at least two smaller regions.FIG.34show the result of the determination step of the above example. As a specific example, the output of the determination is two smaller regions. InFIG.34, the image in the left is input, the middle image is the result during the process and the last one is the output.

Referring now toFIG.35, therein is shown an example of a further stage3501of processing by the acne separation module410in an embodiment. Continuing with the example from the determination process, a determination process for the small regions can also be applied. The determination process can repeat until a determination process returns empty for the output regions.

FIG.35depicts an example for two smaller regions give the two smaller regions given in the earlier figure and depicts how the result look on the original image. As a specific example, after applying the step one for each output (the image on the left and the middle one), the acne separation module generates the segmented image407identifying a plurality of the acne pimple409that are adjacent by applying a marker308on each of the plurality of the acne pimple409with overlapping of the markers308, as shown in the result as the image on the right. The following are our pseudocode algorithm 1 and 2.

Algorithm 1 To determine whether this is a regionor more. Return new regions and radiusInput = contour, shape of image, Outputs = contours, r_accumulates = min_area, t = draw_thickness, r_accumulate = 0img = draw_contour(contour, thickness = fill)a = area(contour)if a ≤ s then return [ ], 0elsecontours = find_contour(img)while length(contours) == 1 domake contour thinner with amount tr_accumulate += tcontours = find_contour(img)end whileend ifif length(contours) == 0 or 1 then return [ ], 0elsereturn contours, r_accumulateend if

Algorithm 2 To find all center and radius of acne on a given imageInput = image, Outputs = centers_list, radius_listcontours_list = find_contour(image)regions_list = [ ]radius_did_list = [ ]for contour ∈ contours_list doa = area(contour)temp_contour_list = [ ]temp_r = [ ]small_regions, radius = Algorithm1(contour,image_shape)if length(small_regions) == 0 thenregions_list += small_regionsradius_did_list += radiuselsefor region ∈ small_regions dotemp_contour_list += regiontemp_r += radiusend forwhile length(temp_contour_list) != 0 dosub_contour = temp_contour_list[0]sub_r = temp_r[0]small_regions, radius = Algorithm1(sub_contour,imageshape)if length(small_regions) == 0 thenregions_list += small_regionsradius_did_list += radiuselsetemp_contour_list += sub_contourtemp_r += radius + sub_rend ifDelete temp_contour_list[0], temp_r[0]end whileend ifend forcenters_list = [ ]radius_list = [ ]for region, radius_did ∈ zip(regions_list, radius_list) docenter, radius = minEnclosingCircle(region)centers_list += centerradius_list += radius + radius_didend forreturn centers_list, radius_list

The acne area score module process inputs from the skin segmentation module, the acne segmentation module, the acne classification module, or a combination thereof along with the processing by the acne separation module. The pseudocode is as below Algorithm 3 and 4.

Algorithm 3 To compute the area of each acneInput = Center and Radius, Acne segmentation, Outputs = areaacne = draw_circle(Center, Radius)acne = acne ∩ acne_segmentationa = area(acne)return a

Algorithm 4 To compute the area of each type of acneInput = Acne classification, Acne segmentationOutputs = area_listarea_list = [ ]for list_of_acne ∈ Acne_classification_output doacne = zeros_like(image_shape)for Center, Radius ∈ list_of_acne dodraw_circle(Center, Radius)end foracne = acne ∩ acne_segmentationa = area(acne)area_list += aend forreturn area_list

Referring now toFIG.36, therein is shown an example of the performance graph3601of the compute system with acne diagnostic mechanism116ofFIG.1in an embodiment.FIG.36depicts an example of an embodiment with ROC. The ROC curve is for classification of inflammation vs non-inflammation acne. Area under the curve (AUC) 0.84 (over 1.00)FIG.36in image case where the compute system100ofFIG.1, the acne diagnostic module116ofFIG.1, or a combination thereof can compute the average true positive rate (TPR) and false positive rate (FPR) for each image and aggregate them to compute AUC.

Referring now toFIG.37, therein is shown an example of the further performance graph3701of the compute system100ofFIG.1with acne diagnostic mechanism in an embodiment.FIG.37depicts an example with the receiver operating characteristic (ROC) curve in a lesion analysis where each acne is treated independently. In the lesion case where the compute system100, the acne diagnostic module116ofFIG.1, or a combination thereof treat each acne independently.FIG.37depicts an example with the area under the curve (AUC) equal to 0.69. The AUC indicates the ability to diagnose patients with and without the disease or condition based on the test, where 0.5 suggests no discrimination, 0.7 to 0.8 is considered acceptable, 0.8 to 0.9 is considered excellent, and more than 0.9 is considered outstanding. The reason the compute system100, the acne diagnostic module116, or a combination thereof computes the AUC score in two cases is that in some image, the quality is not good so the investigators may make mistakes. If the mistakes are on a bad quality image and those images have a high number of acne, the overall score is affected too much.

Referring now toFIG.38, therein is shown an example of the yet further performance graph3801of the compute system100ofFIG.1with acne diagnostic mechanism relative to investigators in an embodiment.FIG.38depicts an example where the different number in acne count between acne AI118ofFIG.1and investigators is compared.FIG.38depicts an example as a histogram plot, where 89.06% images have a difference in acne count less than 2 and 96.09% images have the difference less than 3. For each image, the compute system100, the acne diagnostic module116ofFIG.1, or a combination thereof can count the number of acne by acne AI118and by investigator and then compare the different in number in these two. In all cases the acne AI118is more capable than the investigators.

Referring now toFIG.39, therein is shown an example chart3901of the classification and number of acne in an embodiment.FIG.39depicts the example chart3901, that classifies the level of acne based on number of acne (does not take into account the type of acne). In an example of a public dataset ACNE04, ANCE04 includes 1457 images, 18,983 lesions (number of lesions labeled by experts), images classified by acne severity (level 0 mild indicated in blue, level 1 moderate indicated in green, level 2 severe indicated in brown, level 3 very severe indicated in tan), corresponding to the number of acne lesions on the image being analyzed, 95.09% of labeled lesions detected by the compute system100, the acne diagnostic module116, or a combination thereof.

Analysis of the ACNE04 dataset, the compute system100, the acne diagnostic module116, or a combination thereof determined that bounding boxes from experts are not lesion centered and/or too big with respect to the lesion; sometimes lesions are in the same bounding box; lesion are sometimes missed by experts; some images are low quality; and some images are not acne.

Referring now toFIG.40, therein is shown an example of another performance graph4001of the compute system100ofFIG.1with acne diagnostic mechanism in an embodiment.FIG.40depicts an example of ROC curve representing the True Positive (TP) rate in function of the False Positive (FP) rate for different thresholds of classification. Higher the AUC (Area Under the Curve) is better as the classifier. The example depicts ROC curve based on TP 0.67, where the acne diagnostic mechanism detects a labeled lesion, and FP 0.19 where the acne diagnostic mechanism detects something but it is not a labeled lesion.

The high FP rate can be explained by the fact that some of the acne are not labeled in the dataset but the acne diagnostic mechanism still detects it, showing good performance. The not so high TP rate show that the acne diagnostic mechanism missed some lesions which is a weakness. However, it is also partially due to the annotations of the experts, who sometimes consider scar or discoloration as acne. The acne diagnostic mechanism performs well in detection when the lesion is not completely flat but elevated.

Referring now toFIG.41, therein is shown an example of a graphical representation of severity scores4101of the compute system100ofFIG.1with the acne diagnostic mechanism426ofFIG.4in an embodiment. ACNE04 dataset has 4 levels of severity and the acne AI118ofFIG.1of the acne diagnostic module116gives a continuous severity score for each image with a value from 0 to 100. ICC 0.74 with 95% CI [0.72, 0.77].

FIG.41shows the boxplots of AI severity4101of images given their ACNE04 severity level. The AI severity increases following the ACN04 severity level. For level 0, AI severity is low, and for level 3, severity is high. Other observation is that the dispersion of values is more important for higher levels. This can be explained because ACNE04 level of severity only depends on the number of lesions while our severity is calculated using the number, size and type of acne (comedones, cysts, papules, etc.).

Referring now toFIG.42, therein is shown an example of a further graphical representation of severity scores4201of the compute system100ofFIG.1with acne diagnostic mechanism in an embodiment.FIG.42depicts an example of a scatter plot on the severity level by ACNE04 (value from 0-3) and severity detected by Acne AI118ofFIG.1(value from 0-100).FIG.42depicts the severity 0 by ACNE04, severity detected by Acne AI118is low and in the severity 3 by ACNE04, severity detected by Acne AI118is high.

Referring now toFIG.43, therein is shown an example of a yet further graphical representation of severity scores4301of the compute system100ofFIG.1with acne diagnostic mechanism in an embodiment.FIG.43depicts an example of ROC curve for level 0 and 3 (minimum and maximum) following ACNE04. Since we do not have a conversion between our system and their system, a plot for middle level cannot be done. ROC curve and AUC of level 0 depicted in green and level 3 depicted in tan. The accuracy of level 0 is 0.88 and level 3 is 0.97. Best threshold for severity 0 is 7.7 with TP 0.83 and FP 0.3. Best threshold for severity 3 is 37.1 with TP 0.91 and FP 0.06.

Referring now toFIG.44, therein is shown an example of a block diagram of data augmentation4401of the compute system100ofFIG.1with acne diagnostic mechanism in an embodiment. Data augmentation4401adds diversity to the image dataset by applying random transformations to the images under certain rules. By doing this, the final model is more likely to generalize, which means that it will work properly under different conditions with better performance and accuracy, not only the type of image that it was trained on. Data augmentations4401to the image dataset applied during the training process and the original images are modified (flipped, rotated, add some random noise, add changes in brightness, etc.) before being sent as an input image4402to the acne diagnostic mechanism. The data augmentation4401is applied without the human intervention and using random or selected parameters or both. An augmentation4404is applied to the input image4402. An augmented image4406is created for submission to a deep neural network4408for processing. A predicted result4410is output from the deep neural network4408.

Referring now toFIG.45, there in shown an example of images for flipping augmentation4501for data augmentation4401ofFIG.44of the compute system100ofFIG.1with acne diagnostic mechanism in an embodiment.FIG.45depicts examples of images with transformation to flip an original image4502. The flip of the original image4501can be horizontal flipping4504or vertical flipping4506or composition of both4508(horizontal and vertical). The orientation of the flip is chosen at random between: none, vertical, horizontal, both in order to randomize the training.

Referring now toFIG.46, there in shown an example of images for rotation augmentation4601for data augmentation4401ofFIG.4of the compute system100ofFIG.1with acne diagnostic mechanism in an embodiment.FIG.46depicts examples of images with transformation of rotation. This transformation applies a rotation of angle α4602randomly chosen between 0 and 180°. The acne diagnostic mechanism is capable of recognizing skin, body parts and skin lesions from pictures taken under real life conditions. Images taken in real life do not always appear with the same angle or viewpoint. For example, two pictures of a hand taken at different moments do not necessary show the hand in the same position. The rotation transformation allows generation of pictures with multiple angles, making it possible to train the model using more realistic data.

Referring now toFIG.47, there in shown an example of images for brightness augmentation4701for data augmentation4401pfFIG.4of the compute system100of FIG.1with acne diagnostic mechanism in an embodiment.FIG.47depicts examples of images with transformation of varying brightness augmentation4701. In real conditions, a person can take a picture under different light exposure. As a consequence, the image can have more or less brightness. The acne diagnostic mechanism functions and performs correctly under different lighting conditions. The acne diagnostic mechanism can be trained by adding different level of brightness to input images4702. At each brightness augmentation4701with this function, the brightness of the image changes by an amount that is randomly chosen in the range [−Blim, +Blim], where Blim represents the extremes of brightness that still allows the input images4702to be recognized. To emphasize its effect, the examples depicted inFIG.47were generated setting Blim=0.3.

Referring now toFIG.48, there in shown an example of images for resize augmentation4801for data augmentation4401ofFIG.4of the compute system100ofFIG.1with acne diagnostic mechanism in an embodiment.FIG.48depicts examples of an original image4802with transformation of different or rescaling of the image. As examples, the images are taken with different devices, at different distances or mimic that effect. The images being augmented or part of the dataset include images with different sizes. The re-scaling transformation changes the size of the original image4802(in the same way as a zoom in or zoom out).

As an example, the resize augmentation4801is done by resizing image to a random size chosen in the range [l1*size, l2*size], where size is the model input size, and l1, l2are lower limit and upper limit. For example l1=0.8, l2=1.2. If the resized image's size is lower than model size, random padding to get a squared image: fill left, right, upper or bottom border with as much zeros as needed to make the image square. Otherwise, random crop to the model's input size: choose a random point in the image and consider it as the center of a size*size square.FIG.48shows an example of resizing to a smaller size than the model's input size as shown in the difference in size of the images.

Referring now toFIG.49, there in shown an example of images for pad augmentation4901for data augmentation4401ofFIG.4of the compute system100ofFIG.1with acne diagnostic mechanism in an embodiment.FIG.49depicts another example of resizing or rescaling when the image size is smaller than the size*size (in this example 384×384), random pad is added to increase the size of the resize image4904to create a padded resized image4902. The padded resized image4902can have a border added to the perimeter of the resize image4904.

Referring now toFIG.50, there in shown an example of images for resize augmentation5001for data augmentation4401ofFIG.4of the compute system100ofFIG.1with acne diagnostic mechanism in an embodiment.FIG.50depicts examples of resizing/rescaling to resize to a bigger size than the model's input size, such as 384×384.FIG.50is an example with random crop to the model's input size or greater. In this example, the random resize5002included one dimension that is below the input standard.

Referring now toFIG.51, there in shown an example of images for crop augmentation5101for data augmentation4401ofFIG.4of the compute system100ofFIG.1with acne diagnostic mechanism in an embodiment.FIG.51depicts examples of resizing/rescaling5104to resize to a bigger size that can meet the model's input size, such as 384×384. The random crop5102can be padded and cropped to meet the minimum input size of 384×384.

Referring now toFIG.52, there in shown an example of images for shifting augmentation5201for data augmentation4401ofFIG.4of the compute system100ofFIG.1ofFIG.1with acne diagnostic mechanism in an embodiment.FIG.52depicts examples of images with transformation of shifting of the image.FIG.52depicts an example of the images shifting to the right or to the left.FIG.52depicts shifting augmentation5201with different levels. The same translation can be done from the right to left direction. The maximum number of pixels to shift the image is defined by Nmaxand each time the augmentation is called, the actual number of pixels to shift the image is randomly sampled in the range [0, Nmax].

Referring now toFIG.53, there in shown an example of images for distortion augmentation5301for data augmentation4401ofFIG.4of the compute system100ofFIG.1with acne diagnostic mechanism in an embodiment.FIG.53depicts examples of images with transformation of distortion of the image. In photography, distortion is particularly associated with zoom lenses, particularly large-range zooms, but may also be found in prime lenses. This transformation mimics this effect using albumentation's optical distortion function as well as the elastic transform function. The original image4502can be subjected to optical distortion5302, elastic distortion5304, or a combined transformation5306.

Referring now toFIG.54, there in shown an example of images for cutout augmentation5401for data augmentation4401ofFIG.4of the compute system100ofFIG.1with acne diagnostic mechanism in an embodiment.FIG.54depicts examples of images with transformation of cut-out of the image. The data augmentation4401with cut-out allows to mask random patches of the image with cutout augmentation5402. After applying this augmentation, the resulting image includes with many different small black squares on it. The size of the square is set randomly with the maximum value depends on the model's input size.

Referring now toFIG.55, there in shown an example of images for noise augmentation5501for data augmentation4401ofFIG.4of the compute system100ofFIG.1with acne diagnostic mechanism in an embodiment.FIG.55depicts examples of images with transformation of noise, such as Gaussian noise, to the image. This augmentation seeks at mimicking the camera noise that is sometimes visible when pictures are taken with bad quality cameras (like smart-phones cameras for example). The term ‘Gaussian’ refers to the fact that each pixel will be perturbed with respect to a Gaussian distribution of mean μ and variance σ. To add such noise into images, we first choose the maximum amount of noise desired, denoted by σmax, and then we randomly sample its intensity between 0 and σmax. The specific examples shown inFIG.55is with Gaussian noise augmentation with different levels of noise. σmax=50.

Referring now toFIG.56, there in shown an example of images for blur augmentation5601for data augmentation4401ofFIG.4of the compute system100ofFIG.1with acne diagnostic mechanism in an embodiment.FIG.56depicts examples of images with transformation of blur, such as Gaussian blur, to the image. This augmentation seeks at adding blur to the image. As a more specific example, a specific type of blur called Gaussian blur is used for data augmentation. Mathematically, applying a Gaussian blur to an image is the same as convolving the image with a Gaussian kernel, whose size defines the level of blur. The greater the kernel size, the more blur we add. In the training pipeline, the kernel size is randomly chosen between 0 and kmax=27. The specific examples shown inFIG.56is with Gaussian blur augmentation with different kernel sizes.

Referring now toFIG.57, there in shown an example of images for hue/saturation augmentation5701for data augmentation4401ofFIG.4of the compute system100ofFIG.1with acne diagnostic mechanism in an embodiment.FIG.57depicts examples of images with transformation of hue, saturation, or both to the image. This data augmentation randomly change hue, saturation and value of the input image. As examples, there are 3 parameters that are randomly sampled for each image, defining the intensity of this augmentation: range for changing hue between −20 and 20: range for changing saturation between −30 and 30; range for changing value of pixels between −20 and 20. The specific examples shown inFIG.57is with different hue/saturation and value changes.

Referring now toFIG.58, there in shown an example of images for RGB augmentation5801for data augmentation4401ofFIG.4of the compute system100ofFIG.1with acne diagnostic mechanism in an embodiment.FIG.58depicts examples of images with transformation of RGB shift to the image. This data augmentation4401perturbs each of the three color channels' pixels by a randomly chosen number in the range [−Nmax,c, +Nmax,c], where c denotes the channel (R. G or B). During the training, we set Nmax,R=Nmax,B=Nmax,B=2. However, to emphasize its effect, the following examples were generated by setting Nmax,R=Nmax,B=Nmax,B=20.

Referring now toFIG.59, there in shown an example of images for Contrast Limited Adaptive Histogram Equalization (CLAHE) augmentation5901for data augmentation of the compute system with acne diagnostic mechanism in an embodiment.FIG.59depicts examples of images with CLAHE transformation5902to the image. CLAHE allows the equalization of images. CLAHE is a variant of Adaptive histogram equalization (AHE) which takes care of over-amplification of the contrast. CLAHE operates on small regions in the image, called tiles, rather than the entire image. The neighboring tiles are then combined using bilinear interpolation to remove the artificial boundaries. This algorithm can be applied to improve the contrast of images.

Referring now toFIG.60, there in shown an example of images for brightness-contrast augmentation6001for data augmentation4401ofFIG.4of the compute system100ofFIG.1with acne diagnostic mechanism in an embodiment.FIG.58depicts examples of images with transformation of brightness-contrast augmentation6001to the image. In addition to changing brightness of the image, this augmentation also changes its contrast by an amount randomly chosen in the range [−Clim, +Clim]. During the training phase, we set Clim=0.15, but to emphasize its effect, the following examples were generated with Clim=0.3. The specific examples shown inFIG.60is with different amount of contrast.

Referring now toFIG.61, therein is shown an exemplary block diagram of the compute system100in an embodiment. The compute system100can include the first device102, the network104, and the second device106. The first device102can send information in a first device transmission6108over the network104to the second device106. The second device106can send information in a second device transmission6110over the network104to the first device102or the vehicle201ofFIG.2.

For illustrative purposes, the compute system100is shown with the first device102as a client device, although it is understood that the compute system100can include the first device102as a different type of device.

Also, for illustrative purposes, the compute system100is shown with the second device106as a server, although it is understood that the compute system100can include the second device106as a different type of device. For example, the second device106can be a client device. By way of an example, the compute system100can be implemented entirely on the first device102.

Also, for illustrative purposes, the compute system100is shown with interaction between the first device102and the second device106. However, it is understood that the first device102can be a part of or the entirety of an autonomous vehicle, a smart vehicle, or a combination thereof. Similarly, the second device106can similarly interact with the first device102representing the autonomous vehicle, the intelligent vehicle, or a combination thereof.

For brevity of description in this embodiment of the present invention, the first device102will be described as a client device and the second device106will be described as a server device. The embodiment of the present invention is not limited to this selection for the type of devices. The selection is an example of an embodiment of the present invention.

The first device102can include a first control circuit6112, a first storage circuit6114, a first communication circuit6116, a first interface circuit6118, and a first location circuit6120. The first control circuit6112can include a first control interface6122. The first control circuit6112can execute a first software6126to provide the intelligence of the compute system100.

The first control circuit6112can be implemented in a number of different manners. For example, the first control circuit6112can be a processor, an application specific integrated circuit (ASIC) an embedded processor, a microprocessor, a hardware control logic, a hardware finite state machine (FSM), a digital signal processor (DSP), or a combination thereof. The first control interface6122can be used for communication between the first control circuit6112and other functional units or circuits in the first device102. The first control interface6122can also be used for communication that is external to the first device102.

The first control interface6122can receive information from the other functional units/circuits or from external sources, or can transmit information to the other functional units/circuits or to external destinations. The external sources and the external destinations refer to sources and destinations external to the first device102.

The first control interface6122can be implemented in different ways and can include different implementations depending on which functional units/circuits or external units/circuits are being interfaced with the first control interface6122. For example, the first control interface6122can be implemented with a pressure sensor, an inertial sensor, a microelectromechanical system (MEMS), optical circuitry, waveguides, wireless circuitry, wireline circuitry, or a combination thereof.

The first storage circuit6114can store the first software6126. The first storage circuit6114can also store the relevant information, such as data representing incoming images, data representing previously presented image, sound files, or a combination thereof.

The first storage circuit6114can be a volatile memory, a nonvolatile memory, an internal memory, an external memory, or a combination thereof. For example, the first storage circuit6114can be a nonvolatile storage such as non-volatile random-access memory (NVRAM), Flash memory, disk storage, or a volatile storage such as static random-access memory (SRAM).

The first storage circuit6114can include a first storage interface6124. The first storage interface6124can be used for communication between the first storage circuit6114and other functional units or circuits in the first device102. The first storage interface6124can also be used for communication that is external to the first device102.

The first storage interface6124can receive information from the other functional units/circuits or from external sources, or can transmit information to the other functional units/circuits or to external destinations. The external sources and the external destinations refer to sources and destinations external to the first device102. The first storage interface6124can receive input from and source data to the acne diagnostic module116.

The first storage interface6124can include different implementations depending on which functional units/circuits or external units/circuits are being interfaced with the first storage circuit6114. The first storage interface6124can be implemented with technologies and techniques similar to the implementation of the first control interface6122.

The first communication circuit6116can enable external communication to and from the first device102. For example, the first communication circuit6116can permit the first device102to communicate with the second device106and the network104.

The first communication circuit6116can also function as a communication hub allowing the first device102to function as part of the network104and not limited to be an endpoint or terminal circuit to the network104. The first communication circuit6116can include active and passive components, such as microelectronics or an antenna, for interaction with the network104.

The first communication circuit6116can include a first communication interface6128. The first communication interface6128can be used for communication between the first communication circuit6116and other functional units or circuits in the first device102. The first communication interface6128can receive information from the second device106for distribution to the other functional units/circuits or can transmit information to the other functional units or circuits.

The first communication interface6128can include different implementations depending on which functional units or circuits are being interfaced with the first communication circuit6116. The first communication interface6128can be implemented with technologies and techniques similar to the implementation of the first control interface6122.

The first interface circuit6118allows the user112ofFIG.1to interface and interact with the first device102. The first interface circuit6118can include an input device and an output device. Examples of the input device of the first interface circuit6118can include a keypad, a touchpad, soft-keys, a keyboard, a microphone, an infrared sensor for receiving remote signals, or any combination thereof to provide data and communication inputs.

The first interface circuit6118can include a first display interface6130. The first display interface6130can include an output device. The first display interface6130can include a projector, a video screen, a touch screen, a speaker, a microphone, a keyboard, and combinations thereof.

The first control circuit6112can operate the first interface circuit6118to display information generated by the compute system100and receive input from the user112. The first control circuit6112can also execute the first software6126for the other functions of the compute system100, including receiving location information from the first location circuit6120. The first control circuit6112can further execute the first software6126for interaction with the network104via the first communication circuit6116. The first control circuit6112can operate the acne diagnostic mechanism115ofFIG.1.

The first control circuit6112can also receive location information from the first location circuit6120. The first control circuit6112can operate the acne diagnostic module116.

The first location circuit6120can be implemented in many ways. For example, the first location circuit6120can function as at least a part of the global positioning system, an inertial compute system, a cellular-tower location system, a gyroscope, or any combination thereof. Also, for example, the first location circuit6120can utilize components such as an accelerometer, gyroscope, or global positioning system (GPS) receiver.

The first location circuit6120can include a first location interface6132. The first location interface6132can be used for communication between the first location circuit6120and other functional units or circuits in the first device102, including the environmental sensors210.

The first location interface6132can receive information from the other functional units/circuits or from external sources, or can transmit information to the other functional units/circuits or to external destinations. The external sources and the external destinations refer to sources and destinations external to the first device102. The first location interface6132can receive the global positioning location from the global positioning system (not shown).

The first location interface6132can include different implementations depending on which functional units/circuits or external units/circuits are being interfaced with the first location circuit6120. The first location interface6132can be implemented with technologies and techniques similar to the implementation of the first control circuit6112.

The second device106can be optimized for implementing an embodiment of the present invention in a multiple device embodiment with the first device102. The second device106can provide the additional or higher performance processing power compared to the first device102. The second device106can include a second control circuit6134, a second communication circuit6136, a second user interface6138, and a second storage circuit6146.

The second user interface6138allows an operator (not shown) to interface and interact with the second device106. The second user interface6138can include an input device and an output device. Examples of the input device of the second user interface6138can include a keypad, a touchpad, soft-keys, a keyboard, a microphone, or any combination thereof to provide data and communication inputs. Examples of the output device of the second user interface6138can include a second display interface6140. The second display interface6140can include a display, a projector, a video screen, a speaker, or any combination thereof.

The second control circuit6134can execute a second software6142to provide the intelligence of the second device106of the compute system100. The second software6142can operate in conjunction with the first software6126. The second control circuit6134can provide additional performance compared to the first control circuit6112.

The second control circuit6134can operate the second user interface6138to display information. The second control circuit6134can also execute the second software6142for the other functions of the compute system100, including operating the second communication circuit6136to communicate with the first device102over the network104.

The second control circuit6134can be implemented in a number of different manners. For example, the second control circuit6134can be a processor, an embedded processor, a microprocessor, hardware control logic, a hardware finite state machine (FSM), a digital signal processor (DSP), or a combination thereof.

The second control circuit6134can include a second control interface6144. The second control interface6144can be used for communication between the second control circuit6134and other functional units or circuits in the second device106. The second control interface6144can also be used for communication that is external to the second device106.

The second control interface6144can receive information from the other functional units/circuits or from external sources, or can transmit information to the other functional units/circuits or to external destinations. The external sources and the external destinations refer to sources and destinations external to the second device106.

The second control interface6144can be implemented in different ways and can include different implementations depending on which functional units/circuits or external units/circuits are being interfaced with the second control interface6144. For example, the second control interface6144can be implemented with a pressure sensor, an inertial sensor, a microelectromechanical system (MEMS), optical circuitry, waveguides, wireless circuitry, wireline circuitry, or a combination thereof.

The second storage circuit6146can store the second software6142. The second storage circuit6146can also store the information such as data representing incoming images, data representing previously presented image, sound files, or a combination thereof. The second storage circuit6146can be sized to provide the additional storage capacity to supplement the first storage circuit6114.

For illustrative purposes, the second storage circuit6146is shown as a single element, although it is understood that the second storage circuit6146can be a distribution of storage elements. Also, for illustrative purposes, the compute system100is shown with the second storage circuit6146as a single hierarchy storage system, although it is understood that the compute system100can include the second storage circuit6146in a different configuration. For example, the second storage circuit6146can be formed with different storage technologies forming a memory hierarchal system including different levels of caching, main memory, rotating media, or off-line storage.

The second storage circuit6146can be a controller of a volatile memory, a nonvolatile memory, an internal memory, an external memory, or a combination thereof. For example, the second storage circuit6146can be a controller of a nonvolatile storage such as non-volatile random-access memory (NVRAM), Flash memory, disk storage, or a volatile storage such as static random access memory (SRAM).

The second storage interface6148can receive information from the other functional units/circuits or from external sources, or can transmit information to the other functional units/circuits or to external destinations. The external sources and the external destinations refer to sources and destinations external to the second device106.

The second storage interface6148can include different implementations depending on which functional units/circuits or external units/circuits are being interfaced with the second storage circuit6146. The second storage interface6148can be implemented with technologies and techniques similar to the implementation of the second control interface6144.

The second communication circuit6136can enable external communication to and from the second device106. For example, the second communication circuit6136can permit the second device106to communicate with the first device102over the network104.

The second communication circuit6136can also function as a communication hub allowing the second device106to function as part of the network104and not limited to be an endpoint or terminal unit or circuit to the network104. The second communication circuit6136can include active and passive components, such as microelectronics or an antenna, for interaction with the network104.

The second communication circuit6136can include a second communication interface6150. The second communication interface6150can be used for communication between the second communication circuit6136and other functional units or circuits in the second device106. The second communication interface6150can receive information from the other functional units/circuits or can transmit information to the other functional units or circuits.

The second communication interface6150can include different implementations depending on which functional units or circuits are being interfaced with the second communication circuit6136. The second communication interface6150can be implemented with technologies and techniques similar to the implementation of the second control interface6144.

The second communication circuit6136can couple with the network104to send information to the first device102. The first device102can receive information in the first communication circuit6116from the second device transmission6110of the network104. The compute system100can be executed by the first control circuit6112, the second control circuit6134, or a combination thereof. For illustrative purposes, the second device106is shown with the partition containing the second user interface6138, the second storage circuit6146, the second control circuit6134, and the second communication circuit6136, although it is understood that the second device106can include a different partition. For example, the second software6142can be partitioned differently such that some or all of its function can be in the second control circuit6134and the second communication circuit6136. Also, the second device106can include other functional units or circuits not shown inFIG.61for clarity.

The functional units or circuits in the first device102can work individually and independently of the other functional units or circuits. The first device102can work individually and independently from the second device106and the network104.

The functional units or circuits in the second device106can work individually and independently of the other functional units or circuits. The second device106can work individually and independently from the first device102and the network104.

The functional units or circuits described above can be implemented in hardware. For example, one or more of the functional units or circuits can be implemented using a gate array, an application specific integrated circuit (ASIC), circuitry, a processor, a computer, integrated circuit, integrated circuit cores, a pressure sensor, an inertial sensor, a microelectromechanical system (MEMS), a passive device, a physical non-transitory memory medium containing instructions for performing the software function, a portion therein, or a combination thereof.

For illustrative purposes, the compute system100is described by operation of the first device102and the second device106. It is understood that the first device102and the second device106can operate any of the modules and functions of the compute system100.

Referring now toFIG.62, therein is shown a flow chart of a method6200of operation of a compute system100ofFIG.1in an embodiment of the present invention. The method6200includes: detecting a skin area in a patient image in a block6202; segmenting the skin area into a segmented image having an acne pimple at the center in a block6204; generating a target pixel array from the segmented image includes identifying a plurality of the acne pimples that are adjacent in the segmented image in a block6206; identifying an acne characterization of the acne pimples including an area of each acne and an acne score in a block6208; and assembling a user interface display from the acne characterization for displaying on a device in a block6210.

The resulting method, process, apparatus, device, product, and/or system is straightforward, cost-effective, uncomplicated, highly versatile, accurate, sensitive, and effective, and can be implemented by adapting known components for ready, efficient, and economical manufacturing, application, and utilization. Another important aspect of an embodiment of the present invention is that it valuably supports and services the historical trend of reducing costs, simplifying systems, and increasing performance.

These and other valuable aspects of an embodiment of the present invention consequently further the state of the technology to at least the next level.

While the invention has been described in conjunction with a specific best mode, it is to be understood that many alternatives, modifications, and variations will be apparent to those skilled in the art in light of the foregoing description. Accordingly, it is intended to embrace all such alternatives, modifications, and variations that fall within the scope of the included claims. All matters set forth herein or shown in the accompanying drawings are to be interpreted in an illustrative and non-limiting sense.