EXTENDED-REALITY SKIN-CONDITION-DEVELOPMENT PREDICTION AND VISUALIZATION

In general, this disclosure describes techniques for automatically predicting and visualizing a future development of a skin condition of a patient. In some examples, a computing system is configured to estimate, based on sensor data, a skin-condition type for a skin condition on an affected area of a body of a patient; determine, based on the sensor data and the estimated skin-condition type, modeling data indicative of a typical development of the skin-condition type; generate, based on the sensor data and the modeling data, a 3-dimensional (3-D) model indicative of a predicted future development of the skin condition over time; generate extended reality (XR) imagery of the affected area of the body of the patient overlaid with the 3-D model; and output the XR imagery.

TECHNICAL FIELD

The disclosure relates to medical computing systems.

BACKGROUND

A dermatological patient may suffer from a skin condition, such as a rash, burn, abrasion, outbreak, blemish, bruise, infection, or the like.

SUMMARY

In general, this disclosure describes systems and techniques for automatically estimating or identifying a patient's skin-condition type, predicting a future development of the skin condition over time, and visualizing the predicted future development via extended-reality (“XR”) elements. For example, techniques disclosed herein include generating and outputting XR imagery of a predicted future development of a patient's skin condition. The XR imagery may include “live” or “real-time” augmented reality (AR) imagery of the patient's body overlaid with a virtual three-dimensional (3-D) model of the predicted skin condition, or in other examples, a virtual 3-D model of the predicted skin condition overlaid on the patient's actual body as viewed through a transparent display screen.

As one non-limiting example, the techniques of this disclosure include a computing system configured to capture sensor data (including 2-D image data) indicative of a patient's skin condition, feed the collected data through a deep-learning model configured to estimate the skin-condition type, predict a unique future development of the skin condition, and generate and output XR imagery visualizing the predicted future development of the skin condition. In this way, the techniques described herein may provide one or more technical advantages that provide at least one practical application. For example, the techniques described in this disclosure may be configured to provide more accurate and/or comprehensive visual information to a specialist (e.g., a dermatologist).

In some additional aspects, the techniques of this disclosure describe improved techniques for generating the XR elements as compared to more-typical techniques. As one example, the techniques of this disclosure include generating and rendering XR elements (e.g., three-dimensional virtual models) based on 3-D sensor data as input, thereby enabling more-accurate virtual imagery (e.g., 3-D models) constructed over a framework of curved surfaces, as compared to more-common planar surfaces.

In one example, the techniques described herein include a method performed by a computing system, the method comprising: estimating, based on sensor data, a skin-condition type for a skin condition on an affected area of a body of a patient; determining, based on the sensor data and the estimated skin-condition type, modeling data indicative of a typical development of the skin-condition type; generating, based on the sensor data and the modeling data, a 3-dimensional (3-D) model indicative of a predicted future development of the skin condition over time; generating extended reality (XR) imagery of the affected area of the body of the patient overlaid with the 3-D model; and outputting the XR imagery.

In another example, the techniques described herein include a computing system comprising processing circuitry configured to: estimate, based on sensor data, a skin-condition type for a skin condition on an affected area of a body of a patient; determine, based on the sensor data and the estimated skin-condition type, modeling data indicative of a typical development of the skin-condition type; generate, based on the sensor data and the modeling data, a 3-dimensional (3-D) model indicative of a predicted future development of the skin condition over time; generate extended reality (XR) imagery of the affected area of the body of the patient overlaid with the 3-D model; and output the XR imagery.

In another example, the techniques described herein include a non-transitory computer-readable medium comprising instructions for causing one or more programmable processors to: estimate, based on sensor data, a skin-condition type for a skin condition on an affected area of a body of a patient; determine, based on the sensor data and the estimated skin-condition type, modeling data indicative of a typical development of the skin-condition type; generate, based on the sensor data and the modeling data, a 3-dimensional (3-D) model indicative of a predicted future development of the skin condition over time; generate extended reality (XR) imagery of the affected area of the body of the patient overlaid with the 3-D model; and output the XR imagery.

DETAILED DESCRIPTION

A dermatological patient may suffer from a skin condition, such as a rash, burn, abrasion, outbreak, blemish, bruise, infection, tumor, lesions, necrosis, boils, blisters, discoloration, or the like. In the absence of treatment, or similarly, in the presence of incorrect or ineffective treatment (as a result of, for example, incorrect diagnosis), the condition may grow, spread, or otherwise change over time. Advances in artificial intelligence (AI), deep learning (DL), and machine-learning systems and techniques may enable systems to be trained to estimate (e.g., identify, to a certain probability) the skin-condition type or category based on 2-D imagery of the condition. For example, with the development of high-performance graphics processing units (GPUs) and specialized hardware for AI, the machine-learning field may be developed to implement various pattern-recognition architectures in neural networks (NNs) in order to classify (e.g., categorize, label, or identify) a condition based on a two-dimensional (2-D) image of an affected skin area.

According to techniques of this disclosure, a computing system (e.g., one or more computing devices) may be configured to not only estimate a skin-condition type with greater accuracy and precision than existing techniques (e.g., due to, inter alia, a more comprehensive set of sensor-data input), but also to predict and visualize a future development of the skin condition over time. For example,FIG. 1depicts a conceptual diagram of a skin-condition-prediction system100configured to predict and visualize a future development of a skin condition102on an affected area or region104of a body106of a patient108, in accordance with techniques of this disclosure.

In general, system100represents or includes a computing system110configured to estimate (e.g., determine or identify, to a certain probability), based on sensor data, a skin-condition type, label, or category corresponding to skin condition102. Computing system110may further determine (e.g., retrieve, receive, generate, etc.), based on the sensor data and the estimated type of skin condition102, modeling data indicative of a typical development of the estimated type of skin condition102. Computing system110may then generate, based on the sensor data and the modeling data, a three-dimensional (3-D) model indicative of a predicted future development of skin condition102over time; generate extended-reality (“XR”) imagery112of the patient's affected skin area104overlaid with the 3-D model; and output the XR imagery112for display.

As used herein, the term “extended reality” encompasses a spectrum of user experiences that includes virtual reality (“VR”), mixed reality (“MR”), augmented reality (“AR”), and other user experiences that involve the presentation of at least some perceptible elements as existing in the user's environment that are not present in the user's real-world environment, as explained further below. Thus, the term “extended reality” may be considered a genus for MR, AR, and VR.

“Mixed reality” (MR) refers to the presentation of virtual objects such that a user sees images that include both real, physical objects and virtual objects. Virtual objects may include text, 2-D surfaces, 3-D models, or other user-perceptible elements that are not actually present in the physical, real-world environment in which they are presented as coexisting. In addition, virtual objects described in various examples of this disclosure may include graphics, images, animations or videos, e.g., presented as 3-D virtual objects or 2-D virtual objects. Virtual objects may also be referred to as “virtual elements.” Such elements may or may not be analogs of real-world objects.

In some examples of mixed reality, a camera may capture images of the real world and modify the images to present virtual objects in the context of the real world. In such examples, the modified images may be displayed on a screen, which may be head-mounted, handheld, or otherwise viewable by a user. This type of MR is increasingly common on smartphones, such as where a user can point a smartphone's camera at a sign written in a foreign language and see in the smartphone's screen a translation in the user's own language of the sign superimposed on the sign along with the rest of the scene captured by the camera. In other MR examples, in MR, see-through (e.g., transparent) holographic lenses, which may be referred to as waveguides, may permit the user to view real-world objects, i.e., actual objects in a real-world environment, such as real anatomy, through the holographic lenses and also concurrently view virtual objects.

The Microsoft HOLOLENS™ headset, available from Microsoft Corporation of Redmond, Wash., is an example of a MR device that includes see-through holographic lenses that permit a user to view real-world objects through the lens and concurrently view projected 3D holographic objects. The Microsoft HOLOLENS™ headset, or similar waveguide-based visualization devices, are examples of an MR visualization device that may be used in accordance with some examples of this disclosure. Some holographic lenses may present holographic objects with some degree of transparency through see-through holographic lenses so that the user views real-world objects and virtual, holographic objects. In some examples, some holographic lenses may, at times, completely prevent the user from viewing real-world objects and instead may allow the user to view entirely virtual environments. The term mixed reality may also encompass scenarios where one or more users are able to perceive one or more virtual objects generated by holographic projection. In other words, “mixed reality” may encompass the case where a holographic projector generates holograms of elements that appear to a user to be present in the user's actual physical environment.

In some examples of mixed reality, the positions of some or all presented virtual objects are related to positions of physical objects in the real world. For example, a virtual object may be tethered or “anchored” to a table in the real world, such that the user can see the virtual object when the user looks in the direction of the table but does not see the virtual object when the table is not in the user's field of view. In some examples of mixed reality, the positions of some or all presented virtual objects are unrelated to positions of physical objects in the real world. For instance, a virtual item may always appear in the top-right area of the user's field of vision, regardless of where the user is looking. XR imagery or visualizations may be presented in any of the techniques for presenting MR, such as a smartphone touchscreen.

Augmented reality (“AR”) is similar to MR in the presentation of both real-world and virtual elements, but AR generally refers to presentations that are mostly real, with a few virtual additions to “augment” the real-world presentation. For purposes of this disclosure, MR is considered to include AR. For example, in AR, parts of the user's physical environment that are in shadow can be selectively brightened without brightening other areas of the user's physical environment. This example is also an instance of MR in that the selectively brightened areas may be considered virtual objects superimposed on the parts of the user's physical environment that are in shadow.

Furthermore, the term “virtual reality” (VR) refers to an immersive artificial environment that a user experiences through sensory stimuli (such as sights and sounds) provided by a computer. Thus, in VR, the user may not see any physical objects as they exist in the real world. Video games set in imaginary worlds are a common example of VR. The term “VR” also encompasses scenarios where the user is presented with a fully artificial environment, in which the locations of some virtual objects are based on the locations of corresponding physical objects relative to the user. Walk-through VR attractions are examples of this type of VR. XR imagery or visualizations may be presented using techniques for presenting VR, such as VR goggles.

In accordance with techniques of this disclosure, computing system110is configured to generate and output XR imagery112of a predicted future development of skin condition102of patient108. In some examples, XR imagery112may include “live” or “real-time” composite 2-D imagery of the affected region104of the patient's body106, overlaid with a projection of a virtual 3-D model114of the predicted skin-condition development. In other examples, XR imagery112may include the projection of the virtual 3-D model114displayed relative to the affected area104of the patient's actual body106, as viewed through a transparent display screen.

FIG. 2Ais a block diagram of an example computing system200that operates in accordance with one or more techniques of the present disclosure.FIG. 2Amay illustrate a particular example of computing system110ofFIG. 1. In other words, computing system200includes one or more computing devices, each computing device including one or more processors202, any or all of which are configured to predict and visualize a future development of skin condition102of patient108(FIG. 1).

As detailed further below with respect to the example hardware architectures depicted inFIG. 2BandFIG. 4, computing system200ofFIG. 2Amay include one or more of a workstation, server, mainframe computer, notebook or laptop computer, desktop computer, tablet, smartphone, XR display device, datastore, distributed network, and/or other programmable data-processing apparatuses of any kind. In some examples, a computing system may be or may include any component or system that includes one or more processors or other suitable computing environment for executing software instructions configured to perform the techniques described herein, and, for example, need not necessarily include one or more elements shown inFIG. 2A. As one illustrative example, communication units206, and in some examples, other components such as storage device(s)208, may not necessarily be included within computing system200, in examples in which the techniques of this disclosure may be performed without these components.

As shown in the specific example ofFIG. 2A, computing system200includes one or more processors202, one or more input devices204, one or more communication units206, one or more output devices212, one or more storage devices208, one or more user interface (UI) devices210, and in some examples, but not all examples, one or more sensor modules228(also referred to herein as “sensors228”). Computing system200, in one example, further includes one or more applications222and operating system216(e.g., stored within a computer readable medium, such as storage device(s)208) that are executable by processors202of computing system200.

Each of components202,204,206,208,210,212, and228is coupled (physically, communicatively, and/or operatively) for inter-component communications. In some examples, communication channels214may include a system bus, a network connection, an inter-process communication data structure, or any other method for communicating data. As one example, components202,204,206,208,210,212, and228may be coupled by one or more communication channels214. In some examples, two or more of these components may be distributed across multiple (discrete) computing devices. In some such examples, communication channels214may include wired or wireless data connections between the various computing devices.

Processors202, in one example, are configured to implement functionality and/or process instructions for execution within computing system200. For example, processors202may be capable of processing instructions stored in storage device208. Examples of processors202may include one or more of a microprocessor, a controller, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or equivalent discrete or integrated logic circuitry.

One or more storage devices208(also referred to herein as “memory208”) may be configured to store information within computing system200during operation. Storage device(s)208, in some examples, are described as computer-readable storage media. In some examples, storage device208is a temporary memory, meaning that a primary purpose of storage device208is not long-term storage. Storage device208, in some examples, is described as a volatile memory, meaning that storage device208does not maintain stored contents when the computer is turned off. Examples of volatile memories include random access memories (RAM), dynamic random access memories (DRAM), static random access memories (SRAM), and other forms of volatile memories known in the art.

In some examples, storage device208is used to store program instructions for execution by processors202. Storage device208, in one example, is used by software or applications running on computing system200to temporarily store information during program execution. For example, as shown inFIG. 2A, storage device208is configured to store operating system216, skin-condition-types data218, modeling data220, sensor data226, and various programs or applications222, including a skin-condition modeler224, as detailed further below with respect toFIG. 2C.

Storage devices208, in some examples, also include one or more computer-readable storage media. Storage devices208may be configured to store larger amounts of information than volatile memory. Storage devices208may further be configured for long-term storage of information. In some examples, storage devices208include non-volatile storage elements. Examples of such non-volatile storage elements include magnetic hard discs, optical discs, floppy discs, flash memories, or forms of electrically programmable memories (EPROM) or electrically erasable and programmable (EEPROM) memories.

Computing system200, in some examples, also includes one or more communication units206. Computing system200, in one example, utilizes communication units206to communicate with external devices via one or more networks, such as one or more wired/wireless/mobile networks. Communication unit(s)206may include a network interface card, such as an Ethernet card, an optical transceiver, a radio frequency transceiver, or any other type of device that can send and receive information. Other examples of such network interfaces may include 3G, 4G, 5G and Wi-Fi radios. In some examples, computing system200uses communication unit206to communicate with an external device.

Computing system200, in one example, also includes one or more user-interface (“UI”) devices210. UI devices210, in some examples, are configured to receive input from a user through tactile, audio, or video feedback. Examples of UI device(s)210include a presence-sensitive display, a mouse, a keyboard, a voice-responsive system, a video camera, a microphone, or any other type of device for detecting a command from a user. In some examples, a presence-sensitive display includes a touch-sensitive screen or “touchscreen.”

One or more output devices212may also be included in computing system200. Output device212, in some examples, is configured to provide output to a user using tactile, audio, or video stimuli. Output device212, in one example, includes a presence-sensitive display, a sound card, a video graphics adapter card, or any other type of device for converting a signal into an appropriate form understandable to humans or machines. Additional examples of output device212include a speaker, a cathode ray tube (CRT) monitor, a liquid crystal display (LCD), or any other type of device that can generate intelligible output to a user.

Computing system200may include operating system216. Operating system216, in some examples, controls the operation of components of computing system200. For example, operating system216, in one example, facilitates the communication of one or more applications222with processors202, communication unit206, storage device208, input device204, user interface device210, and output device212.

Application222may also include program instructions and/or data that are executable by computing system200. As detailed below with respect toFIG. 2C, skin-condition modeler224is one example of an application222of computing system200. For instance, skin-condition modeler224may include instructions for causing computing system200to perform techniques described in the present disclosure, for example, to predict and visualize a future development of skin condition102of patient108(FIG. 1).

FIG. 2Bis a block diagram depicting individual computing devices of an example hardware architecture of computing system200ofFIG. 2A. As shown inFIG. 2B, computing system200includes a mobile device230, one or more data-streaming devices232, a local server234, and a cloud server236. The example hardware architecture depicted inFIG. 2Bis intended for illustrative purposes only, and is not intended to be limiting. Other architectures of system200ofFIG. 2Ahaving more, fewer, or different computing devices than those depicted inFIG. 2Bmay likewise be configured to perform techniques of this disclosure.

For instance, other examples of hardware architectures of computing system200may include a physically distinct XR-display device, such as an MR or VR headset. In other examples, such as the example depicted inFIG. 2B, the functionality of an XR-display device may be performed by mobile device230, as detailed further below. As another example, the functionality of local server234may be performed by mobile device230. Accordingly, any of the techniques described herein as being performed by either mobile device230or local server234may, in fact, be performed by the other device or by both devices.

Mobile device238may include virtually any mobile (e.g., lightweight and portable) computing device that is local to a user. For example, mobile device238may include a smartphone, tablet, or the like, that includes sensor modules228(or “sensors228”) and a display screen238. As detailed further below, sensors228are configured to capture sensor data226indicative or descriptive of skin condition102of patient108ofFIG. 1. Sensor modules228of mobile device238may include, as non-limiting examples, an inertial measurement unit (IMU)240, a camera244, and in some examples, but not all examples, depth sensor242.

In the specific example depicted inFIG. 2B, IMU240includes a 9-axis IMU including: a 3-axis gyroscope246configured to generate angular rate data indicating a change in position of mobile device230; a 3-axis accelerometer248configured to capture data indicative of an acceleration of mobile device230due to outside forces; and a 3-axis magnetometer250configured to determine the orientation of mobile device230relative to Earth's magnetic field.

In some examples, depth sensor242may include a time-of-flight (TOF)-based depth sensor configured to measure a distance to an object by reflecting a signal off of the object and measuring the duration between transmission of the initial signal and receipt of the reflected signal. In the specific example depicted inFIG. 2B, depth sensor242includes light-detection-and-ranging (LIDAR)250, configured to generate infrared (IR) depth data configured to indicate the distance between the body106of patient108(FIG. 1) and mobile device230.

Camera244is configured to capture standard red-green-blue (RGB) image data. As one non-limiting example, camera244may include an integrated4-Megapixel camera configured to capture images at about30to60frames per second (FPS).

Display screen238, which is an example of UI device210ofFIG. 2A, is configured to output XR content112(FIG. 1) for display to a user of mobile device230. Display screen238may include a touchscreen, transparent visor, or other similar surface configured to display graphical content.

Data-streaming devices232may be examples of communication channels214ofFIG. 2A. As shown inFIG. 2B, data-streaming devices232may include Wi-Fi252, local-area-network (LAN) connections254, and/or other hardware fabricated according to appropriate data-communication protocols to transfer data between the various computing devices of computing system200.

In some examples, but not all examples, local server234may include any suitable computing device (e.g., having processing circuitry and memory) that is physically or geographically local to a user of mobile device230. As one non-limiting example, local server234may include a CUDA-enabled graphics-processing unit (GPU); an Intel i7+ processor; and installed software including Nvidia's CUDA and CUDNN (10.1 or later), Python, C#, and CUDA C++. In other examples, as referenced above, local server234may be integrated within mobile device230, such that mobile device230may perform the functionality ascribed to both devices. In some such examples, local server230may be conceptualized as a “module” (e.g., one or more applications) running on mobile device230and configured to provide a “service” according to techniques of this disclosure.

Cloud server236includes any computing device(s) (e.g., datastores, server rooms, etc.) that are not geographically local to a user of mobile device230and local server234. For instance, in examples in which mobile device230includes an “activated” smartphone, cloud server236may include remote computing servers managed by the telecommunications network configured to provide cellular data to mobile device230, and/or computing servers managed by developers of applications222(e.g., skin-condition modeler224) running on mobile device230.

FIG. 2Cis a block diagram illustrating example software modules of computing system200ofFIG. 2A, and more specifically, illustrating example sub-modules of skin-condition modeler224. For illustrative purposes, and for ease of understanding, the functionality of the software modules ofFIG. 2Care described with reference to the example hardware architecture depicted inFIG. 2B. As shown inFIG. 2C, skin-condition modeler224includes data collector260, mesh builder262, condition estimator264, development predictor266, model generator268, and XR generator270. In other examples, skin-condition modeler224may include more, fewer, or different software components configured to perform techniques in accordance with this disclosure.

In some examples, but not all examples, skin-condition modeler224is configured to passively receive a comprehensive set of sensor data226describing or otherwise indicative of various aspects of skin condition102. In other examples, skin-condition modeler224includes data collector260, a module configured to actively retrieve, aggregate, and/or correlate sensor data226. For example, data collector260may be in data communication with sensors228that are physically integrated within mobile device230(or other computing device of computing system200) and/or other physically distinct sensor modules that are communicatively coupled to computing system200. In some such examples, data collector260is configured to control sensor modules228, e.g., to command the sensors228to generate and output sensor data226.

As described above with respect toFIG. 2B, sensor data226may include a variety of different types of sensor data, such as, but not limited to, motion and orientation data from IMU240, relative depth data from depth sensor242, and 2-D image data from camera244. In some examples, the 2-D image data includes a plurality of overlapping 2-D images of the affected area104of the body106of patient108that collectively define 3-D, arcuate-shaped imagery.

For instance, as illustrated inFIG. 3A, image data of sensor data226may include a plurality of overlapping 2-D images306A-306D (collectively “2-D images306”) captured by camera244of mobile device230, while mobile device230moves along an arcuate-shaped path of motion, such that, when aligned according to their respective location and orientation of capture, the 2-D images306collectively define a conceptual curved surface308. As one non-limiting example, 2-D images306may be collected by a user, such as a clinician of patient108(e.g., a dermatologist), the patient108themselves, or another user of mobile device230, by moving mobile device230in an arcuate (e.g., curved) motion, as indicated by arrows302inFIG. 3A. For instance, the user may move mobile device230along a curved path302that generally correlates to a curvature304of the affected area104of the patient's body106, in order to capture imagery of skin condition102from multiple angles along the curvature304of the patient's body106. For example, the user may revolve mobile device230along a180-degree arc centered on the affected area104of the patient's body106, while keeping the lens of camera244aimed at (e.g., directed toward) the affected area104.

In alternate examples in which camera244is not integrated within mobile device230, data collector260may be configured to control a specialized image-capture device that is specifically designed to capture 2-D images306along an arcuate path of motion. One illustrative example of such an image-capture device is an orthodontist's dental x-ray machine, which revolves an x-ray emitter and an x-ray detector around the curvature of a patient's head while capturing x-ray imagery at a plurality of different positions along the path of motion.

While camera244captures 2-D images306along curved path302, one or more additional sensors228(e.g., IMU240and/or depth sensor242) simultaneously collect other types of sensor data226that may be correlated to the 2-D images306. For example, data collector260may use IMU data from IMU240to determine, for each 2-D image306, a viewing angle (e.g., orientation) of camera244relative to, for example, Earth's gravity and Earth's magnetic field, and by extension, relative to a prior image and/or a subsequent image of 2-D images306. Similarly, data collector260may use depth data from depth sensor242to determine, for each 2-D image306, a relative distance between the affected area104of patient108(as depicted within each 2-D image) and camera244, and by extension, a relative location of mobile device230when each 2-D image306was captured.

In examples in which the individual types of sensor data are not already (e.g., automatically) associated in this way upon capture, data collector260may be configured to correlate or aggregate the various types of sensor data to produce correlated datasets, wherein each dataset includes sensor data226from different types of sensors228, but that was captured at approximately the same instance in time (e.g., within a threshold range or “window” of time). For instance, data collector260may embedded timestamp data in order to produce the correlated datasets. Data collector260may then transfer a copy of the correlated sensor data226to mesh builder262.

In general, as illustrated inFIG. 3B, mesh builder262is configured to use sensor data226to generate, based on sensor data226(e.g., based at least in part on 2-D images306), a virtual, 3-D curved polygon mask320that graphically represents the patient's affected skin area104. For example, mesh builder262may be configured to analyze the 2-D images306in order to identify a plurality of feature points322within the 2-D images306. Feature points322may include virtually any identifiable object or landmark appearing in at least two overlapping images of 2-D images306. In some examples, feature points322may include, as non-limiting examples, a freckle, an edge or outline of the patient's body106, an edge or outline of skin condition102, or a sub-component of skin condition102, such as an individual bump or spot.

After identifying feature points322, mesh builder262may attempt to match corresponding (e.g., identical) feature points across two or more overlapping images of the 2-D images306. In some examples, but not all examples, mesh builder262may then use the relative (2-D) positions of feature points322within the respective 2-D images to orient (e.g., align) the 2-D images306relative to one another, and by extension, the graphical image content (e.g., the patient's affected skin area104) contained within the 2-D images306.

Mesh builder262may use the correlated sensor data226(e.g., depth-sensor data and/or IMU data), to determine a relative 3-D position of each feature point relative to the other feature points322. Mesh builder322may then draw (e.g., define) a virtual “edge” between each pair of adjacent or proximal feature points322, thereby defining a plurality of 2-D polygons324or “tiles” that collectively define 3-D polygon mask320having a curvature that accurately represents (e.g., highly conforms to) the curved geometry304of the affected skin area104of the patient's body.

In this way, mesh builder262reduces an amount of distortion that would otherwise appear in any single 2-D image306depicting skin condition102. For example, analogous to how projecting the surface of a globe onto a 2-D map of planet Earth results in increasingly distorted continents at latitudes farther from the Equator, capturing a 2-D image306of a curved area104of a patient's body106inherently distorts and/or obscures any portion of the curved area that is not directly tangent to an optical axis of the camera244. Accordingly, any skin-condition-estimation technique based directly on captured 2-D images naturally introduces a significant amount of error when attempting to recognize a distorted pattern or texture of the skin condition. However, in the techniques described herein, mesh builder262essentially assembles 3-D polygon mesh320by identifying and extracting relatively un-distorted sections within 2-D images306(e.g., portions of 2-D images306that were oriented generally perpendicular to the optical axis of camera244at the time of capture), and assembling the extracted un-distorted image sections into a relatively high-resolution virtual 3-D model of affected skin area104. Mesh builder262may then transfer a copy of 3-D polygon mask320to condition estimator264.

In general, condition estimator264is configured to determine, based at least in part on 3-D polygon mask320derived from sensor data226, a skin-condition “type” (e.g., category or label) that matches, represents, defines, or otherwise applies to the patient's skin condition102, to within a certain (e.g., above-threshold) probability. For example, as used herein, a skin-condition “type” may refer to, as non-limiting examples: (1) a broad or general category of skin conditions (e.g., “rash” or “blemish”); (2) a specific medical name for a skin condition or a group of related skin conditions (e.g., “folliculitis”); (3) a determinable cause of a skin condition (e.g., “mosquito bite” or “scabies”); or (4) any other similar label corresponding to a set of objective descriptive parameters of (e.g., criteria for) a known skin condition, such that a determined applicable label provides useful information about the patient's skin condition102.

In some examples, condition estimator264may be configured to generate, based on 3-D polygon mask320, “revised” 2-D imagery that more accurately depicts the patient's skin condition102(e.g., with significantly reduced image distortion, as described above) than any individual 2-D image of sensor data226, and then estimate an applicable skin-condition type based on the revised 2-D imagery.

For instance, as illustrated conceptually inFIG. 3C, condition estimator264may decompose the surface (e.g., the color and texture data overlying the virtual polygon structure) of 3-D polygon mask320into revised 2-D imagery326. For example, as described above, the “faces” of polygons324are extracted sections of 2-D images306that most-accurately depict (e.g., with the least distortion) the texture, color, etc., of the patient's affected skin area104. Because the curvature304of the patient's body, and by extension, the corresponding curvature of 3-D polygon mask320, is not particularly relevant to identifying the skin-condition type, condition estimator264is configured to analyze the un-distorted image data of the individual 2-D polygons324, irrespective of the relative orientations between the polygons. Accordingly, in some examples, condition estimator264may “flatten” the polygons324onto a single planar surface, such as into a single common 2-D image or imagery326, in order to perform texture-based and pattern-based analysis of the 2-D polygons324. As referenced above, in this way (e.g., via a 2-D-to-3-D-to-2-D image-conversion technique), condition estimator264produces a substantially high-resolution (e.g., minimal-distortion) representation of skin condition102on which to base an estimation of a matching skin-condition type.

In some examples, but not all examples, when flattening 3-D polygon mask320into 2-D imagery326, condition estimator264may be configured to intentionally re-introduce a minor amount of distortion of polygons324. For example, in order to extrapolate (e.g., approximate) a shape (e.g., perimeter or outline) of skin condition102for purposes of estimating the skin-condition type (such as for smaller, local sub-sections of the affected area104), condition estimator264may “fill-in” the gaps between individual polygons324, such as by replicating the texture or pattern of the adjacent polygons into the gaps. In other examples, condition estimator264may analyze the texture, pattern, and/or color each polygon individually of the other polygons324, thereby abrogating the need to extrapolate pixels between consecutive polygons.

In some examples, but not all examples, prior to determining a matching skin-condition type, condition estimator264may be configured to automatically identify (e.g., locate) the affected area104of the patient's body106, either within the original 2-D images306from camera244or on the surface of the 3-D polygon mask320. For example, in response to user input, condition estimator264may automatically perform texture-and-color analysis on 2-D images306(e.g., “image data”) in order to locate the affected area104within the 2-D images306or within 3-D polygon mask320, as appropriate. For instance, condition estimator264may apply one or more pattern-recognition algorithms to the image data in order to identify and return an area or areas of the image data that have characteristics typical of skin conditions, including, as non-limiting examples, reddish or darkish coloration, a raised texture indicating hives or bumps, or any other abrupt transition in continuity of color or pattern on the patient's body, indicating a rash or lesion.

In other examples, such as examples in which sensors228include an infrared-based depth sensor250, condition estimator234may identify (e.g., locate) the affected area based on infrared data. For example, the patient's body108may appear “warmer” than the surrounding environment within the infrared data. Accordingly, condition estimator264may use the infrared data to “narrow down” the set of potential skin-condition locations to areas including the patient's body108, and then use other image-recognition techniques to particularly locate the affected skin area104.

In other examples, condition estimator264may identify the affected area104of the patient's body106based on user input. For example, skin-condition modeler224may prompt the user to indicate, such as by using a finger or by drawing a bounding box on display screen236of mobile device230, the location of affected area104within one of 2-D images306or on 3-D polygon mask320displayed on display screen238.

Condition estimator264may determine a matching skin-condition type, such as by comparing 2-D imagery326(and/or sensor data226) to a set of skin-condition-types data218(e.g., retrieved from storage device(s)208ofFIG. 2A). In some examples, skin-condition-types data218may include, for each of a plurality of different types of known skin conditions, data indicative of a typical physical appearance or other common physical attributes of the respective skin condition. In some examples, skin-condition-types data218may include, for each type of skin condition, an objectively defined range of values for each of a plurality of skin-condition parameters. For instance, example skin-condition parameters may include a relative coloring, a pattern (e.g., color pattern), a texture (e.g., physical pattern), a size, a shape, or any other objectively identifiable and measurable quality of the skin condition. As one illustrative example, skin-condition-types data218may describe a particular type of skin condition that includes a bumpy-texture parameter, wherein the dataset for that skin-condition type includes a range of values defining a typical density of bumps per unit surface area of skin, or a range of values defining typical diameters of each bump.

In some examples, the “typical” value or values for a skin-condition parameter includes a simple numerical range (e.g., from 6-10 bumps per square inch). In some such examples, by comparing 2-D imagery326to skin-condition-types data218, condition estimator264may return a plurality of different “candidate” skin-condition types, wherein the patient's skin condition102satisfies the criteria (e.g., falls within the ranges of parameter values) for every candidate skin-condition type.

In other examples, the “typical” value or values for a skin-condition parameter includes a Gaussian or “normal” probability distribution indicating relative probabilities of different values, such as based on a number of standard deviations from a most-probable value. In some such examples, condition estimator264may be configured to select or identify a single best-matched skin-condition type, wherein the patient's skin condition102most-approximates the most-probable value across the various indicated parameters for the best-matched skin-condition type.

In some examples, skin-condition-types data218may include one or more parameters based on other sensor data226, such as infrared data from depth-sensor242. As one illustrative example, infrared data may indicate a particularly “warm” region of the patient's body108, which, as indicated within skin-condition-types data218, may be indicative of a skin-condition type such as “recent burn” or other typically exothermic skin condition.

In the above-described examples, condition estimator264identifies one or more matching types of skin conditions based on objective, articulable criteria that may be readily available to a user of computing system200, if desired. In other words, computing system200may be configured to output a report articulating the objective basis for the determined skin-condition type.

In other examples, condition estimator264may include one or more artificial-intelligence (AI), deep-learning, or machine-learning models or algorithms configured to determine or estimate a skin-condition type that matches the patient's skin condition102based on 2-D imagery326. In general, a computing system uses a machine-learning algorithm to build a model based on a set of training data such that the model “learns” how to make predictions, inferences, or decisions to perform a specific task without being explicitly programmed to perform the specific task. Once trained, the computing system applies or executes the trained model to perform the specific task based on new data. Examples of machine-learning algorithms and/or computer frameworks for machine-learning algorithms used to build the models include a linear-regression algorithm, a logistic-regression algorithm, a decision-tree algorithm, a support vector machine (SVM) algorithm, a k-Nearest-Neighbors (kNN) algorithm, a gradient-boosting algorithm, a random-forest algorithm, or an artificial neural network (ANN), such as a four-dimensional convolutional neural network (CNN). For example, a gradient-boosting model may comprise a series of trees where each subsequent tree minimizes a predictive error of the preceding tree. Accordingly, in some examples in which condition estimator264uses a machine-learning model to determine a matching skin-condition type, the basis for the determination may be sufficiently encapsulated within the machine-learning model so as not be readily apparent (e.g., not clearly objectively articulable) to the user. Upon determining one or more matching skin-condition types for skin condition102, condition estimator264is configured to transfer the determined skin-condition type(s) to development predictor266.

In general, development predictor266is configured to predict, based at least in part on the determined skin-condition type, a unique future development of the patient's skin condition102over time. For example, development predictor266may receive the determined skin-condition types from condition estimator264, and either a copy of 3-D polygon mask320from mesh builder262, a copy of revised 2-D imagery326from condition estimator264, or both.

Based on the determined skin-condition types, development predictor266determines (e.g., generates, receives, or retrieves from storage device(s)208) a corresponding set of modeling data220for each determined skin-condition type. Modeling data220describes an average or “typical” developmental behavior of each skin-condition type. The typical developmental behavior may include, as non-limiting examples, a typical growth rate, a typical growth pattern, a typical growth direction, a typical change in relative severity, a typical change in coloration, typical growth regions on patients' bodies, a typical change in texture, or any other description of a known, statistically probable change in the respective skin-condition over time.

In some examples, modeling data220may include multiple different “typical” developmental datasets based on different variables. As one illustrative example, modeling data220may include, for a particular skin-condition type, a first dataset describing a typical development of the skin-condition type in the absence of medical treatment, and a second dataset describing a typical development of the skin-condition type in response to effective medical treatment, or any other similar developmental scenario based on controllable variables.

Development predictor266may then determine, based on the current parameter values of the patient's skin condition102(e.g., indicated by 3-D polygon mesh320and/or revised 2-D imagery326), and based on the typical development of the determined skin-condition type (e.g., indicated by modeling data220), a set of predicted future parameter values of the patient's skin condition at various points in time. In other words, polygons324(of 3-D polygon mask320and/or 2-D imagery326) represent (e.g., encode) a set of initial conditions that are unique to patient108. On the other hand, modeling data220represents (e.g., encodes) a most-probable rate-of-change for each skin-condition parameter as experienced by many prior patients. Conceptually, development predictor266is configured to apply the “rate of change” information (e.g., modeling data220) to the “initial condition” information (e.g., polygons324), in order to predict a unique future development of patient's102.

In one specific example, development predictor266is configured to use modeling data220and polygons324to produce, for each descriptive skin-condition parameter of skin condition102, a mathematical function that models a change in the parameter over time. Each mathematical function may be configured to receive, as an independent variable, a value representing a future point in time (e.g., a value of “2” representing two weeks into the future), and output, based on the independent variable, a corresponding predicted future value for the respective skin-condition parameter.

In some such examples, development predictor266may be configured to automatically generate, based on a set of stored, predetermined values for the independent time variable, a set of predicted future states of development of skin condition102, wherein each future state of development includes a predicted future dataset of associated values for each skin-condition parameter at the respective predetermined point in time indicated by each predetermined time value. In some such examples, development predictor266may be configured to generate, for each predicted future dataset, a respective plurality of “future” polygons, wherein each set of future polygons graphically depicts a developmental stage of skin condition102in a way that satisfies the predicted future dataset.

In other examples, development predictor266includes a neural-network-based model trained to predict the future development of skin condition102based on polygons324and modeling data220as input. For example, development predictor266may apply a custom neural-network in order to graphically predict the developmental stages of skin condition102, or in other words, to automatically generate and output each set of future polygons. Development predictor266may then transfer the mathematical developmental functions, the predicted future datasets, and/or the pluralities of future polygons, to model generator268.

In general, model generator268is configured to generate a virtual 3-D developmental model that includes a plurality of predicted growth-stage models, each growth-stage model graphically depicting a predicted future development of skin condition102at a different point in time. As one example, model generator268is configured to receive the various plurality of predicted future polygons, and assemble each set of future polygons into a 3-D growth-stage model. For instance, while decomposing 3-D virtual mesh320into individual polygons324, condition estimator264may have selected a “reference” polygon from among individual polygons324, and then generated a reference dataset describing the relative and orientations of all of the other polygons324relative to the reference polygon. Accordingly, each set of future polygons may include a respective reference polygon that corresponds to the original reference polygon of polygons324. Therefore, model generator268may be configured to use the reference dataset to re-align all of the other future polygons relative to the reference polygon of the respective set, thereby constructing a set of virtual 3-D growth-stage models, collectively making up a 3-D developmental model330(FIG. 3D) for skin condition102. Model generator268may then transfer the 3-D developmental model330, comprising the set of growth-stage models, to XR generator270.

In general, XR generator270is configured to generate and output extended-reality (XR) content (e.g., XR imagery112ofFIG. 1) that includes 3-D developmental model330or other graphical imagery derived therefrom. XR generator270may be configured to generate and output different types of XR content based on the particular type of XR device being used to display the content. As one example, when outputting content for display on a transparent visor of an MR headset, XR generator270may generate a 2-D projection of 3-D developmental model330, and anchor the 2-D projection onto the visor relative to the location of the patient's affected area104, as viewed from the perspective of a user wearing the MR headset. In other examples, when outputting content for display on a (non-transparent) display screen of a virtual-reality (VR) headset, XR generator270may generate a 2-D projection of 3-D developmental model330overlaid onto a virtual model of the patient's affected skin area104, or a virtual avatar of patient108. In other examples, such as the example illustrated inFIG. 3D, when outputting content to display screen238of mobile device230, XR generator270may generate composite 2-D imagery346based on real-time 2-D images332and 3-D developmental model330.

In accordance with techniques of this disclosure, XR generator270is configured to generate XR content through a distance-based object-rendering approach. For example, as illustrated and described with respect toFIG. 3D, XR generator270is configured to receive updated, current, or real-time 2-D imagery332of the patient's affected area104from camera244. XR generator270then identifies feature points322within 2-D imagery332, which may include some or all of the same feature points322identified by mesh builder262, as described above. Based on the relative locations of feature points322within 2-D imagery332(e.g., the distances to one another and from the camera244), XR generator270defines a virtual axis334within 2-D imagery332.

XR generator270may continue to receive updated or real-time sensor data226, such as IMU data and depth-sensor data. Based on updated sensor data226, XR generator270determines and monitors the relative location and orientation of virtual axis334, in order to determine and monitor a relative distance and orientation between the camera224and the patient's affected skin area104, as depicted within current 2-D imagery332. Based on virtual axis334and the monitored relative distance, XR generator236determines (e.g., selects or identifies) an augmentation surface340an area within 2-D imagery332on which to overlay virtual content, such as 3-D developmental model330. In some examples, but not all examples, augmentation surface340includes the patient's affected skin area104, which, as described above, may include the same feature points322previously identified by mesh builder262.

Based on the monitored relative location and orientation of virtual axis334within current imagery332, XR generator270determines a corresponding size and relative orientation at which to generate a 2-D projection of 3-D developmental model330(e.g., to align developmental model330with virtual axis334). For example, if XR generator270determines that virtual axis334is getting “farther away” from camera244, as indicated by current imagery332, XR generator270generates a relatively smaller 2-D projection of 3-D developmental model330, and conversely, a relatively smaller 2-D projection when virtual axis is nearer to camera244.

XR generator270may then generate a composite image346by overlaying the 2-D projection of 3-D developmental model330onto augmentation surface340within current imagery332. For example, XR generator270may identify corresponding (e.g., matching) feature points322within both of the current imagery332and the 2-D projection of 3-D developmental model330, and overlay the 2-D projection onto current imagery332such that the corresponding pairs of feature points322overlap. In other words, XR generator270may position each growth-stage model by matching feature points322in the initial 2-D image332with the feature points322in the graphical texture of 3-D developmental model330, and anchoring the 3-D developmental model330above the pre-rendered mesh of target augmentation surface340. In some examples, XR generator may perform an iterative alignment process, by repeatedly adjusting the position of the 2-D projection relative to the 2-D image so as to reduce or minimize an error (e.g., a discrepancy) between corresponding matched feature points.

XR generator270then outputs composite image346to display screen238of mobile device230. In this way, XR generator270(e.g., via a graphics processing unit (GPU) of mobile device230), renders XR (e.g., AR) content and displays real-time AR developmental stages of skin condition102overtop of the patient's affected skin area104.

In some examples, skin-condition modeler224is configured to identify and correct for anomalies or other errors, such as while estimating a skin-condition type, or while predicting and visualizing the future development of skin condition102. For example, skin-condition modeler224may receive user input (e.g., feedback from a dermatologist or other user) indicating an anomaly, such as an incorrectly estimated skin-condition type or an implausible development (e.g., excessive or insufficient growth, change in coloration, or the like) within 3-D developmental model330. As one example, a user may submit a manual correction for one or more of the individual growth-stage models of 3-D developmental model330. In examples in which condition estimator264includes a machine-learned model trained to estimate the skin-condition type, and/or examples in which development predictor266includes a machine-learned model trained to generate the growth-stage models, upon receiving a manual correction or other user feedback, skin-condition modeler224may be configured to automatically perform batch-wise (e.g., complete) retraining of either or both of these skin-condition-predictive models, using the user's feedback as new training data. In some such examples, in which the “magnitude” of the user's correction (e.g., the magnitude of the difference between the user's indication of the “correct” developmental pattern and the automatically generated “incorrect” developmental pattern) exceeds a pre-determined threshold, skin-condition modeler224may be configured to generate and output a notification that the machine-learning model is operating outside acceptable variance limits, and that the model may need to be updated (as compared to merely retrained) by the developer.

FIG. 4is a conceptual diagram depicting an example of the skin-condition-prediction system100ofFIG. 1. More specifically,FIG. 4depicts an example including computing system400that includes two computing devices: a “local” computing device230and a “remote” or “cloud” computing server236. This example configuration is advantageous in that the cloud server236may include greater processing power, and therefore may be better-suited to handle the more-computationally-intensive, but less-time-sensitive techniques of a skin-condition-prediction process, such as estimating a skin-condition type. Conversely, the local device230can better-handle the less-resource-intensive, but more-time-sensitive techniques, such as updating an orientation of virtual objects in real-time. Computing system400may be an example of computing system200ofFIG. 2A, in that each individual computing unit230,236may include any or all of the example components of computing system200.

Local device230includes virtually any suitable computing device that is physically (e.g., geographically) local to the user, such as a smartphone, laptop, a desktop computer, a tablet computer, a wearable computing device (e.g., a smartwatch, etc.), or the like. Local device230is configured to receive or capture, from various sensors228, sensor data226including 2-D images306depicting a skin condition102on an affected area104of a body106of a patient108from multiple different perspectives. Other types of sensors may include a depth sensor (e.g., LIDAR and/or infrared-based depth sensing), and a 9-axis IMU240. In some examples, local device230may be configured to wirelessly transfer the sensor data226to cloud computing system236. In other examples, local device230retrains the sensor data226and performs any or all of the functionality of cloud-computing system236described below.

Cloud computing system236(also referred to herein as “CCS236”) is configured to receive the sensor data226, including the 2-D images from mobile device230. CCS236compares the 2-D image(s), or other 2-D imagery derived therefrom, to stored models of skin conditions in order to determine which condition classification best matches the condition in the 2-D imagery. In some examples, CCS236feeds the 2-D imagery into a neural network (e.g., a convolutional neural network (CNN)) trained to estimate or identify a matching skin-condition type. In some such examples, CCS236may be configured to map each pixel of the 2-D imagery to a different input neuron in a 2-D array of input neurons in order to perform pixel-based pattern and texture recognition.

CCS236may then determine (e.g., retrieve, receive, generate, etc.) modeling data based on the determined skin-condition classification, and may generate a set of predicted growth-stage models of skin condition102, e.g., characterizing, via colored texture data, a predicted direction of growth, a predicted coloration, a predicted relative severity, etc., of the skin condition102. CCS236may then construct the growth-stage models over a 3-D curved polygon mesh (collectively forming a 3-D developmental model330) and may send the 3-D mesh (along with the colored texture data of the growth-stage models) back to local device230.

Local device230is configured to monitor (e.g., determine, at regular periodic intervals) a location and orientation of a virtual axis (e.g., axis334ofFIG. 3D) and a relative distance between, for example, a camera or other sensor of local device230and the patient's body108, in order to identify a “plane of augmentation” or augmentation surface340, or in other words, a surface depicted within the 2-D image(s) on which to overlay virtual content. Local device230is further configured to anchor the 3-D developmental model330to the plane of augmentation based on a set of identified anchor points322along the plane of augmentation340. Local device230is further configured to monitor the augmented surface area based on the monitored depth and movement data recorded by the integrated IMU240, and to update the relative position of the 3-D developmental model accordingly. For example, local device230may be responsible for: (1) determining and capturing a relative distance of the patient's area of interest (e.g., affected skin area104) from the local device230; (2) determining and capturing movement subtleties based on IMU data; and (3) controlling display settings, such as adjusting a brightness, contrast, hue, and saturation according to lighting and environmental conditions. In this way, computing system400is configured to animate a predicted future progress of the skin condition104

FIG. 5is a flowchart illustrating an example skin-condition-prediction process, in accordance with one or more aspects of the techniques disclosed. The techniques ofFIG. 5are described primarily with respect to the example hardware architecture of computing system200ofFIG. 2Band the example software modules ofFIG. 2C, however, any suitable computing system may perform the techniques herein.

A computing system200having one or more processors is configured to estimate a skin-condition type or category for a skin condition102on an affected area104of a body106of a patient108(510). For example, the computing system may receive 2-D image data306depicting the affected skin area104from multiple perspectives, and then perform a 2-D-to-3-D-to-2-D image-conversion process in order to produce a graphical depiction of, for example, the size, shape, texture, pattern, and/or coloration of the skin condition102. Computing system200may then identify one or more probable skin-condition types based on the 2-D image data and the stored skin-condition-type data218indicative of known types of skin conditions. For example, computing system200may perform the 2-D-to-3-D-to-2-D process described elsewhere in this disclosure and apply a machine-learning model to the refined 2D data to estimate the skin-condition type.

Based on the identified skin-condition type(s), computing system200may determine (e.g., retrieve or generate) modeling data220describing a typical developmental behavior for the estimated skin-condition type(s) (520). For example, the data may indicate a typical change in size, shape, coloration, texture, or relative severity, of the respective type of skin condition.

Based on the modeling data, computing system200generates a 3-D developmental model330indicating (e.g., graphically depicting) a predicted future development (e.g., at least a predicted direction of growth) of the patient's skin condition102(530). For example, computing system200may apply the refined 2-D data and the modeling data into a machine-learning model trained to generate a plurality of virtual growth-stage models indicating a development of the skin condition at different pre-determined points of time in the future.

The computing system200may use the 3-D developmental model330to generate extended-reality (XR) imagery or other XR content (540). For example, the computing system200may generate composite imagery346depicting the patient's affected skin area104overlaid with a 2-D projection of the 3-D developmental model330. The computing system200may output the XR imagery346to a display device, such as a display screen238of a mobile device230(550). The computing system200may update the XR content in real-time based on a motion of the mobile device230relative to the affected skin area104(as indicated by an integrated IMU240), in order to create the appearance of the 3-D developmental model330“anchored” to the patient's affected skin area104.

FIG. 6is a flowchart illustrating an example dermatological-condition-prediction process, in accordance with one or more aspects of the techniques disclosed. As one non-limiting example, the techniques of this disclosure include a computing system configured to capture sensor data (including 2-D image data) for a patient's skin condition, feed the collected data through a deep-learning model configured to estimate the type of skin condition and predict its future development, and generate and output extended-reality imagery visualizing the predicted future development. The techniques ofFIG. 6are described primarily with respect to the example hardware architecture of computing system200ofFIG. 2B, however, any suitable computing system may perform the techniques herein.

A user (e.g., a patient108or a clinician of patient108) of a mobile device230activates a skin-condition-visualization application, such as skin-condition modeler224ofFIG. 2A, running on mobile device230(602). While activated, mobile device230may be configured to actively stream data, such as sensor data226via data-streaming device(s)232ofFIG. 2B. In other examples, mobile device230may be configured to periodically transfer data (e.g., sensor data226) via data-streaming device(s)232, or after after the data has been captured.

In response to a prompt, the user may select an “Automatic Capture” mode or a “Manual Capture” mode. Upon selecting the “Manual Capture” mode, the user may be further prompted to select a target area within a 2-D image306depicting a skin condition102on an affected skin area104on the body106of a patient108. Upon selecting the “Automatic Capture” mode, skin-condition modeler224may attempt to automatically locate the skin condition102within the 2-D image306.

In response to a prompt (e.g., appearing on display screen238), the user may then move the mobile device230around the affected skin area104(604). While the mobile device is in motion, an integrated camera244captures 2-D images306of the affected skin area104, while other integrated sensors228, such as a 9-axis IMU240and a depth sensor242, capture additional sensor data226describing the relative position, orientation, and/or motion of mobile device230at any given point in time.

Using the 2-D images306and the other sensor data226, skin-condition modeler224generates a 3-D polygon mesh320(606), such as a curved 3-D surface made up of a plurality of 2-D polygons324(so as to mimic the curvature304of the patient's body) overlaid with a graphical texture representing the affected area104of the patient's body.

Skin-condition modeler224may then make a copy of 3-D polygon mesh320and deconstruct the mesh320into the individual 2-D polygons324. For example, skin-condition modeler224may “separate” the 3-D mesh320from the 2-D polygons or “tiles”324that make up the outer surface of the 3-D mesh (608). Skin-condition modeler224may then flatten the tiles324onto a common 2-D plane, and fill in any gaps between adjacent tiles, thereby producing revised 2-D imagery326depicting the size, shape, color, texture, and pattern of the patient's skin condition102. In some examples, but not all examples, skin-condition modeler may be configured to apply revised 2-D imagery326to a “super-resolution” neural network, trained to increase the resolution of 2-D imagery326even further (e.g., by extrapolating particularly high-resolution patterns and textures into lower-resolution areas, smoothing pixel edges, etc.) (610).

In some examples, skin-condition modeler224may prompt the user to input or select a type, category, or label for skin condition102, if known to the user. In other examples, an AI or deep-learning model, such as a neural engine, analyzes the color, texture, and pattern within the revised 2-D imagery326in order to “identify” a type or category to which skin condition102most-likely belongs (612). Based on a typical developmental behavior of the identified type of skin condition, the neural engine predicts a unique (e.g., patient-specific) future development of skin condition102. For example, the neural engine may use the surrounding affected skin area104(as depicted on tiles324) as a reference, e.g., a starting point or set of initial conditions, to apply to the typical developmental behavior in order to generate a plurality of virtual growth-stage models depicting the predicted future development of skin condition102.

In examples in which the virtual growth-stage models each includes a respective 2-D image based on revised 2-D imagery326(e.g., based on individual tiles324), the neural engine may then convert the virtual growth-stage models into curved 3-D growth-stage models by rearranging (e.g., reassembling) individual tiles relative to a designated reference tile (614).

Skin-condition modeler224generates a subsequent 3-D mesh (which may substantially conform to the shape and/or structure of the original 3-D mesh), and reduces noise in the 3-D mesh, such as by averaging-out above-threshold variations in the curvature of the surface of the subsequent 3-D mesh (616). In some examples, skin-condition modeler224may “smooth” the 3-D mesh into a curved surface by first determining (e.g., extrapolating) a curvature of the 3-D mesh, and then simultaneously increasing the number and reducing the size of the individual polygons making up the 3-D mesh, thereby increasing the “resolution” of the 3-D mesh in order to better-approximate the appearance of a smooth curve (618).

Skin-condition modeler224may identify a centerpoint of the subsequent 3-D mesh and designate the centerpoint as a point of reference (620). For example, skin-condition modeler224may define a virtual axis334passing through the centerpoint, and use the axis334as a basis for orientation and alignment of 3-D mesh330relative to subsequent 2-D imagery332.

Skin-condition modeler224may identify, based on virtual axis334and subsequent 2-D imagery332captured by camera244, a plane of augmentation340, or in other words, a “surface” depicted within the 2-D images332upon which virtual objects will be shown or overlaid (622).

Skin-condition modeler224may reduce an amount of noise (e.g., average-out excessive variation) within sensor data226(624), and then feed sensor data226, the subsequent 3-D mesh, the subsequent 2-D imagery332, the augmentation plane340, and the virtual growth stage models into an augmentation engine (e.g., XR generator270ofFIG. 2C) configured to generate and output XR content346(626). For example, the XR content may include composite imagery depicting the patient's affected skin area104overlaid with a 2-D projection of 3-D developmental model330, thereby modeling a predicted future progression of the skin condition102over time. Skin-condition modeler224may perform this dermatological-condition-prediction process in real-time, such that skin-condition modeler224may continue to generate and output this type of XR content in this way as the user continues to move camera244of mobile device230around the affected skin area104(604).