Robust blur estimation for shapes with known structural elements

Various implementations disclosed herein assess the blurriness of portions of images depicting shapes such as codes or text that have known structural elements. This may involve determining whether a portion of an image of a code or text is sufficiently clear (not blurry) to be accurately interpreted. Blur may be assessed based on spatial frequency of statistical analysis. Blur may be assessed using a machine learning model that is trained using target blur metrics determined based on spatial frequency (e.g., analysis of high frequency portions of discrete cosine transforms of image portions) or statistical analysis (e.g., based on corner/edge detection in image portions).

TECHNICAL FIELD

The present disclosure generally relates to estimating image blur and to systems, methods, and devices that estimate image blur to facilitate the interpretation of shapes such as codes and text in images of physical environments.

BACKGROUND

Interpreting shapes such as codes and text can depend upon the blurriness of the images in which the shapes are depicted. Blurriness may result from a variety of factors including, but not limited to, distance from the image sensor, motion of the image sensor or physical environment, and/or the image sensor not being focused on the shape within the physical environment. Existing techniques for estimating blurriness in processes that interpret shapes in images may not be sufficiently accurate or efficient.

SUMMARY

Various implementations disclosed herein assess the blurriness of portions of images depicting shapes, such as codes or text, that have known structural elements. Assessing blur may involve determining a blur value that is specifically tailored to be indicative of blur that will affect the interpretability of a shape of a set of shapes having common structural characteristics but different attributes or colors. Assessing blur may involve determining whether a portion of an image of a code or text is sufficiently clear (not blurry) to be accurately interpreted. Blur may be assessed based on spatial frequency or statistical analysis.

In one exemplary implementation, a processor executes instructions stored in a computer-readable medium to perform a method. The method obtains an image of a physical environment; (e.g., via an image sensor) and determines a portion of the image having a shape, where the shape is one of a plurality of shapes having common structural characteristics (e.g., a code or text that differs from other codes and text with respect to certain attributes and/or color combinations). The method assesses blur (e.g., whether the image is clear or not) of the portion of the image by determining a blur value corresponding to the interpretability of the shape in the image. The method, in accordance with assessing the blur of the portion of the image, interprets the portion of the image to interpret the shape, for example, by recognizing text or decoding a code.

Blur may be assessed using a machine learning model that is trained using target blur metrics determined based on spatial frequency (e.g., analysis of spatial frequencies in the frequency domain using, for example, the discrete cosine or Fourier transforms of image portions) or statistical analysis (e.g., based on corner/edge detection in image portions).

A machine learning model may be trained by processes that take into account knowledge that the shapes, such as codes or text, that are to be interpreted have known structural elements, while also having some variations. A code, for example, may have lines or arcs having certain shapes characteristics according to a code format, but may also having variable size code elements and/or use different color combinations. As another example, text may include a variety of letters having similar characteristics based on the nature of text, but may have variability in font, size, and color. Such known structural characteristics and variabilities can be accounting for in training a machine learning model to assess the blurriness of an image with respect to the purpose of interpreting shapes having those structural characteristics and variabilities.

In some implementations, a machine learning model may be trained using target blur metrics that assess blur based on the spatial frequencies that are most indicative of the blurriness of an image of something having the known structural characteristics and variabilities. In some implementations, a processor of an electronic device executes instructions stored in a non-transitory computer-readable medium to perform a method. The method obtains an image of a physical environment and determines a portion of the image having a code or text. The method assesses a blur (e.g., whether the imaged content in the photo is captured as crisp as expected or not) of the portion of the image using a machine learning model, where the machine learning model is trained using target blur metrics determined based on spatial frequency. For example, a target/ground truth blur metric for each training image may be determined by comparing a discrete cosine transform (DCT) spatial high frequency confidence of an image portion without blur to a DCT spatial high frequency confidence of the image portion with blur. The method, in accordance with assessing the blur of the portion of the image, interprets the portion of the image to decode the code or recognize the text. For example, the method may only interpret the portion of the image if the machine learning model determines a particular classification (e.g., not blurred) or if image has an assessed numerical blur value that is below a threshold value.

In some implementations, a machine learning model may be trained using target blur metrics that assess blur based on statistical analysis (e.g., of corners and edges) of image characteristics that are most indicative of the blurriness of an image of something having the known structural characteristics and variabilities. In some implementations, a processor of an electronic device executes instructions stored in a non-transitory computer-readable medium to perform a method. The method obtains an image of a physical environment and determines a portion of the image comprising a code or text. The method assesses a blur (e.g., whether the image is clear or not) of the portion of the image using a machine learning model, where the machine learning model is trained using target blur metrics determined based on statistical analysis. For example, a target/ground truth blur metric for each training image may be determined by comparing a transform corresponding to edge or corner detection (e.g., Laplacian edge detection) of the training image without blur to a transform corresponding to edge or corner detection of the training image with blur. The method, in accordance with assessing the blur of the image portion, interprets the portion of the image to decode the code or recognize the text. For example, method may only interpret the portion of the image if the machine learning model determines a particular classification (e.g., not blurred) or a blur value below a threshold value.

DESCRIPTION

FIG.1illustrates an example of an electronic device105used by a user110within a physical environment100. A physical environment refers to a physical world that people can interact with and/or sense without the aid of electronic systems. Physical environments, such as a physical park, include physical articles, such as physical trees, physical buildings, and physical people. People can directly sense and/or interact with the physical environment, such as through sight, touch, hearing, taste, and smell.

In the example ofFIG.1, the device105is illustrated as a single device. Some implementations of the device105are hand-held. For example, the device105may be a mobile phone, a tablet, a laptop, and so forth. In some implementations, the device105is worn by a user. For example, the device105may be a watch, a head-mounted device (HMD), and so forth. In some implementations, functions of the device105are accomplished via two or more devices, for example additionally including an optional base station. Other examples include a laptop, desktop, server, or other such devices that includes additional capabilities in terms of power, CPU capabilities, GPU capabilities, storage capabilities, memory capabilities, and the like. The multiple devices that may be used to accomplish the functions of the device105may communicate with one another via wired or wireless communications.

InFIG.1, the physical environment100includes two wall hangings120,130and a flower in a vase140. The first wall hanging120includes text125, the second wall hanging130includes a code135(e.g., a barcode or other visual marker), and the flower in the vase140includes a code145(e.g., a barcode or other visual marker). In some implementations, the electronic device105is configured to capture images and interpret the images to identify shapes such as those of text125and codes135,145. In some implementations, the electronic device105captures one or more images of the physical environment100, including one or more of the text125or codes135,145. The electronic device105may identify the shapes in one or more images and interpret the corresponding portions of the one or more images, e.g., to recognize the text125or decode data encoded in the codes135,145.

FIG.2illustrates an example of the electronic device105acquiring an image in the physical environment100. In this example, the image200, which is shown as displayed on electronic device105, includes a depiction240of the flower in the vase140and a depiction245of the code145. In other implementations, an image such as image200is captured and assessed, but not displayed on the electronic device200. The image200may be evaluated to identify that a portion of the image200(e.g., a patch or region of pixels) depicts the code145. This portion of the image is evaluated to assess its blurriness with respect to the purpose of interpreting shapes having certain structural characteristics and variabilities, e.g., the structural characteristics and variabilities of codes of a particular type or format. Such blurriness is assessed using one or more of the techniques disclosed herein, for example, using a machine learning model. In one example, the machine learning model is trained to do so using target blur metrics that assess blur based on the spatial frequencies that are most indicative of the blurriness of an image of something having the known structural characteristics and variabilities. Such frequencies may be the prevalent or dominant frequencies in the shape structure that are determined to diminish with increasing blur. In another example, the machine learning model may be trained using target blur metrics that assess blur based on statistical analysis (e.g., of corners and edges) of image characteristics that are most indicative of the blurriness of an image of something having the known structural characteristics and variabilities.

FIG.3illustrates an example of relatively crisp portion of an image acquired by the electronic device105with a depiction310of an exemplary code. In contrast,FIG.4illustrates an example of relatively blurry portion of an image acquired by the electronic device105that depicts a depiction410of the same exemplary code. In these examples, the exemplary code is a visual marker that includes markings arranged to form a shape (e.g., arcs forming rings). Each ring may define a template (e.g., grid) of arcs and gaps used to convey information in the visual marker. The template may be selectively filled with sub-markings (e.g., representing a “1” bit) or left as gaps (e.g., representing a “0” bit) to convey information. Visual markers using such an encoding format may have similar structural characteristics (e.g., having similar ring, arc, and gap characteristics) while also being variable (e.g., having differing numbers and placements of arcs and gaps and/or different color usage for background and arc colors).

In some implementations, a machine learning model is trained to assess the blurriness of an image using training data that is specific to the type/format of shapes (e.g., codes or text) that will be interpreted from the image. Such training may use sample images of shapes of the shape type/format that share the common structural characteristics of the type/format of the shape and/or that differ with respect to variable aspects of the type/format of shape. For example, training data may include images of codes that use the same format as the exemplary code depicted inFIGS.3and4with similarly shaped rings, with different numbers of arcs and gaps and different positions of arcs and gaps corresponding to different data encodings and/or in different color combinations. Using such training data may provide a machine learning model that is able to more accurately and/or more efficiently assess the blurriness of an image with respect to the purpose of interpreting shapes of a given type or format having those structural characteristics and variabilities.

In some implementations, a machine learning model is trained to assess the blurriness of an image using training data that includes target blur metrics (e.g., ground truth blur values) that assess blur based on the spatial frequencies that are most indicative of the blurriness of an image of something having the known structural characteristics and variabilities.FIGS.5-6illustrate an example of generating target blur metrics and using those metrics in training a machine learning model to assess the blurriness of an image.

FIG.5illustrates target blur metrics (e.g., ground truth blur values) generated for different versions502a-hof an image (e.g., an image portion depicting a shape). In this example, different amounts of blur are applied to an image of a shape (e.g., a relatively clear image of a code such as is depicted inFIG.3). Blur may be generated via low pass filtering, convolution of an image with a filter (e.g., box or Guassian), blurring using large or small blur kernels, or other any other blurring technique or combination of techniques. In this example, Gaussian blur is added in differing amounts in the various versions of the image. Version502ahas no blur added, version502bhas blur added using blur value 0.5, version502chas blur added using blur value 1.0, version502dhas blur added using blur value 1.5, version502ehas blur added using blur value 2.0, version502fhas blur added using blur value 2.5, version502ghas blur added using blur value 3.0, and version502hhas blur added using blur value 3.5. In alternative examples, additional or alternative techniques may be used to generate or otherwise simulate blur that may occur based on distance between a shape and an image sensor, motion of the image sensor or the shape, and/or the image sensor not being focused on the shape within the physical environment. Blur may be generated via a technique that simulates directional/movement blurs, e.g., using a blur kernel of limited height (e.g., 1 pixel) to simulate horizontal motion blur, using a blur kernel of limited width (e.g., 1 pixel) to simulate vertical motion blur, etc. In some implementations, different types of blur are applied to different images to simulate different real-world sources of blur, e.g., some images with horizontal motion blur, some images with vertical motion blur, some images with diagonal motion blur, some images with blur caused by movement to the left, some images with blur caused by movement to the right, etc.

Spatial frequency representations are determined for each of these versions502a-hof the image. In this example, a discrete cosine transform (DCT)504a-his generated each of these versions502a-hof the image.

In this example, spatial frequency bands of the image506a-hare generated using the discrete cosine transforms (DCT)504a-hand the high frequency components are evaluated to determine target blur metrics508a-hcorresponding to each of the versions502a-hof the image, respectively. The target blur metrics508a-h, for example, may be determined by comparing a DCT high frequency components of an image portion without blur to a DCT high frequency components of the image portion with blur. In this example, the high frequency portion each of the spatial frequency representation (e.g., the DCT) is used because changes in the high frequency portion correspond to blurring that is likely to negatively affect the desired ability to interpret shapes having the known structural characteristics and variabilities of this example. In some implementations, for example, involving text or codes having different formats and structures, a different portion or potions (e.g. a middle frequency range, varying orientations, etc.) of the spatial frequency representations is/are used to determine the target blur metrics based on the known structural characteristics and variabilities of the shapes that will be interpreted.

The process illustrated inFIG.5may be repeated for additional images of shapes of a particular type (e.g., other exemplary codes having the same format). Accordingly, target blur metrics for various levels of blur for many training images of shapes of a given type/format that share (and thus represent) known structural characteristics but that have differing aspects (e.g., variabilities of color, etc.) may be created.

FIG.6illustrates an example training process in which generated blur estimates (e.g., target blur metrics508a-hofFIG.5) are used as ground truth to train a machine learning model to output blur estimates for input images. In this example, training images502a-hare used as training input data610to machine learning model620. For each training image, the machine learning model620produces output630including a blur assessment632, e.g., a classification such as blurred or not blurred or a numerical prediction on a scale from, e.g. 0-10 of a level of blur. The training process uses the output630and ground truth training data615, e.g., the target blur metrics508a-hfromFIG.5, to determine an adjustment650to the machine learning model620. During the course of training over multiple iteration using images of shapes of a particular type (e.g., exemplary codes having the same format) but variations and corresponding target blur metrics determined based on spatial frequency, the machine learning model620is adjusted (and thus learns) to accurately and efficiently produce blur assessments632using images of shapes, e.g., such as image portion310or image portion410ofFIGS.3and4respectively. Because the training data was based on spatial frequency, the machine learning model is effectively trained in a way that takes into account spatial frequency and thus accounts for the known structural characteristics and variabilities of the shapes that will be interpreted based on the machine learning models blur assessment632.

FIG.7is a flowchart illustrating an exemplary method of assessing the blur of a portion of an image of a physical environment. In some implementations, the method700is performed by a device (e.g., electronic device105ofFIG.1). The method700can be performed using an electronic device or by multiple devices in communication with one another. In some implementations, the method700is performed by processing logic, including hardware, firmware, software, or a combination thereof. In some implementations, the method700is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory).

At block710, the method700obtains an image of a physical environment. An image sensor at an electronic device captures the image of the physical environment. In some implementations, the sensor can be an RGB camera, a depth sensor, an RGB-D camera, a monochrome camera, a 2D camera, an IR camera, and/or the any other sensor providing data used to generate an image (or data which allows to generate an image) of a physical environment. In some implementations, combinations of sensors are used. In some implementations, the image is color. In some implementations the image is grayscale.

At block720, the method700determines a portion of the image comprising a shape (e.g., a code or text). The shape the shape may be one of a plurality of shapes having common structural characteristics. The determining may involve determining that the obtained image includes a depiction of a shape, e.g., text or a code. For example, such a shape may be visible on a surface of an object in the physical environment. The shape may include text of single or variable font. The shape may include a one-dimensional (1D) code, a two-dimensional (2D) code, or a (3D) three-dimensional code. The shape may be printed on the surface of the object (e.g., in black and white or color), colored etched, painted, powdered, drawn, sprayed, or the like onto the surface of the object, displayed by a display, or projected by a projector on the object in the physical environment. In some implementations, an electronic device (e.g., including the image sensor) detects the shape (e.g., text or code) in the image of the physical environment using an algorithm or machine learning model. In some implementations, the shape is detected based on identifying a pattern, outline, pixel combination, or characteristic(s) of a selected portion of the image. Based on identifying the shape within the image, a portion of the image (e.g., a subset of some or all of the pixels of the image) are selected for further analysis.

At block730, the method700assesses blur of the portion of the image using a machine learning model, where the machine learning model is trained to assess blur using target blur metrics determined based on spatial frequency. Assessing the blur of the portion of the image may involve assessing whether the image is sufficiently clear or not for an intended purpose, e.g., for interpreting the shape with a desired degree of accuracy. Assessing the blur may involve determining a level of blur (e.g., on a numerical scale) that can be compared against a threshold or other criteria to determine whether the image is sufficiently clear or not for an intended purpose, e.g., for interpreting the shape with a desired degree of accuracy.

The machine learning model may be trained using images depicting a plurality of shapes having common structural characteristics.

The machine learning model may be trained using a target blur metric for each of a plurality of training images of codes or text, e.g., of codes or text have different shapes and/or different color combinations. The machine learning model may be trained using a target blur metrics as ground truth for each of multiple training image. The target blur metrics for the training images may be determined based on evaluating spatial frequencies of blurred and cleared versions of a training image. The target blur metric for each of the plurality of training images may be determined by comparing a spatial frequency range confidence of a clear version of each training image with the spatial frequency range confidence of a blurred version of each training image. For example, this may involve comparing a DCT high frequency components of an image portion without blur to a DCT high frequency components of the image portion with blur.

In some implementations, the machine learning model is trained by obtaining a clear version of a training image and generating a blurred version of the training image using image processing operations, as, for example, a Gaussian blur filter operator. The training further involves filtering spatial frequencies of the clear version and blurred version of the training image, where the filtering is based on generating DCTs that transform the clear version and the blurred version of the training image from the spatial domain to the frequency domain. The training further involves determining the target blur metric based on the filtered spatial frequencies of the clear version and the blurred version of the training image. The machine learning model is trained to predict a blur score using the blurred version of the training image as training input, where a ground truth of the training is based on the determined target blur metric. In some implementations, image processing operations such as Laplacian can be applied to find edges and corners and a statistical operation such as variance can be used to identify if the image is crisp or blurred. The variance can be used for instance as a blur metric to train the machine learning model.

In some implementations, the machine learning model a neural network (e.g., an artificial neural network), decision tree, support vector machine, Bayesian network, or the like.

At block740, the method700, in accordance with assessing the blur of the portion of the image, interprets the portion of the image to decode or recognize the shape. For example, this may involve determining whether to interpret the portion of the image if the ML model determines a particular classification (e.g., not blurred) or a blur value below a threshold value. In live image capture scenario in which a stream of live images of an environment are obtained, each image may be assessed with respect to blur as (e.g., on an ongoing basis) as the respective image is received. If a shape is detected in an image and the image is sufficiently clear, the shape is interpreted. If a shape is detected in an image and the image is too blurry for interpretation, then the device can wait and evaluate the next obtained image in the same way. In such an implementation, the device is able to interpret the shape at the earliest appropriate opportunity, e.g., as soon as an image is received having adequate blur/clarity characteristics for interpretation. Doing so may thus provide a desirable balance of ensuring accurate and fast detection and interpretation of shapes (e.g., text and codes) within a physical environment as well as to minimize overall power consumption since potential complex processing steps can be potentially omitted if the image is estimated as too blurry for further processing. In an alternative implementation, blur is assessed for a set of images and the image having the best clarity (e.g., the least blur) for interpretation is selected for interpretation. Such an implementation may prioritize accuracy over fast detection and interpretation.

FIG.8is a flowchart illustrating an exemplary method of assessing the blur of a portion of an image of a physical environment. In some implementations, the method800is performed by a device (e.g., electronic device105ofFIG.1). The method800can be performed using an electronic device or by multiple devices in communication with one another. In some implementations, the method800is performed by processing logic, including hardware, firmware, software, or a combination thereof. In some implementations, the method800is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory).

At block810, the method800obtains an image of a physical environment. An image sensor at an electronic device captures the image of the physical environment. In some implementations, the sensor can be an RGB camera, a depth sensor, an RGB-D camera, a monochrome camera, a 2D camera, an IR camera, and/or the any other sensor providing data used to generate an image of a physical environment. In some implementations, combinations of sensors are used. In some implementations, the image is color. In some implementations the image is grayscale.

At block820, the method800determines a portion of the image comprising a shape (e.g., a code or text). The shape the shape may be one of a plurality of shapes having common structural characteristics. The determining may involve determining that the obtained image includes a depiction of a shape, e.g., text or a code. For example, such a shape may be visible on a surface of an object in the physical environment. The shape may include text of single or variable font. The shape may include a one-dimensional (1D) code, a two-dimensional (2D) code, or a (3D) three-dimensional code. The shape may printed on the surface of the object (e.g., in black and white or color), colored etched, painted, powdered, drawn, sprayed, or the like onto the surface of the object, displayed by a display, or projected by a projector on the object in the physical environment. In some implementations, an electronic device (e.g., including the image sensor) detects the shape (e.g., text or code) in the image of the physical environment using an algorithm or machine learning model. In some implementations, the shape is detected based on identifying a pattern, outline, pixel combination, or characteristic(s) of a selected portion of the image. Based on identifying the shape within the image, a portion of the image (e.g., a subset of some or all of the pixels of the image) are selected for further analysis.

At block830, the method800assesses blur of the portion of the image using a machine learning model, where the machine learning model is trained using target blur metrics determined based on statistical analysis. Assessing the blur of the portion of the image may involve assessing whether the image is sufficiently clear or not for an intended purpose, e.g., for interpreting the shape with a desired degree of accuracy. Assessing the blur may involve determining a level of blur (e.g., on a numerical scale) that can be compared against a threshold or other criteria to determine whether the image is sufficiently clear or not for an intended purpose, e.g., for interpreting the shape with a desired degree of accuracy.

The machine learning model may be trained using images depicting a plurality of shapes having common structural characteristics.

The target blur metrics used to train the machine learning model may be determined by comparing a transform corresponding to edge or corner detection (e.g., Laplacian edge detection) of a training image without blur to a transform corresponding to edge or corner detection of the training image with blur. Thus, the target blur metric for each of the plurality of training images may be determined by comparing a first transform corresponding to edge or corner detection of a clear version of a respective training image to a second transform corresponding to edge or corner detection of a blurred version of the respective training image. The target blur metric for each of the plurality of training images may be determined using a Laplacian. When using a Laplacian, a reference image may or may not be used. The Laplacian may highlight regions of an image containing rapid intensity changes, such as corners and edges. Performing a statistical analysis such as variance may provide an estimate for how prevalent the edges/corners are. A high variance indicates a crisp image, while a low variance indicates a blurred image. This is because generally if an image contains high variance then there is a wide spread of edge-like and non-edge like regions. On the other hand, if there is very low variance, then generally there is only a small number of edges in the image. Comparing the variance of a crisp image with its blurred counterpart can provide a relative metric of blurriness. In some implementations, the machine learning model is a neural network (e.g., an artificial neural network), decision tree, support vector machine, Bayesian network, or the like.

At block840, the method800, in accordance with assessing the blur of the portion of the image, interprets the portion of the image to decode or recognize the shape. For example, this may involve determining whether to interpret the portion of the image if the ML model determines a particular classification (e.g., not blurred) or a blur value below a threshold value. In live image capture scenario in which a stream of live images of an environment are obtained, each image may be assessed with respect to blur as (e.g., on an ongoing basis) as the respective image is received. If a shape is detected in an image and the image is sufficiently clear, the shape is interpreted. If a shape is detected in an image and the image is too blurry for interpretation, then the device can wait and evaluate the next obtained image in the same way. In such an implementation, the device is able to interpret the shape at the earliest appropriate opportunity, e.g., as soon as an image is received having adequate blur/clarity characteristics for interpretation. Doing so may thus provide a desirable balance of ensuring accurate and fast detection and interpretation of shapes (e.g., text and codes) within a physical environment. In an alternative implementation, blur is assessed for a set of images and the image having the best clarity (e.g., the least blur) for interpretation is selected for interpretation. Such an implementation may prioritize accuracy over fast detection and interpretation.

FIG.9is a block diagram illustrating exemplary components of the device105configured in accordance with some implementations. While certain specific features are illustrated, those skilled in the art will appreciate from the present disclosure that various other features have not been illustrated for the sake of brevity, and so as not to obscure more pertinent aspects of the implementations disclosed herein. To that end, as a non-limiting example, in some implementations the device105includes one or more processing units902(e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, and/or the like), one or more input/output (I/O) devices and sensors906, one or more communication interfaces908(e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, SPI, I2C, and/or the like type interface), one or more programming (e.g., I/O) interfaces910, one or more displays912, one or more interior and/or exterior facing image sensor systems914, a memory920, and one or more communication buses904for interconnecting these and various other components.

In some implementations, the one or more displays912are configured to present a view of a physical environment or a graphical environment (e.g. a 3D environment) to the user. In some implementations, the one or more displays912correspond to holographic, digital light processing (DLP), liquid-crystal display (LCD), liquid-crystal on silicon (LCoS), organic light-emitting field-effect transitory (OLET), organic light-emitting diode (OLED), surface-conduction electron-emitter display (SED), field-emission display (FED), quantum-dot light-emitting diode (QD-LED), micro-electro-mechanical system (MEMS), and/or the like display types. In some implementations, the one or more displays912correspond to diffractive, reflective, polarized, holographic, etc. waveguide displays. In one example, the device105includes a single display. In another example, the device105includes a display for each eye of the user.

In some implementations, the one or more image sensor systems914are configured to obtain image data that corresponds to at least a portion of the physical environment100. For example, the one or more image sensor systems914include one or more RGB cameras (e.g., with a complimentary metal-oxide-semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), monochrome cameras, IR cameras, depth cameras, and/or the like. In various implementations, the one or more image sensor systems914further include illumination sources that emit light, such as a flash. In various implementations, the one or more image sensor systems914further include an on-camera image signal processor (ISP) configured to execute a plurality of processing operations on the image data.

The memory920includes high-speed random-access memory, such as DRAM, SRAM, DDR RAM, or other random-access solid-state memory devices. In some implementations, the memory920includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory920optionally includes one or more storage devices remotely located from the one or more processing units902. The memory920includes a non-transitory computer readable storage medium.

In some implementations, the memory920or the non-transitory computer readable storage medium of the memory920stores an optional operating system930and one or more instruction set(s)940. The operating system930includes procedures for handling various basic system services and for performing hardware dependent tasks. In some implementations, the instruction set(s)940include executable software defined by binary information stored in the form of electrical charge. In some implementations, the instruction set(s)940are software that is executable by the one or more processing units902to carry out one or more of the techniques described herein.

The instruction set(s)940include a ground truth instruction set942, a training instruction set944, and a blur estimation instruction set946. The instruction set(s)940may be embodied a single software executable or multiple software executables.

In some implementations, the ground truth instruction set942is executable by the processing unit(s)902(e.g. a CPU) to generate training images (e.g., clear and blurred versions of an image of a shape) and/or target blur metrics associated with those training images as disclosed herein. To these ends, in various implementations, it includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the training instruction set944is executable by the processing unit(s)902(e.g., a CPU) to train a machine learning model, for example, using the ground truth data produced via execution of ground truth instruction set942, as discussed herein. To these ends, in various implementations, it includes instructions and/or logic therefor, and heuristics and metadata therefor.

In some implementations, the blur estimation instruction set946is executable by the processing unit(s)902(e.g., a CPU) to generate a blur assessment, for example, using a machine learning model trained by execution of training instruction set944, as discussed herein. To these ends, in various implementations, it includes instructions and/or logic therefor, and heuristics and metadata therefor.

Although the instruction set(s)940are shown as residing on a single device, it should be understood that in other implementations, any combination of the elements may be located in separate computing devices. Moreover,FIG.9is intended more as functional description of the various features which are present in a particular implementation as opposed to a structural schematic of the implementations described herein. As recognized by those of ordinary skill in the art, items shown separately could be combined and some items could be separated. The actual number of instructions sets and how features are allocated among them may vary from one implementation to another and may depend in part on the particular combination of hardware, software, and/or firmware chosen for a particular implementation.

The terminology used herein is for the purpose of describing particular implementations only and is not intended to be limiting of the claims. As used in the description of the implementations and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms “comprises” or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, objects, or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, objects, components, or groups thereof.