Patent Description:
The publication "<NPL>, discloses an AI model to estimate and correct distortion artefacts from an image acquired by a camera located below a windshield, using a machine learning approach.

Examples are disclosed that relate to machine-learning based methods of image restoration for through-display imaging. A method of training a machine learning model, according to claim <NUM> is disclosed. computing device comprising a display, a camera positioned behind the display, a logic subsystem, and a storage subsystem storing instructions executable by the logic subsystem according to claim <NUM> is disclosed.

User-facing cameras positioned in the bezel portion of a display may provide an unintuitive user experience for video conference/chat applications, as a user must maintain eye contact with the camera rather than a displayed image of a communication partner in order to maintain a semblance of "eye contact" with the communication partner. Similarly, a camera positioned adjacent to a display may be cumbersome during self-portrait capture, as a user may be unable to both gaze at the camera and a displayed camera stream. Placement of a camera in a device bezel also may pose constraints on bezel width and display area for a given display body size.

To address such issues, a camera may be positioned behind a device display, and capture images through the display. This may allow the use of a thinner bezel, and thus allow for a larger display area, for a given display body size. Such a behind-display camera also may provide for more natural gaze interactions for video conference/chat applications (as the camera may be positioned near or at a displayed image of a communication partner) and a more intuitive self-portrait experience.

However, capturing images through a display may be challenging, as positioning the display in front of a camera aperture may result in lower light transmission, lens occlusion, and diffraction, all of which may degrade image quality. For example, to allow imaging to be performed through an organic light emitting diode (OLED) display, an OLED pixel array may comprise a slit formed in each RGB pixel. Thus, light that enters the camera through the array of slits is diffracted by the slit pattern. The resulting images may be low-illuminated, noisy, and blurry, and thus may be unsuitable for face processing and other machine vision tasks.

Various methods of deconvolution have been used to compensate for blur in an image. Such methods may be combined with denoising algorithms to perform deblurring and denoising. Deconvolution, in the Fourier transform domain, boosts degraded signals by a multiplier to improve image quality. However, in instances where a camera acquires an image through a sufficiently narrow slit or other obstacle (e.g., a slit that comprises an area less than <NUM>% of the pixel area, and/or comprises a width that is less than <NUM>% of the pixel pitch), information may be completely lost. For example, for an image captured from behind an array of sufficiently narrow slits, a Fourier transform of the image may include a gap that contains little to no frequency data. The narrower the slit in the pixel structure, the broader this gap of missing data becomes. Further, the gap appears in the middle of the frequency band, where the eye may be most sensitive to missing information. Thus, the missing data may render the image quality unsuitable for self-portrait capture or video conferencing applications. When image information is totally lost in this manner, methods of deconvolution are unable to recover the missing information, as multiplying a signal of <NUM> by a multiplier returns a <NUM> signal.

Machine learning-based methods are used in some image upscaling applications. For example, convolutional neural networks have been used to upscale high-definition (or lower resolution) video content for display via a <NUM> or other ultra-high definition television by generating additional high frequency information (harmonics) and combining the harmonics with the high-definition video content. However, such techniques "fill in" additional detail-level image information at the extremities of a frequency band, and information at the middle of the frequency band is intact.

Accordingly, examples are disclosed that relate to machine learning-based methods of image restoration. Briefly, the disclosed examples provide a data acquisition pipeline for data preparation, namely, gathering training image pairs comprising degraded (blurry, noisy, and/or hazy) and undegraded (sharp) images to train an image restoration machine learning model. The term "image restoration" as used herein may encompass deblurring and denoising, as well as dehazing and/or color restoration/mapping processes, in various examples. Further, the term "training image pairs" and the like signifies that at least two versions of an image (one undegraded and one degraded) are used in training a model, and does not exclude the use of three or more versions of an image that include such a pair. The disclosed examples also provide a mechanism to generate synthetic training data utilizing computational methods to understand the modeling and degradation process of through-display imaging, in addition or alternatively to manually gathering undegraded/degraded image sets. Synthetic data may synthesize degradations caused by a display in a manner that is consistent with real data, and may also help the image restoration machine learning model generalize to more display aperture patterns. The image restoration machine learning model is trained to achieve a real-time inference from through-display image frames. When deployed on a live video system comprising a behind-display camera, the image restoration machine learning model receives input of an image acquired by the camera, generates frequency information that is missing in the image, denoises the image, and outputs a restored image. In some examples, the image restoration machine learning model may be configured to output restored images at a frame rate sufficiently close to video capture, e.g. to output restored, plausible image frames in real-time or near real-time.

Prior to discussing these examples, <FIG> depicts an example computing device <NUM> in the form of a smartphone comprising a camera <NUM> positioned behind a display <NUM>. Due to the camera position behind the display <NUM>, a size of the display is not constrained by placement of the camera and placement of the camera <NUM> is not constrained to an area adjacent to the display <NUM>. Thus, the display <NUM> shown in <FIG> comprises a size that extends substantially an entire area of a display side of the computing device <NUM>, and the camera is positioned at or near an estimated level at which the eyes of a video conference participant may appear on the display <NUM>, and/or the estimated location of a user's head displayed in a self-portrait imaging mode. Such a camera placement may help to provide more intuitive user experiences. In other examples, the camera <NUM> may be positioned in any other suitable location behind the display <NUM>.

While depicted as a smartphone in <FIG>, in other examples a computing device having a behind-display camera may take any other suitable form. Examples of such computing devices include tablet computers, laptop computers, televisions, video conferencing devices, computer monitors, and desktop computers (including all-in-one desktop computers).

Continuing with <FIG>, the camera <NUM> comprises a visible light (RGB or grayscale) image sensor and/or a near infrared (NIR) image sensor (e.g. RGB-IR). In some examples, the camera <NUM> may comprise a complementary metal oxide semiconductor (CMOS) image sensor and microlens array. In other examples, the camera may comprise a charge-coupled device (CCD). Further, in other examples, other optics in addition to or alternatively to a microlens array may be used. Additionally, in some examples a computing device may include multiple cameras in a stereoscopic arrangement, e.g. to obtain both intensity image data and depth image data, and/or may include one or more depth cameras (e.g. time-of-flight and/or structured light depth sensors). In yet other examples, when the numerical aperture is limited due to constraints on the total optical track length (TOTR), a computing device may include an array of cameras to increase a light collecting capability of a behind-display imaging system and thereby achieve higher signal-to-noise ratios without increasing a total optical track length (TOTR) of the imaging system. Some such examples are described in more detail below.

The display <NUM> comprises a plurality of pixels arranged in a pixel array. In some examples, the pixel array may take the form of an OLED, such as a transparent OLED (tOLED) or a pentile OLED (pOLED). In other examples, the display <NUM> may take the form of a liquid crystal display (LCD). While LCDs use backlights for illumination, pixels of OLEDs produce their own light and thus may have reduced reflection effects for through-display imaging compared to LCDs. Further, tOLEDs utilize transparent material to enable light to be emitted from both sides of the display. It will be understood that while examples are described herein in the context of tOLED and pOLED displays, the disclosed methods and systems are also applicable to other display contexts.

In some examples, the display <NUM> comprises a flexible/bendable display. A flexible/bendable display may comprise, e.g., a polyamide substrate on which the OLED is formed. Such a substrate may appear yellow in transmission. Thus, images captured through a polyamide-containing display panel by a behind-display camera may also appear yellow. An image restoration machine learning model as described herein may be trained to perform color restoration as part of an image restoration process, e.g. based on training data comprising images captured through the yellow-hued display panel and images not captured through the yellow-hued display panel.

<FIG> shows a schematic depiction of a plurality of pixels of an example tOLED configured for through-display imaging. The depicted tOLED display <NUM> comprises a vertical stripe pixel layout, and each pixel <NUM> comprises a vertically oriented slit through which a camera views the outside world. In some examples, an open area fraction of each pixel <NUM> may be on the order of <NUM>-<NUM>% of the pixel area, resulting in an approximately <NUM>% light transmission rate. In other examples, the open area fraction of each pixel and/or the light transmission rate may have values other than these, whether higher or lower. The narrower the slit for a transmission area of each pixel <NUM>, the less light enters the camera positioned behind the slit. Such loss of light may result in unwanted noise, as well as in diffraction-related blurring and loss of frequency information. In other examples, any other suitable array of slits may be used. The term "slit" as used herein refers to any opening in a pixel through which imaging may be performed, and does not imply any particular shape or orientation of the opening.

<FIG> shows an optical transmission microscope image of an example pOLED display <NUM> suitable for use as display <NUM>. The pOLED display <NUM> comprises a pentile pixel layout comprising a <NUM>% open area fraction. The pOLED sub-pixel design follows a RGBG matrix structure, and demonstrates a more complex pixel layout than the tOLED display <NUM>. While the open fraction area of the pOLED display <NUM> is <NUM>% larger than the open fraction area of the tOLED display <NUM>, light transmission for the pOLED display <NUM> may be significantly lower. The light transmission rate for the example pOLED display <NUM>, measured with a spectrophotometer and white light source, was ~<NUM>%. This relatively low value may be attributed to various factors, such as transparent traces that may scatter and diffract light, external Fresnel reflections, a circular polarizer, and/or a substrate (e.g., various glasses or plastics, which may absorb in some regions of the visible spectrum, for example) of the display panel. The low light transmission translates to less light reaching the camera lens, causing a relatively lower signal-to-noise ratio (SNR) compared to the tOLED display <NUM>.

As mentioned above, various problems that affect image quality may arise due to through-display imaging, including low light transmission rate (dark visual field), high-level complicated noise, severe diffraction effects (blur), color shift (color transform), and/or other complicated degradations, as examples. In the examples of <FIG> and <FIG>, images captured from behind the tOLED display <NUM> may exhibit degradation due to blur (horizontal) and noise, and images captured from behind the pOLED display <NUM> may experience degradation due to noise, haze, and low-light transmission.

<FIG> shows example diffraction patterns obtained by illuminating an example tOLED display <NUM> and an example pOLED display <NUM> with coherent red light in a wavelength of <NUM> nanometers (nm), revealing the point spread function (PSF) of each through-display imaging system. The tOLED display <NUM> comprises a grating-like structure of vertical slits, and the resulting diffraction pattern <NUM> extends primarily in a horizontal direction. The pOLED display <NUM> comprises a pentile structure and the diffraction pattern <NUM> extends in vertical and horizontal directions.

<FIG> shows an example white light PSF <NUM> for a camera positioned behind an example tOLED display having vertically oriented slits in pixels, such that the tOLED display functions as a mask in the optical path. The white light PSF <NUM> shows a spread over approximately <NUM> pixels in the x-direction (horizontal), which may be due to the grating-like nature of the mask. <FIG> shows a plot of the enclosed energy in the x and y directions as a function of arbitrary distance for an optical system comprising the example tOLED display in the optical path.

Combined degradation effects caused by through-display imaging through each of a tOLED display and a pOLED display were further characterized by plotting the modulation transfer function (MTF) for each of an optical system comprising an example tOLED display and an optical system comprising an example pOLED display. <FIG> shows MTF characteristics obtained by displaying a periodic sinusoidal pattern to the imaging system comprising the tOLED display and the imaging system comprising the pOLED display. Each MTF shows the influences of diffraction on image contrast. For each display type studied, the measured MTF takes into account the effect of finite lens aperture, lens performance, finite pixel size, noise, non-linearities, and quantization (spatial and bit depth). As shown in <FIG>, for the tOLED display, contrasts along the horizontal direction were mostly lost for some spatial frequencies. For pOLED, the contrast levels degraded when compared to a display-free camera, but did not reach zero for mid-frequency pattern. The plotting shown in <FIG> is consistent with the diffraction patterns revealed by the PSFs.

<FIG> shows a first image <NUM> of a real scene <NUM> as captured by a camera <NUM> looking through an example tOLED display having vertically oriented slits in pixels, and shows a second image <NUM> of the real scene <NUM> as captured by a camera <NUM> looking through an example pOLED display. The first image <NUM> is degraded by blur and noise, among other potential factors. The second image <NUM> is degraded by low light, color transform, high noise, and blur, among other potential factors. Neither image <NUM>, <NUM> plausibly portrays the real scene <NUM>.

Various other sources of degradation may affect images captured from behind a display panel. <FIG> depicts a first image <NUM> of a real-world environment that was captured by a camera <NUM> without a display panel disposed between the camera lens and the real-world background, and a second image <NUM> that was captured by the camera <NUM> positioned behind an example tOLED display having vertically oriented slits in pixels. Bands (one of which is shown at <NUM>) resembling the vertical stripe pixel layout of the tOLED display are visually perceptible in the second image <NUM>.

A degradation pipeline may be used to model the combined degradation effects on an image captured via a behind-display camera. Given a degradation-free image x, a degraded observation y may be formulated according to Equation (<NUM>), where β is the light transmission rate, k is the point spread function, and n is noise.

The image restoration techniques described herein utilize machine-learned methods to restore images captured via a behind-display camera. Prior to deploying and implementing an image restoration machine learning model (e.g. in a live imaging context), the image restoration machine learning model is trained via input of training data comprising undegraded/degraded image pairs, where each undegraded/degraded image pair comprises an undegraded image and a degraded image that represents an appearance of the undegraded image to a behind-display camera.

<FIG> shows aspects of an example data acquisition pipeline <NUM> for generating training data comprising undegraded images and degraded images. The data acquisition pipeline <NUM> includes a display device <NUM>, a camera <NUM> positioned a predetermined distance from the display device <NUM>, and a display sample <NUM> disposed in the optical path between the camera <NUM> and the display <NUM>. The camera <NUM> in this example is positioned at a centerline of the display device <NUM> and oriented/adjusted to capture images of the display device <NUM>, but may have any other suitable position in other examples. The distance between the camera <NUM> and the display device <NUM> is selected such that the camera <NUM> images the full monitor range of the display device <NUM>. In some examples, the data acquisition pipeline also may include a black box cover <NUM> surrounding the display device <NUM>, camera <NUM>, and display sample <NUM>, which may help to prevent light of sources other than the display device <NUM> from reaching the camera aperture.

The display device <NUM> is configured to display images at a suitably high-resolution. For example, the display device <NUM> may comprise a <NUM> LCD or other ultra-high definition (UHD) monitor or television. In other examples, the display device may have any other suitable resolution. In some examples, during data collection, a standard red, green, blue (sRGB) color space of the display device <NUM> may be set such that the true color shown on the display device <NUM> follows a gamma curve with approximately y = <NUM>. In this manner, the sRGB color shown on the display device <NUM> may be mapped back to the irradiance of the original images using an sRGB transfer function.

The camera <NUM> comprises a visible light (RGB or grayscale) machine vision camera. In one experiment, the camera <NUM> comprised a <NUM> RGB machine vision camera, but may be any other suitable camera in other examples. Any suitable images may be selected for display in the data acquisition pipeline <NUM> for image capture via the camera <NUM>. Suitable images include images comprising visual information from within a frequency band (e.g., F<NUM> in the Fourier transform of the image) that would be degraded or lost due to diffraction from the display sample <NUM>. As examples, photographs of buildings, people, and/or landscapes may be suitable for inclusion in the data acquisition pipeline <NUM>, as such images comprise medium-frequency information that would be lost due to diffraction.

In some examples, the data acquisition pipeline <NUM> may utilize synthetic images having pseudo-random patterns of black and white lines, which may help to provide the image restoration machine learning model with isolated examples of sharp lines/edges. In one specific example, images from the DIV2K dataset (<NPL>) may be used as training images.

To obtain a collection of undegraded images for the training dataset, the camera <NUM> captures images displayed on the display device <NUM>. The data acquisition pipeline <NUM> may simulate a real-time video system. In one specific example, a frame rate of the camera <NUM> is set to <NUM> frames per second (fps) and a maximum shutter speed of <NUM> milliseconds (ms), which may help to capture images having a suitably high SNR. In another example, multiple images may be acquired of each image displayed on the display device, and the multiple images may be averaged to obtain undegraded images. Prior to image acquisition, the camera <NUM> may be calibrated (e.g. by measuring a full-screen white image and a full-screen black image displayed via the display device <NUM>, respectively, to set white and black levels).

In some examples, the undegraded images are captured while no display sample <NUM> is positioned in the optical path between the camera <NUM> and the display device <NUM>. In other examples, undegraded images are captured while a display sample <NUM> comprising an optically transparent plate is positioned in the optical path. The use of the display panel comprising the optically transparent plate may act as a proxy for a total thickness of the display and cover glass. As such, when included, the optically transparent plate may comprise a same thickness as a display sample(s) used for degraded image acquisition. Further, in some examples, computing and saving the average of a plurality (e.g., <NUM>-<NUM>) of repeated captured frames may help to form a raw, undegraded image and reduce or eliminate image noises in the undegraded image set.

To obtain a collection of degraded images for the training dataset, the camera <NUM> captures the same plurality of images of the display device <NUM>, but the display sample <NUM> positioned in the optical path comprises a display panel (tOLED, pOLED, etc.) or a mask based upon a display panel design, and the specific gain value is adjusted. This training data generation procedure creates a <NUM>-<NUM> correspondence between undegraded and degraded images without having to rescale, crop, or otherwise process the images acquired, which may result in a simpler machine-learning model that is amenable to lightweight hardware implementations.

As mentioned above, the specific gain value is set for degraded image acquisition. In an example, for images acquired through a tOLED display having vertically oriented slits in pixels, the gain value may be calibrated to <NUM> dB to match the intensity range of the undegraded images. The collected undegraded/degraded pairs are naturally well aligned spatially in pixel-level. In some examples, the data for training may be further normalized or warped, e.g. to make a point spread function relatively consistent over an entire image and thereby help to simplify an image restoration machine learning model. In other examples, further normalization and/or warping may be omitted. For images acquired through a pOLED display, because of the color substrate of the pOLED, some lights may be filtered out and break out the white balance. In some examples, the original intensity level of the images is maintained without re-adjusting the white balance. As the light transmission rate for the pOLED display may be low (~<NUM>%), the gain value may be set to, or close to, a maximum of the camera (e.g. <NUM> dB), to obtain higher signal values.

In some examples, the data acquisition pipeline <NUM> obtains a third set of training data images-noise-free degraded images. Noise-free degraded images may be formed by, during degraded image acquisition, averaging a plurality (e.g., <NUM>-<NUM>) of repeated captures. Further, in some examples, obtaining each set of training data images (undegraded, degraded, and optionally noise-free degraded) may comprise recording both <NUM>-bit raw sensor data and <NUM>-bit sRGB data after camera processing. <FIG> depicts examples of training data collected using the data acquisition pipeline <NUM> of <FIG>. The training data for tOLED degradation includes an undegraded image <NUM>, a noise-free degraded image <NUM>, and a noisy degraded image <NUM>. The training data for pOLED degradation likewise includes an undegraded image <NUM>, a noise-free degraded image <NUM>, and a noisy degraded image <NUM>.

In some examples, training data may be generated using computerized models of image degradation, in addition or alternatively to manually generating training data via the data acquisition pipeline <NUM>. Computerized generation of training data may help to build larger training datasets in less time compared to manual training data acquisition. Further, such methods may help to generalize the image restoration machine learning model to a variety of different display (and thus degradation) types.

As mentioned above, training image data comprising degraded and undegraded image pairs are input as training data into the image restoration machine learning model to train the image restoration machine learning model. The image restoration machine learning model is trained to identify one or more missing frequencies in the frequency domain and generate frequency information to "fill in" the missing frequency information (e.g. perform texture analysis, edge detection, style transfer, etc.). When restoring an image for video conferencing, the restored image does not necessarily need to be a scientifically accurate image for the human eye to perceive it as a plausible image.

In some examples, the image restoration machine learning model comprises a neural network, which is flexible and may combine various aspects of algorithms into a shared framework. <FIG> depicts an example image restoration machine learning model in the form of a neural network <NUM> configured to restore images degraded due to through-tOLED imaging. The neural network <NUM> is a multiscale convolutional network comprising a pre-computed regularized one-dimensional kernel. As mentioned above with reference to <FIG>, the PSF for an optical system comprising the example tOLED display positioned between the camera and an image scene is approximately <NUM> pixels. The blur in an image captured by the optical system may be larger in scale than features in the image, as the blur arises from a slit pattern of the tOLED display positioned between the camera and the real-world scene. To deblur the image, the neural network may utilize a kernel size larger than the PSF. Degraded training data images may also be pre-processed such that the PSF is relatively consistent over the field. In the example of <FIG>, the neural network <NUM> comprises a one-dimensional deconvolution kernel (1x121) based at least on the slit shape and size of the tOLED pixels. While described with reference to image degradation caused by the tOLED display <NUM> of <FIG>, the neural network <NUM> may similarly be used to restore images degraded by the presence of a different type of display positioned in the optical path. A differently configured kernel may be selected for a display comprising different diffraction behavior than a tOLED having vertically oriented slits in pixels.

The neural network <NUM> further comprises convolution shapes trained to perform aspects of image restoration (deblurring, denoising, and optionally dehazing and/or color mapping).

Image data comprising at least pairs of degraded images and undegraded images are input as training data for training the neural network. Provided a degraded image that, in the frequency domain, is missing a frequency band centered on F<NUM>, and without wishing to be bound by theory, training of the neural network may configure the neural network to identify a presence of harmonic frequencies (e.g., <MAT>, or other frequencies centered around multiples or submultiples of F<NUM>, such as <MAT> where m and n are integers), infer that F<NUM> is a missing frequency in the degraded image based on the harmonics identified, and reconstruct the image to include the missing frequency F<NUM>. Where the neural network is configured spatially rather than in frequency space, the recovery functionality may be implemented via spatial filters followed by non-linear operations, along with subsampling and super-sampling of the images at the various stages of the neural network. Further, in some examples, the network may, through training, synthesize image detail in regions of the degraded image where the network would expect such details to exist in the undegraded image.

<FIG> depicts a plot of an example loss function <NUM> for the neural network <NUM> of <FIG>. The depicted loss function is a combination of least absolute deviations (L1) loss and texture loss, which is broadly a cross-correlation of difference between patches of the images. In some examples, the loss function may additionally or alternatively include lease square errors (L2) loss. Prior to computing the loss, the output of the neural network <NUM> was passed through a filter to emulate the human visual response. In other examples, the output of the neural network <NUM> may be passed through one or more layers of another pre-trained network (e.g., VGG-<NUM>, of Simonyan, K. , and Zisserman, A. , "Very Deep Convolutional Networks for Large-Scale Image Recognition" International Conference on Learning Representations, <NUM>, made publicly available by Oxford University, UK, http://www. uk/~vgg/research/very_deep/) before computing the loss, which may also help to provide human vision emulation.

<FIG> depict experimental image restoration results obtained using the neural network <NUM> of <FIG>. In this example, monochrome image <NUM> is an image of a real-world scene captured via a camera unobstructed by a display, monochrome image <NUM> is a degraded image captured via a camera positioned behind a tOLED display with respect to the real-world scene, and monochrome image <NUM> is a restored image output by the neural network <NUM> in response to input of the degraded image <NUM>. Averaged amplitude spectra of each image are shown in <FIG>.

The image restoration machine learning model may recover an RGB image using a similar network architecture to the neural network used for monochrome images, as indicated by the example original RGB image <NUM>, degraded RGB image <NUM>, and restored RGB image <NUM> shown in <FIG>. Input color images may be noisier and include more artifacts than monochrome images.

In some examples, spatial frequency response for a behind-display imaging system may be improved by illuminating the behind-display imaging system with other wavelengths of light, in addition to visible light. <FIG> depicts the MTF curves of red, green, and blue wavelength bands of visible light for a behind-display imaging system comprising an example tOLED display having vertically oriented slits in pixels. The MTF is diffraction-limited and scales with wavelength. Adding wavelengths of infrared (IR) light may at least partially "fill in" missing portions of the MTF visible light curve, as shown in <FIG>.

In a convolutional neural network, pixel nodes at a level in the network are not fully connected to those of an adjacent level, but rather are connected to spatially proximate pixel nodes. Matching the deconvolution kernel at least to the spread of the point-spread function followed by a relatively lightweight series of convolutions (e.g., to perform denoising), as shown in the example of <FIG>, may achieve acceptable output images at video frame rates and with relatively low network complexity. This may provide for a relatively lightweight framework from a compute and memory usage standpoint, and thereby may allow real-time processing or near-real-time processing on a video stream. However, the smaller receptive field for such a neural network may not react to longer range features in images. As one possible solution, a fully connected network may be used, in which every pixel in the output image is connected to every pixel in the input image. While the image restoration provided by such a network may be superior to convolutional networks with a smaller receptive field, the memory and compute requirements are much higher.

Thus, a machine learning model may be configured to look at more scale lengths in an input image, without creating an unduly large kernel, by looking at downsampled versions of the input image alongside a full resolution version of the image. As one example architecture, a u-shaped neural network may allow for a larger receptive field than a convolutional neural network while being less memory and compute-intensive than a fully connected network.

<FIG> is a block diagram illustrating an example u-shaped neural network <NUM> suitable for use as the image restoration machine learning model. The u-shaped neural network <NUM> is configured to progressively reduce resolution of an input image, which allows the u-shaped network to look at a larger receptive field of the input image. Further, the u-shaped neural network <NUM> is configured to preserve high frequency information in the image at each downsampling/extraction stage. This preserved feature information is used in the upsampling process, such that the u-shaped neural network <NUM> outputs an image comprising a same size (resolution) as the input image without loss of high-frequency information. In contrast, a downsampling path of a convolutional neural network discards high frequency information.

The u-shaped neural network <NUM> comprises two sub-encoders (as opposed to the single encoder in existing UNet architectures). One sub-encoder <NUM> computes residual details to add, and the other sub-encoder <NUM> learns content encoding from degraded images. By splitting the encoder into two (or more) sub-encoders, the u-shaped neural network <NUM> may have fewer parameters compared with doubling the width of each layer and may more efficiently infer and learn compared to a single encoder network.

The u-shaped neural network <NUM> is configured to receive input of a four-channel raw sensor data observation y and to output a restored three-channel image x. Training the u-shaped neural network <NUM> from paired undegraded/degraded images comprises applying the L1 loss from image reconstruction to the model <IMG>(y;θ). The objective function may be represented according to Equation (<NUM>), where θ is the learnable parameters of the network, <IMG> represents the network, yi is the observation of degraded images, xi is the ground-truth undegraded image, and N is the total number of images inside a training batch. In some examples, training the u-shaped neural network also comprises applying SSIM and perception loss (VGG Loss) for an ablation study (which examines the functions of selected components in a neural network by replacing or removing such components and observing performance changes of the network <NUM>).

One example learning method for the example u-shaped neural network comprises splitting a training dataset into a training partition and a validation and testing partition prior to inputting training data into the u-shaped neural network <NUM>. The images in the training partition are augmented using raw image augmentation (e.g., as described by <NPL>) while preserving the Bayer pattern. Training, in this example, further comprises using the Adam optimizer with a learning rate of 1e-<NUM> and a decay factor of <NUM> after <NUM> epochs and stopping training at epoch <NUM>.

<FIG> and <FIG> show restoration results on real paired data for tOLED and pOLED degradation types, respectively, achieved via the u-shaped neural network <NUM>. Quantitative results are also listed below in Table <NUM>, which compare the network <NUM> with a baseline real blind denoising algorithm Pixel-shuffle Downsampling (PD) (described by <NPL>), and an unsupervised Wiener filter (WF) (described by <NPL>) given a pre-computed PSF. The performance of the u-shaped neural network <NUM> is reported in peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), learned perceptual image patch similarity (LPIPS) (described by <NPL>), and inference time. The u-shaped neural network <NUM> directly trained on paired data demonstrated better restoration quality. The results were also compared with another Resnet-based structure for super-resolution, EDSR (described by <NPL>). Though EDSR achieved a comparable performance to the U-shaped neural network <NUM>, it required longer inference time.

An ablation study of the u-shaped neural network <NUM> image processing pipeline on loss function and image format was conducted and quantitative results are shown in Table <NUM>, below. Replacing raw data with sRGB data increased inference time and decreased PSNR. Adding SSIM or perception loss increased the perceptive performance. The results also compare the u-shaped neural network <NUM> to a different UNet structure, and demonstrate that the u-shaped neural network <NUM> comprising separate subencoders have an intermediate parameter size, inference time, and relatively better performance in PSNR.

Accordingly, the disclosed examples provide learning-based methods that may help to restore complicated combined image degradation resulting from through-display imaging. In some examples, an image restoration machine learning model may also be trained to perform tasks other than image restoration. For example, an image restoration machine learning model may be trained to perform recognition/classification tasks in addition to image restoration. In one specific example, image degradation caused by covering a camera with a display panel (e.g. tOLED, pOLED, LCD, etc.) may decrease a recall rate for face detection to as small as zero. After image restoration according to the methods described herein, the recall rate may be recovered to an acceptable level.

Aspects of the image restoration machine learning model may be implemented locally in hardware of a computing device, e.g. as stored instructions executable by a logic subsystem of the computing device, and/or may be located remotely from the computing device (e.g. as a cloud-based service). Examples of computing hardware suitable for implementing the image restoration machine learning model are described below with reference to <FIG>. In any instance, the image restoration machine learning model (e.g., CNN <NUM> or u-shaped NN <NUM>) may be configured to perform image processing at or near video frame rates during use of a behind-display camera for live video applications. Further, the image restoration machine learning model may optionally utilize frame-to-frame averaging, which may help to improve output image quality.

In the examples described above, neural networks are disclosed performing deblurring and denoising, wherein denoising helps to correct for attenuation of the optical signal arising from the through-display imaging. Some displays may attenuate the optical signal significantly. For example, some pOLED displays may have a transmittance as low as <NUM>%. This significantly reduces the optical signal, and hence the signal to noise ratio of the system, which may be as low as one in some indoor environments.

The signal on an image sensor is proportional to the transmittance of the display and optics, the square of the lens numerical aperture (<NUM>/(<NUM> f/#)) and the pixel area. Given the screen transmittance, the free parameters for a behind-display camera are the lens numerical aperture and the pixel area. However, the pixel area in a camera is constrained by the field of view of the camera and the available thickness behind the screen to accommodate the camera. Computing devices may have little available interior space in a thickness dimension. For example, mobile devices may have only a few millimeters (e.g. approximately <NUM> in one example) of interior space from the rear of the display to the opposite interior surface of the case. In contrast, an image sensor and associated printed circuit board may have a thickness of around <NUM>, plus around <NUM> microns of clearance for tolerance purposes, in some examples. As such, the available total optical track TOTR (distance from first lens surface to image plane) is only <NUM> in this illustrative example. The focal length may be approximately TOTR/<NUM> or less for f/<NUM> or f/<NUM> lenses. As increasing the aperture to admit more light (lower f-number) increases the TOTR for a given focal length (e.g. lens elements may be thicker, and/or extra elements may be needed for aberration control), the aperture size is constrained by this dimension. As such, it may be difficult to achieve improved performance by increasing the numerical aperture of a camera. Further, as the field of view of the camera is fixed and the resolution of the output image is also specified, pixel dimensions, and thus the light collecting power of the system, are similarly constrained.

Thus, as mentioned above, a computing device may include an array of cameras to increase a light collecting capability of a behind-display imaging system and thereby achieve higher signal-to-noise ratios without increasing a TOTR of the imaging system (see, e.g. <NPL>). Any suitable number of cameras in any suitable arrangement may be used, some examples of which are described below. Each camera of the array provides an image that is at or near the resolution of the output image, and the resulting image data is combined using a neural network, such as those described above. Such a neural network may be trained to manage the image disparity from the camera array as well as to perform denoising and deblurring. Image disparity arises due to the cameras being separated. Although the images can be registered together, each camera will have a slightly different view of the scene. Simply correcting the disparity by translation of the image may result in a local correction, but objects at different depths and at different points in the field will have different disparities. As such, a simple single geometric transformation for each camera may not correct for all such disparities. In contrast, a neural network trained on ground truth images compared to images acquired via such a camera array may correct such disparities globally in an image.

In some examples, the cameras of a camera array may comprise separate camera modules. In other examples, a camera array, or a portion of a camera array, may be constructed in an integrated fashion so that all cameras of the integrated array form images on a same image sensor. For example, the camera lenses may be formed in a single integrated block, or may be provided as individual camera lens assemblies that are positioned in front of a same image sensor. As a more specific example, <FIG> shows schematic representations of (a) an example single <NUM>-element lens design, at <NUM>; (b) four such lens assemblies in an example 2x2 camera array, at <NUM> (other array sizes and geometries may be used in other examples); (c) an exploded view, at <NUM>, of an integrated camera array comprising a layered construction in which each layer of the stack comprises one of the elements of lens design <NUM> arrayed in a 2x2 fashion; and (d) the integrated camera array of <NUM> in an assembled stack construction, at <NUM>.

The use of a camera array (such as camera array <NUM> and integrated camera array <NUM>) may provide advantages over a single behind-display camera in settings in which the diffraction of the light through the display occurs in the presence of bright light sources. Common use scenarios for cameras on computing devices are self-portraits ("selfies") and videoconferencing, both of which may occur in uncontrolled lighting environments. As an example, in a home or office setting, the illuminance on the subject's face may be on the order of fifty lux, such that the facial luminance may be on the order of ten nits. However, there may be light sources in the room having a luminance on the order of <NUM>,<NUM> - <NUM>,<NUM> nits. In such environments, an auto gain system in the host device firmware or in the ISP of the camera may set the exposure level such that the face is correctly exposed. However, even though the relative diffraction efficiency through the screen in the high orders (at large angles) may be very low, e.g. <NUM>%, the image illuminance due to diffraction from the light source may exceed that from the face. In cases where the light source is in the field of view and forms part of the image, a neural network may be trained to deal with the light source. In other cases, the light source itself may lie outside of the field of view of the camera, yet the diffracted light appears inside the field of view. In this case there is less information for the neural network regarding the distribution of the light sources, which may complicate image recovery. <FIG> schematically illustrates an example of such a scenario, where a bright light source <NUM> is outside of a field of view <NUM> of a camera <NUM> (which represents a behind-display camera) while a main subject <NUM> is within the field of view <NUM>. <FIG> shows an example image <NUM> image acquired under conditions in which a strong light source is just outside the field of view. In image <NUM>, a diffraction pattern <NUM> arising from the out-of-view light source can be seen extending across an upper portion of the image.

To mitigate the effects of light sources that are out of the field of view of a behind-display camera, at least one camera in the array may be configured to identify such out-of-view light sources. In some examples, the camera may be a wide-angle camera, such that the width of the field of view is sufficient to capture higher diffraction orders from the screen. <FIG> schematically shows an example of such a camera array <NUM>, which comprises four narrower angle cameras <NUM> configured to sense a main subject <NUM>, and one wider angle camera <NUM> configured to sense light sources out of view of the narrower angle cameras <NUM>, such as light source <NUM>.

The additional width of the field of view <NUM> for wider angle camera <NUM> may be selected based at least partially upon a diffraction efficiency of the screen as a function of angle (E(θ)), and also at least partially upon a luminance range, which can also be called the capture range (R). The capture range R is the ratio of the lowest scene luminance to the maximum light luminance. The maximum additional angle θmax is given by the solution for θ in E(θ) = R. In other words, the wider angle camera lens may be selected in some examples to capture all light that has relative diffraction efficiency greater than R.

Diffraction peaks will occur at values of θ given by the grating equation (sin(θ) = m λ/d, where m is integer, λ is the wavelength and d the pixel pitch). The relative diffraction efficiency versus angle is determined by the individual slit width, a, as follows.

Substituting for the values of sin(θ) from the grating equation, the relative diffraction efficiencies are given by the following.

As an example, to accommodate a range of intensities of <NUM>,<NUM>: <NUM>, E(m) = <NUM>. By way of a more specific example, consider an array of pixel pitch <NUM> with slits of width <NUM>. The solution for the largest m is m = <NUM>. The <NUM>th diffraction order occurs at an angle of <NUM>°, indicating that the wide-angle lens may be selected to accommodate a total angle equal to the original lens plus twice <NUM>°. Where the narrower angle camera lenses have a field of view of +/- <NUM>° horizontal and around +/- <NUM>° vertical, the wider angle lens may be selected to accommodate a field of view of <NUM>° x <NUM>°. This example demonstrates that, although the first order diffraction angle may be small for typical display devices, a wider angle lens may be selected to have a larger field of view than a simple first diffraction order calculation may indicate.

In such examples, the wider angle image may be acquired with lower gain than the narrower angle cameras. However, in a single-sensor implementation, the exposure time and analog gain are fixed for the whole sensor. In this case an attenuating optical filter (a neutral density filter) may be used over the wider angle camera aperture so that the bright lights remain within the capture range of the sensor.

The use of a camera array affords the opportunity to utilize subtractive color. The conventional approach to implementing color on a sensor is to place a red/green/blue (RGB) (or red/green/green/blue (RGGB)) filter array over the sensor pixels in the so-called Bayer pattern. The image is then recovered by interpolation over this array of pixel values. RGB is an additive color scheme in which the individual spectra color filter spectra do not significantly overlap. CMY is an alternative coding scheme in which the spectra do significantly overlap. An advantage of CMY is that each of the C, M and Y filters is roughly twice as wide as an individual R, G or B. As such, there is an increase in optical throughput compared to RGB. This may provide a signal to noise advantage over RGB filters. In some examples, CMY filters may be implemented at the pixel level, while in other examples the C, M and Y filters may be placed over individual cameras in a camera array, rather than in a Bayer-like matrix dispersed over individual pixels. An RGB implementation may have the filters disposed in a like manner. The RGB and/or the CMY implementations may be augmented by the addition of a clear channel (W) which accepts all visible wavelengths. This may afford signal to noise advantages, but also may pose challenges in reconstructing accurate color. However, in the context of a behind-display camera array with reduced signal levels, the improved signal to noise ratio may be favored over preserving nuances of color. <FIG> schematically illustrates various example color filter arrangements shown in a 3x3 array that represents a 3x3 camera array. In this figure, RBG denotes a Bayer color filter array on a camera, whereas individual letters indicate individual color filters located over a corresponding camera. More particularly, filter arrays <NUM> and <NUM> illustrate various example Bayer array arrangements, <NUM> and <NUM> illustrate various individual RGB and RGBW filter arrangements, and <NUM>-<NUM> illustrate various CMY and CMYW arrangements. In other examples, arrays of other sizes/dimensions may utilize any other suitable arrangement of color filters.

<FIG> shows an example computer-implemented method <NUM> for training a machine learning model. The method includes, at step <NUM>, acquiring training image pairs. In some examples, the training image pairs are acquired using a mask positioned in front of a camera, such as in the data acquisition pipeline <NUM> described above, as indicated at <NUM>. In other examples, the training image pairs comprise synthesized data, as indicated at <NUM>. While described herein in the context of image pairs, in some examples, the training image pair is a part of a set of three or more corresponding training images, as indicated at <NUM>. Further, in some examples, a noise-free image is acquired as an average of a plurality of repeated captured frames, as indicated at step <NUM>. Additionally, in some examples, the acquiring the degraded image comprises acquiring the degraded image via a camera array, as indicated at <NUM>. In some such examples, a plurality of lenses focus an image on a same image sensor, while in other examples each of two or more cameras includes a separate image sensor. Further, in some such examples, the camera array may comprise one or more wider angle cameras to locate light sources outside of a field of view of one or more narrower angle cameras of the camera array.

At <NUM>, method <NUM> comprises providing the training image pairs to a machine learning model. In some examples, the training image pairs are provided to a convolutional neural network as in step <NUM>. In some examples, the training image pairs are provided to a U-shaped neural network, as in step <NUM>. At <NUM>, method <NUM> comprises training the machine learning model to generate missing frequency information and/or deblur images. Further, in some examples, the machine learning model may be trained to correct for disparities between images from different cameras of a camera array, as indicated at <NUM>.

<FIG> shows an example method <NUM> for restoring a degraded image via a machine learning model that has been trained, for example, according to method <NUM>. At <NUM>, method <NUM> comprises acquires a degraded image via a behind-display camera. In various examples, method <NUM> comprises acquiring a degraded image from a camera behind a tOLED display <NUM>, a camera behind a pOLED display <NUM>, or an LCD display <NUM>. Further, in various examples, method <NUM> includes acquiring the degraded image via a camera array positioned behind the display <NUM>, which may provide a greater signal to noise ratio than the use of a single camera behind the display.

Method <NUM> further comprises providing the degraded image to a machine learning model, at <NUM>, which outputs the restored image at <NUM>. In some examples the restored image is output in real time or near-real time, for example, to a video application <NUM>, such as a video conferencing application.

In some examples, the machine learning model is hosted by remote service <NUM>. In other examples, the machine learning model may be local to the imaging device, as indicated at <NUM>. In yet other examples, computer-executable logic to perform the image restoration may be distributed between the local device and a remote system. Any suitable machine learning model architecture may be used to perform the image recovery. In some examples, a convolutional neural network <NUM> is used, while in other examples, a U-shaped neural network <NUM> is used. In some such examples, the U-shaped neural network comprises two or more subencoders <NUM>.

Computing system <NUM> includes a logic subsystem <NUM> and a storage subsystem <NUM>.

Logic subsystem <NUM> includes one or more physical devices configured to execute instructions. For example, the logic subsystem may be configured to execute instructions that are part of one or more applications, services, programs, routines, libraries, objects, components, data structures, or other logical constructs.

The logic subsystem <NUM> may include one or more processors configured to execute software instructions. Additionally or alternatively, the logic subsystem may include one or more hardware or firmware logic subsystems configured to execute hardware or firmware instructions. Processors of the logic subsystem may be single-core or multicore, and the instructions executed thereon may be configured for sequential, parallel, and/or distributed processing. Individual components of the logic subsystem optionally may be distributed among two or more separate devices, which may be remotely located and/or configured for coordinated processing. Aspects of the logic subsystem may be virtualized and executed by remotely accessible, networked computing devices configured in a cloud-computing configuration.

Storage subsystem <NUM> includes one or more physical devices configured to hold instructions executable by the logic subsystem to implement the methods and processes described herein. When such methods and processes are implemented, the state of storage subsystem <NUM> may be transformed-e.g., to hold different data.

Storage subsystem <NUM> may include removable and/or built-in devices. Storage subsystem <NUM> may include optical memory (e.g., CD, DVD, HD-DVD, Blu-Ray Disc, etc.), semiconductor memory (e.g., RAM, EPROM, EEPROM, etc.), and/or magnetic memory (e.g., hard-disk drive, floppy-disk drive, tape drive, MRAM, etc.), among others. Storage subsystem <NUM> may include volatile, nonvolatile, dynamic, static, read/write, read-only, random-access, sequential-access, location-addressable, file-addressable, and/or content-addressable devices.

It will be appreciated that storage subsystem <NUM> includes one or more physical devices.

The term "program" may be used to describe an aspect of computing system <NUM> implemented to perform a particular function. In some cases, a program may be instantiated via logic subsystem <NUM> executing instructions held by storage subsystem <NUM>. It will be understood that different programs may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Likewise, the same program may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The term "program" may encompass individual or groups of executable files, data files, libraries, drivers, scripts, database records, etc..

When included, display subsystem <NUM> may be used to present a visual representation of data held by storage subsystem <NUM> This visual representation may take the form of a graphical user interface (GUI). As the herein described methods and processes change the data held by the storage subsystem, and thus transform the state of the storage subsystem, the state of display subsystem <NUM> may likewise be transformed to visually represent changes in the underlying data. Such display devices may be combined with logic subsystem <NUM> and/or storage subsystem <NUM> in a shared enclosure, or such display devices may be peripheral display devices.

When included, input subsystem <NUM> may comprise or interface with one or more user-input devices such as a keyboard, mouse, touch screen, game controller, and/or camera (e.g. a behind-display camera). Example NUI componentry may include a microphone for speech and/or voice recognition; an infrared, color, stereoscopic, and/or depth camera (behind-display or other) for machine vision and/or gesture recognition; a head tracker, eye tracker, accelerometer, and/or gyroscope for motion detection and/or intent recognition; as well as electric-field sensing componentry for assessing brain activity.

As non-limiting examples, the communication subsystem <NUM> may be configured for communication via a wireless telephone network, or a wired or wireless local- or wide-area network. In some embodiments, the communication subsystem <NUM> may allow computing system <NUM> to send and/or receive messages to and/or from other devices via a network such as the Internet.

Claim 1:
A method of training a machine learning model, the method comprising:
inputting training image pairs into the machine learning model, each training image pair comprising an undegraded image and a degraded image that represents an appearance of the undegraded image to a behind-display camera which is positioned behind a display comprising a plurality of pixels and a plurality of openings in pixels through which light enters the camera , wherein the openings are sufficiently narrow so that a transform of the degraded image into the frequency domain includes a gap that contains little to no frequency data in at least one frequency band due to diffraction; and
training the machine learning model using the training image pairs to generate frequency information that is missing from a transform into the frequency domain of the degraded images.