Simultaneously correcting image degradations of multiple types in an image of a face

Described herein are technologies related to correcting image degradations in images. An image is received, and values for features that are usable to correct for image degradation associated with blur, noise, and low light conditions are generated by separate encoders based upon the received image. A fusion network learned by way of network architecture search fuses these values, and the fused values are employed to generate an improved image, such that the image degradations associated with blur, noise, and low light conditions are simultaneously corrected.

BACKGROUND

Videoconferencing applications are becoming increasingly popular, where two or more people communicate in real-time by way of networked computers. In operation, a first client computing device that executes a videoconferencing application is employed by a first user who is participating in a videoconference, and a camera that is included in or coupled to the client computing device captures images of the face of the first user. The first client computing device transmits the images to a second client computing device operated by a second user who is participating in the videoconference, whereupon the images of the face of the first user are presented on a display that is coupled to or included within the second client computing device. To optimize the experience of the users participating in videoconferences, it is desirable that images of faces of other participants in the videoconference shown to such users are of high quality.

Images of faces captured during videoconferences, however, often include one or more types of degradation; for example, an image of a face of a videoconference participant may be blurry (which may be caused by motion of the videoconference participant relative to the camera that captures images of the face of such participant). In another example, the image the face of the videoconference participant may include undesirable noise, where the noise in the image may be caused by noise in a communications channel between client computing devices, may be introduced when the image is compressed, etc. In still yet another example, an environment of a videoconference participant may have poor lighting conditions (e.g., too little light), and therefore an image of the face of the videoconference participant may include degradations associated with poor lighting conditions.

Computer-executable software has been developed to improve quality of an image of a face of a user. For instance, first computer-executable software has been developed to correct for blurring in images, where such first software includes a first encoder/decoder pair; second computer-executable software has been developed to correct for noise in images, where such second software includes a second encoder/decoder pair, and third computer-executable software has been developed to correct for degradations associated with low lighting conditions, where such third software includes a third encoder/decoder pair. The aforementioned software, however, are collectively ill-suited for use in a videoconferencing environment, because each of the first, second, and third software operates independently from the others. More specifically, for example, using the above-described software to correct for multiple types of image degradation requires that an image of a face of a user must first be provided to the first software (to correct for blurring), and the resultant corrected image must then be provided to the second software (to correct for noise), and the resultant further corrected image must thereafter be provided to the third software (to correct for low lighting conditions evident in the image).

Utilizing such a sequential approach, however, is not well-suited for videoconferencing applications, as: 1) image correction performed sequentially, as described above, may not be able to be completed quickly enough to allow for each frame in a video feed to be subject to correction, and 2) a correction made by one software module may introduce an image degradation of a type that was previously corrected for by another software module. Thus, the second software module that is configured to correct for noise in an image of a face of a user may introduce degradations associated with blur into the image, where blur was previously corrected for by the first software module.

SUMMARY

Described herein are various technologies relating to simultaneous correction of many types of degradation in an image (e.g., an image that includes a face of a person). Exemplary types of image degradation include blur, noise, and degradation caused by poor lighting conditions when the image was captured (such as low lighting conditions). In an example, prior to an image being subjected to correction for different types of image degradation, an identity vector for a person captured in the image can be constructed based upon several “clean” images that include the face of the person. The identity vector can include several values for features of the face of the person that are invariant (e.g., the values in the identity vector are approximately the same regardless as to whether the person has recently shaved or has facial hair, as the person ages, as hairstyle of the person changes, etc.).

Subsequent to the identity vector for the person being generated, an image that includes the face of the person is received, wherein such image may have multiple types of image degradation associated therewith (e.g., blur, noise, and/or degradation associated with low lighting conditions). Optionally, a classifier receives the image, wherein the classifier is configured to output values that are indicative of relative amounts of blur, noise, and/or degradation caused by lowlight conditions in the image. The image (and optionally the identity vector and outputs of the classifier) is provided to multiple different encoders that are respectively configured to output values for features that are employed to correct for different types of image degradation in the image. For example, the multiple encoders include: 1) a first encoder that is configured to output first values for first features in the image, wherein the first features are usable to correct for blur in the image; 2) a second encoder that is configured to output second values for second features in the image, wherein the second features are usable to correct for noise in the image; and 3) a third encoder that is configured to output third values for third features in the image, wherein the third features are usable to correct for image degradation caused by poor lighting conditions when the image was captured.

Therefore, in an example, the first encoder is provided with the image that includes the face of the person, the identity vector for the person, and a value output by the classifier that is indicative of an amount of image degradation associated with blur in the image relative to amounts of image degradation associated with noise and poor lighting conditions. Similarly, the second encoder is provided with the image that includes the face of the person, the identity vector for the person, and the value output by the classifier that is indicative of an amount of image degradation associated with noise in the image relative to amounts of image degradation associated with blur and poor lighting conditions. Finally, the third encoder receives the image that includes the face of the person, the identity vector for the person, and the value output by the classifier that is indicative of an amount of image degradation associated with poor lighting conditions in the image relative to amounts of image degradation associated with blur and noise. The three different encoders can be provided with their respective inputs in parallel and can generate their respective output feature values in parallel.

A computer-implemented fusion network receives a concatenation of the feature values output by the multiple encoders and fuses such values together to form a fused encoding of the image. As will be described in greater detail herein, the fusion network includes multiple repeatable cells, with each cell including different processing blocks that generate their own outputs. In an example, connections between cells and between blocks within cells are learned by way of network architecture search (NAS).

A decoder receives the fused encoding of the image and outputs an improved image, where the improved image includes corrections for the image degradations of multiple types that existed in the image that was provided to the encoders. It can therefore be ascertained that image degradations of multiple types are corrected for simultaneously utilizing the approach described herein. In contrast, conventional approaches for improving different types of degradation in an image require for corrections to be made sequentially rather than in parallel, such that the conventional approaches require more processing resources than the approach described herein.

In an exemplary application of the technologies described herein, frames of a video generated during a videoconference can be improved through such technologies. Because different types of image degradation are corrected for simultaneously, the technologies described herein can be employed to improve each frame of video in a videoconference, thereby improving the user experience with a videoconferencing application.

DETAILED DESCRIPTION

Described herein are various technologies pertaining to improving an image of a face of a person by simultaneously correcting for multiple different types of degradation in the image, wherein the different types of degradation in the image can include (but are not limited to) blur in the image, noise in the image, and degradation associated with the image being captured in poor lighting conditions (e.g., low lighting conditions). Briefly, an in accordance with technologies that will be described in greater detail below, an image of a face of a person can be provided in parallel to multiple different encoders, where each of the encoders is configured to output values for features that are employed to correct for a respective type of degradation in the image. Thus, for example, a first encoder outputs first values for first features in the image that are used to correct blur in the image, a second encoder outputs second values for second features in the image that are used to correct for noise in the image, and a third encoder outputs third values for third features that are used to correct for degradation associated with poor lighting conditions when the image was captured. Put differently, the first encoder can output a first encoding of the image, the second encoder can output a second encoding of the image, and the third encoder can output a third encoding of the image.

A computer-implemented fusion network receives these multiple different encodings and generates a fused encoding of the image based upon the multiple different encodings. As will be described below, the fusion network can include multiple cells, wherein each cell includes several processing blocks. Network architecture search can be employed to identify connections between cells as well as couplings between processing blocks. A decoder receives the fused encoding of the image and generates an improved image based upon the fused encoding. It can therefore be ascertained that the combination of the encoders, the fusion network, and the decoder simultaneously corrects for multiple different types of image degradation that are present in the image.

In addition, an identity vector for the person can be employed in connection with generating the improved image of the face of the person. The identity vector is analogous to a fingerprint of a person, where a sequence of values in the identity vector invariantly identifies features of the person. The identity vector is generated based upon several “clean” images of the face of the person, which can be collected during a registration phase, where images of the face of the person are captured, and where such images are substantially free of blur, noise, and degradation associated with low lighting conditions. In another example, the “clean” images of the face of the person are acquired from an image library approved by the person. The encoders are trained to account for the identity vector when outputting their respective encodings, such that facial features of the person are considered when noise, blur, and degradation associated with low lighting conditions are corrected. These aspects will be described in greater detail herein.

With reference now toFIG.1, a functional block diagram of an exemplary computing system100that is configured to receive an image, simultaneously correct multiple different types of degradation in the image, and output an improved image (with degradations of the multiple types corrected therein) is illustrated. The computing system100can be a client computing device that, for example, is used in connection with a videoconferencing application. In another example, the computing system100can be or include a server computing device that hosts a videoconferencing application. Other examples are contemplated.

The computing system100includes a processor102and memory104, where the memory104includes data that is accessible to the processor102and instructions that are executed by the processor102. The memory104includes an image106, wherein the image106optionally includes a face108of a person. For example, the image106is generated by a camera of a client computing device when a person is participating in a videoconference by way of the client computing device. Thus, the face108is the face of the person participating in the videoconference.

The memory104can further include an identity vector110for the person whose face108is captured in the image106. The identity vector110comprises a sequence of values that are collectively and invariantly indicative of an identity of the person and are further representative of features of the face108of the person. Generation of the identification vector110is described in greater detail below.

The memory104also includes an image improver module112that is configured to receive the image106and generate an improved image114, where the improved image114includes the face108of the person (with multiple types of image degradation that were existent in the image106corrected). As will be described below, the image improver module112is configured to simultaneously correct for multiple different types of image degradation in the image106when generating the improved image114. This is in direct contrast to conventional approaches, which require different types of image degradation to be addressed sequentially.

The image improver module112optionally includes a classifier module116that is configured to receive the image106and the identity vector110and output values that are representative of relative amounts of different types of image degradation in the image106of the face108of the person. For instance, when the image improver module112is configured to simultaneously correct for three different types of image degradation in the image106, the classifier module116outputs a first value, a second value, and a third value, wherein the first value is representative of an amount of image degradation of the first type relative to amounts of image degradation of the second and third types in the image106, the second value is indicative of an amount of image degradation of the second type relative to amounts of image degradation of the first and third types in the image106, and the third value is indicative of an amount of image degradation of the third type in the image106relative to amounts of image degradation of the first and second types in the image106,

The image improver module112additionally includes several encoders118-120, where each encoder is configured to output a respective image encoding that is usable to correct for a respective type of image degradation in the image106. Pursuant to an example, the encoders118-120include a first encoder118that is configured to output a first image encoding that is usable to correct for blur in the image106of the face108of the person. The encoders118-120can further include a second encoder that is configured to output a second image encoding that is usable to correct for noise in the image106of the face108of the person. The encoders118-120can also include a third encoder that is configured to output a third image encoding that is usable to correct for image degradation caused by poor lighting conditions when the image106of the face108of the person was captured.

Continuing with this example, the first encoder118is provided with the image106, the identity vector110, and the first value output by the classifier module116; the second encoder is provided with the image106, the identity vector110, and the second value output by the classifier module116; and the third encoder is provided with the image106, the identity vector110, and the third value output by the classifier module116. Moreover, the encoders118-120are provided with the image106, the identity vector110, and the appropriate output of the classifier module116in parallel such that the encoders118-120generate encodings of the image106in parallel.

The image improver module112also includes a fusion module122that receives the encodings of the image106output by the encoders118-120and generates a fused encoding of the image106based upon the encodings output by the encoders118-120. The fused encoding is a set of values that, when decoded, result in image degradations of multiple types in the image106being simultaneously corrected. To that end, the fusion module122includes several cells124-126, where each of the cells124-126includes a same set of processing blocks, which will be described in greater detail below. Connections between cells from amongst the cells124-126and connections between processing blocks in the cells124-126can be learned by way of network architecture search.

The image improver module112also comprises a decoder module128that receives output of the fusion module122(the fused encoding) and generates the improved image114based upon the fused encoding. The improved image114may then be caused to be presented on a display (e.g., instead of the image106).

Additional detail pertaining to components of the image improver module112are now set forth. The classifier module116, in example, has a suitable neural network architecture, where the classifier module116is trained based upon training pairs {y, c}, where y is an image and c is a vector of binary numbers, with each element in the vector representative of a respective type of image degradation. Therefore, when the image improver module112is configured to correct image degradations of three different types, the vector c includes three values, with each value being 1 or 0 depending upon whether the respective type of image degradation is existent in the image y. The classifier module116can have a softmax layer, such that output of the classifier module116, when provided with the image106as input, is a distribution of values over the different types of image degradations, where the values sum to 1.

As indicated previously, the encoders118-120may include three different encoders that are respectively configured to output image encodings that are usable to correct for blur, noise, and degradation caused by poor lighting conditions in the image106(although other types of encoders are contemplated) Pursuant to an example, the first encoder118is employed in connection with correcting for blur in images. In an exemplary embodiment, the first encoder118corrects for blur based upon an estimation of a blur kernel given a blurry image, and then employs deconvolution to obtain a deblurred image. In another example, the first encoder118corrects for blur based upon one or more priors, such as sparsity, L0gradient prior, patch prior, manifold prior, and low-rank, where such prior is used to compute a blur kernel from a given blurry image. In yet another example, the first encoder118is or includes a convolutional neural network (CNN), where deblurring performed by the first encoder118can be blind or non-blind. Exemplary non-blind image deblurring methods assume and use some knowledge about the image blur kernel. Moreover, the first encoder118can extract structural information in the form of facial fucidial or key points, face exemplary masks, and/or semantic masks. Other suitable approaches are also contemplated.

The second encoder in the image improver module112can denoise images and can model the image prior over a noisy image and use an estimated amount of noise in the image to denoise the image. In another example, the image improver module112employs blind image denoising techniques, where noise is modeled using technologies such as a non-local Bayes approach and/or a low-rank mixture of Gaussians. Moreover, similar to the first encoder118, the second encoder can be or include a CNN that is configured to output features values that can be employed to denoise an image.

The third encoder can be configured to generate an image encoding that is usable to correct for low light conditions. To that end, the third encoder can employ histogram equalization, matching region templates, contrast statistics, bilateral learning, intermediate high dynamic range (HDR) supervision, reinforcement learning, and adversarial learning. In yet another example, the third encoder employs frequency-based decomposition and an enhancement model to perform low-light enhancement. Other approaches are also contemplated. In summary, each of the encoders118-120is configured to output values for features that are usable to correct for different types of image degradation in images.

Referring now toFIG.2, a functional block diagram of an exemplary implementation of the fusion module122is depicted. In the depiction of the fusion module122illustrated inFIG.2, the fusion module122includes three cells124,202, and204; it is to be understood, however, that the fusion module122can include more or fewer than three cells. The fusion module122acts as a fusion network (Fn(.)) and, as referenced above, fuses feature values (encodings of an image) output by the encoders118-120. Each of the cells124,202, and204is a repeatable module, where the fusion module122includes several interconnected cells to create the fusion network (Fn(.)). A fusion cell Cell(.) can be defined as a directed acyclic graph that includes B processing blocks, where B is an integer that is greater than one. Each block i in the lthfusion cell Flis a two tensors to one tensor mapping structure, defined as a tuple of (I1, I2, O1, O2, M), where I1, I2∈lare selections of input tensors, O1, O2∈are selections of layer types applied to corresponding input tensors, and M∈is a method to combine the outputs O1, O2to form the output tensor Zilof the ithprocessing block in the lthcell (Fil). The output Zlof the fusion cell Flis a concatenation of outputs of all processing blocks in the fusion cell Fl, i.e., {Z1l, Z2l, . . . ZBl}.lis the set of input tensors, wherein such set includes outputs of the previous cell Fl-2and previous previous cell Fl-2. In an example, element wise addition is used as the sole operator for combining the method in. Exemplary types of layersinclude the following: 1) a 3×3 dilated convolutional layer; 2) a 5×5 dilated convolutional layer; 3) a 3×3 separable convolutional layer; 4) a 5×5 separable convolutional layer; 5) an identity layer or skipped connection; 6) a zero connection; 7) a self attention layer that is configured to compute self-attention; 8) a layer with residual-type connections (referred to as Res2Block and described below); 9) a residual operator) referred to as Res-op and described in greater detail below); 10) a deveiling layer (described in greater detail below); and 11) a Fast Fourier Transform (FFT) operator (described in greater detail below).

In the example depicted inFIG.2, the third cell204includes twelve layers206-228(of different layer types). The first layer206is a 3×3 dilated convolutional layer, the second layer208is a self-attention layer, the third layer210is a 5×5 dilated convolutional layer, the fourth layer212is an identity layer (or skipped connection), the fifth layer214is a 3×3 separable convolutional layer, the sixth layer216is a zero connection layer, the seventh layer218is in FFT operator, the eighty layer220is a 5×5 separable convolutional layer, the ninth layer222is a deveiling operator, the tenth layer224is a Res2Block layer, the eleventh layer226is another self-attention layer, and the twelfth layer228is a Res-op layer.FIG.2further depicts exemplary connections between the cells124,202, and204as well as between layers in the third cell204, wherein such connections can be learned by way of network architecture search.

The Res2Block layer224includes a 1×1 convolution, after which features maps are split into s feature map subsets, denoted by xi, where i∈{1, 2, . . . , s}. Each feature subset xihas the same spatial size but

1s
number of channels compared to the input feature map. Except for xi, each xihas a corresponding 3×3 convolution, denoted as Ki(.). yidenotes the output of Ki(.). The future subset xiis added with the output of Ki-1(.), and then fed into of Ki(.). To reduce parameters while increasing s, the 3×3 convolution can be emitted for xi. Thus, yican be written as:

FIG.3is a functional block diagram of the FFT layer218. The exemplary FFT layer218splits the input feature values (xin) into two parts (xin1and xin2)301. The FFT layer218includes a first 3×3 convolutional layer302, a second 3×3 convolutional layer304, and a Weiner deconvolutional layer306. The first convolutional layer302receives xin1and outputs x1, while the second convolutional layer304receives xin2and outputs x2. The Weiner deconvolutional layer306applies Weiner deconvolution to x1and x2to obtain xout:

Xout=X2⁢X1(X22+ϵ)(2)
where X1, X2, and Xoutare Fourier transforms of x1, x2, and xoutrespectively. ∈ resembles inverse of signal-to-noise ratio use during Weiner deconvolution, and in an example, can be set to 0.01.

FIG.4depicts a functional block diagram of the deveiling, layer222. The deveiling layer222includes a convolutional layer402(e.g., a 3×3 convolutional layer) and is configured to learn a latent mask A that enhances features from lowlight conditions as follows:

It can be ascertained that A is a learnable mask that is a function of y, where A is learned using the convolutional layer402given input feature values xin, followed by element-wise multiplication of A with xin, resulting in generation of enhanced feature values xout.

FIG.5is a functional block diagram of the res-op layer228, where the res-op layer228includes two 3×3 convolutional layers502and504, a min operator506and a max operator508. Both convolutional layers502and504are provided with input feature values xinand outputs of the convolutional layers502and504are directed to both the min operator506and the max operator508. Element-wise differences are computed to generate output feature values xoutwith reduced noise compared to xin.

Returning toFIG.2, and as described above, connections between cells and cell layers in the fusion module122can be learned through use of network architecture search. It can be ascertained that the search includes both network-level search (e.g., searching connections between different cells124,202, and204of the fusion module122) as well as cell-level search (e.g., exploring connections between layers of different types in each fusion cell). An exemplary approach for searching for such connections is now described. With respect to the search space, every output tensor Zilof block Filis connected to all input tensors Iilthrough operation Oj→i:
Zil=ΣZjl∈lilOj→l(Zjl)  (4)
Ōj→lcan be defined as an approximation of best search for operators (layers) using continuous relaxation, as follows:
Oj→i(Zjl)=αj=ikOk(Zjl),  (5)
whereαj→ik=1 and αj→ik≥0∀i,j. This can be implemented using softmax. Therefore, using Eq. 4) and Eq. 5), the fusion cell architecture can be summarized as follows:
Zl=Cell(Zl-1,Zl-2,α)  (6)

Referring again toFIG.1, the identity vector110is now described in greater detail. Currently, there exists a large number of images of faces of most people that are “clean” (e.g., the images do not have a substantial amount of blur, noise, or degradations caused by poor lighting conditions). These images are often available on mobile phones, on computers, and/or in a network-accessible data repository. The identity vector for the person whose face108is in the image106can be generated from such images. Further, if several “clean” images of the face of the person are not available, “clean” images of the face of the person can be captured during a registration phase, where the person can be requested to position himself/herself in front of a camera, and several images of the face of the person (e.g., from different angles relative to the camera) can be captured, wherein such images can be employed to construct the identity vector110. As the identity vector110is extracted based upon “clean” images of the face of the person, the identity vector110is a more reliable and a stronger prior when compared to face exemplar masks or semantic maps extracted from blurry images.

Given the degraded image106as input y and a corresponding set of clean imagesC, features of the face in y and {Ci}∈Ccan be computed. These can be denoted as Fyand {FiC}. Thereafter, adaptive instance normalization can be applied as follows:

F_iC=σ⁡(Fy)⁢(FiC-μ⁡(FiC)σ⁡(FiC))+μ⁡(Fy)(7)
where σ(.) and μ(.) denote standard deviation and mean, respectively. Adaptive instance normalization can be applied before passing the identity vector110as input to the encoders118-120, since the clean images {Ci}∈Cmay have different styles (as such images may be taken in different scenarios and thus may contain information that is not usable for an image restoration task). The mean of theseFiCcan be defined as the identity vector110Iiden=mean({FiC}) and used as input along with y to the encoders118-120.

Referring now toFIG.6, a functional block diagram of a training computing system600that is configured to train the image improver module112is illustrated. The computing system600includes computer-readable storage602that comprises several clean images604of faces of several different people, where the clean images604are labeled to identify people whose faces are included in the clean images604. The computer readable storage602also includes training images606, where the training images606include one or more types of image degradation, but also include faces of the same people as captured in the clean images604. The training images606may be the clean images604with some type of degradation added thereto. Alternatively, the clean images604may be the training images606with type(s) of image degradation removed therefrom.

The training computing system600further includes a processor608and memory610that includes instructions that are executed by the processor. The memory610includes an identity vector generator module612that is configured to generate identity vectors for different people whose faces are captured in the clean images604. The training computing system600further includes a training module614that trains the image improver module112using a combination of three losses: L2-loss, perceptual loss (loss that is based upon differences between high-level image feature representations extracted from pretrained CNNs), and identity loss (iden). L2-loss is computed between a restored lean face image {circumflex over (x)} and a ground-truth image x,mse=∥{circumflex over (x)}−x∥22. With respect to perceptual loss, feature values extracted for {circumflex over (x)} and x are denoted as F{circumflex over (x)}and Fx, respectively. Perceptual loss is then defined as follows:

ℒper=1NHW⁢∑i⁢∑j⁢∑k⁢Fx^-Fx,(8)
where N, H, and W are number of channels, height, and width of F{circumflex over (x)}.

Identity loss can be defined as follows. Given a degraded image y and a corresponding set of clean imagesC, {FiC} can be computed using a suitable algorithm for generating the identify vector110. {circumflex over (x)} is a restored clean facial image corresponding to y. The identity loss can be defined as follows:

ℒiden=1P⁢∑{FiC}⁢arc⁢⁢cos⁡(〈Fx^,FiC〉)(9)
where F{circumflex over (x)}are feature value extracted using {circumflex over (x)} and P is a number of clean images inC(in the clean images604) belonging to the same identity.

Thus, the identity loss with respect to an image can be computed by providing an improved image616to the identity vector generator module610, which generates an identity vector for a face in the improved image616. The training module614can train the image improver module612to minimize the overall loss (including the identity loss) based upon the identity vector for the face in the improved image616.

The training module614can train the classifier module116and the encoders118-120based upon {y, x, c}. With more specificity, the classifier module116can be trained using {y, c}, where c is a vector of length three: c={b, n, l}, and further where b, n, and l are binary numbers (i.e., b, n, and l are only 1 if image y contains blur, noise, and lowlight degradation, respectively, and 0 otherwise. Training using {y, c} produces class label ĉ, which indicates degradations present in y.

The encoders118-120can be initially trained to address corresponding individual tasks of the blurring, denoising, and lowlight enhancement, respectively. Given a degraded image y, class ĉ, (output by classifier module116), and an identity vector lidenas input to the encoders118-120, the image improver module112outputs a restored face image {circumflex over (x)}. As noted above, in an example, the number of layers in each fusion cell is set to 12. The alpha's (α) and the weights of the encoders118-120and the fusion module122can be updated alternately. The image improver module112can be trained usingfinalwith the Adam optimizer and batch size of 40. In an example, the learning rate is set to be 0.0005.

Referring now toFIG.7, a functional block diagram of an exemplary system700that supports videoconferencing is illustrated. The system700includes a first client computing device702and a second client computing device704that are in communication with one another by way of a network706. A first user708operates the first client computing device702while a second user710operates the second client computing device704. The first client computing device702includes a processor712and memory714that is operably coupled to the processor712. The memory714includes an operating system716, where applications installed on the client computing device702operate in conjunction with the operating system716. The client computing device702further includes a display718upon which graphical data can be presented. In the exemplary system700, the memory714includes a video conferencing application720that is executed by the processor712. The videoconferencing application720is configured to receive video of the user708captured by a camera (not shown) and transmit the video to the second client computing device704by way of the network706. Similarly, the second client computing device704captures video that includes images of the face of the second person user710and transmits video of the second user710to the first client computing device702by way of the network706.

The operating system716has a driver722installed therein, where the driver722includes the image improver module112. While the image improver module112is illustrated as being included in the driver722, it is to be understood that the image improver module112may be included in the videoconferencing application, may be included in a kernel of the operating system716, etc.

In an exemplary operation, the first client computing device702receives an identity vector for the second user710. For instance, the second client computing device704may have the identity vector stored thereon or may have access to the identity vector, and the second client computing device704transmits the identity vector for the second user710to the first client computing device702by way of the network706. Additionally, the first client computing device702can receive video that includes several video frames724, wherein the frames include the face of the second user710. Such video frames724are provided to the videoconferencing application720, and the videoconferencing application720calls the driver722and directs each of the video frames724as well as the identity vector for the second user710to the driver722. The image improver module112can simultaneously correct for multiple types of image degradation in each frame of the video frames724(as described above) and can return improved video frames726to the videoconferencing application720. The videoconferencing application720may then display the improved video frames726on the display718as video728. Thus, the videoconferencing application is able to depict the video728, which includes higher quality frames than would otherwise be presented on the display718.

Referring now toFIG.8, a functional block diagram of an exemplary client computing device800is depicted. The client computing device800includes a processor802and memory804that is operably coupled to the processor802. The client computing device800further includes a display806upon which graphical data is presented. The memory804of the client computing device800has an operating system808loaded therein, and a video playing/editing application810is executed by the processor802in conjunction with the operating system808. The video playing/editing application810can access several video frames812. Such video frames812may be a part of a home film, an older movie or television program, a video streamed from a streaming service, etc., where the video frames and812may include faces of different people.

The operating system data808has a driver814installed therein, where the driver814includes the image improver module112. In another example, the image improver module112is included in the video playing/editing application810. Upon the video playing/editing application810receives a request to play a video that includes the video frames812, the video editing/playing application810calls the driver814, and the video frames812are provided to the image improver module112. The image improver module112denoises, deblurs, and improves degradations associated with low lighting conditions in the video frames812and generates improved video frames816as described above. These improved video frames816are then played by the video playing/editing application810on the display806as video818.

While a few exemplary applications of the image improver module112have been set forth herein, it is to be understood that other applications are contemplated. For example, a user may have several images that were captured with older technology, and thus may have more noise, blur, etc. than what is desired by the user. The image improver module112can be provided with such images, and the image improver module112can output improved images (where the image improver module112deblurs, denoises, and corrects degradations associated with low lighting conditions). Other applications are also contemplated.

Now turning solely toFIG.9, a flow diagram illustrating an exemplary methodology for improving images is illustrated. The methodology900starts at902, and at904an image is received, where a portion of the image includes the face of a person. At906, the portion of the image that includes the face is extracted from the image. In an example, the image is provided to a computer-implemented neural network that is configured to identify locations of faces in images and create bounding boxes around such faces. Such a neural network may be employed in connection with extracting the portion of the image that includes the face from the image.

At908, the portion of the image is provided in parallel to a first encoder, a second encoder, and a third encode. As described previously, the first encoder outputs first feature values, the second encoder outputs second feature values, and the third encoder outputs third feature values. The first features may be features that are usable to correct for noise in the portion of the image, the second features may be usable to correct for blur in the portion of the image, and the third features may be usable to correct for poor lighting conditions when the image was captured.

At910, using a fusion network identified by way of a network architecture search, the first feature values, the second feature values, and the third feature values are fused to form fused feature values. At912, based upon the fused feature values, an improved portion of the image is generated, and at914the extracted portion of the image is replaced with the improved portion of the image in the image. The result, then, is that the face of the person in the improved image is deblurred, denoised, and has improved lighting conditions associated therewith when compared to the originally received image. The methodology900completes at916.

With reference now toFIG.10, a flow diagram illustrating an exemplary methodology1000for improving image frames generated during a videoconference is illustrated. The methodology1000starts at1002, and at1004a frame of a video for a videoconference is received. At1006, a determination is made regarding whether the video frame includes a face of a person; if it is determined at1006that the video frame includes a face of a person, then at1008an improved frame is generated by simultaneously correcting for image degradations of multiple different types in the frame. At1010, the improved frame is displayed as part of the videoconference.

After the improved frame is displayed and/or upon the determination of1006that the frame fails to include a face, at1012a determination is made as to whether the videoconference has been completed. If the videoconference has not completed, the methodology1000returns to1004, where a next frame of video is received. If the videoconference is determined to have completed at1012, the methodology completes at1014.

Various examples are now set forth.

Example 1: A computing system that is configured to simultaneously correct for multiple types of degradation in computer-readable images, the computing system comprises a processor and memory. The memory stores instructions that, when executed by the processor, cause the processor to perform acts comprising: providing a computer-readable image to a first encoder, the first encoder configured to output a first encoding of the computer-readable image that, when decoded, is configured to correct for a first type of image degradation in the computer-readable image; in parallel with providing the computer-readable image to the first encoder, providing the computer-readable image to a second encoder, the second encoder configured to output a second encoding of the computer-readable image that, when decoded, is configured to correct for a second type of image degradation in the computer-readable image; using a computer-implemented fusion network and based upon the first encoding of the computer-readable image and the second encoding of the computer-readable image, generating a fused encoding of the computer-readable image; and decoding the fused encoding of the computer-readable image to generate an improved computer-readable image, wherein the first type of image degradation and the second type of image degradation are simultaneously corrected during the decoding of the fused encoding of the computer-readable image.

Example 2: The computing system according to Example 1, the acts further comprising: in parallel with providing the computer-readable image to the first encoder and the second encoder, providing the computer-readable image to a third encoder, the third encoder configured to output a third encoding of the computer-readable image that, when decoded, is configured to correct for a third type of image degradation in the computer-readable image, wherein the fused encoding is based further upon the third encoding of the computer-readable image, and further wherein the first type of image degradation, the second type of image degradation, and the third type of image degradation are simultaneously corrected during the decoding of the fused encoding of the computer-readable image.

Example 3: The computing system according to Example 2, wherein the first type of image degradation is blur, the second type of image degradation is noise, and the third type of image degradation is low lighting when the computer-readable image was captured by a camera.

Example 4: The computing system according to any of Examples 1-3, the acts further comprising: prior to providing the computer-readable image to the first encoder and the second encoder, determining that the computer-readable image includes the face of the person, Wherein the computer-readable image is provided to the first encoder and the second encoder only upon determining that the computer-readable image includes the face of the person.

Example 5: The computing system according to Example 4, the acts further comprising: receiving an identity vector for the person, the identity vector comprises values that are representative of features of the face of the person, wherein the first encoder generates the first encoding of the computer-readable image based upon the identity vector, and further wherein the second encoder generates the second encoding of the computer-readable image based upon the identity vector.

Example 6: The computing system according to Example 5, the acts further comprising: generating the identity vector based upon a plurality of images that include the face of the person.

Example 7: The computing system according to any of Examples 1-6, the acts further comprising: prior to providing the computer-readable image to the first encoder and the second encoder, providing the computer-readable image to a classifier, wherein the classifier, based upon the computer-readable image, outputs: a first value that is indicative of an amount of image degradation of the first type of image degradation in the computer-readable image; and a second value that is indicative of an amount of image degradation of the second type of image degradation in the computer-readable image, wherein the first encoding of the computer-readable image is based upon the first value and the second encoding of the computer-readable image is based upon the second value.

Example 8: The computing system according to any of Examples 1-7, wherein the computer-readable image is a frame of video generated during a live video teleconference, and wherein the acts of providing, providing, using, and decoding are performed on each frame in the video.

Example 9: The computing system according to any of Examples 1-8, wherein the fusion network comprises multiple repeatable cells, wherein each cell comprises two layers, and further wherein outputs of the two layers are directed to a tensor.

Example 10: The computing system according to any of Examples 1-9 being a client computing device that executes a videoconferencing application, wherein the image is received by the videoconferencing application from a second client computing device over a network connection between the client computing device and the second client computing device.

Example 11: The computing system according to any of Examples 1-10 being a client computing device that executes a videoconferencing application, wherein the image is generated by a camera of the client computing device, the acts further comprising transmitting the improved computer-readable image to a second client computing device that is in network communication with the client computing device as part of a videoconference facilitated by the video conferencing application.

Example 12: A method performed by at least one processor of a computing system, the method comprising: receiving a computer-readable image of a face of a person, wherein the computer-readable image of the face of the person comprises image degradations for: noise in the computer-readable image; blur in the computer-readable image; and lighting conditions when the computer-readable image was captured; and simultaneously correcting the image degradations to generate an improved computer-readable image, wherein the image degradations for noise, blur, and lighting conditions are each at least partially corrected in the computer-readable image.

Example 13: The method according to Example 12, wherein simultaneously correcting the image degradations to generate the improved computer-readable image comprises: providing the computer-readable image to a first encoder, wherein the first encoder outputs a first encoding of the computer-readable image based upon the computer-readable image, and further wherein the first encoding encodes a first correction to be made to the computer-readable image to correct for the noise in the computer-readable image; in parallel with providing the computer-readable image to the first encoder, providing the computer-readable image to a second encoder, wherein the second encoder outputs a second encoding of the computer-readable image based upon the computer-readable image, and further wherein the second encoding encodes a second correction to be made to the computer-readable image to correct for the blur in the computer-readable image; in parallel with providing the computer-readable image to the first encoder and the second encoder, providing the computer-readable image to a third encoder, wherein the third encoder outputs a third encoding of the computer-readable image based upon the computer-readable image, and further wherein the third encoding encodes a third correction to be made to the computer-readable image to correct for the lighting conditions when the computer-readable image was captured; fusing the first encoding, the second encoding, and the third encoding to form a fused encoding of the computer-readable image; and decoding the fused encoding to generate the improved computer-readable image.

Example 14: The method according to any of Examples 12-13, further comprising: receiving an identity vector for the face of the person, wherein the identity vector comprises values that invariantly represent features of the face of the person, wherein the improved computer-readable image is generated based upon the identity vector for the face of the person.

Example 15: The method according to any of Examples 12-14, further comprising: providing the computer-readable image to a classifier, wherein the classifier is configured to output a first value that is indicative of a magnitude of image degradation associated with noise in the computer-readable image, a second value that is indicative of a magnitude of image degradation associated with blur in the computer-readable image, and a third value that is indicative of a magnitude of image degradation associated with lighting conditions in the computer-readable image, wherein the image degradations are simultaneously corrected based upon the first value, the second value, and the third value output by the classifier.

Example 16: The method according to any of Examples 12-15, wherein the computer-readable image is a frame of a video, and further wherein the acts of receiving and simultaneously correcting are performed on each frame of the video in real-time.

Example 17: The method according to Example 16, wherein the computing system is a server computing device that receives the video from a client computing device that is in network communication with the computing system.

Example 18: A computer-readable storage medium comprising instructions that, when executed by a processor, cause the processor to perform acts comprising: providing a computer-readable image to a first encoder, the first encoder configured to output a first encoding of the computer-readable image that, when decoded, is configured to correct for a first type of image degradation in the computer-readable image; in parallel with providing the computer-readable image to the first encoder, providing the computer-readable image to a second encoder, the second encoder configured to output a second encoding of the computer-readable image that, when decoded, is configured to correct for a second type of image degradation in the computer-readable image; using a computer-implemented fusion network and based upon the first encoding of the computer-readable image and the second encoding of the computer-readable image, generating a fused encoding of the computer-readable image; and decoding the fused encoding of the computer-readable image to generate an improved computer-readable image, wherein the first type of image degradation and the second type of image degradation are simultaneously corrected during the decoding of the fused encoding of the computer-readable image.

Example 19: The computer-readable storage medium according to Example 18, the acts further comprising: in parallel with providing the computer-readable image to the first encoder and the second encoder, providing the computer-readable image to a third encoder, the third encoder configured to output a third encoding of the computer-readable image that, when decoded, is configured to correct for a third type of image degradation in the computer-readable image, wherein the fused encoding is based further upon the third encoding of the computer-readable image, and further wherein the first type of image degradation, the second type of image degradation, and the third type of image degradation are simultaneously corrected during the decoding of the fused encoding of the computer-readable image.

Example 20: The computer-readable storage medium according to Example 19, wherein the first type of image degradation is blurring, the second type of image degradation is noise, and the third type of image degradation is low lighting when the computer-readable image was captured by a camera.

Example 21: A computing system configured to perform a method according to any of Examples 12-17.

Example 22: A computing system configured with the computer-readable storage medium of any of Examples 18-20.

Referring now toFIG.11, a high-level illustration of an exemplary computing device1100that can be used in accordance with the systems and methodologies disclosed herein is illustrated. For instance, the computing device1100may be used in a system that is configured to simultaneously correct image degradations of multiple types in an image. By way of another example, the computing device1100can be used in a system that supports computer-implemented videoconferencing. The computing device1100includes at least one processor1102that executes instructions that are stored in a memory1104. The instructions may be, for instance, instructions for implementing functionality described as being carried out by one or more components discussed above or instructions for implementing one or more of the methods described above. The processor1102may access the memory1104by way of a system bus1106. In addition to storing executable instructions, the memory1104may also store identity vectors, images, feature values, video, etc.

The computing device1100additionally includes a data store1108that is accessible by the processor1102by way of the system bus1106. The data store1108may include executable instructions, images, identity vectors, video, etc. The computing device1100also includes an input interface1110that allows external devices to communicate with the computing device1100. For instance, the input interface1110may be used to receive instructions from an external computer device, from a user, etc. The computing device1100also includes an output interface1112that interfaces the computing device1100with one or more external devices. For example, the computing device1100may display text, images, etc. by way of the output interface1112.