AI-BASED CHARACTERISTIC GUIDANCE METHOD AND SYSTEM FOR ENHANCING QUALITY OF DIFFUSION MODELS

The invention provides artificial intelligence-based characteristic guidance method and system for enhancing quality of a diffusion model in generating a data from a noisy data based on a condition information. The method comprises: generating a nonlinear precorrection vector; performing, by a regularization module, a context regularization iteration to obtain an updated nonlinear correction vector and a nonlinear correction gradient; checking if a convergence criterion is met; and denoising the noisy data based on the updated nonlinear correction vector to generate the data. By using the regularization module, the provided characteristic guidance method not only greatly improves the stability of data generation by the diffusion model, but also provides enhanced control over context through two context modes: the detail enhancement mode and the context enhancement mode. The present invention can enhance the semantic characteristics of prompts and mitigate irregularities in image generation.

FIELD OF THE INVENTION

The present invention generally relates to guided generative diffusion technologies. More specifically the present invention relates to characteristic guidance method and system for enhancing quality of diffusion model.

BACKGROUND OF THE INVENTION

Diffusion models, such as denoising diffusion probabilistic model (DDPM), models distributions of data ρ(x) by recovering an original data x0˜p(x) from one of its noise contaminated versions xi. The noise contaminated data is a linear combination of the original image and a gaussian noise:

where n is the total diffusion steps, αi is the contamination weight at a time ti∈[0, T] in the forward diffusion process, and ∈i is a standard Gaussian random noise.

The DDPM trains a denoising neural network ϵ(x, ti) to predict and remove the noise from the noisy data under an objective of minimizing a denoising loss function:

In the ideal scenario where diffusion time steps among ti are infinitesimal, the optimal solution for the denoising objective (2) is defined

The forward diffusion process represented by Eq. (1) places stringent constraints on permissible forms of the denoising neural networks. First, ϵ(x, ti) is proportional to the score function s(x, ti)=∇xi log ρ(xi). Second, the score function is a solution of the score FP equation, which could be rewritten in terms of

in which

and σ(t)=√{square root over (1−e−t)}. These constraints lay the foundation for duality between forward and backward diffusion processes, which is essential for successful sampling from the distribution ρ(x).

Conditional DDPMs, which generate data based on a given condition c, model the conditional distribution p(x|c) with a denoising neural network represented as ϵθ(x|c, ti). However, the training data in practice might only have weak or noised in-formation about condition c, therefore we need a way to enhance the control strength in this situation.

Guidance is a technique for conditional data generation that trades off control strength and image diversity. It generally aims to sample from the distribution:

where ω>0 is the guidance scale. When ω is large, this distribution concentrates on samples that have the highest conditional likelihood p(clx).

Classifier guidance, requiring an additional classifier, faces implementation challenges in non-classification tasks like text-to-image generation. Classifier-free guidance circumvents this by linearly combining conditional and unconditional DDPM weighted by the guidance scale parameter ω. More specifically, classifier free guidance use the following guided denoising neural network ϵCF, deduced by using ϵ∝∇ log p, to approximately sample from p(x|c, ω):

It exactly computes ϵ(x|c, ti, ω), the denoising neural network of p(x|c, ω), at time ti=0. However, it is not a good approximation for ti>0 as it deviates from the FP equation (4) when ω is large, leading to unstable data generation, such as overly saturated and un-natural images. Though techniques like dynamic thresholding can handle color issues by clipping out-of-range pixel values, a systematic solution for general sampling tasks, including latent space diffusion and those beyond image generation, remains absent.

Characteristic guidance is a systematic solution to the large guidance scale issue by correcting the non-linearity neglected in classifier-free guidance. It is a kind of non-linear corrected classifier-free guidance:

in which x1=x+ωΔx, x2=x+(1+ω)Δx, and Δx is a non-linear correction term. It is evident that when Δx=0, the characteristic guidance is equivalent to the classifier-free guidance (6).

The correction Δx is determined from the training-free non-linear relation

where σi=√{square root over (1−αi)} is a scale parameter and the operator P is the projection module works as an orthogonal projection operator. Equation (8) can be solved using the fixed-point iteration method.

The projection module P acts channel-wisely and is specified by a vector g:

For pixel space diffusion model, characteristic guidance uses the operator g=1 as projection to the channel-wise mean. For latent space diffusion model, characteristic guidance uses g=Δx−(ϵθ(x, ti)−ϵθ(x|c, ti)) σi. For low dimensional cases that are not images, characteristic guidance uses the operator P to be identity.

However, existing characteristic guidance approaches are still subject to some stability issues.

SUMMARY OF THE INVENTION

It is an objective of the present invention to provide an improved characteristic guidance for data generation by diffusion model with enhanced stability over original characteristic guidance approaches.

In accordance with a first aspect of the present invention, an artificial intelligence-based characteristic guidance method is provided for enhancing quality of a diffusion model in generating a new data from a noisy data based on a condition information. The method comprises:

In accordance with a second aspect of the present invention, an artificial intelligence-based characteristic guidance system is provided for enhancing quality of a diffusion model in generating a new data from a noisy data based on a condition information. The system comprises: a server; and one or more client devices connected to the server through a network. The server and the one or more client devices are configured to share computation resources with one another in a distributive manner to perform the artificial intelligence-based method according to the first aspect of the present invention.

By using the regularization module, the provided characteristic guidance method not only greatly improves the stability of data generation by the diffusion model, but also provides enhanced control over context through two context modes: the detail enhancement mode and the context enhancement mode. The present invention can enhance the semantic characteristics of prompts and mitigate irregularities in image generation.

DETAILED DESCRIPTION

In the following description, details of the present invention are set forth as preferred embodiments. It will be apparent to those skilled in the art that modifications, including additions and/or substitutions may be made without departing from the scope and spirit of the invention. Specific details may be omitted so as not to obscure the invention; however, the disclosure is written to enable one skilled in the art to practice the teachings herein without undue experimentation.

FIG. 1 illustrates a flowchart for an artificial intelligence-based method S100 for enhancing quality of a diffusion model in generating, based on a condition information c, a new data xi-1 from a noisy data xi (also denoted as x in the following description) at a diffusion time ti in accordance with one embodiment of the present invention. As shown, the method S100 comprises the following steps:

In step S108, the target regularization value λgt is correlated with a diffusion time of the diffusion model at each step of the context regularization iteration, and has a highest value when the diffusion time is equal to a total diffusion time of the diffusion model. For example, the target regularization value λgt may be obtained by:

λ
   gt
  
  =
  
   A
   ⁢
     
   
    exp
    ⁡
    (
    
     
      
       t
       i
      
      -
      
       t
       n
      
     
     B
    
    )

where A represents the amplitude of regularization, B represents the effective range of regularization, ti represents the current diffusion time of the diffusion model at step i, and tn represents the diffusion time at the final step n (i.e., the total diffusion time) of the diffusion model.

In some embodiments, the target regularization value λgt is subject to a detail enhancement mode wherein A is in a range from 85 to 100 and B is in a range from 0.6 to 0.9.

In some embodiments, the target regularization value λgt is subject to a context enhancement mode wherein A is in a range from 1 to 5 and B is in a range from 1 to 5.

In step S110, the convergence criterion may be set as:

where η represents a tolerance threshold for convergence serving as a measure to judge if the nonlinearity in the data (e.g. image/audio/signal) have been consistently corrected.

FIG. 2 illustrates a process flowchart for step S106 of generating a nonlinear precorrection vector in accordance with one embodiment of the present invention. As shown, step S106 may comprise:

In step S1061, the conditionally noised data xc may be defined as xc=x+ωΔx; and the unconditionally noised data xu may be defined as xu=x+(1+ω)Δx.

For steps S1062 and S1063, the unconditional denoise neural network and the conditional denoise neural network may be obtained by training a denoise neural network ϵ(x, t) respectively by a unconditional diffusion model and a conditional diffusion model, which generates new data based on the given condition c, to predict and remove noise from the noisy data xi under an objective of minimizing a denoising loss function. In some embodiments, the denoising loss function may be defined as Eq. (2), that is,

In step S1064, the nonlinear precorrection vector Δxl(k) may be computed by:

FIG. 3 illustrates a process flowchart for step S108 of performing the context regularization iteration in accordance with one embodiment of the present invention. As shown, step S108 may comprise:

In step S1081, the projection loss may be formulated as:

In step S1083, the nonlinear correction gradient gΔx(k) is constructed by:

In step S1084, the regularization gradient gλ(k) is constructed by:

FIG. 4 illustrates a flowchart for step S114 of denoising the noisy data xi based on the updated nonlinear correction vector to generate the new data xi-1. As shown, step S114 may comprise:

In step S1142, the noise ϵiCH may be predicted by:

FIG. 5 illustrates a block diagram for a characteristic guidance system 500 configured for performing the characteristic guidance method of S100. The system 500 may include a server 510; and one or more client devices 520 connected to the server 510 through a network 530.

The server 510 and each client device 520 may be equipped with a CPU, data storage, and a GPU. Additionally, the server 510 may include a specialized transmission manager comprising a dedicated network routing system.

In one embodiment, the generation of images or audio is facilitated by a content application implemented on either the client device, the server system, or both. This application adopts the DDPM model, wherein the traditional denoise module is substituted with a characteristic guidance denoise module featuring context regularization iteration. The DDPM model's components—denoising module, steps, and parameters—are stored on the system's data storage and accessed by the device's CPU and GPU. The DDPM process can be distributed between the client device and the server system based on their computational capabilities, typically with the denoising module implemented in the server and the denoising steps stored in the client device.

The process starts when a generation command is activated on a client device. The content application initiates a DDPM process, generating noised image/audio signals and the necessary DDPM parameters. The DDPM process then proceeds to denoise these signals step by step. At each generation step, the noised signal is sent through network to the server system's transmission manager, which forwards it to the server's content application for further denoising. The processed signal is then sent back to the client device through the transmission manager. The final, denoised output is stored on the client device, completing the generation process.

This streamlined approach demonstrates the system's ability to leverage distributed computing resources for efficient image or audio generation, highlighting its potential in applications requiring high-quality media production with sophisticated context-aware enhancements.

In a text-to-image generation task, the characteristic guidance system can produce high-quality images that boast enhanced details and richer context from text-based prompts. As depicted in FIG. 6A, the process begins when the user inputs a prompt directly or generates one through a natural language processing (NLP) model. Following this, the characteristic guidance system takes the prompt as a conditional input to start the image generation process. Subsequently, the generated image is saved in a specific format and securely stored within the system's storage. This streamlined workflow ensures that users can easily convert textual descriptions into vivid, contextually rich visual representations with minimal effort.

In the image-to-image generation task, the characteristic guidance system facilitates the transformation of initial images into refined versions with enhanced details and context. As depicted in FIG. 6B, the process initiates with the loading of an image from the system's storage. This image is then analyzed by an NLP-based model, which converts it into a descriptive text prompt that encapsulates the essence and context of the original image. Both the original image and its corresponding text prompt are then fed as conditional inputs into the characteristic guidance system. Mirroring the text-to-image task, the system leverages this dual-input to generate a new image that retains the core attributes of the original while infusing it with additional clarity and contextual depth. The final image is saved in a specified format and stored securely within the system's storage.

In an audio generation task, the characteristic guidance system could be used to generate clear and contextually appropriate audio. Text-to-Audio Generation transforms textual prompts into clear, contextually relevant audio. Similar to image generation task, users input text, then the characteristic guidance system take input as condition to generate audio that captures the prompt's essence, suitable for applications ranging from audiobook creation to dialogue generation.

FIG. 7 shows images generated by the characteristic guidance method of the present invention under the two context control modes, the detail enhancement mode and the context enhancement mode, in comparison with a classifier-free guidance method. As shown, both modes have the ability of enhancing the semantic characteristics of prompts and mitigates irregularities in image generation. They also allow the user to control context changes more accurately.

FIGS. 8A and 8B illustrate convergence behaviors of the original characteristic guidance system and the characteristic guidance system of the present invention (under the detail enhancement mode). As shown, the present invention succeeded in ensuring the convergence of every iteration step while the original characteristic guidance fail to converge at the first few steps.

The functional units and modules of the artificial intelligence-based quality-enhancement system in accordance with the embodiments disclosed herein may be implemented using computing devices, computer processors, or electronic circuitries including but not limited to application specific integrated circuits (ASIC), field programmable gate arrays (FPGA), microcontrollers, and other programmable logic devices configured or programmed according to the teachings of the present disclosure. Computer instructions or software codes running in the computing devices, computer processors, or programmable logic devices can readily be prepared by practitioners skilled in the software or electronic art based on the teachings of the present disclosure.

All or portions of the methods in accordance to the embodiments may be executed in one or more computing devices including server computers, personal computers, laptop computers, mobile computing devices such as smartphones and tablet computers.

The embodiments may include computer storage media, transient and non-transient memory devices having computer instructions or software codes stored therein, which can be used to program or configure the computing devices, computer processors, or electronic circuitries to perform any of the processes of the present invention. The storage media, transient and non-transient memory devices can include, but are not limited to, floppy disks, optical discs, Blu-ray Disc, DVD, CD-ROMs, and magneto-optical disks, ROMs, RAMs, flash memory devices, or any type of media or devices suitable for storing instructions, codes, and/or data.

Each of the functional units and modules in accordance with various embodiments also may be implemented in distributed computing environments and/or Cloud computing environments, wherein the whole or portions of machine instructions are executed in distributed fashion by one or more processing devices interconnected by a communication network, such as an intranet, Wide Area Network (WAN), Local Area Network (LAN), the Internet, and other forms of data transmission medium.