Repository: AUTOMATIC1111/stable-diffusion-webui

Issue:
Img2Img alt should divide by sigma[-1], not std
https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/53651696dc1492f3b7cc009d64a55532bc787aa7/scripts/img2imgalt.py#L59

While the noise used in stable diffusion is generated from a normal distribution, that doesn't mean that it is always perfectly normal. Normalizing the reverse-generated noise results in incorrect saturation in the output image.

I've been able to solve the saturation issue by dividing by the first sigma. I spent some time verifying the img2imgalt technique by attempting to re-derive it. Dividing by the first sigma appears to be the correct approach.

See my contribution to another repo here:
code: https://github.com/sd-webui/stable-diffusion-webui/blob/17748cbc9c34df44d0381c42e4f0fe1903089438/scripts/sd_utils.py#L525
original pr: https://github.com/sd-webui/stable-diffusion-webui/pull/1070/files#diff-2e278c1b9a8c0e308b8272729de19c973ac24710b3467dfb9c877db5d1cf7a3f


Edit:
The changes by MartinCairnsSQL below are _also_ required. They're from the original author's gist.
https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/736#issuecomment-1252538516


Discussion:
Dumb/random question, but why does it only work with Euler, and would it be possible to make it work with Euler_A/something else?
It only works with euler because the code is just the euler sampler in reverse. To support other samplers, they have to be similarly derived. It shouldn't be too hard for some of them, but it's still not super easy.
This just gives me blurry images for settings that used to work. What settings reproduce original images by the author with your code?
It's gives me blurry images too
I had a look at the diffs in the method and found that the variables sigma_in, t & d also have changes.  I made the changes to a copy of the script and tested a few images and found the original version had too much contrast for the same set of prompts other than that the sigma change didn't blur the image with the extra changes.

[img2imgalt_sigma.zip](https://github.com/AUTOMATIC1111/stable-diffusion-webui/files/9609118/img2imgalt_sigma.zip)

https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/c2775528c9e5336b37987832795621fa49ff97d3/scripts/img2imgalt.py#L35
```
        sigma_in = torch.cat([sigmas[i - 1] * s_in] * 2)
```

https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/c2775528c9e5336b37987832795621fa49ff97d3/scripts/img2imgalt.py#L39
```
        if i == 1:
            t = dnw.sigma_to_t(torch.cat([sigmas[i] * s_in] * 2))
        else:
            t = dnw.sigma_to_t(sigma_in)
```

https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/c2775528c9e5336b37987832795621fa49ff97d3/scripts/img2imgalt.py#L46
```
        if i == 1:
            d = (x - denoised) / (2 * sigmas[i])
        else:
            d = (x - denoised) / sigmas[i - 1]
```
These changes look good.

Would it be possible to combine img2imgalt with masked inpaint in some way?
Separating the original prompt from the img2imgalt prompt we are essentially trying to tell the program which parts are the ones we want changed. But sometimes it is hard for the program to key in on those areas. Trying to change the hair color it would only do parts of the hair, and I would have to throw the settings too far out of whack, producing jibbirish in the process, in order to encapsulate the full hair. If I could do masked assist, or even automatically using a tool as txt2mask, would that not yield a better outcome?

Resolving patch:
diff --git a/scripts/img2imgalt.py b/scripts/img2imgalt.py
--- a/scripts/img2imgalt.py
+++ b/scripts/img2imgalt.py
@@ -59,7 +59,55 @@ def find_noise_for_image(p, cond, uncond, cfg_scale, steps):
     return x / x.std()
 
 
-Cached = namedtuple("Cached", ["noise", "cfg_scale", "steps", "latent", "original_prompt", "original_negative_prompt"])
+Cached = namedtuple("Cached", ["noise", "cfg_scale", "steps", "latent", "original_prompt", "original_negative_prompt", "sigma_adjustment"])
+
+
+# Based on changes suggested by briansemrau in https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/736
+def find_noise_for_image_sigma_adjustment(p, cond, uncond, cfg_scale, steps):
+    x = p.init_latent
+
+    s_in = x.new_ones([x.shape[0]])
+    dnw = K.external.CompVisDenoiser(shared.sd_model)
+    sigmas = dnw.get_sigmas(steps).flip(0)
+
+    shared.state.sampling_steps = steps
+
+    for i in trange(1, len(sigmas)):
+        shared.state.sampling_step += 1
+
+        x_in = torch.cat([x] * 2)
+        sigma_in = torch.cat([sigmas[i - 1] * s_in] * 2)
+        cond_in = torch.cat([uncond, cond])
+
+        c_out, c_in = [K.utils.append_dims(k, x_in.ndim) for k in dnw.get_scalings(sigma_in)]
+
+        if i == 1:
+            t = dnw.sigma_to_t(torch.cat([sigmas[i] * s_in] * 2))
+        else:
+            t = dnw.sigma_to_t(sigma_in)
+
+        eps = shared.sd_model.apply_model(x_in * c_in, t, cond=cond_in)
+        denoised_uncond, denoised_cond = (x_in + eps * c_out).chunk(2)
+
+        denoised = denoised_uncond + (denoised_cond - denoised_uncond) * cfg_scale
+
+        if i == 1:
+            d = (x - denoised) / (2 * sigmas[i])
+        else:
+            d = (x - denoised) / sigmas[i - 1]
+
+        dt = sigmas[i] - sigmas[i - 1]
+        x = x + d * dt
+
+        sd_samplers.store_latent(x)
+
+        # This shouldn't be necessary, but solved some VRAM issues
+        del x_in, sigma_in, cond_in, c_out, c_in, t,
+        del eps, denoised_uncond, denoised_cond, denoised, d, dt
+
+    shared.state.nextjob()
+
+    return x / sigmas[-1]
 
 
 class Script(scripts.Script):
@@ -78,9 +126,10 @@ def ui(self, is_img2img):
         cfg = gr.Slider(label="Decode CFG scale", minimum=0.0, maximum=15.0, step=0.1, value=1.0)
         st = gr.Slider(label="Decode steps", minimum=1, maximum=150, step=1, value=50)
         randomness = gr.Slider(label="Randomness", minimum=0.0, maximum=1.0, step=0.01, value=0.0)
-        return [original_prompt, original_negative_prompt, cfg, st, randomness]
+        sigma_adjustment = gr.Checkbox(label="Sigma adjustment for finding noise for image", value=False)
+        return [original_prompt, original_negative_prompt, cfg, st, randomness, sigma_adjustment]
 
-    def run(self, p, original_prompt, original_negative_prompt, cfg, st, randomness):
+    def run(self, p, original_prompt, original_negative_prompt, cfg, st, randomness, sigma_adjustment):
         p.batch_size = 1
         p.batch_count = 1
 
@@ -88,7 +137,10 @@ def run(self, p, original_prompt, original_negative_prompt, cfg, st, randomness)
         def sample_extra(conditioning, unconditional_conditioning, seeds, subseeds, subseed_strength):
             lat = (p.init_latent.cpu().numpy() * 10).astype(int)
 
-            same_params = self.cache is not None and self.cache.cfg_scale == cfg and self.cache.steps == st and self.cache.original_prompt == original_prompt and self.cache.original_negative_prompt == original_negative_prompt
+            same_params = self.cache is not None and self.cache.cfg_scale == cfg and self.cache.steps == st \
+                                and self.cache.original_prompt == original_prompt \
+                                and self.cache.original_negative_prompt == original_negative_prompt \
+                                and self.cache.sigma_adjustment == sigma_adjustment
             same_everything = same_params and self.cache.latent.shape == lat.shape and np.abs(self.cache.latent-lat).sum() < 100
 
             if same_everything:
@@ -97,8 +149,11 @@ def sample_extra(conditioning, unconditional_conditioning, seeds, subseeds, subs
                 shared.state.job_count += 1
                 cond = p.sd_model.get_learned_conditioning(p.batch_size * [original_prompt])
                 uncond = p.sd_model.get_learned_conditioning(p.batch_size * [original_negative_prompt])
-                rec_noise = find_noise_for_image(p, cond, uncond, cfg, st)
-                self.cache = Cached(rec_noise, cfg, st, lat, original_prompt, original_negative_prompt)
+                if sigma_adjustment:
+                    rec_noise = find_noise_for_image_sigma_adjustment(p, cond, uncond, cfg, st)
+                else:
+                    rec_noise = find_noise_for_image(p, cond, uncond, cfg, st)
+                self.cache = Cached(rec_noise, cfg, st, lat, original_prompt, original_negative_prompt, sigma_adjustment)
 
             rand_noise = processing.create_random_tensors(p.init_latent.shape[1:], [p.seed + x + 1 for x in range(p.init_latent.shape[0])])
             
@@ -121,6 +176,7 @@ def sample_extra(conditioning, unconditional_conditioning, seeds, subseeds, subs
         p.extra_generation_params["Decode CFG scale"] = cfg
         p.extra_generation_params["Decode steps"] = st
         p.extra_generation_params["Randomness"] = randomness
+        p.extra_generation_params["Sigma Adjustment"] = sigma_adjustment
 
         processed = processing.process_images(p)
 


Test patch: