Repository: AUTOMATIC1111/stable-diffusion-webui

Issue:
[Bug]: Error when generating with 'Highres. fix' and 'Upscale latent space image when doing hires. fix'
### Is there an existing issue for this?

- [X] I have searched the existing issues and checked the recent builds/commits

### What happened?

Attempting to use 'Highres. fix' with 'Upscale latent space image when doing hires. fix' results in the error message "UnboundLocalError: local variable 'decoded_samples' referenced before assignment" when it attempts to do the second batch of steps in the console.

When trying to generate "a cat" using stable diffusion 1.5, console prints:
```
Error completing request4:21,  7.00it/s]
Arguments: ('a cat', '', 'None', 'None', 28, 0, False, False, 1, 1, 7, -1.0, -1.0, 0, 0, 0, False, 1024, 1024, True, 0.7, 0, 0, 0, False, False, None, '', 1, '', 0, '', True, False, False) {}
Traceback (most recent call last):
  File "...\modules\ui.py", line 221, in f
    res = list(func(*args, **kwargs))
  File "...\webui.py", line 63, in f
    res = func(*args, **kwargs)
  File "...\modules\txt2img.py", line 48, in txt2img
    processed = process_images(p)
  File "...\modules\processing.py", line 426, in process_images
    res = process_images_inner(p)
  File "...\modules\processing.py", line 519, in process_images_inner
    samples_ddim = p.sample(conditioning=c, unconditional_conditioning=uc, seeds=seeds, subseeds=subseeds, subseed_strength=p.subseed_strength)
  File "...\modules\processing.py", line 688, in sample
    decoded_samples,
UnboundLocalError: local variable 'decoded_samples' referenced before assignment
```

### Steps to reproduce the problem

1. Enable 'Upscale latent space image when doing hires. fix' in Settings
2. Enable 'Highres. fix'
3. Generate

### What should have happened?

The image should have generated

### Commit where the problem happens

d885a4a57b72152745ca76192ef1bdda29e6461d

### What platforms do you use to access UI ?

Windows

### What browsers do you use to access the UI ?

Google Chrome
[Bug]: Error when generating with 'Highres. fix' and 'Upscale latent space image when doing hires. fix'
### Is there an existing issue for this?

- [X] I have searched the existing issues and checked the recent builds/commits

### What happened?

Attempting to use 'Highres. fix' with 'Upscale latent space image when doing hires. fix' results in the error message "UnboundLocalError: local variable 'decoded_samples' referenced before assignment" when it attempts to do the second batch of steps in the console.

When trying to generate "a cat" using stable diffusion 1.5, console prints:
```
Error completing request4:21,  7.00it/s]
Arguments: ('a cat', '', 'None', 'None', 28, 0, False, False, 1, 1, 7, -1.0, -1.0, 0, 0, 0, False, 1024, 1024, True, 0.7, 0, 0, 0, False, False, None, '', 1, '', 0, '', True, False, False) {}
Traceback (most recent call last):
  File "...\modules\ui.py", line 221, in f
    res = list(func(*args, **kwargs))
  File "...\webui.py", line 63, in f
    res = func(*args, **kwargs)
  File "...\modules\txt2img.py", line 48, in txt2img
    processed = process_images(p)
  File "...\modules\processing.py", line 426, in process_images
    res = process_images_inner(p)
  File "...\modules\processing.py", line 519, in process_images_inner
    samples_ddim = p.sample(conditioning=c, unconditional_conditioning=uc, seeds=seeds, subseeds=subseeds, subseed_strength=p.subseed_strength)
  File "...\modules\processing.py", line 688, in sample
    decoded_samples,
UnboundLocalError: local variable 'decoded_samples' referenced before assignment
```

### Steps to reproduce the problem

1. Enable 'Upscale latent space image when doing hires. fix' in Settings
2. Enable 'Highres. fix'
3. Generate

### What should have happened?

The image should have generated

### Commit where the problem happens

d885a4a57b72152745ca76192ef1bdda29e6461d

### What platforms do you use to access UI ?

Windows

### What browsers do you use to access the UI ?

Google Chrome


Discussion:
I just encountered this bug as well. The problem is specifically when using the highres fix with the "Upscale latent space image when doing hires. fix" option ticked on the settings page. It seems that when that is checked, in the conditional beginning at https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/78b879b4426d3730acdf3f4737d37a2ee29cb023/modules/processing.py#L654
the later-referenced variable `decoded_samples` is indeed never set in the true branch.
+1 seeing this as well, looks like the bug was introduced in this commit https://github.com/AUTOMATIC1111/stable-diffusion-webui/commit/26a3fd2fe9314330336fb0e28d1e9ca7d2abe10e

Specifically caused by enabling Scale Latent
Yeah just tried it without scaling latent space and it worked, I'll update the issue to include that
+1, I also just ran into this problem.
Same issue here. Log:

```
Traceback (most recent call last):
  File "C:\Users\user\Desktop\automatic1111\modules\ui.py", line 185, in f
    res = list(func(*args, **kwargs))
  File "C:\Users\user\Desktop\automatic1111\webui.py", line 63, in f
    res = func(*args, **kwargs)
  File "C:\Users\user\Desktop\automatic1111\modules\txt2img.py", line 48, in txt2img
    processed = process_images(p)
  File "C:\Users\user\Desktop\automatic1111\modules\processing.py", line 426, in process_images
    res = process_images_inner(p)
  File "C:\Users\user\Desktop\automatic1111\modules\processing.py", line 519, in process_images_inner
    samples_ddim = p.sample(conditioning=c, unconditional_conditioning=uc, seeds=seeds, subseeds=subseeds, subseed_strength=p.subseed_strength)
  File "C:\Users\user\Desktop\automatic1111\modules\processing.py", line 688, in sample
    decoded_samples,
UnboundLocalError: local variable 'decoded_samples' referenced before assignment
```
Also have this issue, same error above. 

I also have this issue.
> Yeah just tried it without scaling latent space and it worked, I'll update the issue to include that

Scale latent has been essential for higher quality generations. Is there a way to fix this without removing it?
Temporary fix is to just revert to commit a0a7024c679056dd66beb1832e52041b10143130 
I just encountered this bug as well. The problem is specifically when using the highres fix with the "Upscale latent space image when doing hires. fix" option ticked on the settings page. It seems that when that is checked, in the conditional beginning at https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/78b879b4426d3730acdf3f4737d37a2ee29cb023/modules/processing.py#L654
the later-referenced variable `decoded_samples` is indeed never set in the true branch.
+1 seeing this as well, looks like the bug was introduced in this commit https://github.com/AUTOMATIC1111/stable-diffusion-webui/commit/26a3fd2fe9314330336fb0e28d1e9ca7d2abe10e

Specifically caused by enabling Scale Latent
Yeah just tried it without scaling latent space and it worked, I'll update the issue to include that
+1, I also just ran into this problem.
Same issue here. Log:

```
Traceback (most recent call last):
  File "C:\Users\user\Desktop\automatic1111\modules\ui.py", line 185, in f
    res = list(func(*args, **kwargs))
  File "C:\Users\user\Desktop\automatic1111\webui.py", line 63, in f
    res = func(*args, **kwargs)
  File "C:\Users\user\Desktop\automatic1111\modules\txt2img.py", line 48, in txt2img
    processed = process_images(p)
  File "C:\Users\user\Desktop\automatic1111\modules\processing.py", line 426, in process_images
    res = process_images_inner(p)
  File "C:\Users\user\Desktop\automatic1111\modules\processing.py", line 519, in process_images_inner
    samples_ddim = p.sample(conditioning=c, unconditional_conditioning=uc, seeds=seeds, subseeds=subseeds, subseed_strength=p.subseed_strength)
  File "C:\Users\user\Desktop\automatic1111\modules\processing.py", line 688, in sample
    decoded_samples,
UnboundLocalError: local variable 'decoded_samples' referenced before assignment
```
Also have this issue, same error above. 

I also have this issue.
> Yeah just tried it without scaling latent space and it worked, I'll update the issue to include that

Scale latent has been essential for higher quality generations. Is there a way to fix this without removing it?
Temporary fix is to just revert to commit a0a7024c679056dd66beb1832e52041b10143130 

Resolving patch:
diff --git a/modules/masking.py b/modules/masking.py
--- a/modules/masking.py
+++ b/modules/masking.py
@@ -49,7 +49,7 @@ def expand_crop_region(crop_region, processing_width, processing_height, image_w
     ratio_processing = processing_width / processing_height
 
     if ratio_crop_region > ratio_processing:
-        desired_height = (x2 - x1) * ratio_processing
+        desired_height = (x2 - x1) / ratio_processing
         desired_height_diff = int(desired_height - (y2-y1))
         y1 -= desired_height_diff//2
         y2 += desired_height_diff - desired_height_diff//2
diff --git a/modules/processing.py b/modules/processing.py
--- a/modules/processing.py
+++ b/modules/processing.py
@@ -134,11 +134,7 @@ def txt2img_image_conditioning(self, x, width=None, height=None):
             # Dummy zero conditioning if we're not using inpainting model.
             # Still takes up a bit of memory, but no encoder call.
             # Pretty sure we can just make this a 1x1 image since its not going to be used besides its batch size.
-            return torch.zeros(
-                x.shape[0], 5, 1, 1, 
-                dtype=x.dtype, 
-                device=x.device
-            )
+            return x.new_zeros(x.shape[0], 5, 1, 1)
 
         height = height or self.height
         width = width or self.width
@@ -156,11 +152,7 @@ def txt2img_image_conditioning(self, x, width=None, height=None):
     def img2img_image_conditioning(self, source_image, latent_image, image_mask = None):
         if self.sampler.conditioning_key not in {'hybrid', 'concat'}:
             # Dummy zero conditioning if we're not using inpainting model.
-            return torch.zeros(
-                latent_image.shape[0], 5, 1, 1,
-                dtype=latent_image.dtype,
-                device=latent_image.device
-            )
+            return latent_image.new_zeros(latent_image.shape[0], 5, 1, 1)
 
         # Handle the different mask inputs
         if image_mask is not None:
@@ -174,11 +166,11 @@ def img2img_image_conditioning(self, source_image, latent_image, image_mask = No
                 # Inpainting model uses a discretized mask as input, so we round to either 1.0 or 0.0
                 conditioning_mask = torch.round(conditioning_mask)
         else:
-            conditioning_mask = torch.ones(1, 1, *source_image.shape[-2:])
+            conditioning_mask = source_image.new_ones(1, 1, *source_image.shape[-2:])
 
         # Create another latent image, this time with a masked version of the original input.
         # Smoothly interpolate between the masked and unmasked latent conditioning image using a parameter.
-        conditioning_mask = conditioning_mask.to(source_image.device)
+        conditioning_mask = conditioning_mask.to(source_image.device).to(source_image.dtype)
         conditioning_image = torch.lerp(
             source_image,
             source_image * (1.0 - conditioning_mask),
@@ -674,6 +666,13 @@ def save_intermediate(image, index):
 
         if opts.use_scale_latent_for_hires_fix:
             samples = torch.nn.functional.interpolate(samples, size=(self.height // opt_f, self.width // opt_f), mode="bilinear")
+            
+            # Avoid making the inpainting conditioning unless necessary as 
+            # this does need some extra compute to decode / encode the image again.
+            if getattr(self, "inpainting_mask_weight", shared.opts.inpainting_mask_weight) < 1.0:
+                image_conditioning = self.img2img_image_conditioning(decode_first_stage(self.sd_model, samples), samples)
+            else:
+                image_conditioning = self.txt2img_image_conditioning(samples)
 
             for i in range(samples.shape[0]):
                 save_intermediate(samples, i)
@@ -700,14 +699,14 @@ def save_intermediate(image, index):
 
             samples = self.sd_model.get_first_stage_encoding(self.sd_model.encode_first_stage(decoded_samples))
 
+            image_conditioning = self.img2img_image_conditioning(decoded_samples, samples)
+
         shared.state.nextjob()
 
         self.sampler = sd_samplers.create_sampler_with_index(sd_samplers.samplers, self.sampler_index, self.sd_model)
 
         noise = create_random_tensors(samples.shape[1:], seeds=seeds, subseeds=subseeds, subseed_strength=subseed_strength, seed_resize_from_h=self.seed_resize_from_h, seed_resize_from_w=self.seed_resize_from_w, p=self)
 
-        image_conditioning = self.txt2img_image_conditioning(x)
-
         # GC now before running the next img2img to prevent running out of memory
         x = None
         devices.torch_gc()


Test patch: