Repository: AUTOMATIC1111/stable-diffusion-webui

Issue:
Problems with LDSR upscaling
When using the new LDSR upscaling feature, be it via SD upscaling in img2img or directly in "Extras", black rectangles appear in the outputs on the right and at the bottom when upscaling a 230x219 photo. 

![image 7](https://user-images.githubusercontent.com/111315671/191547996-47082733-9e6a-49d0-9ba5-6df01c2bb175.png)
original

![00021](https://user-images.githubusercontent.com/111315671/191548044-7799ad8f-02d1-4ac5-9699-4957d25a9442.png)
upscaled

no issues when running it on a 512x704 image

![00151-747985091-four squirrels by Tom Whalen](https://user-images.githubusercontent.com/111315671/191548282-fd4c4572-21d9-45ee-9e76-db5060dd6469.png)
original

![00020](https://user-images.githubusercontent.com/111315671/191548167-f08ba06b-6c4f-4c5f-b3ac-2bb9662f871b.png)
upscaled


Discussion:
Same issue here. Only seems to happen for images with non-square aspect ratios.
I’m having the same trouble when upscaling perfectly square images. It seems that aspect ratio does not matter. And the thickness of both horizontal and vertical black bars seems to somehow correlate with the chosen Resize factor and the resulting dimensions. Here’s an example (orange frames do not belong to images):
![ldsr_black_rects](https://user-images.githubusercontent.com/5103155/193427848-c177853b-7fea-413e-b008-704b6dc92796.png)

Resolving patch:
diff --git a/modules/ldsr_model_arch.py b/modules/ldsr_model_arch.py
--- a/modules/ldsr_model_arch.py
+++ b/modules/ldsr_model_arch.py
@@ -101,8 +101,8 @@ def super_resolution(self, image, steps=100, target_scale=2, half_attention=Fals
         down_sample_rate = target_scale / 4
         wd = width_og * down_sample_rate
         hd = height_og * down_sample_rate
-        width_downsampled_pre = int(wd)
-        height_downsampled_pre = int(hd)
+        width_downsampled_pre = int(np.ceil(wd))
+        height_downsampled_pre = int(np.ceil(hd))
 
         if down_sample_rate != 1:
             print(
@@ -110,7 +110,12 @@ def super_resolution(self, image, steps=100, target_scale=2, half_attention=Fals
             im_og = im_og.resize((width_downsampled_pre, height_downsampled_pre), Image.LANCZOS)
         else:
             print(f"Down sample rate is 1 from {target_scale} / 4 (Not downsampling)")
-        logs = self.run(model["model"], im_og, diffusion_steps, eta)
+        
+        # pad width and height to multiples of 64, pads with the edge values of image to avoid artifacts
+        pad_w, pad_h = np.max(((2, 2), np.ceil(np.array(im_og.size) / 64).astype(int)), axis=0) * 64 - im_og.size
+        im_padded = Image.fromarray(np.pad(np.array(im_og), ((0, pad_h), (0, pad_w), (0, 0)), mode='edge'))
+        
+        logs = self.run(model["model"], im_padded, diffusion_steps, eta)
 
         sample = logs["sample"]
         sample = sample.detach().cpu()
@@ -120,6 +125,9 @@ def super_resolution(self, image, steps=100, target_scale=2, half_attention=Fals
         sample = np.transpose(sample, (0, 2, 3, 1))
         a = Image.fromarray(sample[0])
 
+        # remove padding
+        a = a.crop((0, 0) + tuple(np.array(im_og.size) * 4))
+
         del model
         gc.collect()
         torch.cuda.empty_cache()


Test patch: