Wrong color information when plotting stylegan3 generated images with motplotlib

username: clbr
cooked: <p>Hey there!<br>
I’ve been playing around with plotting images generated from StyleGAN3, but there seems to be something funky going on with the color channels that i can’t really pin down.</p>
<p>Generating an image with:</p>
<pre><code class="lang-auto">device = torch.device('cuda')
# open pickle file and load generator
network_pkl = './models/stylegan3-r-afhqv2-512x512.pkl'
with dnnlib.util.open_url(network_pkl) as f:
    G = legacy.load_network_pkl(f, force_fp16=True)['G_ema'].cuda()

noise_mode = 'const'
trunc = 1
seed = 10
z = torch.from_numpy(np.random.RandomState(seed).randn(1, G.z_dim)).to(device)
# returns [b, c, 512, 512] tensor
img = G(z, truncation_psi=trunc, noise_mode=noise_mode, c=None)
# permute to [b, 512, 512, c] and scale to 0-255
img = (img.permute(0, 2, 3, 1) * 127.5 + 128).clamp(0, 255).to(torch.uint8)
# remove batch channel
img = img.squeeze(0) 
# move tensors to cpu
img_cpu = img.cpu()
# plot with pyplot
plt.imshow(img_cpu.numpy())
# save with pillow
PIL.Image.fromarray(img_cpu.numpy(), 'RGB').save(f'seed{seed}.png')
</code></pre>
<p>From my understanding the generator of StyleGAN3 returns an image as a tensor of <code>[batch_size, channels, height, width]</code> with channels being <code>RGB</code>. The plotted image seems to be have completely wrong color information.</p>
<p>From <code>plt.imshow(img_cpu.numpy())</code>:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/1/e11f41c6d3228f932d8bbb31ac32ec88f917bb88.jpeg" alt="Screenshot from 2021-10-18 15-34-19" data-base62-sha1="w7woHYmdaMbpN7VJuXcywuLqjk4" width="280" height="253"></p>
<p>Saving with <code>PIL.Image.fromarray(img_cpu.numpy(), 'RGB').save(f'seed{seed}.png')</code>:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/d/0d4b8afb40250715df53e29042a07bfeb9b91b96.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/0d4b8afb40250715df53e29042a07bfeb9b91b96" title="Screenshot from 2021-10-18 15-34-32"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/d/0d4b8afb40250715df53e29042a07bfeb9b91b96.jpeg" alt="Screenshot from 2021-10-18 15-34-32" data-base62-sha1="1TC3xczIGH0RhxsfQqSyubvBciy" width="497" height="500" data-dominant-color="5D6256"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-10-18 15-34-32</span><span class="informations">516×519 87.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I swapped around the channels to <code>BGR</code>, <code>GRB</code> etc. but couldn’t get the plot to display the correct image. Would appreciate any help with this :).</p>
<p>Cheers</p>
<p>Edit: Issue wasn’t code related. Was using jetbrains DataSpell ide which is in early preview and still has some bugs.</p>

username: ptrblck
cooked: <p>That’s quite strange, as I can load your second image via <code>PIL.Image.open</code> and visualize it in <code>PIL</code> as well as <code>matplotlib</code> to get the same output.<br>
Did you change any colormaps in <code>matplotlib</code> or loaded any other template (e.g. via <code>seaborn</code>)?</p>

username: clbr
cooked: <p>Nope, haven’t changed anything.</p>
<pre><code class="lang-auto">import PIL.Image as Image
import matplotlib.pyplot as plt

with Image.open("seed10.png") as im:
    plt.imshow(im)
</code></pre>
<p>This gives me the same image. With <code>seed10.png</code> being the correct image above.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/a/4ae3d53c0478460d3299f82dfd9779b17ef908c8.jpeg" alt="Screenshot from 2021-10-19 10-42-15" data-base62-sha1="aGvsGqHdZ8FtPMqhFHGpNAWPltC" width="278" height="255"></p>
<p>Also tried grabbing the screencapture above as <code>.jpeg</code>, gives me the same result.<br>
This is with matplotlib 3.4.2 and pillow 8.4.0.</p>
<p>But i guess this isn’t really a pytorch issue anymore. <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>

username: ptrblck
cooked: <p>Yeah, it’s unrelated to PyTorch, but still interesting as I haven’t seen the issue before.<br>
It doesn’t look like a perfectly inverted image though.<br>
Using your code snippet I get the same image back and inverting it naively via:</p>
<pre data-code-wrap="python"><code class="lang-python">plt.imshow(np.array(img)*(-1)+255)
</code></pre>
<p>gives<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/4/d46d7b6af0af9eb8ba86dcb1db18dc1905178e30.png" alt="image" data-base62-sha1="ujdMRhhrQfAq4AJou6HVS7BT45O" width="259" height="252"></p>

username: ptrblck
cooked: <p>Could you run this test and check the colors in your setup, please?</p>
<pre><code class="lang-python">red = np.concatenate((
    np.full((224, 224, 1), 255).astype(np.uint8),
    np.zeros((224, 224, 1)).astype(np.uint8),
    np.zeros((224, 224, 1)).astype(np.uint8)
), axis=2)

green = np.concatenate((
    np.zeros((224, 224, 1)).astype(np.uint8),
    np.full((224, 224, 1), 255).astype(np.uint8),
    np.zeros((224, 224, 1)).astype(np.uint8)
), axis=2)

blue = np.concatenate((
    np.zeros((224, 224, 1)).astype(np.uint8),
    np.zeros((224, 224, 1)).astype(np.uint8),
    np.full((224, 224, 1), 255).astype(np.uint8),
), axis=2)

plt.imshow(red)
plt.imshow(green)
plt.imshow(blue)
</code></pre>

username: clbr
cooked: <p>Looks like i found the culprit <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20">. Was using jetbrains early preview of DataSpell.</p>
<p>DataSpell:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/f/efa47663d77f1aebb0318a2075a85a2e52109837.png" alt="Screenshot from 2021-10-19 10-59-42" data-base62-sha1="ybYtquMqlzOnSexOyCo6naKb5NZ" width="537" height="447"></p>
<p>PyCharm:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/7/979e849b63d108d8a7c2c92b0fdc53933be41c63.png" alt="Screenshot from 2021-10-19 10-56-54" data-base62-sha1="lDhIIAKJ0lOiiT0ktZlixfDOMi7" width="395" height="399"></p>

username: ptrblck
cooked: <p>Good to hear you’ve found the issue.<br>
I’m not familiar with this IDE, but do you have an idea why colors seems to be inverted or at least remapped? That seems to be unexpected (unless it’s of course a bug, then it’ll get fixed I guess).</p>
<p>EDIT: Also, just out of interest: could you still post the red, green, blue outputs?</p>

username: clbr
cooked: <p>If i would have to guess its probably a bug with how the code editor is rendered.<br>
Cheers for you effort btw. i’ll post the outputs in a sec <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Edit: here the the individual outputs in DataSpell, excuse the rather large image.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/5/4529ca935e0f579c8641ed9a14eae818199dd0f3.png" data-download-href="https://discuss.pytorch.org/uploads/default/4529ca935e0f579c8641ed9a14eae818199dd0f3" title="Screenshot from 2021-10-19 11-08-00"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/5/4529ca935e0f579c8641ed9a14eae818199dd0f3_2_227x500.png" alt="Screenshot from 2021-10-19 11-08-00" data-base62-sha1="9RQtWog7D4fArptwZAQxiJFBSKL" width="227" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/5/4529ca935e0f579c8641ed9a14eae818199dd0f3_2_227x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/5/4529ca935e0f579c8641ed9a14eae818199dd0f3_2_340x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/5/4529ca935e0f579c8641ed9a14eae818199dd0f3_2_454x1000.png 2x" data-dominant-color="3E413E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-10-19 11-08-00</span><span class="informations">485×1067 28.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Seems like they don’t have issues rendering the individual colors, no clue then. <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=12" title=":joy:" class="emoji" alt=":joy:" loading="lazy" width="20" height="20"></p>

username: ptrblck
cooked: <p>Yeah, weird indeed. I was thinking if the <a href="https://www.jetbrains.com/help/dataspell/configuring-colors-and-fonts.html">color scheme</a> might interact with <code>matplotlib</code> and override the colors or rather be globally applied to the entire IDE. In any case, you might forward the issue to the IDE devs. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=10" title=":wink:" class="emoji" alt=":wink:"></p>

username: clbr
cooked: <p>Will do. And actually you are right. With the light theme the image is plotted correctly.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/9/997ad1bc7125d2d0176f2f08b8976f89062d8ccc.png" alt="Screenshot from 2021-10-19 11-13-49" data-base62-sha1="lTKbUl6dXX4f6dzeKljztotqJXe" width="335" height="344"></p>

username: ptrblck
cooked: <p>OK, cool.<br>
To your previous pictures: note that the colors are also not “true” RGB. Look at this quick and dirty check:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/4/b44c56eae9e83b56726328e48d43311e7948ca56.png" data-download-href="https://discuss.pytorch.org/uploads/default/b44c56eae9e83b56726328e48d43311e7948ca56" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/4/b44c56eae9e83b56726328e48d43311e7948ca56_2_690x437.png" alt="image" data-base62-sha1="pIZv5cjsAjz9WiaM0vVuAq4vSD4" width="690" height="437" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/4/b44c56eae9e83b56726328e48d43311e7948ca56_2_690x437.png, https://discuss.pytorch.org/uploads/default/original/3X/b/4/b44c56eae9e83b56726328e48d43311e7948ca56.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/4/b44c56eae9e83b56726328e48d43311e7948ca56.png 2x" data-dominant-color="52403D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">736×467 54.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: clbr
cooked: <p>Well… <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=12" title=":joy:" class="emoji" alt=":joy:" loading="lazy" width="20" height="20"><br>
Seems to be correct here as well:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/d/8d4f33885e2194b919d09263581ee2ff99c72a0b.png" data-download-href="https://discuss.pytorch.org/uploads/default/8d4f33885e2194b919d09263581ee2ff99c72a0b" title="Screenshot from 2021-10-19 11-16-17"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/d/8d4f33885e2194b919d09263581ee2ff99c72a0b_2_171x500.png" alt="Screenshot from 2021-10-19 11-16-17" data-base62-sha1="ka50Hzs6ds2APqRSITKzMIx3cVl" width="171" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/d/8d4f33885e2194b919d09263581ee2ff99c72a0b_2_171x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/d/8d4f33885e2194b919d09263581ee2ff99c72a0b_2_256x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/d/8d4f33885e2194b919d09263581ee2ff99c72a0b_2_342x1000.png 2x" data-dominant-color="AEB7AE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-10-19 11-16-17</span><span class="informations">351×1026 23.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: WoodieDudy
cooked: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/2/e217e774a6be4b53010a6f2f3110363544a6c430.png" alt="image" data-base62-sha1="wg77BxM3kS3ei4zi7EOG3rG9wgo" width="649" height="369"><br>
It also can be fixed by unchecking “invert image outputs for the dark themes” box</p>

username: barista
cooked: <p>Oh man, this feature which is active by default has cost me hours of my life. Why would I want the image outputs to be inverted? It’s great to have more features but they should leave this off by default.</p>

username: ptrblck
cooked: <p>Yes, I totally agree and don’t know the reason why image outputs should be remapped using the IDE color scheme by default (it also took some time and imagination to figure out what might be causing the issue).<br>
Maybe <a class="mention" href="/u/clbr">@clbr</a> forwarded the feedback to the IDE devs so that the default might be changed?</p>