cycleGAN with blank background

username: Alan_lam
cooked: <p>I am building a cycleGAN with a u-net structure to improve the image quality of Cone Beam Computer Tomography (CBCT), with the Fan Beam Computer Tomography FBCT) being the target images. Because I want to mainly enhance the quality in the lung region, I crop the lung volumes out of the original images and assign value 0 to the remaining regions other than the lung (given that the lung pixel value ranges from -1000 to 0). Scaling and normalization are done to the array with the range of -1000 to 0. Please see the following images as an example for my dataset:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/c/cc79403b1ad2f185050c97d442fd021edd6d0dfe.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/cc79403b1ad2f185050c97d442fd021edd6d0dfe" title="CBCT"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/c/cc79403b1ad2f185050c97d442fd021edd6d0dfe_2_500x500.jpeg" alt="CBCT" data-base62-sha1="taRbkNGTu7STcRhJXjBVH96zCsS" width="500" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/c/cc79403b1ad2f185050c97d442fd021edd6d0dfe_2_500x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/c/c/cc79403b1ad2f185050c97d442fd021edd6d0dfe_2_750x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/c/cc79403b1ad2f185050c97d442fd021edd6d0dfe_2_1000x1000.jpeg 2x" data-dominant-color="E2E2E2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">CBCT</span><span class="informations">2000×2000 156 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
In the example above, the left most image is the CBCT and the right most image is the FBCT. The middle one is the generated image from the CBCT by the cycleGAN model. This is actually an example from the very early epoch of the training.</p>
<p>But when the model goes on training, it somehow gradually loses its ability to capture the anatomy of the images, and eventually generate a blank image with all value 0. (see image below)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/d/fd6b2f6207cacf918bc1a6db660ce267e5abadff.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/fd6b2f6207cacf918bc1a6db660ce267e5abadff" title="Image53-60_E9"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/d/fd6b2f6207cacf918bc1a6db660ce267e5abadff_2_500x500.jpeg" alt="Image53-60_E9" data-base62-sha1="A9QrsXOtNCpeqDx2Nj8nh7jRZDN" width="500" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/d/fd6b2f6207cacf918bc1a6db660ce267e5abadff_2_500x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/f/d/fd6b2f6207cacf918bc1a6db660ce267e5abadff_2_750x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/d/fd6b2f6207cacf918bc1a6db660ce267e5abadff_2_1000x1000.jpeg 2x" data-dominant-color="E9E9E9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Image53-60_E9</span><span class="informations">2000×2000 115 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/3/d3246c2d8097d12075d9c5515c4a8d142497161d.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/d3246c2d8097d12075d9c5515c4a8d142497161d" title="Image18-0_E17"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/3/d3246c2d8097d12075d9c5515c4a8d142497161d_2_500x500.jpeg" alt="Image18-0_E17" data-base62-sha1="u7QMra2pnOf8JjPz1HWdE4vl0IJ" width="500" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/3/d3246c2d8097d12075d9c5515c4a8d142497161d_2_500x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/d/3/d3246c2d8097d12075d9c5515c4a8d142497161d_2_750x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/3/d3246c2d8097d12075d9c5515c4a8d142497161d_2_1000x1000.jpeg 2x" data-dominant-color="F1F1F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Image18-0_E17</span><span class="informations">2000×2000 96.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The loss along the training is climbing back up after several epoch and it starts to lose the anatomy information.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/9/598e07e1d02660d09b7ce119bb0ad69efe7ac5cd.png" data-download-href="https://discuss.pytorch.org/uploads/default/598e07e1d02660d09b7ce119bb0ad69efe7ac5cd" title="Untitled"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/9/598e07e1d02660d09b7ce119bb0ad69efe7ac5cd_2_690x358.png" alt="Untitled" data-base62-sha1="cMeNfNA5eAmOUD9mh0kftw85f65" width="690" height="358" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/9/598e07e1d02660d09b7ce119bb0ad69efe7ac5cd_2_690x358.png, https://discuss.pytorch.org/uploads/default/original/3X/5/9/598e07e1d02660d09b7ce119bb0ad69efe7ac5cd.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/9/598e07e1d02660d09b7ce119bb0ad69efe7ac5cd.png 2x" data-dominant-color="FDFCFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Untitled</span><span class="informations">867×451 19.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>What makes me curious is that such loss of anatomy does not occur when I simply input the whole CBCT and FBCT images as the dataset, without doing lung segmentation nor value assignment to the regions outside the lung. If un-segmented images are given, the model actually successfully translate the CBCT into mimicking the FBCT quality. I do the segmentation since I want the model to only concentrate in the lung region to see if it performs better.</p>
<p>I wonder if this is the consequences with that the background has extremely high value than the region of interest (i.e. background value: 0;  lung value: -1000 to 0). Is there any work published on cycleGAN training with images containing blank background? If yes, is there any special measure when assigning value to the background, or when doing normalization and scaling? I can’t really find any so far.</p>
<p>Any insight is appreciated. Thank you.</p>