Style Transfer - how to choose the cnn layers?

username: laro
cooked: <p>I read this tutorial:<br>
<a href="https://pytorch.org/tutorials/advanced/neural_style_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Neural Transfer Using PyTorch — PyTorch Tutorials 1.12.0+cu102 documentation</a></p>
<p>In that tutorial they choosed:</p>
<ul>
<li>conv layer <span class="hashtag">#4</span> for <strong>content_layers</strong>
</li>
<li>conv layers: 1, 2, 3, 4, 5 for <strong>style_layers</strong>
</li>
</ul>
<ol>
<li>Why they choosed the 4 conv layer for <strong>content_layers</strong> and <strong>style_layers</strong>  ?</li>
<li>I read that <strong>content_layers</strong> are the deep conv layers,<br>
so how can conv layer <span class="hashtag">#5</span> choosed for <strong>style_layers</strong> ? (which is deeper than conv layer <span class="hashtag">#4</span> which choosed for <strong>content_layers</strong>) ?</li>
<li>I saw another medium post about <strong>Style Transfer</strong>:<br>
<a href="https://medium.com/analytics-vidhya/style-transfer-pytorch-84cf2e9ba86d" class="inline-onebox" rel="noopener nofollow ugc">Style Transfer using Pytorch. I have recreated the style transfer… | by Alex Diaz | Analytics Vidhya | Medium</a><br>
In that post the author choosed other conv layers for <strong>content_layers</strong> and <strong>style_layers</strong><br>
Are there any rules or ideas behind the conv layers we choose ?</li>
</ol>

username: Andrei_Cristea
cooked: <p>Hi Amit -</p>
<p>I doubt you will find too much theory on this since it will often come down to the specific architecture and images the model was trained on.</p>
<p>However, I recommend you look into feature visualization (there are several approaches here which you’ll find after a quick search) and simply spend a little time visualizing each of your layers. After a little while it should become apparent which is more “style” and which is more “content”.</p>
<p>Good luck!</p>

username: laro
cooked: <p>I took your advice and looked on the feature visualization:</p>
<p>The source image was:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/6/468b33af41414c79dac0d12514bb186a02902340.png" alt="src" data-base62-sha1="a43F1eyQqbgHl5PFlYowW0y3B8Q" width="541" height="416"></p>
<p>According to the article in: <a href="https://medium.com/analytics-vidhya/style-transfer-pytorch-84cf2e9ba86d" class="inline-onebox" rel="noopener nofollow ugc">Style Transfer using Pytorch. I have recreated the style transfer… | by Alex Diaz | Analytics Vidhya | Medium</a><br>
the content layer is layer <span class="hashtag-raw">#21</span> (conv4_2) and it’s first 4 fatures:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/9/d90b32fa969f38b90d5d6663cffb4ff39b8912ea.png" data-download-href="https://discuss.pytorch.org/uploads/default/d90b32fa969f38b90d5d6663cffb4ff39b8912ea" title="content"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/9/d90b32fa969f38b90d5d6663cffb4ff39b8912ea_2_690x138.png" alt="content" data-base62-sha1="uY3Bw6tGTfg37tJsT7EBvBPFYXE" width="690" height="138" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/9/d90b32fa969f38b90d5d6663cffb4ff39b8912ea_2_690x138.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/9/d90b32fa969f38b90d5d6663cffb4ff39b8912ea_2_1035x207.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/9/d90b32fa969f38b90d5d6663cffb4ff39b8912ea.png 2x" data-dominant-color="C9B983"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">content</span><span class="informations">1254×251 190 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The other conv layers are style layers.<br>
The first 4 features of the first 3 layers:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/8/784bdf4e7699c63ce45f14a671add80fa3364543.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/784bdf4e7699c63ce45f14a671add80fa3364543" title="style"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/8/784bdf4e7699c63ce45f14a671add80fa3364543_2_690x395.jpeg" alt="style" data-base62-sha1="habQPqlDZlLuF29RVMyXBJPt0MX" width="690" height="395" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/8/784bdf4e7699c63ce45f14a671add80fa3364543_2_690x395.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/7/8/784bdf4e7699c63ce45f14a671add80fa3364543_2_1035x592.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/8/784bdf4e7699c63ce45f14a671add80fa3364543.jpeg 2x" data-dominant-color="9D9393"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">style</span><span class="informations">1264×725 177 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<ul>
<li>And it seems that there is no much difference between those layers (style and content).</li>
<li>To me is seems that we can extract the outline of the objects of an image from the style and content layer (and not just from the content layer… as you can see above)</li>
</ul>
<p>So what am I missing or misunderstanding?</p>

username: Andrei_Cristea
cooked: <p>Cool visualizations! I think what you’re seeing here is which area of the image is activating the different layers the most, which doesn’t exactly help you in understanding what exactly the layer is looking for / what the feature is matching on. <a href="https://distill.pub/2020/circuits/visualizing-weights/" rel="noopener nofollow ugc">This article is a helpful addition to yours with more info.</a>.</p>
<p>Broadly there are at least a couple of approaches you can take to visualizing features the way you are likely looking to do:<br>
(1) go through an actual image dataset and find examples of chunks of image that achieve peak activation of a particular neuron, and then combine those in some way<br>
(2) take a GAN approach of transforming a randomly initiated image in the direction that maximally activates a particular neuron. this can suffer from some high-frequency problems and look funky, so you may have to regularize for that (penalizing high frequency patterns)</p>
<p>I don’t think that there’s a single way to do it that will always work, which is perhaps why there isn’t a good package that does this for you out of the box. Interpretability is extremely exciting, and if you build anything cool and want to share it, the community will appreciate it!</p>