If input does not have any locality, will naive DNN better than CNN?

ljj7975: <p>This idea comes to be randomly.</p>
<p>One of the advantage of CNN is the fact that it captures the locality in the input space through receptive fields.</p>
<p>If the input data does not have such relationship among features,<br>
Would DNN provide better results?</p>
<p>Is there any previous study on this? I would like to take a look</p>

KFrank: <p>Hi Brandon!</p>
<aside class="quote no-group quote-modified" data-username="ljj7975" data-post="1" data-topic="47301" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ljj7975/48/12561_2.png" class="avatar"> ljj7975:</div>
<blockquote>
<p>…<br>
One of the advantage of CNN is the fact that it captures the locality in the input space through receptive fields.</p>
<p>If the input data does not have such relationship among features,<br>
Would DNN provide better results?<br>
…</p>
</blockquote>
</aside>
<p>First, by “CNN” I assume you mean a “convolutional neural<br>
network,” that is, a network whose first few layers consist of<br>
convolutions.  And by “naive DNN” I assume you mean a<br>
network consisting of fully-connected layers.</p>
<p>Yes, your expectation is correct – a CNN will not work well<br>
on a problem whose input doesn’t have “locality.”</p>
<p>Let’s say that your input consists of 1024 values, and that<br>
the relationship between, say, inputs <span class="hashtag-raw">#7</span> and <span class="hashtag-raw">#505</span> is just as<br>
likely to carry useful information as the relationship between<br>
inputs <span class="hashtag-raw">#7</span> and <span class="hashtag-raw">#8</span>.  The constrained structure of the CNN – its<br>
“assumption” of locality – is going to make it difficult for the<br>
network to discover the relationship between inputs <span class="hashtag-raw">#7</span> and<br>
<span class="hashtag-raw">#505</span>.</p>
<p>(You can implement a convolutional layer as a fully-connected<br>
layer where the fully-connected weight matrix consists of a<br>
bunch of copies of the convolutional block, with zeros elsewhere.<br>
The weights that would mix inputs <span class="hashtag-raw">#7</span> and <span class="hashtag-raw">#505</span> together and<br>
potentially discover these inputs’ relationships will be zero in<br>
this case – so no mixing, at least in this layer.)</p>
<p>Also note that in addition to this constraint of locality, a<br>
convolutional layer also has translational invariance.  (That<br>
is, the convolutional block is translated unchanged across<br>
the input, which is to say, it is <em>convolved</em> with the  input.)</p>
<p>For example, let’s say your input is a 2-d image.  We expect<br>
that edges within the image that form potentially useful<br>
features have more or less the same structure and meaning<br>
in the lower left corner of the image as in the upper right.<br>
Or we may expect to be able to detect the image of a dog<br>
equally well in the lower left and upper right corners of the<br>
image.</p>
<p>If the local, translationally-invariant structure of the convolutional<br>
layers match the structure of the input data, then you come out<br>
ahead.  If it doesn’t match, then you come out behind, and you<br>
are better off training fully-connected layers that can learn the<br>
(non-local, non-translationally-invariant) structure of the input<br>
data.</p>
<p>Best.</p>
<p>K. Frank</p>

ljj7975: <p>Thank you very much !<br>
It’s the best answer</p>