Differeces between Multi-dimension input to the nn.Linear layer and Iterative forwarding

username: obin
cooked: <p>Hi.<br>
I recently realized that <strong>nn.Linear</strong> can handle multi-dimension tensor.<br>
However, it seems that I do not fully understand the computation process in the linear layer (or matrix multiplication something…)</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/9/4919fee9b1d6e3751b17ca0bdd1d27238a525cd4.png" data-download-href="https://discuss.pytorch.org/uploads/default/4919fee9b1d6e3751b17ca0bdd1d27238a525cd4" title="multi_dim_layer"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/9/4919fee9b1d6e3751b17ca0bdd1d27238a525cd4_2_690x266.png" alt="multi_dim_layer" data-base62-sha1="aqGy9ZLtbNC0yg9WD2wDoUWUkcY" width="690" height="266" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/9/4919fee9b1d6e3751b17ca0bdd1d27238a525cd4_2_690x266.png, https://discuss.pytorch.org/uploads/default/original/3X/4/9/4919fee9b1d6e3751b17ca0bdd1d27238a525cd4.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/9/4919fee9b1d6e3751b17ca0bdd1d27238a525cd4.png 2x" data-dominant-color="EEEEEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">multi_dim_layer</span><span class="informations">778×301 38.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>In the example code above, I have (3 * 100 = 300) numbers of 512-dimension vectors.<br>
<strong>result1</strong> is the result of directly forwarding the multi dimension tensor to the linear layer.<br>
<strong>result2</strong> is the result of iteratively(100) forwarding (3 X 512) tensor to the linear layer and stacked them.</p>
<p>The two results are similar but <strong>not the same</strong> as you can see in the output of the code.<br>
Why the two computation yields different results?</p>

username: ptrblck
cooked: <p>The difference is most likely created due to the limited floating point precision and your code looks correct in comparing these use cases. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: chof
cooked: <p>Hi ptrblck, if here 100 is assumed to be the time_step, what fc_layer(rand_vector) is doing is same as TimeDistributed(Dense(256)) in tensorflow, right?<br>
I am actually referring to this problem: <a href="https://stackoverflow.com/questions/61372645/how-to-implement-time-distributed-dense-tdd-layer-in-pytorch" rel="nofollow noopener">https://stackoverflow.com/questions/61372645/how-to-implement-time-distributed-dense-tdd-layer-in-pytorch</a>, It seems to me that what nn.linear() does is actually the fashion of <strong>time-distributed dense</strong>, why are people still looking for time-distributed dense function? I would appreciate your correction. Thanks alot.</p>

username: ptrblck
cooked: <p>I’m not sure how the TDD layer works in TensorFlow, but as given in the code snippet, the linear layer will be applied to all samples in dim1 separately.<br>
If that’s what TDD does, then it should be equal. A verification with some constant inputs would be nice, in case you have a TF installation <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>