Why does the Linear module seems to do unnecessary transposing?

Brando_Miranda: <p>I was looking at the code for <code>torch.nn.Linear(in_features, out_features, bias=True)</code> and it seems that it store the matrix one way but then decides that to compute stuff its necessary to transpose (though the transposing seems it could have been avoided). Why does it store a matrix with dimensions (code <a href="http://pytorch.org/docs/master/_modules/torch/nn/modules/linear.html#Linear" rel="nofollow noopener">http://pytorch.org/docs/master/_modules/torch/nn/modules/linear.html#Linear</a>):</p>
<pre><code>self.weight = Parameter(torch.Tensor(out_features, in_features))
</code></pre>
<p>to then go ahead an compute the linear transform as follows:</p>
<pre><code>def forward(self, input):
    return F.linear(input, self.weight, self.bias)
</code></pre>
<p>which points to (<a href="http://pytorch.org/docs/master/_modules/torch/nn/functional.html#linear" rel="nofollow noopener">http://pytorch.org/docs/master/_modules/torch/nn/functional.html#linear</a>):</p>
<pre><code>output = input.matmul(weight.t())
</code></pre>
<p>can’t this just be avoided by respecting the order of how the dimensions were given and then do a matrix multiply without the transpose?</p>
<pre><code>def __init__(self, in_features, out_features, bias=True):
....
    self.weight = Parameter(torch.Tensor(in_features,out_features))
</code></pre>
<p>then just do:</p>
<pre><code>input.matmul(weight)
</code></pre>
<p>and we avoid having to rotate data? Maybe rotating the data doesn’t happen as I think in hardware and the data is not moved at all or something, but it just seems really unnecessary.</p>
<p>Besides even if it wasn’t inefficient, its really weird to me that the data is represented as row vectors (i.e. one row is a data point so the rows span the space of data points in the original raw dimension) but then the weight vector is stored as a D_out x D_in which seems to imply that the D_out is the target dimension where we land to so it seems odd to start thinking about row vectors to then randomly switch to column vectors. Why was this done?</p>
<p>Plus when one does <code>linear.weight</code> it was surprising to me to discover that the shape of the parameters were all switched from what I initially write when I created my linear layer. Maybe its just me but it seems super odd and confusing.</p>

L0SG: <p>I was also thinking about this, and found this issue:</p><aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/pytorch/issues/2159">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/issues/2159" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/2159" target="_blank" rel="noopener nofollow ugc">Efficient forward pass in nn.Linear</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2017-07-20" data-time="07:05:29" data-timezone="UTC">07:05AM - 20 Jul 17 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2017-07-20" data-time="07:26:18" data-timezone="UTC">07:26AM - 20 Jul 17 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/yunjey" target="_blank" rel="noopener nofollow ugc">
          <img alt="yunjey" src="https://avatars.githubusercontent.com/u/15663219?v=4" class="onebox-avatar-inline" width="20" height="20">
          yunjey
        </a>
      </div>
    </div>

    <div class="labels">
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">Hi, 

Maybe we can improve the operation speed in `nn.Linear`.

`nn.Linear` <span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">creates a weight matrix of shape (out_features, in_features) as you can see [here](https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/linear.py#L39).
Then, as you can see in [F.linear](https://github.com/pytorch/pytorch/blob/master/torch/nn/functional.py#L550-L558), input matrix is multiplied by the transpose of the weight matrix.

I think the transpose operation is unnecessary. What if we create a weight matrix of shape (in_features, out_features)? Then, the input matrix can be multiplied by the weight matrix (we don't need to calculate transpose of the matrix).

Does this sound right? If i'm right, can i do pull request?

Thanks</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>From what i understand, transposing in forward pass has no overhead. But backward pass will be less efficient if</p>
<blockquote>
<p>input.matmul(weight)</p>
</blockquote>

Brando_Miranda: <p>why is it less efficient in backward pass?</p>

L0SG: <p>I’m afraid I cannot answer the question since I don’t know about details of CuDNN implementation. Apparently RNN, which uses same matrix multiple times, exploits pre-transposing for performance improvement.</p>
<pre><code>https://devblogs.nvidia.com/parallelforall/optimizing-recurrent-neural-networks-cudnn-5/
</code></pre>
<blockquote>
<p>When performing a GEMM the standard BLAS API allows you to transpose either of the two input matrices. Some of the four combinations of transpose/not-transposed run slightly faster or slower than others. Depending on the way that the equations are mapped to the computation, a slower version of the GEMM may be used. By performing a transpose operation up-front on the weight matrix, each step can be made slightly faster. This comes at the cost of the transpose, but that is fairly cheap, so if the transposed matrix is to be used for more than a few iterations it is often worth it.</p>
</blockquote>
<p>It would be nice if someone else can elaborate this in more general models, including feed-forward network.</p>

smth: <p>Transposition is free for gemm calls, because BLAS libraries (that implement general matrix multiply (gemm)) support both row major and column major matrices, and transpositions.</p>
<p>So it’s okay to have that transpose call, it’s practically a free operation.</p>

Robert_Bamler: <p>Adding to smth’s response, storing the second matrix of a matrix multiplication in transposed form may even increase efficiency. This is because the multiplication routine can access the memory in a more contiguous way, leading to fewer cache misses. See, e.g.,<br>
<a href="https://stackoverflow.com/questions/18796801/increasing-the-data-locality-in-matrix-multiplication" rel="nofollow noopener">https://stackoverflow.com/questions/18796801/increasing-the-data-locality-in-matrix-multiplication</a></p>
<p>Storing the second matrix in transposed form can easily lead to a ~5x speedup in a naive matrix multiplication implementation. The effect will be much smaller in pytorch because the underlying matrix multiplication routine is certainly more clever than the one in the above link.</p>

Jimmy_Xiaoke_Shen: <p>This is a very good explanation. It explains that by using the transpose, we will have less catch missing. It will be perfect if anyone can futher analysis whehter the transpose have any influence to the backward process.</p>

KUSHAL_JAIN: <p>Is this behavior only restricted for <code>nn.Linear</code> layers or is it implemented in all <code>nn</code> modules. I specifically want to know if the internal weight matrices are transposed for an <code>RNN</code> layer. I can see that the <code>weight_ih</code> that is input to hidden matrix is transposed while storing but I cannot be sure about the <code>weight_hh</code> since it’s a square matrix. I need to know since I am updating weight manually for each connection and transposed matrices might imply that I am updating the wrong connections. Basically, I want to know “which neuron led the neuron in subsequent layer to fire”.</p>