Fast Walsh-Hadamard transform

username: LoryPack
cooked: <p>Following my previous question, (<a href="https://discuss.pytorch.org/t/fastfood-random-projection/19163" class="inline-onebox">Fastfood random projection</a>), I am implementing Fast Walsh-Hadamard transform (FWHT) in Pytorch extending autograd Function, in order to be able to propagate gradients through it. This is done very easily: as the Walsh-Hadamard transform is equal to its inverse, the same transformation is applied in the forward and backward pass.</p>
<p>The point is that I found a big time difference when computing the transformation itself operating on Pytorch tensors or on Numpy array (20x slower with tensors). Specifically, the following piece of code operates on tensor:</p>
<pre><code class="lang-auto">class FWHT_OLD(Function):
    
    def transform(tensor):
        """ Simple implementation of FWHT, receiving as input a torch Tensor. """
        bit = length = len(tensor)
        result = torch.tensor(tensor)

        for _ in range(int(np.log2(length))): 
            bit &gt;&gt;= 1
            for i in range(length):  # d loops
                if i &amp; bit == 0: 
                    j = i | bit
                    temp = torch.tensor(result[i])  # otherwise it copies by reference!
                    result[i] += result[j]
                    result[j] = temp - result[j]

        result /= np.sqrt(length)  # normalize
        return result

    @staticmethod
    def forward(ctx, input):
        return FWHT_OLD.transform(input) 
    
    @staticmethod
    def backward(ctx, grad_output):
        return FWHT_OLD.transform(grad_output)  
    
</code></pre>
<p>While the following does the same operation but converting the tensor to a numpy array (only the lines with a comment changed with respect to previous one):</p>
<pre><code class="lang-auto">class FWHT(Function):
    
    def transform(tensor):
        """ Simple implementation of FWHT, receiving as input a torch Tensor. """
        bit = length = len(tensor)
        result = tensor.detach().numpy()  # transform to numpy

        for _ in range(int(np.log2(length))): 
            bit &gt;&gt;= 1
            for i in range(length): 
                if i &amp; bit == 0: 
                    j = i | bit
                    temp = result[i]  # this copies by value
                    result[i] += result[j]
                    result[j] = temp - result[j]

        result /= np.sqrt(length) 
        return torch.from_numpy(result)  # transform back to torch

    @staticmethod
    def forward(ctx, input):
        return FWHT.transform(input) 
    
    @staticmethod
    def backward(ctx, grad_output):
        return FWHT.transform(grad_output)  
    
</code></pre>
<p>In the following picture there is a time analysis of the performance of the algorithm in forward and backward pass. The yellow line is the comparison with a pure numpy code.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/7/79302c7cd780292708d15aa85aeaf0c451a487b3.png" alt="fwht_time" data-base62-sha1="hi4Z7j1TddPD6Zi7ihQVAk7vxvR" width="432" height="288"></p>
<p>Does anybody have a clue on why this happens?</p>
<p>DISCLAIMER: My code was inspired from <a href="https://gist.github.com/dougalsutherland/1a3c70e57dd1f64010ab" rel="noopener nofollow ugc">https://gist.github.com/dougalsutherland/1a3c70e57dd1f64010ab</a></p>

username: kealexanderwang
cooked: <p>Did you ever figure out the speed issue?</p>