How to implement MLPconv of Network in Network?

BarCodeReader: <p>I did not see any pytorch implementation about this, so I want to give a try.</p>
<p>from the paper, it is said that, they apply a tiny MLP layer(or few layers) after the conv kernel. So this MLP layer is acting locally instead of globally, I saw some code just using conv2d with kernel size 1 attached to conv2d with kernel size 3, which is different from the original paper.</p>
<p>Because the convolution in CNN, before going to ReLU, is just a elementwise-multiplication with a sum action, so the MLPconv should be something like below:</p>
<pre><code class="lang-auto"># suppose we have a conv kernel K, which is 3x3 in size, and
# it is right about to do the element-wise multiplication with the image patch P, 
# which is also 3x3 in size(think about a 3x3 area of an image with NxM size). 
# Here I use * represents element-wise multiplication.

mlp1 = nn.Linear(9,9)
mlp2 = nn.Linear(9,1)
relu = F.relu()

# mlpconv action then is equal to a convolution action with 
# each of the step doing below calculation:
relu( mlp2( relu( mlp1( (P*K).view(-1) )))) 
</code></pre>
<p>but how to do this in pytorch???</p>
<p>I need some ideas and your help.</p>

ptrblck: <p>How would the input channels be treated?<br>
Would the linear layer perform an operation such as:</p>
<pre><code class="lang-python"># input patch
patch = torch.randn(in_channels, height, width).view(-1)
lin = nn.Linear(in_channels * height * width, 1)
out = lin(patch)
</code></pre>
<p>If so, then this would be equal to a conv layer without overlapping windows, no?</p>

BarCodeReader: <p>ok i implemented, with a sliding window…now they are updating…both mlp and conv2d</p>
<p>but it is really ugly…I still look forward your opnions</p>

SwayStar123: <p>Here is a simple implementation of it</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/SwayStar123/MLPConv/blob/main/mlpconv.py">
  <header class="source">

      <a href="https://github.com/SwayStar123/MLPConv/blob/main/mlpconv.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/SwayStar123/MLPConv/blob/main/mlpconv.py" target="_blank" rel="noopener nofollow ugc">SwayStar123/MLPConv/blob/main/mlpconv.py</a></h4>


      <pre><code class="lang-py">import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Union, Tuple

class MLPConv2d(nn.Module):
    """
    A convolutional layer where each convolution operation is performed by an MLP.

    Args:
        in_channels (int): Number of channels in the input image.
        out_channels (int): Number of output channels.
        kernel_size (int or tuple): Size of the convolving kernel.
        stride (int or tuple, optional): Stride of the convolution. Default: 1
        padding (int or tuple, optional): Zero-padding added to both sides of the input. Default: 0
        dilation (int or tuple, optional): Spacing between kernel elements. Default: 1
        num_layers (int, optional): Number of layers in the MLP. Default: 2
        hidden_size (int, optional): Number of neurons in the hidden layers of the MLP. Default: out_channels
        activation (callable, optional): Activation function. Default: nn.ReLU
        bias (bool, optional): If True, adds a learnable bias to the output. Default: True
</code></pre>



  This file has been truncated. <a href="https://github.com/SwayStar123/MLPConv/blob/main/mlpconv.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


qq-me: <p>MLPConv is the same as pointwise convolution, it is is equivalent to a convolution with 1x1 kernel</p>