Rewrite Keras code to PyTorch

username: thecho7
cooked: <p>Greetings,<br>
What I want to do is rewriting the Keras code into PyTorch.<br>
The only problem is how to implement Depthwise and Separable 2D convolution in PyTorch.<br>
Here’s the original code from <a href="https://github.com/vlawhern/arl-eegmodels/blob/master/EEGModels.py" rel="nofollow noopener">https://github.com/vlawhern/arl-eegmodels/blob/master/EEGModels.py</a></p>
<pre><code>input1   = Input(shape = (1, Chans, Samples))

##################################################################
block1       = Conv2D(F1, (1, kernLength), padding = 'same',
                               input_shape = (1, Chans, Samples),
                               use_bias = False)(input1)
block1       = BatchNormalization(axis = 1)(block1)
block1       = DepthwiseConv2D((Chans, 1), use_bias = False, 
                               depth_multiplier = D,
                               depthwise_constraint = max_norm(1.))(block1)
block1       = BatchNormalization(axis = 1)(block1)
block1       = Activation('elu')(block1)
block1       = AveragePooling2D((1, 4))(block1)
block1       = dropoutType(dropoutRate)(block1)

block2       = SeparableConv2D(F2, (1, 16),
                               use_bias = False, padding = 'same')(block1)
block2       = BatchNormalization(axis = 1)(block2)
block2       = Activation('elu')(block2)
block2       = AveragePooling2D((1, 8))(block2)
block2       = dropoutType(dropoutRate)(block2)
    
flatten      = Flatten(name = 'flatten')(block2)

dense        = Dense(nb_classes, name = 'dense', 
                     kernel_constraint = max_norm(norm_rate))(flatten)
softmax      = Activation('softmax', name = 'softmax')(dense)

return Model(inputs=input1, outputs=softmax)</code></pre>

username: ErlendFax
cooked: <p>Found on this forum:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.features = nn.Sequential(
            nn.Conv2d(1, 1, kernel_size=3, stride=1, padding=1),
            nn.ReLU(inplace=True),
        )

    def forward(self, x, y):
        x1 = self.features(x)
        x2 = self.features(y)
        x = torch.cat((x1, x2), 1)
        return x
</code></pre>
<p>By using <code>x = torch.cat((x1, x2), 1)</code> one can concatenate multiple towers/blocks. The forward function can take multiple input data. Want to learn more, search for “Pytorch multi input network” or something similar.</p>

username: ptrblck
cooked: <p>You can use the <code>groups</code> argument of <code>nn.Conv2d</code> followed by a pointwise convolution to get <code>SeparableConv2d</code>. Have a look at the <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Conv2d">docs</a> for more information of <code>groups</code>.</p>

username: Bijan1
cooked: <p>Thanks for your response <a class="mention" href="/u/ptrblck">@ptrblck</a>, but how should we define the depthwise_constraint = max_norm(1.) in pytorch as well as other features in keras documentation such as:</p>
<ul>
<li><strong>depthwise_regularizer</strong>: Optional regularizer for the convolution kernel.</li>
<li><strong>bias_regularizer</strong>: Optional regularizer for the bias vector.</li>
<li><strong>activity_regularizer</strong>: Optional regularizer function for the output.</li>
<li><strong>depthwise_constraint</strong>: Optional projection function to be applied to the kernel after being updated by an <code>Optimizer</code> (e.g. used to implement norm constraints or value constraints for layer weights). The function must take as input the unprojected variable and must return the projected variable (which must have the same shape). Constraints are not safe to use when doing asynchronous distributed training.</li>
<li><strong>bias_constraint</strong>: Optional projection function to be applied to the bias after being updated by an <code>Optimizer</code>.</li>
</ul>
<p>Thank you.</p>