Very small weight updates

username: difisi
cooked: <p>I’m trying to apply a sensor-fusion architecture, explained in Vielzeuf, V. et al. - <a href="https://hal.archives-ouvertes.fr/hal-01858560/document" rel="noopener nofollow ugc">this paper</a> - , to my own problem. <strong>The way it works</strong> is it takes M separate 1D inputs, each of which enter M twin CNNs. Then, there is an additional twin (M + 1)-nth “CentralNet” connected to the remaining CNNs in such a way that its layers are a linear combination with learnable weights of itself with the corresponding layers of the other twins, as the following diagram suggests.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/b/bc91d0bef86b4c35bc3c48d0a27369352c125aa4.png" data-download-href="https://discuss.pytorch.org/uploads/default/bc91d0bef86b4c35bc3c48d0a27369352c125aa4" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/b/bc91d0bef86b4c35bc3c48d0a27369352c125aa4_2_517x200.png" alt="image" data-base62-sha1="qUaaHcgxdfit5CXIt8zh8VujPlq" width="517" height="200" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/b/bc91d0bef86b4c35bc3c48d0a27369352c125aa4_2_517x200.png, https://discuss.pytorch.org/uploads/default/optimized/2X/b/bc91d0bef86b4c35bc3c48d0a27369352c125aa4_2_775x300.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/b/bc91d0bef86b4c35bc3c48d0a27369352c125aa4.png 2x" data-dominant-color="F2F2F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">838×325 21.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Where M stands for the modality with numbered by its superscript, C stands for central (CentralNet) and the subscripts denote the number of the layer (Vielzeuf, V. et al.)</p>
<p>I’ve implemented this in PyTorch with the following Model Class:</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self, n_mods, conv_layers, conn_layers, kernels, paddings, pools, drop):
        super().__init__()

        # Number of modalities
        self.n_mods = n_mods

        # Number of convolutional and fully connected layers
        self.conv_len = len(conv_layers) - 1
        self.conn_len = len(conn_layers) - 1

        # CentralNet input, feature and output linear transformations' weights (uniformly distributed)
        self.w_init = nn.Parameter(torch.ones(n_mods) * 1 / n_mods, requires_grad=True)

        w_hidden = torch.ones((self.conv_len + self.conn_len), n_mods + 1) * 1 / (n_mods + 1)
        self.w_hidden = nn.Parameter(w_hidden, requires_grad=True)

        #  Build n_mods copies of the required architecture and append to final fusion_net
        self.fusion_net = nn.ModuleList()
        for mod in range(n_mods + 1):
            mod_net = nn.ModuleList()

            for D_in, D_out, kernel, padding, pool in zip(conv_layers, conv_layers[1:], kernels, paddings, pools):
                mod_net.append(self.conv_block(D_in, D_out, kernel, padding, pool))

            for i, D_in, D_out in zip(range(self.conn_len), conn_layers, conn_layers[1:]):
                mod_net.append(self.conn_block(D_in, D_out, drop, (i + 1) == self.conn_len))

            self.fusion_net.append(mod_net)

    def forward(self, x):
        # Input arrives as list of n_mods (batch_size, 1, signal_len) tensors
        y = list(x)

        # Create virtual input for CentralNet with linear combination of other n_mods inputs - mod = 0 is CentralNet
        y.insert(0, torch.zeros(x[0].shape, requires_grad=False))
        for i in range(self.n_mods):
            y[0] += self.w_init[i] * x[i]

        # Iterate through each set of corresponding layers from all the n_mods + 1 networks
        for block in range(self.conv_len + self.conn_len):
            for mod in range(self.n_mods + 1):
                for layer in self.get_block(mod, block):
                    y[mod] = layer(y[mod])

                # Perform weighted sum in every set of layers
                if mod == 0:
                    y[0] = y[0].clone() * self.w_hidden[block, mod]
                else:
                    y[0] = y[0].clone() + self.w_hidden[block, mod] * y[mod]

            # Flatten last set of convolutional layers
            if block + 1 == self.conv_len:
                y = self.flatten(y)

        return y

    # Repeating convolutional blocks
    def conv_block(self, D_in, D_out, kernel, padding, pool):
        conv_module = nn.ModuleList()
        conv_module.append(nn.Conv1d(D_in, D_out, kernel, padding=padding, bias=False))
        conv_module.append(nn.MaxPool1d(pool, stride=2))
        conv_module.append(nn.ReLU(inplace=False))
        conv_module.append(torch.nn.BatchNorm1d(D_out))
        return conv_module

    # Repeating fully-connected blocks
    def conn_block(self, D_in, D_out, drop, last):
        conn_module = nn.ModuleList()
        conn_module.append(nn.Linear(D_in, D_out, bias=False))
        conn_module.append(nn.ReLU(inplace=False))
        
        # Don't use dropout in last fully-connected layer
        if not last:
            conn_module.append(nn.Dropout(p=drop))
        return conn_module

    def flatten(self, y):
        N, _, _ = y[0].size()
        for i in range(len(y)):
            y[i] = y[i].view(N, -1)
        return y
    
    # Get operations from layer block and modality mod
    def get_block(self, mod, block):
        return list(list(self.fusion_net.children())[mod].children())[block]
</code></pre>
<p>Where <code>n_mods</code> is an integer for the number of input modalities, <code>conv_net</code> a list with the depth size of each convolutional layer (starting with 1), <code>conn_net</code> a list with the number of nodes in each fully connected layer (starting with the flattened length number and ending with 1), <code>kernels</code> is a list with the kernel size for each convolution, similarly, <code>padding</code> is a list with the paddings, <code>pools</code> is a list with the pooling size of for each convolutional block and <code>drop</code> is a float indicating the dropout probability.</p>
<p><strong>My struggle is that</strong> even though the architecture is able to learn as a whole - converge loss and reach high accuracy - the weighted sum parameters barely change from the initial values. The original paper reports how these weights significantly change, giving more importance to some modalities at different stages of the network, as shown:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/e/ed7e87b3a205ef9e448bf0c8d55bb8fe7d1ca691.png" alt="image" data-base62-sha1="xSYfuOJdK2C0CI39DzPWHdUlqSt" width="224" height="155"><br>
(Vielzeuf, V. et al.)</p>
<p>I’m using the sum of BCEWithLogitsLoss for each <code>n_mods + 1</code> modalities’ output as loss function, SGD, L1 normalization for weighted sum parameters, L2 for every other and the same learning rate for both groups (using 0.1 or 0.01 for weighted sum parameters drives them to near 0). <strong>Did I not implement it correctly with the “non-conventional” additional learnable parameters and the weighted sums?</strong></p>

username: Tahir
cooked: <aside class="quote no-group" data-username="difisi" data-post="1" data-topic="50119">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/difisi/48/12342_2.png" class="avatar"> difisi:</div>
<blockquote>
<p><strong>My struggle is that</strong> even though the architecture is able to learn as a whole - converge loss and reach high accuracy</p>
</blockquote>
</aside>
<p>i didn’t understand how you are visualizing or noticing all the weights changes?</p>

username: difisi
cooked: <p>I call <code>model.w_init</code> and <code>model.w_hidden</code> before and after an epoch or more and realise the initial values aren’t too far off from the lastly updated ones and very similar amongst each other. For example, with a learning rate of 0.001, starts at:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/5/5eda79acdee44516a51bd4d9ac5076119e63a5b6.png" alt="image" data-base62-sha1="dx6XJeEsCmpJhVQW7xZju7Ol3tc" width="467" height="198"></p>
<p>and after one epoch:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/9/980c0089c51fe78960c5d3187dcbb0b985796ac5.png" alt="image" data-base62-sha1="lH4hYlLJMbWkUhmuUAbGrtokOFf" width="465" height="202"></p>
<p>With a learning rate of 0.01, ends at:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/c/ce5623f85e7ac5ef739e5d085011f4368270f3ca.png" alt="image" data-base62-sha1="trkUFNQALiLLBQGVfHG7HE3WI1Y" width="471" height="202"></p>

username: Tahir
cooked: <p>It seems like they do change a bit…?</p>

username: difisi
cooked: <p>They either fall to very close to zero or change very little, mostly all in the same direction. This would mean they mostly bear the same importance - not much at all. I’m not sure if it is a result that makes sense, hence thinking there could be some implementation problem. Do you also think the implementation is ok? Then I could be worrying needlessly and the weights are actually evolving correctly</p>

username: Tahir
cooked: <p>The weight changes in individual weight values have to be very small IMO. As long as the model is training and loss is going down, i wont worry.</p>