GRU and soft attention problem

username: nbravo
cooked: <p>Hello everyone,</p>
<p>First of all, I want to thank this community for help me out in a lot of tiny and big problems since I began using Pytorch; sadly I couldn’t find a related thread with the current problem i’m facing. I don’t have much experience with recurrent models and attention mechanisms, that’s why i can’t figure it out how can I solve it yet, that’s why i’m asking to you what may be wrong with my code.</p>
<p>I’m implementing a multimodal model that matches video with corresponding text descriptions by converting video-sentence pairs into a common vector space. The paper of this model is <a href="https://www-nlpir.nist.gov/projects/tvpubs/tv17.papers/DL-61-86.pdf" rel="noopener nofollow ugc">DL-61-86 at TRECVID 2017: Video-to-Text Description</a> (it doesn’t have any public code, that’s why i’m implementing it by my own).</p>
<p>The part that i’m struggling to get right is the RNN and soft attention mechanism in the video branch of the model. It applies a GRU to N frames (N depending of each video) of the video and then uses a soft attention over the N outputs of the GRU to get a 1-D embedding of the video. My problem is with the GRU and trainable weights of the soft attention mechanism Ws, Wm and Wm. They all tend to a constant too quick (first 1-2 iterations) in the training process, ignoring the video branch input and making the output vector of the branch almost zero in every value.<br>
This are the tensorboard histrogram of this weights during training.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/a/3abdb20d61d851f3462b267291367a889851c4ba.png" data-download-href="https://discuss.pytorch.org/uploads/default/3abdb20d61d851f3462b267291367a889851c4ba" title="Screenshot_20191023_203029"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/a/3abdb20d61d851f3462b267291367a889851c4ba_2_690x259.png" alt="Screenshot_20191023_203029" data-base62-sha1="8nE74qMm1iUyRaUlITgUAKJhsEi" width="690" height="259" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/a/3abdb20d61d851f3462b267291367a889851c4ba_2_690x259.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/a/3abdb20d61d851f3462b267291367a889851c4ba_2_1035x388.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/a/3abdb20d61d851f3462b267291367a889851c4ba.png 2x" data-dominant-color="FCF8F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot_20191023_203029</span><span class="informations">1342×505 56.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/f/9f87f2b678eed370a9032a919cbdc89e66f3edff.png" data-download-href="https://discuss.pytorch.org/uploads/default/9f87f2b678eed370a9032a919cbdc89e66f3edff" title="Screenshot_20191023_203111"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/f/9f87f2b678eed370a9032a919cbdc89e66f3edff_2_690x130.png" alt="Screenshot_20191023_203111" data-base62-sha1="mLhbu7zuV32D5fhX67BympL19PV" width="690" height="130" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/f/9f87f2b678eed370a9032a919cbdc89e66f3edff_2_690x130.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/f/9f87f2b678eed370a9032a919cbdc89e66f3edff_2_1035x195.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/f/9f87f2b678eed370a9032a919cbdc89e66f3edff.png 2x" data-dominant-color="FBF7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot_20191023_203111</span><span class="informations">1328×252 30 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>My implementation of the video branch is this, please focus on the soft attention part where I iterate over the batch. I commented the sizes of the layers and tensors in the forward method for easier read.</p>
<pre><code class="lang-auto">def xavier_init(fc):
    r = np.sqrt(.6) / np.sqrt(fc.in_features + fc.out_features)
    fc.weight.data.uniform_(-r, r)
    if fc.bias is not None:
        fc.bias.data.fill_(0)

class VideoBranch(nn.Module):
                          # 2048         2048         512        'gpu:0'
    def __init__(self, resnet152_dim, hidden_dim, video_out_dim, device):
        super(VideoBranch, self).__init__()
        self.device = device
        self.gru = nn.GRU(resnet152_dim, hidden_dim)

        self.w_v = nn.Linear(resnet152_dim, hidden_dim * 2)   # 2048 x 4096
        self.w_s = nn.Linear(hidden_dim * 2, hidden_dim * 2)   # 4096 x 4096
        self.w_m = nn.Linear(hidden_dim * 2, hidden_dim * 2) # 4096 x 4096

        self.dropout_1 = nn.Dropout(0.2)
        self.dense_1 = nn.Linear(hidden_dim * 2, video_out_dim)  # 4096 x 512 
        self.dense_1_bn = nn.BatchNorm1d(video_out_dim)  # 512
        self.dropout_2 = nn.Dropout(0.2)
        self.dense_2 = nn.Linear(video_out_dim, video_out_dim) # 512 x 512
        self.dense_2_bn = nn.BatchNorm1d(video_out_dim) # 512

        xavier_init(self.w_v)
        xavier_init(self.w_s)
        xavier_init(self.w_m)
        xavier_init(self.dense_1)
        xavier_init(self.dense_2)

    def forward(self, resnet152_vectors, n_frames): # [B, M, 2048] , [B,]  
        # M is the maximum of frames extracted from video in the dataset, if a video
        # has N &lt; M frames, its embedding (a list of tensors of dimension 2048) is padded with 
        # M - N zero tensors of size 2048 at the end of the list. This allows to iterate using a 
        # DataLoader (all inputs of the same size)

        # packing the input vectors of the GRU for computation efficiency
        embs = pack_padded_sequence(resnet152_vectors, n_frames, batch_first=True, enforce_sorted=False)
        gru_out_packed, _ = self.gru(embs)
        # Getting the output of the GRU for each video, they are padded to
        gru_out = pad_packed_sequence(gru_out_packed, batch_first=True)[0]

        # soft attention mechanism
        # since every embedding in the batch has different number of frames N, 
        # and the attention mechanism works with the mean of the gru output of each video, 
        # i can't make computations by batch and i have to iterate on every video of the batch, 
        # so first i crop the padded zero tensors the 'pad_packed_sequence' added and I apply the soft 
        # attention video by video. This is the part that i'm worried about because maybe by 
        # iterating over the batch, the gradient is applied multiple times on the trainable weights 
        # of the attention mechanism (Wm, Ws and Wv), modifing its values more than necessary.
        v_a = None
        for i in range(resnet152_vectors.size(0)):
           # s is the concat of the every GRU output (2048-d tensor) of the video with its corresponding input (2048-d tensor).
           # size: [N, 4096]. N different for every video.
            s = torch.cat((gru_out[i, :int(n_frames[i]), :], resnet152_vectors[i, :int(n_frames[i]), :]), 1)
            
            v = torch.mean(resnet152_vectors[i, :int(n_frames[i]), :], dim=0)  # [2048]
            y_s = nn.Tanh()(self.w_s(s))  # [N, 4096]
            y_v = nn.Tanh()(self.w_v(v))  # [4096]
            m = torch.mul(y_s, y_v)       # [N, 4096]
            r = self.w_m(m)               # [N, 4096]
            alpha = nn.Softmax(dim=1)(r)  # [N, 4096]
            
            # v_a is the final vector of the attention where i concat every final 4096-d vector of each
            # video and get a final [B, 4096] to continue making calculation by batch
            if i == 0:
                v_a = torch.sum(torch.mul(alpha, s), dim=0).unsqueeze(0)   # [1, 4096]
            else:
                v_a = torch.cat((v_a, torch.sum(torch.mul(alpha, s), dim=0).unsqueeze(0)), dim=0)

        # latent space conversion
        y = self.dropout_1(v_a) # [B, 4096]
        y = self.dense_1(y)     # [B, 512] and so on...
        y = self.dense_1_bn(y) 
        y = nn.ReLU()(y)
        y = self.dropout_2(y)
        y = self.dense_2(y)
        y = self.dense_2_bn(y)
        y = nn.ReLU()(y)
        return y                #[B, 512]
</code></pre>
<p>I know this may be a hard problem to solve, but I want to discard any implementation mistakes, because i’ve looked up and fixed every other aspect (training data, training process, loss function, etc) and i can’t get this model to work as intended yet.</p>
<p>Thanks you very much for your time.</p>