nn.GRU with mask

pretbc: <p>Hello,</p>
<p>Im wondering how to implement below Keras approach for GRU into Pytorch GRU.</p>
<pre><code class="lang-auto">    frame_features_input = keras.Input((MAX_SEQ_LENGTH, NUM_FEATURES))
    mask_input = keras.Input((MAX_SEQ_LENGTH,), dtype="bool")

    # Refer to the following tutorial to understand the significance of using `mask`:
    # https://keras.io/api/layers/recurrent_layers/gru/
    x = keras.layers.GRU(16, return_sequences=True)(
        frame_features_input, mask=mask_input
    )
</code></pre>
<p>most important part is mask definition.</p>
<p>Keras:</p>
<p>mask: Binary tensor of shape [samples, timesteps] indicating whether a given timestep should be masked (optional). An individual True entry indicates that the corresponding timestep should be utilized, while a False entry indicates that the corresponding timestep should be ignored. Defaults to None.</p>

J_Johnson: <p>Something like this should work:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

# set variables
features = 16
hidden_dim = 32
seq_len = 128
batch_size = 64

model = nn.GRU(features, hidden_dim, bias=False)

# generate an input
dummy_input = torch.rand((seq_len, batch_size, features))

# generate a mask
mask = torch.rand((seq_len, batch_size, 1))&gt;0.5 # 50% masked values
mask = mask.repeat(1,1,features) #this only repeats the same mask across the features dimension

# apply the mask
dummy_input_masked = torch.where(mask, 0, dummy_input) # filling masked values with 0
print(dummy_input_masked)

# make the hidden layer
h0 = torch.rand(1, batch_size, hidden_dim)

#put masked input and hidden layer into the model
output, hn = model(dummy_input_masked, h0)
</code></pre>

pretbc: <p>I don’t know if Your answer cover me needs, becouse You pass at the and masked input , and what Keras Gru do (in my opinion) , is applying some kind of attention based on given mask.</p>
<p>My features</p>
<pre><code class="lang-auto">FEATURE
tensor([[[0.6853, 0.7958, 0.0017,  ..., 0.1823, 0.5553, 0.2935],
         [0.6637, 0.8767, 0.0271,  ..., 0.1365, 0.3762, 0.1650],
         [0.5990, 0.7948, 0.0245,  ..., 0.1462, 0.4408, 0.1912],
         ...,
         [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
         [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
         [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000]],

        [[0.3059, 1.4664, 0.0243,  ..., 0.6032, 0.6973, 0.3338],
         [0.3780, 1.1796, 0.1137,  ..., 0.3834, 0.7203, 0.1014],
         [0.2908, 1.2995, 0.1637,  ..., 0.5579, 0.8582, 0.2847],
         ...,
         [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
         [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
         [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000]]]) torch.Size([2, 112, 2048])
MASK
tensor([[ True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
          True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
          True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
          True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
          True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
          True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
          True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
          True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
          True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
          True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
         False, False, False, False, False, False, False, False, False, False,
         False, False],
        [ True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
          True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
          True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
          True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
          True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
          True,  True,  True,  True,  True,  True,  True,  True,  True,  True,
          True, False, False, False, False, False, False, False, False, False,
         False, False, False, False, False, False, False, False, False, False,
         False, False, False, False, False, False, False, False, False, False,
         False, False, False, False, False, False, False, False, False, False,
         False, False, False, False, False, False, False, False, False, False,
         False, False]]) torch.Size([2, 112])
</code></pre>
<p>My code is based on colab implementation for video sequence analyze - they took features from CNN model ( inception v3) where features are [SEQ_LEN, 2048]</p>
<p>and they wanna pass them to GRU net</p>
<p>link → <a href="https://colab.research.google.com/github/keras-team/keras-io/blob/master/examples/vision/ipynb/video_classification.ipynb#scrollTo=tY9i2oAQnfW_" rel="noopener nofollow ugc">keras_colab_video_cls</a></p>

J_Johnson: <p>It looks like the mask was intended to be used to ignore any frames that were missing:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/4/543e97295795b1d75ce6b35beffde343f0e65e20.png" data-download-href="https://discuss.pytorch.org/uploads/default/543e97295795b1d75ce6b35beffde343f0e65e20" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/4/543e97295795b1d75ce6b35beffde343f0e65e20.png" alt="image" data-base62-sha1="c1gcPFLiyaSNw7raUO2udbpRFba" width="690" height="270" data-dominant-color="F4F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">810×318 33.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/f/af98a06417b8aba6f5a3454893f1a4c2c5606b12.png" data-download-href="https://discuss.pytorch.org/uploads/default/af98a06417b8aba6f5a3454893f1a4c2c5606b12" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/f/af98a06417b8aba6f5a3454893f1a4c2c5606b12_2_442x500.png" alt="image" data-base62-sha1="p3oyBy0JKDA1oQiKLlIr4aVcqSC" width="442" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/f/af98a06417b8aba6f5a3454893f1a4c2c5606b12_2_442x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/f/af98a06417b8aba6f5a3454893f1a4c2c5606b12_2_663x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/f/af98a06417b8aba6f5a3454893f1a4c2c5606b12.png 2x" data-dominant-color="A1A0A0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">741×837 50.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/0/6034ed028bcf3020f4e2e7d0a4c804168b9a09c5.png" data-download-href="https://discuss.pytorch.org/uploads/default/6034ed028bcf3020f4e2e7d0a4c804168b9a09c5" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/0/6034ed028bcf3020f4e2e7d0a4c804168b9a09c5.png" alt="image" data-base62-sha1="dJ5egVq3sv6cy7Mg3VG9YnLcbXv" width="690" height="311" data-dominant-color="EEEFEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">691×312 9.89 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>It does not appear to be an attention mask, like in a Transformer/LLM. In fact, the above is a loop that seems to just determine whether the frame exists or not, and then inserts a 1 if it does, or defaults to 0. You could recreate a similar loop with PyTorch for your dataset, and then output the mask from the dataloader and apply it based on that.</p>

pretbc: <p>I did that</p>
<pre><code class="lang-auto">feature_extractor = timm.create_model('inception_v3', pretrained=True, num_classes=0, global_pool='avg')
feature_extractor.eval()


class LoadData(Dataset):
    def __init__(self, data_frame, feature_extractor, transformer=None):
        self.data_frame = data_frame
        self.feature_extractor = feature_extractor
        self.transformer = transformer

    def __len__(self):
        return len(self.data_frame)

    def __getitem__(self, index):
        dumped_frames_path = self.data_frame.iloc[index]['dumped_frames_path']
        frames = self.load_frames(dumped_frames_path, transformer=self.transformer)
        features = self.extract_features(frames)
        label = int(self.data_frame.iloc[index]['cls_labels'])
        return features, label

    def extract_features(self, frames):
        frame_masks = torch.zeros(size=(1, MAX_SEQ_LENGTH), dtype=torch.bool)
        frame_features = torch.zeros(size=(1, MAX_SEQ_LENGTH, NUM_FEATURES))
        frames = frames[None, ...]
        for i, batch in enumerate(frames):
            video_length = batch.shape[0]
            length = min(MAX_SEQ_LENGTH, video_length)
            with torch.no_grad():
                for j in range(length):
                    frame_features[i, j, :] = self.feature_extractor(batch[None, j, :])
                frame_masks[i, :length] = 1
        frame_masks = frame_masks.squeeze()
        frame_features = frame_features.squeeze()
        return frame_features, frame_masks

    @staticmethod
    def load_frames(path, max_frames=0, transformer=None):
        frames = []
        for img in (p for p in os.listdir(path) if p.endswith('.png')):
            frame = Image.open(os.path.join(path, img))
            if transformer is not None:
                frame = transforms(frame)
            frames.append(frame)
            if len(frames) == max_frames:
                break
        if not frames:
            raise FileNotFoundError(f'Cannot list path: {path}')
        return torch.stack(frames)


train_data = LoadData(train[:100], feature_extractor, transformer=transforms)
train_loader = DataLoader(train_data, batch_size=32, shuffle=True, pin_memory=True, drop_last=True)
</code></pre>
<p>and model</p>
<pre><code class="lang-auto">class RnnGru(nn.Module):
    def __init__(self, input_dim, hidden_dim, n_layers, num_classes, drop_prob=0.4):
        super().__init__()
        self.n_layers = n_layers
        self.hidden_dim = hidden_dim
        self.gru = nn.GRU(input_dim, hidden_dim, n_layers, batch_first=True, dropout=drop_prob)
        self.fc = nn.Linear(hidden_dim, num_classes)
        self.relu = nn.ReLU()

    def forward(self, x):
        h0 = torch.zeros(self.n_layers, x.size(0), self.hidden_dim).to(device)

        out, _ = self.gru(x, h0)
        out = self.relu(out)
        out = self.fc(out[:, -1, :])
        return out
</code></pre>
<p>but still I cannot understand what should I place as model input</p>
<ol>
<li>should I do under training :</li>
</ol>
<pre><code class="lang-auto">dummy_input_masked = torch.where(frame_masks, 0, frame_features)
output, hn = model(dummy_input_masked)
</code></pre>

pretbc: <p>I just read Mask part</p>
<p>so I assume I have to use packing</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/b/cb1a96feea44d57d2e0a4ec791f91dc9e99e3ef1.png" data-download-href="https://discuss.pytorch.org/uploads/default/cb1a96feea44d57d2e0a4ec791f91dc9e99e3ef1" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/b/cb1a96feea44d57d2e0a4ec791f91dc9e99e3ef1_2_690x404.png" alt="image" data-base62-sha1="sYJTua1gzudkMuhXuwrk6AMWrOV" width="690" height="404" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/b/cb1a96feea44d57d2e0a4ec791f91dc9e99e3ef1_2_690x404.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/b/cb1a96feea44d57d2e0a4ec791f91dc9e99e3ef1_2_1035x606.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/b/cb1a96feea44d57d2e0a4ec791f91dc9e99e3ef1_2_1380x808.png 2x" data-dominant-color="F6F5F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1704×1000 72.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

J_Johnson: <p>If you are using the packed sequence route, something like this might work:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

# set variables
features = 16
hidden_dim = 32
seq_len = 128
batch_size = 64

model = nn.GRU(features, hidden_dim, bias=False)

# generate an input
dummy_input = torch.rand((seq_len, batch_size, features))

# generate a mask
mask = torch.rand((seq_len, batch_size, 1))&gt;0.5 # 50% masked values

# get lengths
lengths = torch.sum(mask.squeeze(2), 0)

mask = mask.repeat(1,1,features) #this only repeats the same mask across the features dimension

# apply the mask
dummy_input_masked = torch.where(mask, 0, dummy_input) # filling masked values with 0

# pack the sequence
dummy_input_packed = torch.nn.utils.rnn.pack_padded_sequence(dummy_input_masked, lengths, enforce_sorted=False)

print(dummy_input_packed)

# make the hidden layer
h0 = torch.rand(1, batch_size, hidden_dim)

#put masked input and hidden layer into the model
output, hn = model(dummy_input_packed, h0)

print(output)
</code></pre>