Masking operation on specific dimension

username: kinjad
cooked: <p>Hi,<br>
I have a question when using Pytorch for my experiments. I am not sure if I used a very appropriate title to address this, but I’ll describe it in details below.<br>
I am working on building text models using LSTM. Below is the snapshot of my original code:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/3/325b61f9537ed7e77ba205430470b1f7f58fcfeb.png" data-download-href="https://discuss.pytorch.org/uploads/default/325b61f9537ed7e77ba205430470b1f7f58fcfeb" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/2X/3/325b61f9537ed7e77ba205430470b1f7f58fcfeb.png" alt="image" data-base62-sha1="7btEVkQWjh9uDVCv8Uj0HSELS0b" width="690" height="144" data-dominant-color="060504"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1868×392 21 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Basically what I am doing is to feed the character level embedding of each word inside a sentence into a LSTM and gather the output of the LSTM of the last character of each word.<br>
So the output of the LSTM(which is the forw_lstm_out and back_lstm_out in the code) have a shape of:<br>
batch_size * seq_len_char * char_hidden_dim, where seq_len_char is the padded length of all the characters in a sentence(number of all the characters in a sentence) and char_hidden_dim is the hidden dimension of the LSTM cell. In forw_position, it stores the positions of the last character of each word. So for example, if the sentence is “I almost returned it”, then the forw_position will store something like [0, 7, 16, 18] which correspond to [‘I’, ‘t’, ‘d’, ‘t’]. And forw_position are padded as well in order to be processed in batch. So the shape of forw_position will be batch_size * word_len * char_hidden_dim, where word_len is the padded word length(number of words in a sentence).<br>
So the gathering operation will gather the output of the LSTM of the last character of each word in the sentence to represent the embedding of each word.<br>
What I want to do now is to instead of using the output of the last character, I want to use the average output of all the characters in a word. For example, for word “almost”, instead of using the output for just ‘t’ which is the last character, I want to use the average of the output of all the characters (‘a’, ‘l’, ‘m’, ‘o’, ‘s’, ‘t’)  to represent the embedding for word “almost”.<br>
I thought about using masks like this:<br>
Given LSTM out which has shape of [batch_size * seq_len_char * char_hidden_dim], I will create a mask which has a shape of [batch_size * word_len * seq_len_char * char_hidden_dim]. And in this mask, on the third dimension which is the seq_len_char dimension, I will use 0/1 to indicate the characters of a word. But I am not sure how to exactly perform matrix multiplication for these two matrices. Or is there better way to do this? Could someone help me on this? Thank you so much!</p>