## 3 The Proposed Method  
In this paper, we proposed a simple method to handle variable length sequences for RNNs without information distortion. All the input sequences must be pre-processed with a target length of _max/bmax_ and the position of _end_ (i.e., _max_end_*_ or _bmax_end_*_), or with PackedSequence. With these conditions, there is no information loss, and the original frames are not affected by the padded frames. The bottom of Figure 4 and Algorithm 1 show the overview and pseudo code of the proposed method, respectively. Consider the set of output sequences \(\textbf{H}=[\textbf{H}_{1},\ldots,\textbf{H}_{n}]\) in a mini-batch of size \(n\). All the output sequences in **H** has the same length \(T\), which are computed from the padded input sequences. For each mini-batch, each output sequence \(\textbf{H}_{i}=[\textbf{H}_{i}[0],\ \textbf{H}_{i}[1],\ \ldots,\ \textbf{H}_{i}[T-1]]\) is first truncated to have its original length \(t_{i}=\textbf{sl}[i]\). The truncation is applied at the end of the sequence, that is, the truncated sequence is \(\overset{\wedge}{\textbf{H}}_{i}=\textbf{H}_{i}[:t_{i}]=[\textbf{H}_{i}[0],\ \textbf{H}_{i}[1],\ \ldots,\ \textbf{H}_{i}[t_{i}-1]]\)). Each truncated sequence \(\overset{\wedge}{\textbf{H}}_{i}\) is then used to compute the embedding by temporal pooling. With the baseline methods, the embedding  
Figure 3: The procedure of the PackedSequence object in PyTorch. is computed using \(\mathbf{H}_{i}\) rather than \(\overset{\wedge}{\mathbf{H}_{i}}\). With the proposed method, therefore, we can exclude the effects of the padded frames in the output sequence from the embedding. Regardless of the kinds of pre-processing method (i.e., the baseline and the proposed), we used two kinds of temporal pooling (i.e., temporal_pooling\((\cdot)\) in Algorithm 1) for computing embeddings. One is simply running average of the output sequence \(\mathbf{H}_{i}\) along time axis. In this case, the embedding \(\mathbf{o}_{i}\) becomes either \(\frac{1}{t}\sum_{j=0}^{T-1}\mathbf{H}_{i}[j]\) (for the baseline) or \(\frac{1}{t_{i}}\sum_{j=0}^{t_{i}-1}\mathbf{H}_{i}[j]\) (for the proposed). The other is running weighted sum of the output sequence \(\mathbf{H}_{i}\) along time axis. In this case, the embedding \(\mathbf{o}_{i}\) becomes either \(\sum_{j=0}^{T-1}a_{j}\mathbf{H}_{i}[j]\) (for the baseline) or \(\sum_{j=0}^{t_{i}-1}a_{j}\mathbf{H}_{i}[j]\) (for the proposed), where \(a_{j}\) is the attention weight for the \(j\)-th output frame \(\mathbf{H}_{i}[j]\) obtained by a self-attention mechanism [11; 12]. In the proposed method, computing embeddings from output sequences cannot be performed in mini-batch units (i.e., cannot be computed in parallel), because all the original sequences generally have variable lengths. Also, there is an additional overhead for the output truncation. Therefore, the proposed method is slower than the baseline method. Nonetheless, we expect that the execution time between the baseline and proposed method do not differ significantly. This is due to the fact that most of the time is occupied by the RNN operation itself, which can be processed in mini-batch units (i.e., can be computed in parallel) for both the baseline and proposed method. Figure 4: The overview of the (**top**) conventional and (**bottom**) proposed methods. The yellow frames represent the output from the original input frame. The gray frames represent the output from the padded frame. The green elements represent an embedding. ```
1:Input
2:\(\mathbf{H}=[\mathbf{H}_{1},\dots,\mathbf{H}_{n}]\): list of the output sequences. Each sequence has the shape of \(T\times D\)
3:\(\mathbf{sl}=[t_{1},\dots,t_{n}]\): list of the original lengths of each sequence (\(t_{i}<T\) for all \(i\)). 4:\(n\): mini-batch size. 5:\(T\): target length. 6:\(D\): the dimensionality.