# A Brief Survey on Deep Learning-Based Temporal Knowledge Graph Completion
## 3 Deep Learning-Based TKGC Methods
### CapsNet-Based Methods  
Capsule network (CapsNet) (2018) can be seen as an improved version of CNN, which can encode the intrinsic spatial relationships between a part and the whole in images. It consists of two key components: capsules and a dynamic routing mechanism. A capsule is a group of neurons that processes different information about an object, and the length of its output vector represents the probability that the object represented by the capsule is present in the current input. To guarantee the length of the vector stays between 0 and 1, a squash function is used:  
\[\mathbf{v}_{j}=\frac{||\mathbf{s}_{j}||^{2}}{1+||\mathbf{s}_{j}||^{2}}\frac{ \mathbf{s}_{j}}{||\mathbf{s}_{j}||}, \tag{3}\]  
where \(\mathbf{s}_{j}\) is the input of a capsule, and \(\mathbf{v}_{j}\) is its squashed output.  
The dynamic routing mechanism specifies how capsules communicate with each other between layers. Specifically, all output vectors \(\mathbf{u}_{i}\) of capsules in the lower layer are first multiplied by a weight matrix \(\mathbf{W}_{ij}\). Then, the weighted sum of newly obtained vectors are input into a capsule \(\mathbf{s}_{j}\) in the next layer:  
\[\mathbf{\hat{u}}_{j|i}=\mathbf{W}_{ij}\mathbf{u}_{i},\quad\mathbf{s}_{j}=\sum _{i}c_{ij}\mathbf{\hat{u}}_{j|i}, \tag{4}\]  
where \(c_{ij}\) is the coupling coefficient between capsule \(i\) and capsule \(j\).  
TempCaps (2019) extends the static KGC method CapsE (2018) to TKGC, which consists of a neighbor selector, a dynamic routing aggregator, and a multi-layer perceptron (MLP) decoder. Specifically, it first selects two types of neighboring entities for each entity, namely local entities and global relation entities, through the neighbor selector. Then, it uses the dynamic routing aggregator to learn the embedding of the entity based on the retrieved neighbors. Finally, the score for each candidate entity is generated by feeding the embeddings of the entities in the candidate set into the MLP decoder.  
BiQCap (2019) combines biquaternions and capsule networks for TKGC. Specifically, it first embeds entities, relations, and timestamps into the biquaternions of k-dimension, i.e., \(\mathbf{Q}_{s}\), \(\mathbf{Q}_{r}\), \(\mathbf{Q}_{o}\), \(\mathbf{Q}_{t}\in\mathbb{R}^{k}\), where timestamps are represented as translations of the subject entities, and relations are represented as rotations in the biquaternion space after translation. On this basis, a distance-based score function is designed as \(f(s,r,o,t)=||((\mathbf{Q}_{s}+\mathbf{Q}_{t})\odot\mathbf{Q}_{r})\otimes \mathbf{b}-\mathbf{Q}_{o}|\), where \(\mathbf{b}\in\mathbb{R}^{k}\) is a bias vector, and \(\odot\) and \(\otimes\) denote the Hamilton product and element-wise multiplication, respectively. Then, the trained embeddings of entities, relations, and timestamps from biquaternions are used as the input to the capsule network, and a novel routing algorithm called inverted dot-product attention is proposed, which determines the routing probability directly through the agreement between parent and child capsules.  
Methods in this category have similar characteristics to the CNN-based methods. Since CapsNet is an improved version of CNN, it inherits the characteristics of CNN and can model local and global spatial relationships. Therefore, methods in this category can theoretically better characterize the spatial characteristics of entities in TKGs.