# BIMO: Bootstrap Inter-Intra Modality at Once Unsupervised Learning for Multivariate Time Series
## 2 Materials and Methods  
BIMO's goal is to be easily used in downstream tasks by discovering the most significant modalities for representation learning in all domains of time-series data. This study was inspired by existing work on SOTA contrastive learning-based unsupervised learning methods [15; 23; 27].  
As shown in Figure 1, the proposed BIMO method consists of the main network and two auxiliary networks. The main network consists of an encoder \(f_{\theta}\), a projector \(g_{\theta}\), and a predictor \(q_{\theta}\), and each auxiliary network comprises an encoder and a projector. The main network learns to have a similar distribution between two values \(p^{\prime}_{g}\), \(p^{\prime\prime}_{\lambda}\) from the respective projectors of the two auxiliary networks and a value \(q_{\theta}(p_{\theta})\) from the predictor of the main network.  
It is a significant issue to simultaneously learn both inter- and intra-modality representations. We trained the proposed BIMO method to learn inter-intra modality representation efficiently and stably based on the fundamental concept, i.e., high-level features comprise low-level and intermediate-level features [28].  
An overview of the training process in the proposed BIMO method is given in Algorithm 1. The complexity of the proposed BIMO method is \(O(4N)\), while the complexity of USRL as a existing SOTA method is at least \(O(18N)\).  
While training, we first used a hard constraint in the inter-auxiliary network to learn sufficient low-level coarse information, i.e., the time characteristics within samples, from the intra-auxiliary network. As the number of epochs increased, we gradually applied a hard constraint to the intra-auxiliary network and not to the inter-auxiliary network. Therefore, the proposed BIMO method sufficiently learns fine-grained features, i.e., the correlation between two augmented samples, from the inter-auxiliary network.  
Therefore, BIMO learns low-level features sufficiently at the initial training step, and gradually learns high-level features.  
```
1:Time series set \(X=\{x_{n}\}_{n=1}^{N}\), Number epochs \(M\)
2:Trained \(f_{\theta}\)
3:Initialization \(f_{\theta},g_{\theta},q_{\theta},f_{\lambda},g_{\lambda},f_{\xi},g_{\xi} \leftarrow\) initialize weights
4:\(m\gets 1\)
5:repeat
6:for\(n=1\) to \(N\) with \(s_{n}=size(x_{n})\)do
7: generate \(v\triangleq t(x)\), \(v^{\prime}\triangleq t^{\prime}(x)\) from different augmentation \(t\sim\tau\), \(t^{\prime}\sim\tau\)
8: extract \(s_{v_{\text{sub}}}=size(v)\) in \([1,s_{n}]\)
9: extract \(v_{\text{sub}}\) among subseries of \(v\) of length \(s_{v_{\text{sub}}}\)
10:\(r_{\theta}\gets f_{\theta}(v)\), \(p_{\theta}\gets g_{\theta}(r_{\theta})\), \(q_{\theta}(p_{\theta})\)
11:\(r^{\prime}_{\xi}\gets f_{\xi}(v^{\prime})\), \(p^{\prime}_{\xi}\gets g_{\xi}(r^{\prime}_{\xi})\)
12:\(r^{\prime\prime}_{\lambda}\gets f_{\lambda}(v_{\text{sub}})\), \(p^{\prime\prime}_{\lambda}\gets g_{\lambda}(r^{\prime\prime}_{\lambda})\)
13:\(\mathcal{L}_{inter}\leftarrow\|q_{\theta}(p_{\theta})-\tilde{p}^{\prime}_{ \lambda}\|_{2}^{2}\)
14:\(\mathcal{L}_{intra}\leftarrow\|q_{\theta}(p_{\theta})-\tilde{p}^{\prime\prime}_ {\lambda}\|_{2}^{2}\)
15:\(\mathcal{L}_{BIMO}\leftarrow(1-\frac{1}{m})(\mathcal{L}_{inter})+\frac{1}{m}( \mathcal{L}_{intra})\)
16:\(f_{\theta},g_{\lambda},q_{\theta}\leftarrow\) update weights using \(\mathcal{L}_{BIMO}\)
17:\(f_{\lambda},g_{\lambda},f_{\xi},g_{\xi}\leftarrow\) update weights using moving exponential average
18:endfor
19:\(m\gets m+1\)
20:until m = M
```  
**Algorithm 1** BIMO's training procedure