# A Novel Reconstruction Method for Irregularly Sampled Observation Sequences for Digital Twin
## 3 Irregularly Sampled Observation Sequence Reconstruction Method Based on PRN-NCDE
### Multi-Channel Self-Attention Module  
The multi-channel self-attention (MCSA) module comprises multiple masked multi-head self-attention modules designed to handle the misalignment and irregularities inherent in multivariate observation sequences. Initially, the observation sequence of each dimension within the coal-fired power plant is expanded to a fixed-length sequence at a desired sampling interval. Consequently, for the irregular observation vector \(\mathbf{y}_{n}^{irr}\), certain dimensional observations will be missing. These missing observations can be filled using a constant value \(\tau\), represented as follows:  
\[\mathbf{y}_{n}^{c}=\mathbf{y}_{n}^{irr}\odot\mathbf{m}+\tau\cdot\bar{\mathbf{m}} \tag{3}\]  
where \(\mathbf{y}_{n}^{c}\in\mathbb{R}^{d}\) is the observation vector after filling. \(\mathbf{m}\in\left\{0,1\right\}^{d}\) is an indicator function taking values of either 0 or 1, with 1 denoting the presence of an observation and 0 denoting its absence. \(\bar{\mathbf{m}}\) is the vector formed by inverting elements of \(\mathbf{m}\), and \(\odot\) represents element-wise multiplication. Therefore, the irregular observation sequence can be expanded into a regular sequence with the desired length \(N^{\prime}+1\), represented as follows:  
\[\mathbf{Y}=(\mathbf{y}_{0}^{c},\cdots,\mathbf{y}_{n}^{c},\cdots,\mathbf{y}_{N^{\prime}}^{c}) \tag{4}\]  
where \(\mathbf{Y}\in\mathbb{R}^{d\times(N^{\prime}+1)}\) represents the expanded multivariate sequence containing the filled values \(\tau\). Additionally, due to the non-alignment of multivariate observation sequence in the boiler system, the filled positions of missing values are not entirely consistent. Thus, the number of MCSA self-attention channels should correspond to the dimensionality of observational variable \(d\), and the sequence \(\mathbf{Y}\) must be divided according to variable dimensions (i.e., the rows of \(\mathbf{\gamma}\)), and input into their corresponding self-attention module channels. Taking observational variables in the boiler system as an example, division by variable dimension can be expressed as  
\[\left(\rho_{b},\mathbf{T}_{ss^{\prime}},\cdots,\mathbf{O}_{cp}\right)^{\text{T}}= \text{Block}_{\text{Row}}(\mathbf{Y}=(\mathbf{y}_{0}^{c},\cdots,\mathbf{y}_{n^{\prime}}^{ c},\cdots,\mathbf{y}_{N^{\prime}}^{c})) \tag{5}\]where \(\mathbf{\rho}_{b},\mathbf{T}_{gs},\mathbf{O}_{cp}\in\mathbb{R}^{1\times(N^{\prime}+1)}\) represent the time series with length \(N^{\prime}+1\) of gas density, gas temperature, and gas oxygen content. \(\text{Block}_{\text{Row}}\) represents row-wise partitioning of the matrix. Since the filled positions for each variable are inconsistent, the divided sequences are separately input into corresponding multi-head self-attention modules, represented as follows:  
\[\left(\mathbf{s}_{1}^{{}^{\prime}},\mathbf{s}_{2}^{{}^{\prime}},\cdots,\mathbf{s}_{d}^{{}^{ \prime}}\right)^{\text{T}}=\text{MCSA}\big{(}MH_{1}(\mathbf{\rho}_{b}),MH_{2}\big{(} \mathbf{T}_{gs}\big{)},\cdots,MH_{d}\big{(}\mathbf{O}_{cp}\big{)}\big{)}^{\text{T}} \tag{6}\]  
with  
\[ \begin{array}{l}MH_{i}(\mathbf{s}_{i})=\mathbf{W}_{o}^{i}\big{(}satt\big{(}\mathbf{Q}_{i,1},\mathbf{K}_{i,1},\mathbf{V}_{i,1}\big{)},\cdots,satt\big{(}\mathbf{Q}_{i,H},\mathbf{K}_{i, H},\mathbf{V}_{i,H}\big{)}\big{)}\\ \forall h\in\{1,\cdots,H\},\ \mathbf{Q}_{i,h}=\mathbf{W}_{q}^{i,h}\mathbf{s}_{i},\ \mathbf{K}_{i,h}=\mathbf{W}_{k}^{i,h}\mathbf{s}_{i},\ \mathbf{V}_{i,h}=\mathbf{W}_{v}^{i,h}\mathbf{s}_{i}\end{array} \]  
where \(\mathbf{s}_{i}\) represents the time series of the \(i\)-th variable, \(\mathbf{s}_{i}^{\prime}\) is the output row vector of MCSA, \(MH_{i}\) represents the multi-head attention operation of the \(i\)-th channel, \(\mathbf{W}_{o}^{i}\in\mathbb{R}^{1\times H\cdot\triangleright}\) is the output projection matrix of the \(i\)-th channels and \(satt\big{(}\mathbf{Q}_{i,h},\mathbf{K}_{i,h},\mathbf{V}_{i,h}\big{)}\) is the self-attention operation of the \(h\)-th head of the \(i\)-th channel.