## 3 Materials and Methods
### Cross-Modal Learning Module  
The cross-modal learning module is responsible for processing the input data from multiple modalities and learning shared representations. Each modality is processed by a dedicated subnetwork tailored to the specific data type. Transformers have proven to be highly effective in modeling long-range dependencies and capturing contextual information. In the cross-modal learning module, transformers are used to process data patterns. An illustration of the transformer model's core components where layers were normalized after multiheaded attention is depicted in Figure 2[47]. Transformers excel at learning representations from sequential data and can capture the temporal relationships within cybersecurity modalities like log files, network traffic, and behavior patterns. The foundational constituents of a transformer architecture have been extensively delineated in prior literature [12; 47; 48]. Firstly, the architecture inherently encompasses an Encoder-Decoder Structure, manifesting as two distinctive modules: an encoder tasked with assimilating the input sequence, and a decoder orchestrating the generation of the corresponding output sequence. Secondly, a pivotal mechanism operative within this framework is the Self-Attention Mechanism. This mechanism engenders the capacity for individual elements within the input sequence to selectively attend to other constituent elements within the same sequence. In effect, attention weights are computed, thereby endowing the model with the faculty to emphasize pertinent informational elements during the input processing phase. In tandem with this, the paradigm incorporates the Multi-Head Attention mechanism, which entails the integration of multiple attention layers, colloquially referred to as \"heads\". This arrangement facilitates the discernment of disparate forms of interrelationships existing amongst the elements comprising the input sequence. Concatenation or amalgamation of the outputs stemming from these distinct heads affords a more exhaustive and holistic representation. Subsequently, following the application of the self-attention mechanism, the architecture integrates Feed-Forward Neural Networks. These neural networks serve to further process the representations that have been subjected to the self-attention mechanism, augmenting the model's ability to capture intricate patterns within the data. Furthermore, an intrinsic challenge pertaining to the transformer architecture pertains to its inability to inherently fathom sequential information. To circumvent this, the framework incorporates Positional Encoding. By integrating positional encoding into the input embeddings, the model gains access to crucial positional information. This augmentation equips the transformer with the proficiency to effectively manage and interpret sequential data. The first sublayer obtains the decoder stack's previous output, augments it with positional information, then applies multi-head self-attention to it. While the encoder is  
Figure 2: Transformer model’s core components. meant to attend to all words in the input sequence regardless of their position, the decoder is adjusted to only attend to the words that come before them. As a result, the prediction for a word at position i can only be based on the known outputs for the words preceding it in the sequence. This is accomplished in the multi-head attention mechanism (which implements numerous, single attention functions simultaneously) by applying a mask to the values obtained by the scaled multiplication of matrices Q and K.  
Masking is accomplished by suppressing matrix values that would otherwise correspond to illegal connections [49]:  
\[\text{mask}\Big{(}\text{QK}^{\text{T}}\Big{)}=\text{mask}\left( \begin{bmatrix} \text{e}_{11}&\text{e}_{12}&\dots&\text{e}_{1n}\\ \text{e}_{21}&\text{e}_{22}&\dots&\text{e}_{2n}\\ \vdots&\vdots&\ddots&\vdots\\ \text{e}_{m1}&\text{e}_{m2}&\dots&\text{e}_{mn}\end{bmatrix} \right)=