# Temporal Multimodal Data-Processing Algorithms Based on Algebraic System of Aggregates
## 6 Multi-Image Notion and Formation Algorithm  
The multi-image concept [21] is an essential part of the approach presented in this paper as it enables the required formal description of the sequences of multimodal data about the object under study. **Definition 6**. : _A multi-image is a non-empty aggregate specified as:_  
\[I=\llbracket T,M_{1},\dots,M_{N}\mid\left\langle t_{1},\dots,t_{\tau}\right\rangle,\left\langle a_{1}^{1},\dots,a_{n_{1}}^{1}\right\rangle,\dots,\left\langle a _{1}^{N},\dots,a_{n_{N}}^{N}\right\rangle\rrbracket \tag{13}\]  
_where \(T\) is a set of time values; \(\tau\geq n_{j},j\in[1,\dots,N]\)._  
From a practical point of view, multi-image refers to a structure of consolidated temporal multimodal data sets that describe different aspects of the same object. Let us formulate an algorithm that enables the formation of multi-image for an arbitrary object described through several temporal multimodal data sets. The inputs to the algorithm of the object's multi-image formation are temporal data sets and the data type (modality) of each set. The algorithm consists of seven steps. The result of the algorithm is multi-image, presented in the form of an ordered collection of temporal multimodal data. The first step of the algorithm is the formalization of the object's multi-image data structure. This step is performed according to the requirements for the object's description. Figure 20: An example of applying the singling operation. Multi-image can be defined mathematically, according to (13), or in any other way that allows a researcher to unambiguously specify the sequence of data sets and their modality. At the second step of the algorithm, multi-image specification is decomposed into a set of specifications of partial multi-images as seen in (14). \[I_{j}=\llbracket T,M_{j}\mid\Big{\langle}t_{1},\ldots,t_{\tau_{j}}\Big{\rangle}, \Big{\langle}a_{1},\ldots,a_{n_{j}}\Big{\rangle}\rrbracket \tag{14}\]  
where \(T\) is the set of time values; \(M_{j}\) is the set of data values of \(j\)-modality; \(\tau_{j},n_{j}\in\mathbb{N};\tau_{j}\geq n_{j}\); \(j\in[1,\ldots,N]\); and \(N\) is the number of data sets. At the third step, data are obtained. The procedure for receiving data involves determining the method (protocol, format) of data transmission, determining the time period for data transmission, establishing a connection with the data source and receiving data in a specified way and in a specified format. At the fourth step, the data of each modality are prepared to form the corresponding partial multi-image. A partial multi-image is an aggregate that includes two tuples of elements, namely a tuple of time values and a tuple of elements of a certain modality. Elements are individual values or ordered sets of values (homogeneous or heterogeneous). Data preparation, which is performed at this step, consists of determining (detecting) time values in the set of data obtained at the previous stage. This procedure can be either trivial when the data format requires explicit presentation of time values for the temporal data component, or complex when the time values are presented in an implicit form or are ambiguous. A method for detecting hidden or ill-defined time values must be developed for each specific data format. At the fifth step, partial multi-images are combined into a single multi-image. The multi-image merging procedure includes two actions: normalization and uniting. 1. Normalization of partial multi-images (a normalized multi-image (normalized tuple) is a multi-image (tuple) to the elements of which dummy elements are added; a dummy element is a value that is absent in the tuple before its normalization; an empty element \(\mathcal{O}\) can be used as a dummy element; at other steps of the multi-image processing, dummy elements are ignored): \[