# A New View of Multisensor Data Fusion: Research on Generalized Fusion

Authors: Guo Chen; Zhigui Liu; Guang Yu; Jianhong Liang
Journal: Mathematical Problems in Engineering
 (2021)

---

## Abstract

Multisensor data generalized fusion algorithm is a kind of symbolic computing model with multiple application objects based on sensor generalized integration. It is the theoretical basis of numerical fusion. This paper aims to comprehensively review the generalized fusion algorithms of multisensor data. Firstly, the development and definition of multisensor data fusion are analyzed and the definition of multisensor data generalized fusion is given. Secondly, the classification of multisensor data fusion is discussed, and the generalized integration structure of multisensor and its data acquisition and representation are given, abandoning the research characteristics of object oriented. Then, the principle and architecture of multisensor data fusion are analyzed, and a generalized multisensor data fusion model is presented based on the JDL model. Finally, according to the multisensor data generalized fusion architecture, some related theories and methods are reviewed, and the tensor-based multisensor heterogeneous data generalized fusion algorithm is proposed, and the future work is prospected.


---

## Body

## 1. Introduction

Multisensor data fusion, also known as multisource data fusion [1], is essentially the fusion of heterogeneous data, and the fusion is accurate and inaccurate (with uncertainty) data, similar to natural creatures acquiring information through various senses and comparing, discriminating, and comprehensively analyzing the acquired information with memory or experience to understand the objective world. It played a huge role in several recent local wars; the C3I fusion system [2] of multinational forces attracted the attention of the whole world. As a result, multisensor data fusion technology has become a very active research field in academia.The goal of multisensor data fusion is to process and synthesize multisource data (or information) related to the measured object to obtain a more accurate, more complete, and more reliable, consistent interpretation and description of the measured object than using a single sensor. Most of the current related work is carried out for specific application fields, such as monitoring and detection [3, 4] and environmental perception [5]; according to actual application problems, each establish intuitive fusion criteria and form the so-called optimal fusion scheme on this basis, as a whole, it is characterized by object orientation, which fails to form the basic theoretical framework and generalized algorithm system necessary for this independent discipline.The lack of basic theoretical framework and generalized algorithm system not only hinders scholars’ deep understanding of multisensor data fusion but also prevents scholars from synthesizing and evaluating the fusion system. Generalized fusion algorithm system, as a basic paradigm with universal applicability, must carry out universal research in sensor integration, data collection and representation, fusion framework, data fusion, etc.


### 1.1. Sensor Integration

In a multisensor integrated system platform, the integration method, attributes, and quantity of sensors are the three most basic factors. Any complex system is composed of generalized integrated basic subsystems.


### 1.2. Data Collection and Presentation

Data are generated by the measurement of observation objects by different sensors; different sensors have great differences in the mechanism of data generation. How to collect relatively accurate and complete data and correctly represent is very challenging.


### 1.3. Data Fusion Framework Design

Due to the heterogeneity and heterogeneity of sensors, to design a generalized data fusion model, it is necessary to abandon its application objects and take the sensor integration method and data fusion principle as the theoretical basis.Multisensor data generalized fusion algorithm is a kind of symbolic computing model with multiple application objects, which is based on the sensor generalized integration structure. It is the theoretical basis of numerical fusion, and a basic algorithm to provide modularization support for complex system data fusion.At present, abundant data fusion research results provide important reference for the research of multisensor data generalized fusion algorithm; however, there is a lack of comprehensive review of the research on the generalized fusion algorithm of multisensor data. In order to fill this gap, the basic research on multisensor generalized integration, data acquisition and presentation, multisensor data generalized fusion model, and generalized fusion method is carried out in this paper:(1)
The definition of generalized fusion of multisensor data is proposed, which defines the research methods and objectives of this research direction(2)
The generalized integrated model of multisensor is proposed, which is the basis of designing multisensor system, thus avoiding the blindness of the multisensor system design(3)
The architecture model of multisensor data generalized fusion is proposed; the idea of studying generalized fusion algorithm of multisensor data is determined(4)
The tensor-based generalized fusion algorithm for heterogeneous data of multiple sensors is proposed, which proves the reachability of this research direction and provides a theoretical basis for subsequent research studiesThe organization of this paper is as follows. Section2 discusses the development and concept of multisensor data fusion, abandons the characteristics of object-oriented research in the past, and proposes the definition of generalized fusion of multisensor data. Section 3 introduces the classification of multisensor data fusion. In Section 4, the model of multisensor generalized integration platform and data acquisition and representation are studied in detail. In Section 5, the principle and architecture of multisensor data fusion are studied, and the generalized architecture model of multisensor data fusion is proposed and analyzed in detail. In Section 6, four-level fusion based on the generalized fusion model of multisensor data and the theories and algorithms related to it are reviewed, and the generalized fusion algorithm of multisensor heterogeneous data based on tensor is given. In Section 7, the future development trend and difficulties of the research on the generalized fusion algorithm system of multisensor data are summarized.


## 1.1. Sensor Integration

In a multisensor integrated system platform, the integration method, attributes, and quantity of sensors are the three most basic factors. Any complex system is composed of generalized integrated basic subsystems.


## 1.2. Data Collection and Presentation

Data are generated by the measurement of observation objects by different sensors; different sensors have great differences in the mechanism of data generation. How to collect relatively accurate and complete data and correctly represent is very challenging.


## 1.3. Data Fusion Framework Design

Due to the heterogeneity and heterogeneity of sensors, to design a generalized data fusion model, it is necessary to abandon its application objects and take the sensor integration method and data fusion principle as the theoretical basis.Multisensor data generalized fusion algorithm is a kind of symbolic computing model with multiple application objects, which is based on the sensor generalized integration structure. It is the theoretical basis of numerical fusion, and a basic algorithm to provide modularization support for complex system data fusion.At present, abundant data fusion research results provide important reference for the research of multisensor data generalized fusion algorithm; however, there is a lack of comprehensive review of the research on the generalized fusion algorithm of multisensor data. In order to fill this gap, the basic research on multisensor generalized integration, data acquisition and presentation, multisensor data generalized fusion model, and generalized fusion method is carried out in this paper:(1)
The definition of generalized fusion of multisensor data is proposed, which defines the research methods and objectives of this research direction(2)
The generalized integrated model of multisensor is proposed, which is the basis of designing multisensor system, thus avoiding the blindness of the multisensor system design(3)
The architecture model of multisensor data generalized fusion is proposed; the idea of studying generalized fusion algorithm of multisensor data is determined(4)
The tensor-based generalized fusion algorithm for heterogeneous data of multiple sensors is proposed, which proves the reachability of this research direction and provides a theoretical basis for subsequent research studiesThe organization of this paper is as follows. Section2 discusses the development and concept of multisensor data fusion, abandons the characteristics of object-oriented research in the past, and proposes the definition of generalized fusion of multisensor data. Section 3 introduces the classification of multisensor data fusion. In Section 4, the model of multisensor generalized integration platform and data acquisition and representation are studied in detail. In Section 5, the principle and architecture of multisensor data fusion are studied, and the generalized architecture model of multisensor data fusion is proposed and analyzed in detail. In Section 6, four-level fusion based on the generalized fusion model of multisensor data and the theories and algorithms related to it are reviewed, and the generalized fusion algorithm of multisensor heterogeneous data based on tensor is given. In Section 7, the future development trend and difficulties of the research on the generalized fusion algorithm system of multisensor data are summarized.


## 2. Development and Definition of Multisensor Data Fusion

Multisensor data fusion is an interdisciplinary research field, which involves a wide range of content. With the expanding of application fields, its function and definition connotation have constantly been enriched. People’s research on multisensor data fusion has inherent object-oriented characteristics: researchers in different fields put forward different views based on the expected functions and use functional definitions suitable for specific fields to describe or explain the functions (or purposes) and limitations of multisensor data fusion [6–9]. Therefore, the development process of the definition of multisensor data fusion represents the development process of multisensor data fusion.In the past studies, the functional definition describing multisensor data fusion (function and limitation) is detailed in [10]. Among them, Klein and White [11, 12], Durrant-Whyte [13], and Mastrogiovanni et al. [14] gave representative functional definitions. In short, scholars in various application fields are describing the functions and limitations of multisensor data fusion in different applications. The function description can be summarized into eight aspects; the limited description can be roughly summarized into three aspects, and the statistics are shown in Table 1.Table 1
Multisensor data fusion functional definition description analysis.
Function descriptionLimited description① Obtain (more comprehensive/complete and higher quality) [15, 16] information that is greater than the sum of each contribution part① Source limitation: that is, limiting the data source, such as data or information from sensors② Accurately understand and describe the given scene [17]③ Realize inferences that cannot be achieved with a single sensor [18]② Scenario limitation: that is, the application type or decision-making situation is limited, such as decision-making with strict timing requirements④ Infer events related to the observed object [15, 19]⑤ Improve state estimation, prediction, and risk assessment [13, 16, 20, 21]⑥ Realize precise positioning, tracking, and identification [20, 22–24]⑦ Realize accurate (accuracy, robustness, qualitative, and quantitative) decision-making and action [23, 25]③ Characteristic limitation: that is, to limit the fusion characteristics, such as continuous refinement⑧ Maximize useful information, improve reliability or recognition ability, and minimize the amount of retained data [14, 26–28]In recent studies, scholars believe that multisensor data refers to multisource data [29], including direct data (sensor historical data value) and indirect data (prior knowledge of environment and human input). The data source is not specified to be the same sensor, including heterogeneous sensor, database, and human. The content involved is extensive, covering all possible combination or aggregation methods, and transforms information from different sources and at different times.Scholars also believe that multisensor data fusion is an interdisciplinary research field, using technologies from different fields [30], such as artificial intelligence and information theory. Henrik Boström thinks traditional multisensor data fusion focuses on online sensor data, while modern data fusion should consider other sources. Boström et al. [10], Bloch et al. [31], and Wang et al. [29] gave representative definitions.Based on the current development status of multisensor data fusion and abandoning its application objects, the multisensor data generalized fusion can be defined as follows (Figure1). Using the intelligent computing method, the multimodal time-series data collected by orderly integrated multiple (or class) sensors are sorted, analyzed, and comprehensively processed according to certain criteria, and more accurate and comprehensive inferences than using some information sources alone are gained, and the consistency of the interpretation and description of the measured object of artificial intelligence technology are ultimately obtained.Figure 1
Generalized fusion of multisensor data.


## 3. Classification of Multisensor Data Fusion

Multisensor data refer to the effective use of multisensor resources to obtain the most accurate and inaccurate (uncertain and unknown) data about the detected target and the environment. The uncertainty of multisensor data determines the complexity of data fusion and the diversity of classification, see Table2.(1)
According to the multisensor data fusion method, it can be roughly divided into compression fusion, statistical fusion, feature fusion, knowledge fusion so on.①
Compression and fusion: the data compression process is realized by using a specific compression model, and the input data are usually converted into representations similar to bases and coefficients. The compressed and transformed data remove redundant information, and the scale is effectively reduced. It can be inversely transformed according to the required accuracy and reconstructed to restore the approximate original data result. Compression fusion is an important means to realize data visualization, and data visualization is the main method of data research and analysis [32].②
Statistical fusion: in a multisensor system, the reliability of each sensor directly affects the fusion result. Statistical fusion is the use of statistical method theory to blur the reliability of each sensor, calculate the comprehensive reliability, and then perform data fusion. The method is simple to calculate and its conclusions are relatively stable [33].③
Feature fusion: that is, the feature of the fusion data and the premise of feature fusion are feature extraction. Features are also called target characteristics, which refer to the various characteristics of the target carried in the data obtained by different sensors reflecting the same target. Feature extraction refers to the process of performing various mathematical transformations on data to obtain the indirect target characteristics contained in the data [34].④
Knowledge fusion: knowledge fusion is the process of forming new knowledge by interacting and supporting knowledge from different sources of knowledge. Knowledge fusion can not only fuse data and information but also fuse methods, experience, and even human thoughts [35].(2)
According to the attributes of fusion data, multisensor data fusion can be divided into homogeneous data fusion and heterogeneous data fusion.①
Homogeneous data fusion: it is the consistent representation (interpretation and description) of the fusion process of homogeneous data collected by multiple identical sensors, also known as multisensor homogeneous data fusion.②
Heterogeneous data fusion: that is, the process of consistent representation (interpretation and description) of heterogeneous data collected by multiple different sensors, also known as multisensor heterogeneous data fusion.(3)
According to the abstract level of fusion data, it is divided into signal-level data fusion, feature-level data fusion, and decision-level data fusion.There are essential differences between multisensor data fusion and classical signal processing (single-sensor signal). Multisensor data have complex forms and different abstract levels (signal level, feature level, and decision level).①
Signal-level data fusion: refers to the fusion on the original data layer, that is, the original measurement and report data of various sensors are directly integrated and analyzed without preprocessing. The advantage is that it can maintain as much field data as possible, which is richer, complete, and reliable than other fusion levels. The disadvantages are accurate registration must be performed before pixel-level fusion, the amount of processed data is too large, the processing time is long, and the real-time performance is poor. Pixel-level data fusion is the lowest level of fusion, but it is possible to provide optimal decision-making or optimal recognition. It is often used for multisource image composition, image analysis, and understanding.②
Feature-level data fusion: firstly, extract features from the original data from each sensor (features: direction, speed, edge of the target, etc.), and then perform integrated analysis and processing of the feature information, which belongs to the middle-level fusion. Feature-level data fusion features: it achieves good information compression and is conducive to real-time processing; the extracted features are related to decision analysis, so the fusion result can provide feature information for decision analysis to the maximum.Feature-level data fusion is divided into target state data fusion and target characteristic fusion. Target state data fusion: the main realization is parameter correlation and state vector estimation, which is mainly used in the field of multisensor target tracking. Target feature fusion: that is, using the corresponding technology of pattern recognition, the joint recognition at the feature layer requires that the features be related before fusion, and the feature vectors are classified into meaningful combinations.③
Decision-level data fusion: it is a high-level fusion, and the result of the fusion is the basis for command and control decision-making. In this level of fusion process, each sensor should first establish a preliminary judgment and conclusion on the same target, then perform correlation processing on the decision from each sensor, and finally perform decision-level fusion processing to obtain the final joint judgment. Decision-level fusion has good real-time performance and fault tolerance, but its preprocessing cost is high. At present, network-based signal or information processing often adopts this level of data fusion [36, 37].(4)
According to the time vector and space vector of the fusion data, it can be divided into time fusion, space fusion, and space-time fusion.①
Time fusion refers to the fusion processing of the time domain data of a certain sensor in the system②
Spatial fusion refers to the fusion processing of the measurement values of the related targets at the same sampling time for each sensor in the system③
Spatiotemporal fusion refers to the fusion processing of the measurement values of the relevant targets of the sensors in the system over a period of timeTable 2
Multisensor data fusion classification.
Data fusion methodAbstract level of fusion dataSpatiotemporal vector of fusion dataLimited descriptionCompression and fusionPixel-level data fusionTime fusionHomogeneous data fusionStatistical fusionFeature-level data fusionSpatial fusionFeature fusionDecision-level data fusionTemporal-spatial fusionHeterogeneous data fusionKnowledge fusion


## 4. Multisensor Generalized Integration and Data Acquisition and Output

With the continuous development of intelligent industry, single sensor has been unable to meet the needs of society, and different multisensor integrated systems are needed to match with it, which has shown an increasingly urgent situation. Multisensor integration system is generally a nonlinear system. Its sensor attributes, integration mode, data acquisition, and output directly affect the way and quality of multisensor data fusion.


### 4.1. Multisensor Generalized Integrated System (MGIS)

#### 4.1.1. Multisensor Generalized Integration

According to the attributes and quantity of sensors in the multisensor system, abandoning specific application requirements, Figure2 shows the generalized integration mode of multiple sensors.Figure 2
Multisensor generalized integration.In the multisensor generalized integration method, homogeneous multisensor integration and heterogeneous single-sensor integration are the most basic integration methods, and heterogeneous multisensor integration is an organic combination of different subsystems.


#### 4.1.2. Multisensor Generalized Integrated Structure Model

According to the multisensor generalized integration method, the multisensor generalized integration structure can be divided into①
Homogeneous multisensor integration structure (HMI) refers to a system integrated by multiple sensors with the same attributes, and the data collected by all the sensors represent the same attribute of the monitoring target; its structural model is as follows:(1)ISHMI=S1A1,S2A1,S3A1,…,SnA1,Where ISHMI represents homogeneous multisensor integrated system, S represents the sensor, A1 represents the sensor attribute, and 1, 2, 3, …, n represents the number of sensors.②
Heterogeneous single-sensor integration structure (HSI) refers to a system integrated by a single sensor with different attributes, and the data collected by each sensor represent different attributes of the monitoring target; its structural model is as follows:(2)ISHSI=S1A1,S1A2,S1A3,…,S1AN,where ISHSI represents the heterogeneous single-sensor integrated system, S represents the sensor, A1, A2, A3, …, AN represents sensor attributes, and 1 represents the number of sensors.③
Heterogeneous multisensor integration structure refers to a complex cluster system composed of multiple (or multiple) subsystems. The clustering methods include multiattribute homogeneous multisensor integrated subsystem cluster, single-attribute heterogeneous single-sensor integrated subsystem cluster, and hybrid system cluster.(i)
Multiattribute (homogeneous multisensor integrated) subsystem cluster (M.HMI.SC) structural model is as follows:(3)CSM.HMI.SC=s1A1,s2A1,s3A1,…,snA1ISA1,s1A2,s2A2,s3A2,…,snA2ISA2,s1A3,s2A3,s3A3,…,snA3ISA3,⋮s1AN,s2AN,s3AN,…,snANISAN,where CSM.HMI.SC represents multiattribute homogeneous multisensor integrated subsystem cluster, S represents the sensor, A1, A2, A3, …, AN represents sensor and integration subsystem attribute, 1, 2, 3, …, n represents number of sensors, and IS represents an integration subsystem.(ii)
Single-attribute (heterogeneous single-sensor integrated) subsystem cluster (S.HSI.SC) structural model is as follows:(4)CSS.HSI.SC=s1A1,s1A2,s1A3,…,s1AN1ISA1,s1A1,s1A2,s1A3,…,s1AN2ISA1,s1A1,s1A2,s1A3,…,s1AN3ISA1,⋮s1A1,s1A2,s1A3,…,s1ANnISA1,where CSS.HSI.SC represents single-attribute heterogeneous single-sensor integration subsystem cluster, S represents the sensor, A1, A2, A3, …, AN represents sensor and integration subsystem attribute, 1, 2, 3, …, n represents the number of sensors and integrated subsystems, and IS represents an integration subsystem.(iii)
Hybrid subsystem clusters (HSC) are mainly composed of single-sensor subsystems and homogeneous multisensor subsystems according to application needs. They are generally used in complex systems.


#### 4.1.3. Performance Analysis of Multisensor Generalized Integrated Structure

At present, the complex data fusion technology of multisensor integrated systems is in a high development stage. According to the structure of fusion data sources, the research of data fusion algorithm is a new research idea in the field of data fusion. Different structures of multisensor data acquisition system, their output data properties, and complexity are different, resulting in different data fusion modes, fusion process models, algorithm structures, calculation amounts, and fusion accuracy. Based on the multisensor generalized integration structure model, make “1, 2, and 3” correspond to “low, medium, and high” three levels and represent the corresponding vector advantages and disadvantages. Comparing and analyzing the comprehensive performance of the five structure models, the results are shown in Table3:Table 3
Performance analysis of multisensor generalized integrated system.
MGISHMI.SHIS.SM.HMI.SCS.HSI.SCHSCThe data structure32311Amount of calculation32211Fusion results31223Stability31322Practicability33313Comprehensive performancex¯=∑i=1kwixi,∑i=1kwi=131.82.61.42According to the performance comparison analysis in Table3, it is not difficult to see that the homogeneous multisensor integrated structure has the best performance, and the heterogeneous single-sensor integrated structure has the second best performance; in the cluster system, the cluster structure of multiattribute homogeneous multisensor subsystem and the cluster structure of hybrid subsystem are better than the cluster structure of the single-attribute heterogeneous single-sensor integrated subsystem cluster.


### 4.2. Multisensor Data Acquisition and Output

Multisensor refers to a sensor integrated system platform and a distributed fusion sensor subsystem. The main function is to use it as a signal source to collect data and output. (1) Data collection: data science refers to the data collection of physical sensors as the sensing equipment in the physical space, which is called data collection, that is, the measurement behavior and process of the object being measured by the physical sensor in time sequence. (2) Data output: it refers to the recording method and recording result of the measurement result of the multisensor system. The recording method is divided into real-time recording and abnormal recording. Real-time recording: it refers to the complete recording of sampling results based on time sequence; abnormal recording it refers to recording only abnormal data outside the threshold range based on the given normal threshold range.The data collected by the multisensor generally have the characteristics of heterogeneity, and its heterogeneity arises from differences in expression, source differences, and human factors. Representation difference refers to the diversity of order and dimension (some data are high-order tensors and some data are matrices); source difference refers to different types of sensors or from different detection purposes; human factors refers to the construction of data space, implementation, and technology of data management system [29].


## 4.1. Multisensor Generalized Integrated System (MGIS)

### 4.1.1. Multisensor Generalized Integration

According to the attributes and quantity of sensors in the multisensor system, abandoning specific application requirements, Figure2 shows the generalized integration mode of multiple sensors.Figure 2
Multisensor generalized integration.In the multisensor generalized integration method, homogeneous multisensor integration and heterogeneous single-sensor integration are the most basic integration methods, and heterogeneous multisensor integration is an organic combination of different subsystems.


### 4.1.2. Multisensor Generalized Integrated Structure Model

According to the multisensor generalized integration method, the multisensor generalized integration structure can be divided into①
Homogeneous multisensor integration structure (HMI) refers to a system integrated by multiple sensors with the same attributes, and the data collected by all the sensors represent the same attribute of the monitoring target; its structural model is as follows:(1)ISHMI=S1A1,S2A1,S3A1,…,SnA1,Where ISHMI represents homogeneous multisensor integrated system, S represents the sensor, A1 represents the sensor attribute, and 1, 2, 3, …, n represents the number of sensors.②
Heterogeneous single-sensor integration structure (HSI) refers to a system integrated by a single sensor with different attributes, and the data collected by each sensor represent different attributes of the monitoring target; its structural model is as follows:(2)ISHSI=S1A1,S1A2,S1A3,…,S1AN,where ISHSI represents the heterogeneous single-sensor integrated system, S represents the sensor, A1, A2, A3, …, AN represents sensor attributes, and 1 represents the number of sensors.③
Heterogeneous multisensor integration structure refers to a complex cluster system composed of multiple (or multiple) subsystems. The clustering methods include multiattribute homogeneous multisensor integrated subsystem cluster, single-attribute heterogeneous single-sensor integrated subsystem cluster, and hybrid system cluster.(i)
Multiattribute (homogeneous multisensor integrated) subsystem cluster (M.HMI.SC) structural model is as follows:(3)CSM.HMI.SC=s1A1,s2A1,s3A1,…,snA1ISA1,s1A2,s2A2,s3A2,…,snA2ISA2,s1A3,s2A3,s3A3,…,snA3ISA3,⋮s1AN,s2AN,s3AN,…,snANISAN,where CSM.HMI.SC represents multiattribute homogeneous multisensor integrated subsystem cluster, S represents the sensor, A1, A2, A3, …, AN represents sensor and integration subsystem attribute, 1, 2, 3, …, n represents number of sensors, and IS represents an integration subsystem.(ii)
Single-attribute (heterogeneous single-sensor integrated) subsystem cluster (S.HSI.SC) structural model is as follows:(4)CSS.HSI.SC=s1A1,s1A2,s1A3,…,s1AN1ISA1,s1A1,s1A2,s1A3,…,s1AN2ISA1,s1A1,s1A2,s1A3,…,s1AN3ISA1,⋮s1A1,s1A2,s1A3,…,s1ANnISA1,where CSS.HSI.SC represents single-attribute heterogeneous single-sensor integration subsystem cluster, S represents the sensor, A1, A2, A3, …, AN represents sensor and integration subsystem attribute, 1, 2, 3, …, n represents the number of sensors and integrated subsystems, and IS represents an integration subsystem.(iii)
Hybrid subsystem clusters (HSC) are mainly composed of single-sensor subsystems and homogeneous multisensor subsystems according to application needs. They are generally used in complex systems.


### 4.1.3. Performance Analysis of Multisensor Generalized Integrated Structure

At present, the complex data fusion technology of multisensor integrated systems is in a high development stage. According to the structure of fusion data sources, the research of data fusion algorithm is a new research idea in the field of data fusion. Different structures of multisensor data acquisition system, their output data properties, and complexity are different, resulting in different data fusion modes, fusion process models, algorithm structures, calculation amounts, and fusion accuracy. Based on the multisensor generalized integration structure model, make “1, 2, and 3” correspond to “low, medium, and high” three levels and represent the corresponding vector advantages and disadvantages. Comparing and analyzing the comprehensive performance of the five structure models, the results are shown in Table3:Table 3
Performance analysis of multisensor generalized integrated system.
MGISHMI.SHIS.SM.HMI.SCS.HSI.SCHSCThe data structure32311Amount of calculation32211Fusion results31223Stability31322Practicability33313Comprehensive performancex¯=∑i=1kwixi,∑i=1kwi=131.82.61.42According to the performance comparison analysis in Table3, it is not difficult to see that the homogeneous multisensor integrated structure has the best performance, and the heterogeneous single-sensor integrated structure has the second best performance; in the cluster system, the cluster structure of multiattribute homogeneous multisensor subsystem and the cluster structure of hybrid subsystem are better than the cluster structure of the single-attribute heterogeneous single-sensor integrated subsystem cluster.


## 4.1.1. Multisensor Generalized Integration

According to the attributes and quantity of sensors in the multisensor system, abandoning specific application requirements, Figure2 shows the generalized integration mode of multiple sensors.Figure 2
Multisensor generalized integration.In the multisensor generalized integration method, homogeneous multisensor integration and heterogeneous single-sensor integration are the most basic integration methods, and heterogeneous multisensor integration is an organic combination of different subsystems.


## 4.1.2. Multisensor Generalized Integrated Structure Model

According to the multisensor generalized integration method, the multisensor generalized integration structure can be divided into①
Homogeneous multisensor integration structure (HMI) refers to a system integrated by multiple sensors with the same attributes, and the data collected by all the sensors represent the same attribute of the monitoring target; its structural model is as follows:(1)ISHMI=S1A1,S2A1,S3A1,…,SnA1,Where ISHMI represents homogeneous multisensor integrated system, S represents the sensor, A1 represents the sensor attribute, and 1, 2, 3, …, n represents the number of sensors.②
Heterogeneous single-sensor integration structure (HSI) refers to a system integrated by a single sensor with different attributes, and the data collected by each sensor represent different attributes of the monitoring target; its structural model is as follows:(2)ISHSI=S1A1,S1A2,S1A3,…,S1AN,where ISHSI represents the heterogeneous single-sensor integrated system, S represents the sensor, A1, A2, A3, …, AN represents sensor attributes, and 1 represents the number of sensors.③
Heterogeneous multisensor integration structure refers to a complex cluster system composed of multiple (or multiple) subsystems. The clustering methods include multiattribute homogeneous multisensor integrated subsystem cluster, single-attribute heterogeneous single-sensor integrated subsystem cluster, and hybrid system cluster.(i)
Multiattribute (homogeneous multisensor integrated) subsystem cluster (M.HMI.SC) structural model is as follows:(3)CSM.HMI.SC=s1A1,s2A1,s3A1,…,snA1ISA1,s1A2,s2A2,s3A2,…,snA2ISA2,s1A3,s2A3,s3A3,…,snA3ISA3,⋮s1AN,s2AN,s3AN,…,snANISAN,where CSM.HMI.SC represents multiattribute homogeneous multisensor integrated subsystem cluster, S represents the sensor, A1, A2, A3, …, AN represents sensor and integration subsystem attribute, 1, 2, 3, …, n represents number of sensors, and IS represents an integration subsystem.(ii)
Single-attribute (heterogeneous single-sensor integrated) subsystem cluster (S.HSI.SC) structural model is as follows:(4)CSS.HSI.SC=s1A1,s1A2,s1A3,…,s1AN1ISA1,s1A1,s1A2,s1A3,…,s1AN2ISA1,s1A1,s1A2,s1A3,…,s1AN3ISA1,⋮s1A1,s1A2,s1A3,…,s1ANnISA1,where CSS.HSI.SC represents single-attribute heterogeneous single-sensor integration subsystem cluster, S represents the sensor, A1, A2, A3, …, AN represents sensor and integration subsystem attribute, 1, 2, 3, …, n represents the number of sensors and integrated subsystems, and IS represents an integration subsystem.(iii)
Hybrid subsystem clusters (HSC) are mainly composed of single-sensor subsystems and homogeneous multisensor subsystems according to application needs. They are generally used in complex systems.


## 4.1.3. Performance Analysis of Multisensor Generalized Integrated Structure

At present, the complex data fusion technology of multisensor integrated systems is in a high development stage. According to the structure of fusion data sources, the research of data fusion algorithm is a new research idea in the field of data fusion. Different structures of multisensor data acquisition system, their output data properties, and complexity are different, resulting in different data fusion modes, fusion process models, algorithm structures, calculation amounts, and fusion accuracy. Based on the multisensor generalized integration structure model, make “1, 2, and 3” correspond to “low, medium, and high” three levels and represent the corresponding vector advantages and disadvantages. Comparing and analyzing the comprehensive performance of the five structure models, the results are shown in Table3:Table 3
Performance analysis of multisensor generalized integrated system.
MGISHMI.SHIS.SM.HMI.SCS.HSI.SCHSCThe data structure32311Amount of calculation32211Fusion results31223Stability31322Practicability33313Comprehensive performancex¯=∑i=1kwixi,∑i=1kwi=131.82.61.42According to the performance comparison analysis in Table3, it is not difficult to see that the homogeneous multisensor integrated structure has the best performance, and the heterogeneous single-sensor integrated structure has the second best performance; in the cluster system, the cluster structure of multiattribute homogeneous multisensor subsystem and the cluster structure of hybrid subsystem are better than the cluster structure of the single-attribute heterogeneous single-sensor integrated subsystem cluster.


## 4.2. Multisensor Data Acquisition and Output

Multisensor refers to a sensor integrated system platform and a distributed fusion sensor subsystem. The main function is to use it as a signal source to collect data and output. (1) Data collection: data science refers to the data collection of physical sensors as the sensing equipment in the physical space, which is called data collection, that is, the measurement behavior and process of the object being measured by the physical sensor in time sequence. (2) Data output: it refers to the recording method and recording result of the measurement result of the multisensor system. The recording method is divided into real-time recording and abnormal recording. Real-time recording: it refers to the complete recording of sampling results based on time sequence; abnormal recording it refers to recording only abnormal data outside the threshold range based on the given normal threshold range.The data collected by the multisensor generally have the characteristics of heterogeneity, and its heterogeneity arises from differences in expression, source differences, and human factors. Representation difference refers to the diversity of order and dimension (some data are high-order tensors and some data are matrices); source difference refers to different types of sensors or from different detection purposes; human factors refers to the construction of data space, implementation, and technology of data management system [29].


## 5. Multisensor Data Fusion Principle and Architecture

Humans and animals are born and use extremely natural and reasonable multisensor data fusion capabilities, such as the observation, smell, and inquiry of traditional Chinese medicine. Bat’s judgment of prey is the most primitive multisensor data fusion.


### 5.1. Principles of Multisensor Data Fusion

In the field of automation research, multisensor data fusion technology is derived from the way of imitating human and animal cognition of the world, which is essentially similar to that humans or animals obtain information through various senses, and the acquired information with memory or experience is compared and distinguished. The basic principle is as follows: just like the comprehensive processing of information by the human brain, make full use of multiple sensors (multi-information sources) integrated in an orderly manner. First, the data collected by each sensor is the observation data. We carry out consistency expression, then combine with multiple sensors in space or time redundant or complementary data according to a certain criterion, and finally obtain a consistent explanation or description of the measured object. The specific expression is as follows [38, 39]:(1)
Orderly integraten sensors of N different types to collect and observe data related to the target (N, n = 2, 3, 4, …)(2)
Consistent representation of homogeneous data collected by each sensor(3)
Perform feature extraction on various heterogeneous data (such as output vector, imaging data, discrete or continuous time function data, or a direct attribute description) and extract the feature vectorYi representing the observation data(4)
Perform pattern recognition processing on the feature vectorYi (such as clustering algorithm, adaptive neural network, and tensor expansion operator), to complete the description of various sensors about the target(5)
Knowledge fusion: group and correlate the description data of various sensors about the target and then use the fusion algorithm to synthesize to obtain a consistent interpretation and description of the target


### 5.2. Multisensor Data Fusion Architecture

Multisensor data fusion architecture refers to the whole process of multisensor data fusion, the components of the fusion system, the main functions of each part, the relationship between each part, the relationship between the subsystems and the system, the fusion location, etc. [40, 41].


#### 5.2.1. Typical Multisensor Data Fusion Architecture Model

①
Multisensor integrated fusion structure model: in 1988, Luo and Kay proposed the model, as shown in Figure3 [42].The model is composed of 4 parts: sensor, data fusion, database auxiliary system, and fusion level. The sensor part is composed ofnn≥2 sensors; the data fusion part is described as a progressive fusion method; the database auxiliary system part is described as the intervention or impact on each fusion; the fusion level part is described as the fusion that the model can be used for level.②
Thomopoulos structural model: in 1990, Thomopoulos proposed the model, as shown in Figure4(a) [43].The model is composed of three parts: sensor, data fusion, and database. The data fusion part is described as three levels of fusion, and each level of fusion supports (or influences) each other; the data fusion part supports (or influences) each other with the sensor part and supports (or influences) each other with the database part.③
Waterfall model: in 1998, Harris et al. proposed the model, as shown in Figure4(b) [44].The model consists of three parts: sensors, data fusion, and control. The data fusion part is described as a five-level fusion, a process of incremental advancement, and high-level fusion is based on the results of low-level fusion.④
Mixed model: in 2009, Bedworth and O’Brien proposed the model, as shown in Figure5 [45]:The model consists of four parts: observation, orientation, decision-making, and action. Observation includes data collection and processing; the orientation part includes feature extraction and pattern recognition; the decision-making part includes state estimation and decision fusion; the action part includes control and resource allocation.Figure 3
Multisensor integration and fusion model.Figure 4
(a) Thomopoulos fusion model. (b) Waterfall fall model.
(a)(b)Figure 5
Mixed model.


#### 5.2.2. Multisensor Data Generalized Fusion Model

According to the principle of multisensor data fusion, abandoning application objects, and improving the JDL information fusion model (Steinberg version in 1999) [21], a generalized model of multisensor data fusion is obtained: it consists of data source, level 4 data fusion, human-computer interaction, and data management7. Its functions and relationships are shown in Figure 6.Figure 6
Multisensor data fusion generalized model.


#### 5.2.3. Multisensor Data Fusion Generalized Architecture Model Analysis

①
Data source includes (1) physical sensor integrated system platform (organic physics system or sensor and integrated system platform), (2) distributed fusion sensor subsystem, and (3) reference data, geographic information, supporting databases, etc.②
Human-computer interaction includes (1) manual input of commands, information requests, manual inference and evaluation, manual operator reports, etc., and (2) a mechanism for integrating system alarms, displaying location, and identity information to dynamically cover and deliver results geographically. It includes both multimedia methods of human interaction (graphics, sound, tactile interfaces, etc.) as well as methods to attract human attention and help overcome cognitive limitations.③
Level 1 (source data fusion): based on pixel-level or signal-level data association and representation, it prepares for signal/target observable state estimation or prediction. This means the data source signal is compressed under the condition of ensuring the data acquired by the sensor as little as possible, so as to retain the effective information to the maximum extent for higher level data fusion.④
Level 2 (feature and state estimation): based on the fusion results of data sources, it estimates and predicts the state, attribute, feature, event, or action feature vectors of the target related to heterogeneous data, and according to the feature vector, it estimates and predicts the relationship between entities (data), impact of association and perception, and physical environment and constructs the state trend.⑤
Level 3 (situation fusion): based on the results of feature fusion, it analyzes the advantages and disadvantages of various plans, actions, and state trends and estimates and predicts the interaction between plans and actions to be taken, the impact on the overall situation, and the possible results. Finally, it combined with the support data, the decision data are obtained.⑥
Level 4 (knowledge fusion and process optimization): knowledge fusion is the fusion of sensor data and supporting database data. Process optimization refers to the adaptive data collection and processing; it is responsible for monitoring all links in the entire fusion process and forming a more effective resource allocation plan to support mission goals. It is the feedback part of the whole system, thought of as a process that manages other processes, and is shown outside the fusion process. The main functions are to (i) monitor the performance of each link in the data fusion process and provide it with real-time and long-term control information, (ii) identify what information is needed to improve the multilevel fusion results (inference, location, identity, etc.), (iii) determine the collection of relevant information of the specific source (which type of sensor, which specific sensor, which database, etc.), and (iv) allocate data, realize knowledge fusion, and complete task goals.⑦
Data management: it is the most extensive support function required for data fusion processing. This function provides access and management of the fusion database, including data retrieval, storage, archiving, compression, relational query, and data protection. Database management in data fusion systems is particularly difficult because the amount of data managed is large and diverse (images, signals, vectors, and textures).Among them, the two parts of human-computer interaction and process optimization run through the whole process of data fusion; source data fusion belongs to pixel-level fusion; situation fusion belongs to decision-level fusion; support database refers to soft sensor data; fusion database contains fusion rules and fusion results.


## 5.1. Principles of Multisensor Data Fusion

In the field of automation research, multisensor data fusion technology is derived from the way of imitating human and animal cognition of the world, which is essentially similar to that humans or animals obtain information through various senses, and the acquired information with memory or experience is compared and distinguished. The basic principle is as follows: just like the comprehensive processing of information by the human brain, make full use of multiple sensors (multi-information sources) integrated in an orderly manner. First, the data collected by each sensor is the observation data. We carry out consistency expression, then combine with multiple sensors in space or time redundant or complementary data according to a certain criterion, and finally obtain a consistent explanation or description of the measured object. The specific expression is as follows [38, 39]:(1)
Orderly integraten sensors of N different types to collect and observe data related to the target (N, n = 2, 3, 4, …)(2)
Consistent representation of homogeneous data collected by each sensor(3)
Perform feature extraction on various heterogeneous data (such as output vector, imaging data, discrete or continuous time function data, or a direct attribute description) and extract the feature vectorYi representing the observation data(4)
Perform pattern recognition processing on the feature vectorYi (such as clustering algorithm, adaptive neural network, and tensor expansion operator), to complete the description of various sensors about the target(5)
Knowledge fusion: group and correlate the description data of various sensors about the target and then use the fusion algorithm to synthesize to obtain a consistent interpretation and description of the target


## 5.2. Multisensor Data Fusion Architecture

Multisensor data fusion architecture refers to the whole process of multisensor data fusion, the components of the fusion system, the main functions of each part, the relationship between each part, the relationship between the subsystems and the system, the fusion location, etc. [40, 41].


### 5.2.1. Typical Multisensor Data Fusion Architecture Model

①
Multisensor integrated fusion structure model: in 1988, Luo and Kay proposed the model, as shown in Figure3 [42].The model is composed of 4 parts: sensor, data fusion, database auxiliary system, and fusion level. The sensor part is composed ofnn≥2 sensors; the data fusion part is described as a progressive fusion method; the database auxiliary system part is described as the intervention or impact on each fusion; the fusion level part is described as the fusion that the model can be used for level.②
Thomopoulos structural model: in 1990, Thomopoulos proposed the model, as shown in Figure4(a) [43].The model is composed of three parts: sensor, data fusion, and database. The data fusion part is described as three levels of fusion, and each level of fusion supports (or influences) each other; the data fusion part supports (or influences) each other with the sensor part and supports (or influences) each other with the database part.③
Waterfall model: in 1998, Harris et al. proposed the model, as shown in Figure4(b) [44].The model consists of three parts: sensors, data fusion, and control. The data fusion part is described as a five-level fusion, a process of incremental advancement, and high-level fusion is based on the results of low-level fusion.④
Mixed model: in 2009, Bedworth and O’Brien proposed the model, as shown in Figure5 [45]:The model consists of four parts: observation, orientation, decision-making, and action. Observation includes data collection and processing; the orientation part includes feature extraction and pattern recognition; the decision-making part includes state estimation and decision fusion; the action part includes control and resource allocation.Figure 3
Multisensor integration and fusion model.Figure 4
(a) Thomopoulos fusion model. (b) Waterfall fall model.
(a)(b)Figure 5
Mixed model.


### 5.2.2. Multisensor Data Generalized Fusion Model

According to the principle of multisensor data fusion, abandoning application objects, and improving the JDL information fusion model (Steinberg version in 1999) [21], a generalized model of multisensor data fusion is obtained: it consists of data source, level 4 data fusion, human-computer interaction, and data management7. Its functions and relationships are shown in Figure 6.Figure 6
Multisensor data fusion generalized model.


### 5.2.3. Multisensor Data Fusion Generalized Architecture Model Analysis

①
Data source includes (1) physical sensor integrated system platform (organic physics system or sensor and integrated system platform), (2) distributed fusion sensor subsystem, and (3) reference data, geographic information, supporting databases, etc.②
Human-computer interaction includes (1) manual input of commands, information requests, manual inference and evaluation, manual operator reports, etc., and (2) a mechanism for integrating system alarms, displaying location, and identity information to dynamically cover and deliver results geographically. It includes both multimedia methods of human interaction (graphics, sound, tactile interfaces, etc.) as well as methods to attract human attention and help overcome cognitive limitations.③
Level 1 (source data fusion): based on pixel-level or signal-level data association and representation, it prepares for signal/target observable state estimation or prediction. This means the data source signal is compressed under the condition of ensuring the data acquired by the sensor as little as possible, so as to retain the effective information to the maximum extent for higher level data fusion.④
Level 2 (feature and state estimation): based on the fusion results of data sources, it estimates and predicts the state, attribute, feature, event, or action feature vectors of the target related to heterogeneous data, and according to the feature vector, it estimates and predicts the relationship between entities (data), impact of association and perception, and physical environment and constructs the state trend.⑤
Level 3 (situation fusion): based on the results of feature fusion, it analyzes the advantages and disadvantages of various plans, actions, and state trends and estimates and predicts the interaction between plans and actions to be taken, the impact on the overall situation, and the possible results. Finally, it combined with the support data, the decision data are obtained.⑥
Level 4 (knowledge fusion and process optimization): knowledge fusion is the fusion of sensor data and supporting database data. Process optimization refers to the adaptive data collection and processing; it is responsible for monitoring all links in the entire fusion process and forming a more effective resource allocation plan to support mission goals. It is the feedback part of the whole system, thought of as a process that manages other processes, and is shown outside the fusion process. The main functions are to (i) monitor the performance of each link in the data fusion process and provide it with real-time and long-term control information, (ii) identify what information is needed to improve the multilevel fusion results (inference, location, identity, etc.), (iii) determine the collection of relevant information of the specific source (which type of sensor, which specific sensor, which database, etc.), and (iv) allocate data, realize knowledge fusion, and complete task goals.⑦
Data management: it is the most extensive support function required for data fusion processing. This function provides access and management of the fusion database, including data retrieval, storage, archiving, compression, relational query, and data protection. Database management in data fusion systems is particularly difficult because the amount of data managed is large and diverse (images, signals, vectors, and textures).Among them, the two parts of human-computer interaction and process optimization run through the whole process of data fusion; source data fusion belongs to pixel-level fusion; situation fusion belongs to decision-level fusion; support database refers to soft sensor data; fusion database contains fusion rules and fusion results.


## 5.2.1. Typical Multisensor Data Fusion Architecture Model

①
Multisensor integrated fusion structure model: in 1988, Luo and Kay proposed the model, as shown in Figure3 [42].The model is composed of 4 parts: sensor, data fusion, database auxiliary system, and fusion level. The sensor part is composed ofnn≥2 sensors; the data fusion part is described as a progressive fusion method; the database auxiliary system part is described as the intervention or impact on each fusion; the fusion level part is described as the fusion that the model can be used for level.②
Thomopoulos structural model: in 1990, Thomopoulos proposed the model, as shown in Figure4(a) [43].The model is composed of three parts: sensor, data fusion, and database. The data fusion part is described as three levels of fusion, and each level of fusion supports (or influences) each other; the data fusion part supports (or influences) each other with the sensor part and supports (or influences) each other with the database part.③
Waterfall model: in 1998, Harris et al. proposed the model, as shown in Figure4(b) [44].The model consists of three parts: sensors, data fusion, and control. The data fusion part is described as a five-level fusion, a process of incremental advancement, and high-level fusion is based on the results of low-level fusion.④
Mixed model: in 2009, Bedworth and O’Brien proposed the model, as shown in Figure5 [45]:The model consists of four parts: observation, orientation, decision-making, and action. Observation includes data collection and processing; the orientation part includes feature extraction and pattern recognition; the decision-making part includes state estimation and decision fusion; the action part includes control and resource allocation.Figure 3
Multisensor integration and fusion model.Figure 4
(a) Thomopoulos fusion model. (b) Waterfall fall model.
(a)(b)Figure 5
Mixed model.


## 5.2.2. Multisensor Data Generalized Fusion Model

According to the principle of multisensor data fusion, abandoning application objects, and improving the JDL information fusion model (Steinberg version in 1999) [21], a generalized model of multisensor data fusion is obtained: it consists of data source, level 4 data fusion, human-computer interaction, and data management7. Its functions and relationships are shown in Figure 6.Figure 6
Multisensor data fusion generalized model.


## 5.2.3. Multisensor Data Fusion Generalized Architecture Model Analysis

①
Data source includes (1) physical sensor integrated system platform (organic physics system or sensor and integrated system platform), (2) distributed fusion sensor subsystem, and (3) reference data, geographic information, supporting databases, etc.②
Human-computer interaction includes (1) manual input of commands, information requests, manual inference and evaluation, manual operator reports, etc., and (2) a mechanism for integrating system alarms, displaying location, and identity information to dynamically cover and deliver results geographically. It includes both multimedia methods of human interaction (graphics, sound, tactile interfaces, etc.) as well as methods to attract human attention and help overcome cognitive limitations.③
Level 1 (source data fusion): based on pixel-level or signal-level data association and representation, it prepares for signal/target observable state estimation or prediction. This means the data source signal is compressed under the condition of ensuring the data acquired by the sensor as little as possible, so as to retain the effective information to the maximum extent for higher level data fusion.④
Level 2 (feature and state estimation): based on the fusion results of data sources, it estimates and predicts the state, attribute, feature, event, or action feature vectors of the target related to heterogeneous data, and according to the feature vector, it estimates and predicts the relationship between entities (data), impact of association and perception, and physical environment and constructs the state trend.⑤
Level 3 (situation fusion): based on the results of feature fusion, it analyzes the advantages and disadvantages of various plans, actions, and state trends and estimates and predicts the interaction between plans and actions to be taken, the impact on the overall situation, and the possible results. Finally, it combined with the support data, the decision data are obtained.⑥
Level 4 (knowledge fusion and process optimization): knowledge fusion is the fusion of sensor data and supporting database data. Process optimization refers to the adaptive data collection and processing; it is responsible for monitoring all links in the entire fusion process and forming a more effective resource allocation plan to support mission goals. It is the feedback part of the whole system, thought of as a process that manages other processes, and is shown outside the fusion process. The main functions are to (i) monitor the performance of each link in the data fusion process and provide it with real-time and long-term control information, (ii) identify what information is needed to improve the multilevel fusion results (inference, location, identity, etc.), (iii) determine the collection of relevant information of the specific source (which type of sensor, which specific sensor, which database, etc.), and (iv) allocate data, realize knowledge fusion, and complete task goals.⑦
Data management: it is the most extensive support function required for data fusion processing. This function provides access and management of the fusion database, including data retrieval, storage, archiving, compression, relational query, and data protection. Database management in data fusion systems is particularly difficult because the amount of data managed is large and diverse (images, signals, vectors, and textures).Among them, the two parts of human-computer interaction and process optimization run through the whole process of data fusion; source data fusion belongs to pixel-level fusion; situation fusion belongs to decision-level fusion; support database refers to soft sensor data; fusion database contains fusion rules and fusion results.


## 6. Multisensor Data Fusion Theory and Algorithm

Multisensor is used to obtain the consistent interpretation or description of the measured object, which is mainly realized by data fusion algorithm. At present, the research results of multisensor data fusion are very rich, which provide an important reference for the research of multisensor data-generalized fusion algorithm. Next, according to the 4-level fusion in the multisensor data-generalized fusion model, the related theories and algorithms are sorted out step by step in the following sections.


### 6.1. Source Data Fusion

Source data fusion refers to the data collection and output data sorting with multiple sensors as the signal source, that is, the output data are processed by classification, statistics, compression, and estimation (the homogeneous raw signal output by the same type of sensor). Consistent representation of quality data is gained.


#### 6.1.1. Data Representation

The knowledge and rules discovered from the original data are based on data representation. In recent years, many researchers have discussed and described the work of data representation [46, 47]. The most basic data representation methods include ontology representation, graph representation, tensor representation, and matrix representation [48].①
Ontology representation: ontology is the description of specific domain concepts, also known as the set of concepts [49]. Ontology generation includes two steps: first, mapping the real world (such as entity, attribute, and process) to a set of concepts and then extracting the relationship between concepts. It can represent objects as conceptual models at the semantic level. It simplifies the transformation of knowledge and is the mainstream method of data representation [50, 51].②
Graph representation: it is the representation of natural data with the matrix, which has some limitations. The graph is composed of many points called nodes, which are connected with edges [52]. The commonly used graph representation matrix is adjacency matrix [53].③
Matrix representation: the matrix is also called bidirectional array, which is a parallel description of time domain and space domain. Multichannel signals are generally represented by the matrix [29]. The rows of the matrix contain all sensors or channels, the columns contain all measurement times, and the elements represent signal values. In data mining and machine learning, rectangular arrays describe the attributes or observations of samples as each row corresponds to one sample or observation, and each column corresponds to multiple attributes or observations related to the sample.④
Tensor representation: a tensor is a sequential expansion of a vector. It is a multidimensional array. Each element has multiple indicators. Each indicator represents a model or an order. It is a general tool for representing various heterogeneous data [29]. For example, gait video data can be expressed as a fourth-order tensor composed of pixels, angles, motion, and objects [54]; network link data can be expressed as a third-order tensor [55]; the electronic nose data can be expressed as a third-order tensor [56].


#### 6.1.2. Consistency Test of Homogeneous Data

Homogeneity sensors are arranged in different spatial positions, and their monitoring data have some differences. According to the principle of consistency test, if the difference is greater than the set threshold, the monitoring data are considered as abnormal data, and the accuracy will be seriously affected if it is fused directly [57]. In order to ensure the consistency, continuity, and accuracy of the monitoring data, it is most reasonable to replace the abnormal data with the average value of the normal value in this period. Therefore, only after the homogeneity monitoring data passes the consistency test can the data fusion be carried out and the correct consistency representation be obtained.Homogeneous multisensor data consistency test principle: suppose there aren homogeneous sensors to measure the same attribute of the monitored object, and the measurement results are X1, X2, …, Xn, expressed as Xi (i = 1, 2, …, n), and we perform consistency test on Xi (i = 1, 2, …, n); the test principle is that the difference between two adjacent numbers is less than or equal to the threshold ε; the specific calculation formula is as follows:(5)X2−X1≤ε,X3−X2≤ε,…Xn−Xn−1≤ε.


#### 6.1.3. Weighted Average Fusion Algorithm [58]

The weighted average method is often used for the fusion of homogeneous data of homogeneous sensor systems to monitor dynamic objects. It is a direct fusion method for data sources and the simplest signal-level fusion method. Homogeneous data of a homogeneous sensor system is a description of the same attribute. IfK sensors are used to measure the target, the average value is defined as(6)x¯=∑i=1kwixi,∑i=1kwi=1,where wi represents the weight of the ith sensor. This method is simple and intuitive, but the fusion accuracy is not high, which is suitable for data fusion of the homogeneous multisensor system.


#### 6.1.4. Kalman Filter Fusion Algorithm [59]

This method is a data fusion method based on minimum variance estimation, which is used to estimate homogeneous data with monitoring errors. The goal is to maximize the representation of true values. As proposed in the 1960s, it is the most commonly used technology in target tracking and navigation systems [60]. The disadvantage is that each local sensor requires global estimation and two-way communication, negating some of the advantages of parallelization.


### 6.2. Feature Fusion

Feature fusion is the fusion of data features. Features are also called target characteristics, including the target state, which refers to various characteristics or states of the target carried in the data obtained by different sensors reflecting the same target. There are various interferences in the data acquisition process, and the acquired data may be distorted or unrecoverable. Therefore, a single feature is used as the fusion object, and the fusion result is unreliable.


#### 6.2.1. Feature Extraction

The premise of feature fusion is feature extraction. Feature extraction refers to the process of performing various mathematical transformations on data to obtain the indirect target characteristics contained in the data. Indirect target characteristics refer to the recessive features that can reflect target features (geometry, movement, statistics, etc.) from the side [61]. A large number of theories and practices have shown that when the direct features are not obvious, extracting the indirect features and finding the comprehensive characteristics of the target is the key to multisensor data feature fusion, and it is also an important idea of data fusion in the contemporary information technology field.


#### 6.2.2. Data Association

In a distributed multisensor system, judging whether the information from different subsystems represents the same target, this is data association (interconnection). The purpose of data association is to distinguish different targets and to solve the problem of overlapping sensor spatial coverage areas. The classic data association algorithms are nearest neighbor method [62], probabilistic data association algorithm (PDA) [63, 64], multiple hypothesis method (MHT) [65, 66], and probabilistic multiple hypothesis algorithm (PMHT) [67, 68].


#### 6.2.3. State Estimation

Multisensor systems are generally nonlinear systems. The optimal solution of nonlinear function filtering can be obtained through Bayesian optimal estimation. Therefore, from the Bayesian theory, the state estimation of the system can be obtained by approximating the nonlinear function of the system or the probability density function of the nonlinear function.There are two types of approximation methods for state estimation of nonlinear systems: one is the approximate linearization of the nonlinear links of the system, retaining the low-order terms and ignoring the high-order terms, that is, the direct linear approximation of the nonlinear function. Among which the most widely used are the extended Kalman filter algorithm (EKF) [69] and divided differential filter (DDF) [70, 71]. The other is to approximate the nonlinear distribution by the sampling method, that is, to approximate the probability density function of the nonlinear function. Such methods include particle filter algorithm (PF) [72], unscented Kalman filter algorithm (UKF) [73], and cubature Kalman filter algorithm (CKF) [74] be used for generalized Kalman filter for special cases, etc.


#### 6.2.4. Pattern Recognition

Pattern recognition is generally used for target feature fusion. The common methods include Bayesian inference, DS evidence theory, generation rule method, clustering algorithm, election method, maximum entropy method, fuzzy set theory, and artificial nerve convergence of networks.①
Bayesian reasoning: it is a conditional probability (or marginal probability) theorem about random events A and B [75]. This method is based on the hypothesis prior probability, the probability of observing different data with a given hypothesis, and the observed data itself. It is the main method used to fuse uncertain and incomplete multisensor data in the early days [75]:(7)PBi|A=PA|BiPBi∑j=1nPBjPA|Bj,i=1,2,…,n.②
D-S evidence theory: it is a mathematical method to fuse uncertain data and an expansion of classical probability theory [76, 77], It has the ability to express uncertainty [78]; the measurement of uncertainty is very close to people’s habits of thought. It can gradually reduce the hypothesis through evidence accumulation and synthesis rules, which is applicable to multisource data fusion [79, 80]. It does not require prior information and uses the “interval” method to describe uncertain information, which is flexible in distinguishing “do not know” from “uncertain” and accurately reflecting the aggregation of evidence. Among them, the relationship between the trust function and the likelihood function is PlsX≥BelX and PlsX=1−BelX¯, as shown in Figure 7.③
Generation rule method: it is the knowledge representation mode proposed by Post according to the string substitution rule [81]. At present, many successful expert systems adopt the production knowledge representation method [78, 82].Basic form:P ⟶ Q or IF P THEN Q. The meaning of the production is as follows: if premise P is satisfied, conclusion Q can be deduced or the operation specified by Q can be performed. P is the premise of the production, also called the antecedent, which is composed of a logical combination of facts and is the conjunction of some facts Ai. Q is a set of conclusions or operations, also called the consequent of the production, which indicates the current proposal. When P is satisfied, the conclusion that should be derived or the action that should be performed is a certain fact B.④
Clustering algorithm: it is a statistical analysis method for studying (samples or indicators) classification problems [83, 84], which is divided into clustering, hierarchical clustering, artificial neural network clustering, nuclear clustering, sequence data clustering, complex network clustering, intelligent search clustering, distributed clustering, parallel clustering, high-dimensional clustering, etc. [85, 86]. Common algorithms are K-MEANS [87–90], K-MEDOIDS [91], Binary-positive [92], VISOM [93], incremental support vector clustering [94], CLARANS [95, 96], Trajectory clustering algorithm [97], DBDC [98], CLIQUE [99], subspace clustering [100, 101], etc.⑤
Election algorithm: this algorithm is a common computing type in distributed systems. It selects a process from the process to perform special tasks. Based on the type of network used, it can be divided into ① election algorithm based on ring topology, ② election algorithm based on fully connected topology [102], ③ election algorithm based on comparison, commonly used election algorithms: election algorithm based on ring topology [103], overbearing election algorithm [102], etc.⑥
Entropy method: entropy refers to the degree of confusion of the system. Information entropy refers to the degree of uncertainty of a random variable [104]. Claude Elwood Shannon used i to mark all possible samples in the probability space, Pi to represent the probability of the occurrence of i samples, and K to represent arbitrary constants related to the selection of units. Based on cybernetics and information theory, he believed that when entropy is the largest, it means that the random variable is the most uncertain, that is, the random variable is the most random, and it is the most difficult to make accurate prediction of its behavior. The calculation formula of information entropy S is as follows:(8)Sp1,p2,…,pn=−K∑i=1npilog2pi.Figure 7
Relationship between trust function and likelihood function.Principle of maximum entropy: the main idea is when only part of the knowledge about the unknown distribution is mastered, the probability distribution that conforms to the maximum entropy value of these knowledge should be selected to infer the unknown distribution [105]. The essence is on the premise of known partial knowledge; the most reasonable inference to the unknown distribution is the most uncertain or random inference in accordance with the known knowledge. Characteristics: all uncertainties are retained to minimize risks. It is the criterion to select the statistical characteristics of random variables that most conforms to the objective situation.


### 6.3. Situation Fusion

Situation fusion: based on the result of feature fusion, situation estimation and impact estimation are performed. The main methods used in situation estimation and impact estimation are artificial neural network, deep learning method, clustering algorithm, fuzzy set theory, decision tree, and other methods.


#### 6.3.1. Artificial Neural Network (ANN)

ANN is a nonprogrammed, nonlinear adaptive, and brain-style parallel distributed information processing system proposed on the basis of modern neuroscience. The essence is through the transformation of the network structure and dynamic behavior, with varying degrees at different levels, to imitate the human brain nervous system to process information [106, 107]. A neural network is a computational model, which is composed of a large number of neurons (nodes) connected to each other. The connection mode of the neurons is different, and the composed network is also different. The neuronal structure is shown in Figure 8(a) [108, 109]. The calculation model of the artificial neural network is shown in Figure 8(b) [110, 111].Figure 8
Artificial neural network. (a) Neuronal structure. (b) Artificial neural network computing model.
(a)(b)In the figure,a1 − an indicates the components of the input vector, w1−wn indicates the weight of each neuron’s synapse, b indicates the bias value, f is the transfer function (usually a nonlinear function), t is the nerve. The output of the element is t=fWA′+b, W is the weight vector, A is the input vector, and A′ is the transpose of the A vector. It can be seen that the function of the neuron is to obtain the scalar result of the nonlinear transfer function after the inner product of the weight vector and the input vector are transposed.


#### 6.3.2. Deep Learning

Deep learning is derived from artificial neural network, which is a general term for a class of pattern analysis methods. It has made rich achievements in data mining, machine learning, natural language processing, and other related fields [112]. The purpose of studying deep learning is to establish a neural network that imitates human brain mechanism to interpret data (such as image, sound, and text) [113]. Deep learning includes supervised learning and unsupervised learning. Classical learning models include convolutional neural network (CNN), deep belief network (DBN), and stack automatically encodes network (SAEN) [114]. At present, deep network has been successfully applied to the fusion of single-mode data (such as text and image) and has also been rapidly developed in the fusion of multimode data (such as video) [115, 116].①
Convolutional neural network model: CNN is a kind of feedforward neural network with convolution calculation and depth structure. It has the ability of representation learning and is one of the representative algorithms of depth learning [117, 118], which is composed of the input layer, the hidden layer, and the output layer. It is a neural network that can be used for supervised learning and unsupervised learning, and its hidden layer has the characteristics of less computation [119–121].②
Deep belief network model: the DBN model can also be interpreted as a Bayesian probability generation model [122], which is a multihidden layer neural network composed of multiple restricted Boltzmann machines (RBM) “series” stacks. In the network and stack, the hidden layer of the previous RBM is the explicit layer of the next RBM, and the output of the previous RBM is the input of the next RBM, see Figure 9(a). In the course of their training, the previous RBM must be fully trained before the current RBM of the layer up to the last layer [123]. After layer-by-layer stacking of RBM, the DBN model can extract features layer by layer from the original data and obtain some high-level expressions [124, 125], see Figure 9(b).③
Stack automatically encodes network model: the structure of SAEN is similar to that of DBN, consisting of a stack of several structural units. The difference between the two is that the structural unit of SAEN is autoencoder, while the structural unit of DBN is RBM. The self-encoder is composed of a three-layer network. The input layer and the hidden layer form an encoder, which converts the input signalx into a. The hidden layer and the output layer constitute a decoder, which transforms the code into an output signal y; a multilayer sparse self-encoder can form a stacked self-encoder. That is, the output of the sparse self-encoder of the previous layer is used as the input of the self-encoder of the subsequent layer [118].Figure 9
(a) Deep belief network. (b) Deep belief network training process.
(a)(b)


#### 6.3.3. Fuzzy Set Theory

Fuzzy set theory refers to the use of mathematics to describe fuzzy concepts and extend the exact set to fuzzy sets from the extension. It is also called fuzzy mathematics. Mathematically, it can eliminate the imprisonment that the computer cannot handle fuzzy concepts [126]. The proposal of “membership function” breaks through the absolute relationship of belonging or not belonging to in the classic set theory and describes the ambiguity of things [127, 128].Definition of ambiguity [128, 129]: a measure of the ambiguity of fuzzy set A, which is reflected as the degree of ambiguity of A, is intuitively defined.Let mapD: F (U) ⟶ [0, 1], where D is the ambiguity function defined on F (U); then, D (A) is the ambiguity of fuzzy set A, which should have the following five characteristics:①
Clarity:D (A) = 0 if and only if A ∈ P (U) (the ambiguity of a classical set is always 0)②
Fuzziness:D (A) = 1 if and only if ∀u ∈ U have A (u) = 0.5 (the fuzzy set with membership degree of 0.5 is the fuzziest)③
Monotonicity: ∀u ∈ U if A (u) ≤ B (u) ≤ 0.5, or A (u) ≥ B (u) ≥ 0.5 and D (A) ≤ D (B)④
Symmetry: ∀A ∈ F (U); there are D (A) = D (Ā) (the complements have the same degree of ambiguity)⑤
Additivity:DA∪B+DA∩B = D (A) + D (B)


#### 6.3.4. Decision Tree

The machine learning technology of generating decision tree from data is called decision tree for short. It is a basic classification and regression method [130]. It is a graph theory method of intuitively using probability analysis; that is, on the basis of known occurrence probability of various situations, it can classify its objects or mine data by constructing decision tree [131]. The output of decision tree is single. When facing complex output, multiple independent decision trees can be established to deal with different outputs.In recent years, multisensor data fusion has developed rapidly; when dealing with situation fusion, many scholars also use nonprobabilistic fusion methods such as random set [132–134], rough set [135–138], fuzzy logic [139–142], and Dempster–Shafer [77, 143, 144] to achieve ideal results.


### 6.4. Knowledge Fusion and Process Optimization

Knowledge fusion is the fusion of sensor data and supporting database data. Process optimization refers to the global optimization process based on knowledge fusion.Knowledge fusion includes selection and automatic reasoning. Selection is mainly reflected in the selection of fusion mode and method, with emphasis on location information fusion and parameter data fusion. Automatic reasoning technology is to interpret the observed data environment, the relationship between observed entities, and the hierarchical grouping of targets or objects according to the actual rules, framework, and scripts of the knowledge fusion process to predict the future behavior of a target or entity.Process optimization is usually realized by “effect theory” [145, 146]; that is, a variety of system evaluation indexes and methods are used to monitor and evaluate the performance of each link (subsystem) and form an effective resource allocation scheme, which is equivalent to the feedback part of the whole system.The effectiveness evaluation of the data fusion system is generally quantitatively evaluated by Monte Carlo simulation [147, 148] or covariance error analysis technology [18, 149]. To optimize the data fusion system, the following basic issues must be considered and solved [150, 151]: (1) choose what algorithm or technology is the most suitable and optimal; (2) choose which fusion framework to use (that is, where the data flow is processed in the fusion process) is most appropriate; (3) select which sensor integration method can extract the maximum amount of information; (4) ensure the actual accuracy that each process of data fusion can achieve; (5) optimize the fusion process in a dynamic sense; (6) deal with the impact of the data collection environment; (7) improve the conditions of system operation.


### 6.5. Multisensor Data-Generalized Fusion-Proposed Method

The data fusion theory and algorithm summarized in Sections6.2 and 6.3 have strong application object, relatively isolated application field, and weak interoperability and lack a unified, reliable, efficient, and flexible idea of generalized fusion; in order to solve this very challenging problem, we tentatively carried out some research:(1)
The generalized fusion method of multisensor homogeneous data is as follows:Homogeneous data fusion of the homogeneous multisensor system has been very mature in the past research. It belongs to the source data level fusion in the generalized fusion model, and its universal algorithm has been described in detail in Section6.1.(2)
Tensor-based generalized fusion method for multisensor heterogeneous data:①
Consistent tensor fusion (CTF):In 2016, Kuang et al. proposed the unified tensor fusion (UTF) model (Figure10) [152]; that is, the stretching method is used to model heterogeneous data as subtensors, and the two-layer (whole layer and inner layer) model is inserted to generate a unified tensor, which better realizes the consistent representation of large (multimodal/heterogeneous) data. It provides a foundation for the research of multisensor heterogeneous data fusion.In Figure10, the global layer is a third-order tensor including time TI, space SP, and state U, which is expressed as It×Is×Iu. The inner layer represents three subtensors of three spatial data, and the subtensors are embedded into the whole layer to obtain a unified tensor.Most of the past studies on tensors are based on single-mode and low-order data. Tensor is a high-order generalization of the matrix and represents the variability of various types of data in a high dimension. Tensor factorization is the joint matrix factorization of coupling different factors by sharing factors [53]. When this idea is applied to multisensor heterogeneous data fusion, multisensor heterogeneous data can be fused by tensor decomposition of factors in each subsystem dataset.Given heterogeneous single-sensor integration platformS consists of four vectors: lidar, depth camera, GPS, and support database. The system model is YHSIS∈sLR,sDC,sGPS,sSD. The frequency domain data fusion process is as follows:Based on the UTF model, firstly, the consistency of heterogeneous data collected by sensors is expressed as the tensor; it can be seen that, after the consistency tensor representation of heterogeneous data, the diversity of order and dimension still exists. Based on the set matrix factorization (CMF) [153] equation proposed by Singh and Gordon, the feature tensor is fused to obtain the feature subtensor. Then, the tensor expansion operator (Teo) is used to obtain the higher order unified tensor: T∈RIu×Ic×It×Ix×⋯×Iy×Iz. Finally, the unified data tensor (UDT) is implemented to achieve heterogeneous fusion, and the consistent tensor fusion calculation model (formulas (9)–(11)) is obtained. Figure 11 shows the consistent tensor fusion process.Consistency tensor representation: the LIDAR point cloud data is represented as the second-order tensorTpoint cloud∈RIx×Iy. The depth camera video data is represented as fourth-order tensor Tvideo∈RIh×Ic×If×Iw; GPS data is expressed as a third-order tensor TGPS∈RIec×Ien×Ier. The database data can be expressed as the third-order tensor TXML∈RIt×Iid×Iy.CMF equation:(9)fA,B,W=X−ABT2+Y−AWT2.Tensor extension operator:(10)f:A×⟶B⟶C,C∈RIt×Is×Iu×I1×I2.Unified data sheet quantification:(11)f:du∪dsemi∪ds⟶Tu∪Tsemi∪Ts︸T,whereX∈RI×J and Y∈RI×K are given matrices with common mode I, A∈RIt×Is×Iu×I1, B∈RIt×Is×Iu×I2, du represents unstructured data, dsemi stands for semistructured data, and ds represents structured data.②
Constrained tensor fusion (CTF)Given heterogeneous single-sensor integration platformS consists of four vectors: lidar, depth camera, GPS, and support database. The system model is YHSIS∈sLR,sDC,sGPS,sSD, and its frequency domain data fusion process is as follows:Migration data fusion is the fusion of time domain and spatial domain data, which can be achieved by adding probability constraints on the basis of consistency tensor fusion (CTF). The so-called increase of probability constraint is to increase the cooperation between multivariable Markov chain (MC) and tensor, transform tensor in multiple steps, fuse the space, time, and supporting data of the system migration process, realize the system migration fusion, and obtain the constrained tensor fusion (CTF) calculation model (equations (12) and (15)). The position sequence in GPS data can constitute a spatial transformation model. The corresponding transformation matrix is the row random matrix, and the sum of elements in each row of the matrix is equal to 1.Spatial data fusion: spatial data fusion is realized by the Markov model. MC includes the following: (i) according to the theory of the discrete stochastic process, the transformation matrix corresponds to the stationary distribution, that is, any initial distribution vector can converge to the steady distribution vector after infinitely multiplying with the transformation matrix; (ii) matrix theory shows that the steady distribution vector is the eigenvector with the largest eigenvalue corresponding to the transformation matrix. Referring to [154], the transformation of spatial tensor is shown in Figure 12.Figure12 shows the mobility model of the platform. The elements in the matrix represent the probability of the platform moving from one point to another.Spatiotemporal data fusion: the fusion of time data and spatial data needs spatiotemporal tensor transformation. Mobile behavior is always related to time, and time is an important element of the mobile behavior model. The motion time is discretely represented asi, and its new state space is composed of pose information and time information and is represented as S=T1,P1,Ti,PJ,…,TI,PJ. From [154], the space-time tensor transformation is shown in Figure 13.Figure13 shows the migration model of the platform. The elements of the tensor represent the probability of the platform moving from one point to another at a certain point. Based on the transformation of space tensor, the fourth-order space-time transition tensor TST can be obtained by combining time information through equation (12).Temporal and spatial information fusion:(12)Tt1l1t2l2=countt1,l1⟶t2,l2countt1,l1.Support database fusion: the influence degree of support data is quantified as the greater the correlation coefficient between fusion data and support data tensor, the greater the influence. The knowledge influence coefficient is described as(13)ρKI=∑i=1nKi−K¯Ii−I¯∑i=1nKi−K¯2∑i=1nIi−I¯2.Figure 10
Unified tensor representation model.Figure 11
Consistency tensor fusion process.Figure 12
Space tensor transformation.Figure 13
Temporal and spatial tensor transformation.The correlation coefficient matrixΓ=RKN×IN can be obtained by calculating the correlation coefficients between global tensors. By normalizing the rows in the correlation coefficient matrix Γ, the influence matrix (IM) Λ can be obtained:(14)Γ=ρK1I1ρK1I2…ρK1InρK2I1ρK2I2…ρK2In⋮⋮⋮⋮ρKnI1ρKnI2…ρKnIn,Λ=λK1I1λK1I2…λK1InλK2I1λK2I2…λK2In⋮⋮⋮⋮λKnI1λKnI2…λKnIn.By fusing the influence matrix of supporting data into the spatiotemporal transformation tensor, the system migration behavior fusion can be obtained:(15)A=Λ⊙TST,where ⊙ is the Kronecker product.


## 6.1. Source Data Fusion

Source data fusion refers to the data collection and output data sorting with multiple sensors as the signal source, that is, the output data are processed by classification, statistics, compression, and estimation (the homogeneous raw signal output by the same type of sensor). Consistent representation of quality data is gained.


### 6.1.1. Data Representation

The knowledge and rules discovered from the original data are based on data representation. In recent years, many researchers have discussed and described the work of data representation [46, 47]. The most basic data representation methods include ontology representation, graph representation, tensor representation, and matrix representation [48].①
Ontology representation: ontology is the description of specific domain concepts, also known as the set of concepts [49]. Ontology generation includes two steps: first, mapping the real world (such as entity, attribute, and process) to a set of concepts and then extracting the relationship between concepts. It can represent objects as conceptual models at the semantic level. It simplifies the transformation of knowledge and is the mainstream method of data representation [50, 51].②
Graph representation: it is the representation of natural data with the matrix, which has some limitations. The graph is composed of many points called nodes, which are connected with edges [52]. The commonly used graph representation matrix is adjacency matrix [53].③
Matrix representation: the matrix is also called bidirectional array, which is a parallel description of time domain and space domain. Multichannel signals are generally represented by the matrix [29]. The rows of the matrix contain all sensors or channels, the columns contain all measurement times, and the elements represent signal values. In data mining and machine learning, rectangular arrays describe the attributes or observations of samples as each row corresponds to one sample or observation, and each column corresponds to multiple attributes or observations related to the sample.④
Tensor representation: a tensor is a sequential expansion of a vector. It is a multidimensional array. Each element has multiple indicators. Each indicator represents a model or an order. It is a general tool for representing various heterogeneous data [29]. For example, gait video data can be expressed as a fourth-order tensor composed of pixels, angles, motion, and objects [54]; network link data can be expressed as a third-order tensor [55]; the electronic nose data can be expressed as a third-order tensor [56].


### 6.1.2. Consistency Test of Homogeneous Data

Homogeneity sensors are arranged in different spatial positions, and their monitoring data have some differences. According to the principle of consistency test, if the difference is greater than the set threshold, the monitoring data are considered as abnormal data, and the accuracy will be seriously affected if it is fused directly [57]. In order to ensure the consistency, continuity, and accuracy of the monitoring data, it is most reasonable to replace the abnormal data with the average value of the normal value in this period. Therefore, only after the homogeneity monitoring data passes the consistency test can the data fusion be carried out and the correct consistency representation be obtained.Homogeneous multisensor data consistency test principle: suppose there aren homogeneous sensors to measure the same attribute of the monitored object, and the measurement results are X1, X2, …, Xn, expressed as Xi (i = 1, 2, …, n), and we perform consistency test on Xi (i = 1, 2, …, n); the test principle is that the difference between two adjacent numbers is less than or equal to the threshold ε; the specific calculation formula is as follows:(5)X2−X1≤ε,X3−X2≤ε,…Xn−Xn−1≤ε.


### 6.1.3. Weighted Average Fusion Algorithm [58]

The weighted average method is often used for the fusion of homogeneous data of homogeneous sensor systems to monitor dynamic objects. It is a direct fusion method for data sources and the simplest signal-level fusion method. Homogeneous data of a homogeneous sensor system is a description of the same attribute. IfK sensors are used to measure the target, the average value is defined as(6)x¯=∑i=1kwixi,∑i=1kwi=1,where wi represents the weight of the ith sensor. This method is simple and intuitive, but the fusion accuracy is not high, which is suitable for data fusion of the homogeneous multisensor system.


### 6.1.4. Kalman Filter Fusion Algorithm [59]

This method is a data fusion method based on minimum variance estimation, which is used to estimate homogeneous data with monitoring errors. The goal is to maximize the representation of true values. As proposed in the 1960s, it is the most commonly used technology in target tracking and navigation systems [60]. The disadvantage is that each local sensor requires global estimation and two-way communication, negating some of the advantages of parallelization.


## 6.1.1. Data Representation

The knowledge and rules discovered from the original data are based on data representation. In recent years, many researchers have discussed and described the work of data representation [46, 47]. The most basic data representation methods include ontology representation, graph representation, tensor representation, and matrix representation [48].①
Ontology representation: ontology is the description of specific domain concepts, also known as the set of concepts [49]. Ontology generation includes two steps: first, mapping the real world (such as entity, attribute, and process) to a set of concepts and then extracting the relationship between concepts. It can represent objects as conceptual models at the semantic level. It simplifies the transformation of knowledge and is the mainstream method of data representation [50, 51].②
Graph representation: it is the representation of natural data with the matrix, which has some limitations. The graph is composed of many points called nodes, which are connected with edges [52]. The commonly used graph representation matrix is adjacency matrix [53].③
Matrix representation: the matrix is also called bidirectional array, which is a parallel description of time domain and space domain. Multichannel signals are generally represented by the matrix [29]. The rows of the matrix contain all sensors or channels, the columns contain all measurement times, and the elements represent signal values. In data mining and machine learning, rectangular arrays describe the attributes or observations of samples as each row corresponds to one sample or observation, and each column corresponds to multiple attributes or observations related to the sample.④
Tensor representation: a tensor is a sequential expansion of a vector. It is a multidimensional array. Each element has multiple indicators. Each indicator represents a model or an order. It is a general tool for representing various heterogeneous data [29]. For example, gait video data can be expressed as a fourth-order tensor composed of pixels, angles, motion, and objects [54]; network link data can be expressed as a third-order tensor [55]; the electronic nose data can be expressed as a third-order tensor [56].


## 6.1.2. Consistency Test of Homogeneous Data

Homogeneity sensors are arranged in different spatial positions, and their monitoring data have some differences. According to the principle of consistency test, if the difference is greater than the set threshold, the monitoring data are considered as abnormal data, and the accuracy will be seriously affected if it is fused directly [57]. In order to ensure the consistency, continuity, and accuracy of the monitoring data, it is most reasonable to replace the abnormal data with the average value of the normal value in this period. Therefore, only after the homogeneity monitoring data passes the consistency test can the data fusion be carried out and the correct consistency representation be obtained.Homogeneous multisensor data consistency test principle: suppose there aren homogeneous sensors to measure the same attribute of the monitored object, and the measurement results are X1, X2, …, Xn, expressed as Xi (i = 1, 2, …, n), and we perform consistency test on Xi (i = 1, 2, …, n); the test principle is that the difference between two adjacent numbers is less than or equal to the threshold ε; the specific calculation formula is as follows:(5)X2−X1≤ε,X3−X2≤ε,…Xn−Xn−1≤ε.


## 6.1.3. Weighted Average Fusion Algorithm [58]

The weighted average method is often used for the fusion of homogeneous data of homogeneous sensor systems to monitor dynamic objects. It is a direct fusion method for data sources and the simplest signal-level fusion method. Homogeneous data of a homogeneous sensor system is a description of the same attribute. IfK sensors are used to measure the target, the average value is defined as(6)x¯=∑i=1kwixi,∑i=1kwi=1,where wi represents the weight of the ith sensor. This method is simple and intuitive, but the fusion accuracy is not high, which is suitable for data fusion of the homogeneous multisensor system.


## 6.1.4. Kalman Filter Fusion Algorithm [59]

This method is a data fusion method based on minimum variance estimation, which is used to estimate homogeneous data with monitoring errors. The goal is to maximize the representation of true values. As proposed in the 1960s, it is the most commonly used technology in target tracking and navigation systems [60]. The disadvantage is that each local sensor requires global estimation and two-way communication, negating some of the advantages of parallelization.


## 6.2. Feature Fusion

Feature fusion is the fusion of data features. Features are also called target characteristics, including the target state, which refers to various characteristics or states of the target carried in the data obtained by different sensors reflecting the same target. There are various interferences in the data acquisition process, and the acquired data may be distorted or unrecoverable. Therefore, a single feature is used as the fusion object, and the fusion result is unreliable.


### 6.2.1. Feature Extraction

The premise of feature fusion is feature extraction. Feature extraction refers to the process of performing various mathematical transformations on data to obtain the indirect target characteristics contained in the data. Indirect target characteristics refer to the recessive features that can reflect target features (geometry, movement, statistics, etc.) from the side [61]. A large number of theories and practices have shown that when the direct features are not obvious, extracting the indirect features and finding the comprehensive characteristics of the target is the key to multisensor data feature fusion, and it is also an important idea of data fusion in the contemporary information technology field.


### 6.2.2. Data Association

In a distributed multisensor system, judging whether the information from different subsystems represents the same target, this is data association (interconnection). The purpose of data association is to distinguish different targets and to solve the problem of overlapping sensor spatial coverage areas. The classic data association algorithms are nearest neighbor method [62], probabilistic data association algorithm (PDA) [63, 64], multiple hypothesis method (MHT) [65, 66], and probabilistic multiple hypothesis algorithm (PMHT) [67, 68].


### 6.2.3. State Estimation

Multisensor systems are generally nonlinear systems. The optimal solution of nonlinear function filtering can be obtained through Bayesian optimal estimation. Therefore, from the Bayesian theory, the state estimation of the system can be obtained by approximating the nonlinear function of the system or the probability density function of the nonlinear function.There are two types of approximation methods for state estimation of nonlinear systems: one is the approximate linearization of the nonlinear links of the system, retaining the low-order terms and ignoring the high-order terms, that is, the direct linear approximation of the nonlinear function. Among which the most widely used are the extended Kalman filter algorithm (EKF) [69] and divided differential filter (DDF) [70, 71]. The other is to approximate the nonlinear distribution by the sampling method, that is, to approximate the probability density function of the nonlinear function. Such methods include particle filter algorithm (PF) [72], unscented Kalman filter algorithm (UKF) [73], and cubature Kalman filter algorithm (CKF) [74] be used for generalized Kalman filter for special cases, etc.


### 6.2.4. Pattern Recognition

Pattern recognition is generally used for target feature fusion. The common methods include Bayesian inference, DS evidence theory, generation rule method, clustering algorithm, election method, maximum entropy method, fuzzy set theory, and artificial nerve convergence of networks.①
Bayesian reasoning: it is a conditional probability (or marginal probability) theorem about random events A and B [75]. This method is based on the hypothesis prior probability, the probability of observing different data with a given hypothesis, and the observed data itself. It is the main method used to fuse uncertain and incomplete multisensor data in the early days [75]:(7)PBi|A=PA|BiPBi∑j=1nPBjPA|Bj,i=1,2,…,n.②
D-S evidence theory: it is a mathematical method to fuse uncertain data and an expansion of classical probability theory [76, 77], It has the ability to express uncertainty [78]; the measurement of uncertainty is very close to people’s habits of thought. It can gradually reduce the hypothesis through evidence accumulation and synthesis rules, which is applicable to multisource data fusion [79, 80]. It does not require prior information and uses the “interval” method to describe uncertain information, which is flexible in distinguishing “do not know” from “uncertain” and accurately reflecting the aggregation of evidence. Among them, the relationship between the trust function and the likelihood function is PlsX≥BelX and PlsX=1−BelX¯, as shown in Figure 7.③
Generation rule method: it is the knowledge representation mode proposed by Post according to the string substitution rule [81]. At present, many successful expert systems adopt the production knowledge representation method [78, 82].Basic form:P ⟶ Q or IF P THEN Q. The meaning of the production is as follows: if premise P is satisfied, conclusion Q can be deduced or the operation specified by Q can be performed. P is the premise of the production, also called the antecedent, which is composed of a logical combination of facts and is the conjunction of some facts Ai. Q is a set of conclusions or operations, also called the consequent of the production, which indicates the current proposal. When P is satisfied, the conclusion that should be derived or the action that should be performed is a certain fact B.④
Clustering algorithm: it is a statistical analysis method for studying (samples or indicators) classification problems [83, 84], which is divided into clustering, hierarchical clustering, artificial neural network clustering, nuclear clustering, sequence data clustering, complex network clustering, intelligent search clustering, distributed clustering, parallel clustering, high-dimensional clustering, etc. [85, 86]. Common algorithms are K-MEANS [87–90], K-MEDOIDS [91], Binary-positive [92], VISOM [93], incremental support vector clustering [94], CLARANS [95, 96], Trajectory clustering algorithm [97], DBDC [98], CLIQUE [99], subspace clustering [100, 101], etc.⑤
Election algorithm: this algorithm is a common computing type in distributed systems. It selects a process from the process to perform special tasks. Based on the type of network used, it can be divided into ① election algorithm based on ring topology, ② election algorithm based on fully connected topology [102], ③ election algorithm based on comparison, commonly used election algorithms: election algorithm based on ring topology [103], overbearing election algorithm [102], etc.⑥
Entropy method: entropy refers to the degree of confusion of the system. Information entropy refers to the degree of uncertainty of a random variable [104]. Claude Elwood Shannon used i to mark all possible samples in the probability space, Pi to represent the probability of the occurrence of i samples, and K to represent arbitrary constants related to the selection of units. Based on cybernetics and information theory, he believed that when entropy is the largest, it means that the random variable is the most uncertain, that is, the random variable is the most random, and it is the most difficult to make accurate prediction of its behavior. The calculation formula of information entropy S is as follows:(8)Sp1,p2,…,pn=−K∑i=1npilog2pi.Figure 7
Relationship between trust function and likelihood function.Principle of maximum entropy: the main idea is when only part of the knowledge about the unknown distribution is mastered, the probability distribution that conforms to the maximum entropy value of these knowledge should be selected to infer the unknown distribution [105]. The essence is on the premise of known partial knowledge; the most reasonable inference to the unknown distribution is the most uncertain or random inference in accordance with the known knowledge. Characteristics: all uncertainties are retained to minimize risks. It is the criterion to select the statistical characteristics of random variables that most conforms to the objective situation.


## 6.2.1. Feature Extraction

The premise of feature fusion is feature extraction. Feature extraction refers to the process of performing various mathematical transformations on data to obtain the indirect target characteristics contained in the data. Indirect target characteristics refer to the recessive features that can reflect target features (geometry, movement, statistics, etc.) from the side [61]. A large number of theories and practices have shown that when the direct features are not obvious, extracting the indirect features and finding the comprehensive characteristics of the target is the key to multisensor data feature fusion, and it is also an important idea of data fusion in the contemporary information technology field.


## 6.2.2. Data Association

In a distributed multisensor system, judging whether the information from different subsystems represents the same target, this is data association (interconnection). The purpose of data association is to distinguish different targets and to solve the problem of overlapping sensor spatial coverage areas. The classic data association algorithms are nearest neighbor method [62], probabilistic data association algorithm (PDA) [63, 64], multiple hypothesis method (MHT) [65, 66], and probabilistic multiple hypothesis algorithm (PMHT) [67, 68].


## 6.2.3. State Estimation

Multisensor systems are generally nonlinear systems. The optimal solution of nonlinear function filtering can be obtained through Bayesian optimal estimation. Therefore, from the Bayesian theory, the state estimation of the system can be obtained by approximating the nonlinear function of the system or the probability density function of the nonlinear function.There are two types of approximation methods for state estimation of nonlinear systems: one is the approximate linearization of the nonlinear links of the system, retaining the low-order terms and ignoring the high-order terms, that is, the direct linear approximation of the nonlinear function. Among which the most widely used are the extended Kalman filter algorithm (EKF) [69] and divided differential filter (DDF) [70, 71]. The other is to approximate the nonlinear distribution by the sampling method, that is, to approximate the probability density function of the nonlinear function. Such methods include particle filter algorithm (PF) [72], unscented Kalman filter algorithm (UKF) [73], and cubature Kalman filter algorithm (CKF) [74] be used for generalized Kalman filter for special cases, etc.


## 6.2.4. Pattern Recognition

Pattern recognition is generally used for target feature fusion. The common methods include Bayesian inference, DS evidence theory, generation rule method, clustering algorithm, election method, maximum entropy method, fuzzy set theory, and artificial nerve convergence of networks.①
Bayesian reasoning: it is a conditional probability (or marginal probability) theorem about random events A and B [75]. This method is based on the hypothesis prior probability, the probability of observing different data with a given hypothesis, and the observed data itself. It is the main method used to fuse uncertain and incomplete multisensor data in the early days [75]:(7)PBi|A=PA|BiPBi∑j=1nPBjPA|Bj,i=1,2,…,n.②
D-S evidence theory: it is a mathematical method to fuse uncertain data and an expansion of classical probability theory [76, 77], It has the ability to express uncertainty [78]; the measurement of uncertainty is very close to people’s habits of thought. It can gradually reduce the hypothesis through evidence accumulation and synthesis rules, which is applicable to multisource data fusion [79, 80]. It does not require prior information and uses the “interval” method to describe uncertain information, which is flexible in distinguishing “do not know” from “uncertain” and accurately reflecting the aggregation of evidence. Among them, the relationship between the trust function and the likelihood function is PlsX≥BelX and PlsX=1−BelX¯, as shown in Figure 7.③
Generation rule method: it is the knowledge representation mode proposed by Post according to the string substitution rule [81]. At present, many successful expert systems adopt the production knowledge representation method [78, 82].Basic form:P ⟶ Q or IF P THEN Q. The meaning of the production is as follows: if premise P is satisfied, conclusion Q can be deduced or the operation specified by Q can be performed. P is the premise of the production, also called the antecedent, which is composed of a logical combination of facts and is the conjunction of some facts Ai. Q is a set of conclusions or operations, also called the consequent of the production, which indicates the current proposal. When P is satisfied, the conclusion that should be derived or the action that should be performed is a certain fact B.④
Clustering algorithm: it is a statistical analysis method for studying (samples or indicators) classification problems [83, 84], which is divided into clustering, hierarchical clustering, artificial neural network clustering, nuclear clustering, sequence data clustering, complex network clustering, intelligent search clustering, distributed clustering, parallel clustering, high-dimensional clustering, etc. [85, 86]. Common algorithms are K-MEANS [87–90], K-MEDOIDS [91], Binary-positive [92], VISOM [93], incremental support vector clustering [94], CLARANS [95, 96], Trajectory clustering algorithm [97], DBDC [98], CLIQUE [99], subspace clustering [100, 101], etc.⑤
Election algorithm: this algorithm is a common computing type in distributed systems. It selects a process from the process to perform special tasks. Based on the type of network used, it can be divided into ① election algorithm based on ring topology, ② election algorithm based on fully connected topology [102], ③ election algorithm based on comparison, commonly used election algorithms: election algorithm based on ring topology [103], overbearing election algorithm [102], etc.⑥
Entropy method: entropy refers to the degree of confusion of the system. Information entropy refers to the degree of uncertainty of a random variable [104]. Claude Elwood Shannon used i to mark all possible samples in the probability space, Pi to represent the probability of the occurrence of i samples, and K to represent arbitrary constants related to the selection of units. Based on cybernetics and information theory, he believed that when entropy is the largest, it means that the random variable is the most uncertain, that is, the random variable is the most random, and it is the most difficult to make accurate prediction of its behavior. The calculation formula of information entropy S is as follows:(8)Sp1,p2,…,pn=−K∑i=1npilog2pi.Figure 7
Relationship between trust function and likelihood function.Principle of maximum entropy: the main idea is when only part of the knowledge about the unknown distribution is mastered, the probability distribution that conforms to the maximum entropy value of these knowledge should be selected to infer the unknown distribution [105]. The essence is on the premise of known partial knowledge; the most reasonable inference to the unknown distribution is the most uncertain or random inference in accordance with the known knowledge. Characteristics: all uncertainties are retained to minimize risks. It is the criterion to select the statistical characteristics of random variables that most conforms to the objective situation.


## 6.3. Situation Fusion

Situation fusion: based on the result of feature fusion, situation estimation and impact estimation are performed. The main methods used in situation estimation and impact estimation are artificial neural network, deep learning method, clustering algorithm, fuzzy set theory, decision tree, and other methods.


### 6.3.1. Artificial Neural Network (ANN)

ANN is a nonprogrammed, nonlinear adaptive, and brain-style parallel distributed information processing system proposed on the basis of modern neuroscience. The essence is through the transformation of the network structure and dynamic behavior, with varying degrees at different levels, to imitate the human brain nervous system to process information [106, 107]. A neural network is a computational model, which is composed of a large number of neurons (nodes) connected to each other. The connection mode of the neurons is different, and the composed network is also different. The neuronal structure is shown in Figure 8(a) [108, 109]. The calculation model of the artificial neural network is shown in Figure 8(b) [110, 111].Figure 8
Artificial neural network. (a) Neuronal structure. (b) Artificial neural network computing model.
(a)(b)In the figure,a1 − an indicates the components of the input vector, w1−wn indicates the weight of each neuron’s synapse, b indicates the bias value, f is the transfer function (usually a nonlinear function), t is the nerve. The output of the element is t=fWA′+b, W is the weight vector, A is the input vector, and A′ is the transpose of the A vector. It can be seen that the function of the neuron is to obtain the scalar result of the nonlinear transfer function after the inner product of the weight vector and the input vector are transposed.


### 6.3.2. Deep Learning

Deep learning is derived from artificial neural network, which is a general term for a class of pattern analysis methods. It has made rich achievements in data mining, machine learning, natural language processing, and other related fields [112]. The purpose of studying deep learning is to establish a neural network that imitates human brain mechanism to interpret data (such as image, sound, and text) [113]. Deep learning includes supervised learning and unsupervised learning. Classical learning models include convolutional neural network (CNN), deep belief network (DBN), and stack automatically encodes network (SAEN) [114]. At present, deep network has been successfully applied to the fusion of single-mode data (such as text and image) and has also been rapidly developed in the fusion of multimode data (such as video) [115, 116].①
Convolutional neural network model: CNN is a kind of feedforward neural network with convolution calculation and depth structure. It has the ability of representation learning and is one of the representative algorithms of depth learning [117, 118], which is composed of the input layer, the hidden layer, and the output layer. It is a neural network that can be used for supervised learning and unsupervised learning, and its hidden layer has the characteristics of less computation [119–121].②
Deep belief network model: the DBN model can also be interpreted as a Bayesian probability generation model [122], which is a multihidden layer neural network composed of multiple restricted Boltzmann machines (RBM) “series” stacks. In the network and stack, the hidden layer of the previous RBM is the explicit layer of the next RBM, and the output of the previous RBM is the input of the next RBM, see Figure 9(a). In the course of their training, the previous RBM must be fully trained before the current RBM of the layer up to the last layer [123]. After layer-by-layer stacking of RBM, the DBN model can extract features layer by layer from the original data and obtain some high-level expressions [124, 125], see Figure 9(b).③
Stack automatically encodes network model: the structure of SAEN is similar to that of DBN, consisting of a stack of several structural units. The difference between the two is that the structural unit of SAEN is autoencoder, while the structural unit of DBN is RBM. The self-encoder is composed of a three-layer network. The input layer and the hidden layer form an encoder, which converts the input signalx into a. The hidden layer and the output layer constitute a decoder, which transforms the code into an output signal y; a multilayer sparse self-encoder can form a stacked self-encoder. That is, the output of the sparse self-encoder of the previous layer is used as the input of the self-encoder of the subsequent layer [118].Figure 9
(a) Deep belief network. (b) Deep belief network training process.
(a)(b)


### 6.3.3. Fuzzy Set Theory

Fuzzy set theory refers to the use of mathematics to describe fuzzy concepts and extend the exact set to fuzzy sets from the extension. It is also called fuzzy mathematics. Mathematically, it can eliminate the imprisonment that the computer cannot handle fuzzy concepts [126]. The proposal of “membership function” breaks through the absolute relationship of belonging or not belonging to in the classic set theory and describes the ambiguity of things [127, 128].Definition of ambiguity [128, 129]: a measure of the ambiguity of fuzzy set A, which is reflected as the degree of ambiguity of A, is intuitively defined.Let mapD: F (U) ⟶ [0, 1], where D is the ambiguity function defined on F (U); then, D (A) is the ambiguity of fuzzy set A, which should have the following five characteristics:①
Clarity:D (A) = 0 if and only if A ∈ P (U) (the ambiguity of a classical set is always 0)②
Fuzziness:D (A) = 1 if and only if ∀u ∈ U have A (u) = 0.5 (the fuzzy set with membership degree of 0.5 is the fuzziest)③
Monotonicity: ∀u ∈ U if A (u) ≤ B (u) ≤ 0.5, or A (u) ≥ B (u) ≥ 0.5 and D (A) ≤ D (B)④
Symmetry: ∀A ∈ F (U); there are D (A) = D (Ā) (the complements have the same degree of ambiguity)⑤
Additivity:DA∪B+DA∩B = D (A) + D (B)


### 6.3.4. Decision Tree

The machine learning technology of generating decision tree from data is called decision tree for short. It is a basic classification and regression method [130]. It is a graph theory method of intuitively using probability analysis; that is, on the basis of known occurrence probability of various situations, it can classify its objects or mine data by constructing decision tree [131]. The output of decision tree is single. When facing complex output, multiple independent decision trees can be established to deal with different outputs.In recent years, multisensor data fusion has developed rapidly; when dealing with situation fusion, many scholars also use nonprobabilistic fusion methods such as random set [132–134], rough set [135–138], fuzzy logic [139–142], and Dempster–Shafer [77, 143, 144] to achieve ideal results.


## 6.3.1. Artificial Neural Network (ANN)

ANN is a nonprogrammed, nonlinear adaptive, and brain-style parallel distributed information processing system proposed on the basis of modern neuroscience. The essence is through the transformation of the network structure and dynamic behavior, with varying degrees at different levels, to imitate the human brain nervous system to process information [106, 107]. A neural network is a computational model, which is composed of a large number of neurons (nodes) connected to each other. The connection mode of the neurons is different, and the composed network is also different. The neuronal structure is shown in Figure 8(a) [108, 109]. The calculation model of the artificial neural network is shown in Figure 8(b) [110, 111].Figure 8
Artificial neural network. (a) Neuronal structure. (b) Artificial neural network computing model.
(a)(b)In the figure,a1 − an indicates the components of the input vector, w1−wn indicates the weight of each neuron’s synapse, b indicates the bias value, f is the transfer function (usually a nonlinear function), t is the nerve. The output of the element is t=fWA′+b, W is the weight vector, A is the input vector, and A′ is the transpose of the A vector. It can be seen that the function of the neuron is to obtain the scalar result of the nonlinear transfer function after the inner product of the weight vector and the input vector are transposed.


## 6.3.2. Deep Learning

Deep learning is derived from artificial neural network, which is a general term for a class of pattern analysis methods. It has made rich achievements in data mining, machine learning, natural language processing, and other related fields [112]. The purpose of studying deep learning is to establish a neural network that imitates human brain mechanism to interpret data (such as image, sound, and text) [113]. Deep learning includes supervised learning and unsupervised learning. Classical learning models include convolutional neural network (CNN), deep belief network (DBN), and stack automatically encodes network (SAEN) [114]. At present, deep network has been successfully applied to the fusion of single-mode data (such as text and image) and has also been rapidly developed in the fusion of multimode data (such as video) [115, 116].①
Convolutional neural network model: CNN is a kind of feedforward neural network with convolution calculation and depth structure. It has the ability of representation learning and is one of the representative algorithms of depth learning [117, 118], which is composed of the input layer, the hidden layer, and the output layer. It is a neural network that can be used for supervised learning and unsupervised learning, and its hidden layer has the characteristics of less computation [119–121].②
Deep belief network model: the DBN model can also be interpreted as a Bayesian probability generation model [122], which is a multihidden layer neural network composed of multiple restricted Boltzmann machines (RBM) “series” stacks. In the network and stack, the hidden layer of the previous RBM is the explicit layer of the next RBM, and the output of the previous RBM is the input of the next RBM, see Figure 9(a). In the course of their training, the previous RBM must be fully trained before the current RBM of the layer up to the last layer [123]. After layer-by-layer stacking of RBM, the DBN model can extract features layer by layer from the original data and obtain some high-level expressions [124, 125], see Figure 9(b).③
Stack automatically encodes network model: the structure of SAEN is similar to that of DBN, consisting of a stack of several structural units. The difference between the two is that the structural unit of SAEN is autoencoder, while the structural unit of DBN is RBM. The self-encoder is composed of a three-layer network. The input layer and the hidden layer form an encoder, which converts the input signalx into a. The hidden layer and the output layer constitute a decoder, which transforms the code into an output signal y; a multilayer sparse self-encoder can form a stacked self-encoder. That is, the output of the sparse self-encoder of the previous layer is used as the input of the self-encoder of the subsequent layer [118].Figure 9
(a) Deep belief network. (b) Deep belief network training process.
(a)(b)


## 6.3.3. Fuzzy Set Theory

Fuzzy set theory refers to the use of mathematics to describe fuzzy concepts and extend the exact set to fuzzy sets from the extension. It is also called fuzzy mathematics. Mathematically, it can eliminate the imprisonment that the computer cannot handle fuzzy concepts [126]. The proposal of “membership function” breaks through the absolute relationship of belonging or not belonging to in the classic set theory and describes the ambiguity of things [127, 128].Definition of ambiguity [128, 129]: a measure of the ambiguity of fuzzy set A, which is reflected as the degree of ambiguity of A, is intuitively defined.Let mapD: F (U) ⟶ [0, 1], where D is the ambiguity function defined on F (U); then, D (A) is the ambiguity of fuzzy set A, which should have the following five characteristics:①
Clarity:D (A) = 0 if and only if A ∈ P (U) (the ambiguity of a classical set is always 0)②
Fuzziness:D (A) = 1 if and only if ∀u ∈ U have A (u) = 0.5 (the fuzzy set with membership degree of 0.5 is the fuzziest)③
Monotonicity: ∀u ∈ U if A (u) ≤ B (u) ≤ 0.5, or A (u) ≥ B (u) ≥ 0.5 and D (A) ≤ D (B)④
Symmetry: ∀A ∈ F (U); there are D (A) = D (Ā) (the complements have the same degree of ambiguity)⑤
Additivity:DA∪B+DA∩B = D (A) + D (B)


## 6.3.4. Decision Tree

The machine learning technology of generating decision tree from data is called decision tree for short. It is a basic classification and regression method [130]. It is a graph theory method of intuitively using probability analysis; that is, on the basis of known occurrence probability of various situations, it can classify its objects or mine data by constructing decision tree [131]. The output of decision tree is single. When facing complex output, multiple independent decision trees can be established to deal with different outputs.In recent years, multisensor data fusion has developed rapidly; when dealing with situation fusion, many scholars also use nonprobabilistic fusion methods such as random set [132–134], rough set [135–138], fuzzy logic [139–142], and Dempster–Shafer [77, 143, 144] to achieve ideal results.


## 6.4. Knowledge Fusion and Process Optimization

Knowledge fusion is the fusion of sensor data and supporting database data. Process optimization refers to the global optimization process based on knowledge fusion.Knowledge fusion includes selection and automatic reasoning. Selection is mainly reflected in the selection of fusion mode and method, with emphasis on location information fusion and parameter data fusion. Automatic reasoning technology is to interpret the observed data environment, the relationship between observed entities, and the hierarchical grouping of targets or objects according to the actual rules, framework, and scripts of the knowledge fusion process to predict the future behavior of a target or entity.Process optimization is usually realized by “effect theory” [145, 146]; that is, a variety of system evaluation indexes and methods are used to monitor and evaluate the performance of each link (subsystem) and form an effective resource allocation scheme, which is equivalent to the feedback part of the whole system.The effectiveness evaluation of the data fusion system is generally quantitatively evaluated by Monte Carlo simulation [147, 148] or covariance error analysis technology [18, 149]. To optimize the data fusion system, the following basic issues must be considered and solved [150, 151]: (1) choose what algorithm or technology is the most suitable and optimal; (2) choose which fusion framework to use (that is, where the data flow is processed in the fusion process) is most appropriate; (3) select which sensor integration method can extract the maximum amount of information; (4) ensure the actual accuracy that each process of data fusion can achieve; (5) optimize the fusion process in a dynamic sense; (6) deal with the impact of the data collection environment; (7) improve the conditions of system operation.


## 6.5. Multisensor Data-Generalized Fusion-Proposed Method

The data fusion theory and algorithm summarized in Sections6.2 and 6.3 have strong application object, relatively isolated application field, and weak interoperability and lack a unified, reliable, efficient, and flexible idea of generalized fusion; in order to solve this very challenging problem, we tentatively carried out some research:(1)
The generalized fusion method of multisensor homogeneous data is as follows:Homogeneous data fusion of the homogeneous multisensor system has been very mature in the past research. It belongs to the source data level fusion in the generalized fusion model, and its universal algorithm has been described in detail in Section6.1.(2)
Tensor-based generalized fusion method for multisensor heterogeneous data:①
Consistent tensor fusion (CTF):In 2016, Kuang et al. proposed the unified tensor fusion (UTF) model (Figure10) [152]; that is, the stretching method is used to model heterogeneous data as subtensors, and the two-layer (whole layer and inner layer) model is inserted to generate a unified tensor, which better realizes the consistent representation of large (multimodal/heterogeneous) data. It provides a foundation for the research of multisensor heterogeneous data fusion.In Figure10, the global layer is a third-order tensor including time TI, space SP, and state U, which is expressed as It×Is×Iu. The inner layer represents three subtensors of three spatial data, and the subtensors are embedded into the whole layer to obtain a unified tensor.Most of the past studies on tensors are based on single-mode and low-order data. Tensor is a high-order generalization of the matrix and represents the variability of various types of data in a high dimension. Tensor factorization is the joint matrix factorization of coupling different factors by sharing factors [53]. When this idea is applied to multisensor heterogeneous data fusion, multisensor heterogeneous data can be fused by tensor decomposition of factors in each subsystem dataset.Given heterogeneous single-sensor integration platformS consists of four vectors: lidar, depth camera, GPS, and support database. The system model is YHSIS∈sLR,sDC,sGPS,sSD. The frequency domain data fusion process is as follows:Based on the UTF model, firstly, the consistency of heterogeneous data collected by sensors is expressed as the tensor; it can be seen that, after the consistency tensor representation of heterogeneous data, the diversity of order and dimension still exists. Based on the set matrix factorization (CMF) [153] equation proposed by Singh and Gordon, the feature tensor is fused to obtain the feature subtensor. Then, the tensor expansion operator (Teo) is used to obtain the higher order unified tensor: T∈RIu×Ic×It×Ix×⋯×Iy×Iz. Finally, the unified data tensor (UDT) is implemented to achieve heterogeneous fusion, and the consistent tensor fusion calculation model (formulas (9)–(11)) is obtained. Figure 11 shows the consistent tensor fusion process.Consistency tensor representation: the LIDAR point cloud data is represented as the second-order tensorTpoint cloud∈RIx×Iy. The depth camera video data is represented as fourth-order tensor Tvideo∈RIh×Ic×If×Iw; GPS data is expressed as a third-order tensor TGPS∈RIec×Ien×Ier. The database data can be expressed as the third-order tensor TXML∈RIt×Iid×Iy.CMF equation:(9)fA,B,W=X−ABT2+Y−AWT2.Tensor extension operator:(10)f:A×⟶B⟶C,C∈RIt×Is×Iu×I1×I2.Unified data sheet quantification:(11)f:du∪dsemi∪ds⟶Tu∪Tsemi∪Ts︸T,whereX∈RI×J and Y∈RI×K are given matrices with common mode I, A∈RIt×Is×Iu×I1, B∈RIt×Is×Iu×I2, du represents unstructured data, dsemi stands for semistructured data, and ds represents structured data.②
Constrained tensor fusion (CTF)Given heterogeneous single-sensor integration platformS consists of four vectors: lidar, depth camera, GPS, and support database. The system model is YHSIS∈sLR,sDC,sGPS,sSD, and its frequency domain data fusion process is as follows:Migration data fusion is the fusion of time domain and spatial domain data, which can be achieved by adding probability constraints on the basis of consistency tensor fusion (CTF). The so-called increase of probability constraint is to increase the cooperation between multivariable Markov chain (MC) and tensor, transform tensor in multiple steps, fuse the space, time, and supporting data of the system migration process, realize the system migration fusion, and obtain the constrained tensor fusion (CTF) calculation model (equations (12) and (15)). The position sequence in GPS data can constitute a spatial transformation model. The corresponding transformation matrix is the row random matrix, and the sum of elements in each row of the matrix is equal to 1.Spatial data fusion: spatial data fusion is realized by the Markov model. MC includes the following: (i) according to the theory of the discrete stochastic process, the transformation matrix corresponds to the stationary distribution, that is, any initial distribution vector can converge to the steady distribution vector after infinitely multiplying with the transformation matrix; (ii) matrix theory shows that the steady distribution vector is the eigenvector with the largest eigenvalue corresponding to the transformation matrix. Referring to [154], the transformation of spatial tensor is shown in Figure 12.Figure12 shows the mobility model of the platform. The elements in the matrix represent the probability of the platform moving from one point to another.Spatiotemporal data fusion: the fusion of time data and spatial data needs spatiotemporal tensor transformation. Mobile behavior is always related to time, and time is an important element of the mobile behavior model. The motion time is discretely represented asi, and its new state space is composed of pose information and time information and is represented as S=T1,P1,Ti,PJ,…,TI,PJ. From [154], the space-time tensor transformation is shown in Figure 13.Figure13 shows the migration model of the platform. The elements of the tensor represent the probability of the platform moving from one point to another at a certain point. Based on the transformation of space tensor, the fourth-order space-time transition tensor TST can be obtained by combining time information through equation (12).Temporal and spatial information fusion:(12)Tt1l1t2l2=countt1,l1⟶t2,l2countt1,l1.Support database fusion: the influence degree of support data is quantified as the greater the correlation coefficient between fusion data and support data tensor, the greater the influence. The knowledge influence coefficient is described as(13)ρKI=∑i=1nKi−K¯Ii−I¯∑i=1nKi−K¯2∑i=1nIi−I¯2.Figure 10
Unified tensor representation model.Figure 11
Consistency tensor fusion process.Figure 12
Space tensor transformation.Figure 13
Temporal and spatial tensor transformation.The correlation coefficient matrixΓ=RKN×IN can be obtained by calculating the correlation coefficients between global tensors. By normalizing the rows in the correlation coefficient matrix Γ, the influence matrix (IM) Λ can be obtained:(14)Γ=ρK1I1ρK1I2…ρK1InρK2I1ρK2I2…ρK2In⋮⋮⋮⋮ρKnI1ρKnI2…ρKnIn,Λ=λK1I1λK1I2…λK1InλK2I1λK2I2…λK2In⋮⋮⋮⋮λKnI1λKnI2…λKnIn.By fusing the influence matrix of supporting data into the spatiotemporal transformation tensor, the system migration behavior fusion can be obtained:(15)A=Λ⊙TST,where ⊙ is the Kronecker product.


## 7. Development Trend and Urgent Difficulties to Be Solved

At present, most of the work on multisensor data fusion in academic circles is carried out for specific applications, without forming the basic theoretical framework and algorithm system. Therefore, the establishment of the basic theoretical framework and the generalized algorithm system of multisensor data fusion is the main trend of the future development of this field. Based on its future development trend, the following problems are urgently needed to be solved:(1)
Establish the optimal management scheme of sensor resources. In a multisensor data fusion system, sensing is the source of fusion data; the number, attributes, and integration methods of sensors directly determine the quality of the fusion data, which is one of the key factors affecting the fusion result. The sensor resource optimization program will optimize the scheduling of sensor resources from three aspects, space management, time management, and mode management, so that they can be used to the fullest and most rationally and achieve the optimal performance of the sensor system.(2)
Evaluation criterion multisensor system is established; avoid blind design fusion system. The fault tolerance or robustness of the multisensor integrated system directly affects the quality of data acquisition and overcomes the difficulties of sensor measurement error modeling, real-time response of the system to complex dynamic environment, establishment of knowledge base, and reasonable arrangement of sensors. It is the key to avoid blind design of the fusion system to design the generalized sensor integration scheme and perfect the generalized fusion architecture of multisensor data.(3)
Establish theories and methods that can fully and effectively utilize multiple sensors to provide redundant information. The more sufficient the amount of information is, the closer the fusion result is to the essence of things. With the help of some new technologies in other fields, theories, and algorithms that can fully and effectively utilize redundant information of multisensors, reduce the impact of data defects (imprecise and uncertain) and alleviate outliers, and false data [155] are developed, which is one of the key factors to improve the accuracy of data fusion.(4)
Establish criteria for judging data fusion to reduce the ambiguity of data association; inconsistent fusion data, also known as data association ambiguity, is one of the main obstacles to overcome in data fusion. In the process of multisensor data fusion, data consistency is the key factor that affects the fusion result. Data association is the key to ensuring the consistency of fused data, that is, to ensure that the fused information is information about the same goal or phenomenon.(5)
Develop and improve the basic theory of data fusion. Academia has conducted extensive research on data fusion technology and has achieved a lot of successful experience, but until today, the theoretical foundation is still incomplete, and effective basic algorithms are still missing. The development and improvement of the basic theory of data fusion are key factors for the rapid development of this field.(6)
Improve the fusion algorithm to improve the fusion performance. Fusion algorithm is the core of data fusion. In the new development, introducing new mathematical methods to improve the fusion algorithm is the long-cherished wish of countless scholars. The introduction of modern statistical theory, random set theory, fuzzy set theory, rough set theory, Bayes theory, evidence theory, support vector machine, and other intelligent computing technologies will bring new development opportunities to the state estimation of nonlinear non-Gauss systems and heterogeneous data fusion.(7)
Establish a knowledge base for data fusion applications. In the field of data fusion, it is necessary to establish databases and knowledge bases, form-optimized storage mechanisms, high-speed parallel retrieval and reasoning mechanisms, etc., and to improve the operating efficiency of the cluster fusion system and the reliability of the fusion results.(8)
Established generalized fusion algorithm system of multisensor data. The generalized algorithm based on the basic integrated structure model of the multisensor should have the advantages of reducing data defects, alleviating abnormal values and false data, processing highly conflicting data, processing data multimodality, processing data correlation, processing data alignment/registration, and processing data association. It also should have other capabilities as to select a fusion framework for complex system data fusion, implement timing operations, process static and dynamic data states [156], and compress data dimensions [30].



---

*Source: 5471242-2021-10-15.xml*