Neural Network System for Multimodal and Multilabel Classification: A Sleep Diagnosis Example
Creators: name: Cheng, Yi-Hsuan
Publisher: name: RMIT University
Year: 2024
Subjects: classificationCode: 460207
subject: Modelling and simulation

description: Machine Learning (ML) and Deep Learning (DL), in particular, include robust and diverse data analysis techniques that allow us to extract information and make decisions. The accuracy of these decisions depends on the amount and quality of data used to train the ML models. Large numbers of expert-labelled data from a single source are not always available. Moreover, decisions based on a single source or type (modality) of information are often not as accurate as decisions considering multiple types of data coming from different sources. Information fusion integrates information from multiple sources to provide a more comprehensive and accurate understanding of a given problem.
Multiple studies indicated that incorporating information fusion into ML models can be beneficial and improve the quality of decisions. However, most existing ML works show only the benefits of cross-modality fusions crafted for specific applications. The study presented in this thesis undertakes a systematic and generalised approach to information fusion, allowing simultaneous fusion across different factors, such as cross-modality and cross-label. Given the extremely high complexity of the multi-fusion task, the methods proposed in this thesis were validated using a single database, the Cyclic Alternating Pattern (CAP) Sleep Database, which consists of a collection of 108 polysomnographic recordings taken at the Sleep Disorders Center of the Ospedale Maggiore of Parma, Italy. It included an extensive collection of sleep data recordings, and it is commonly used in automatic sleep diagnosis research. 
In the context of the CAP Sleep Database, there were two possible types of information fusion. The first type was the fusion of information from different modality sensors such as electrocardiograms- (ECG), electroencephalogram- (EEG), and electromyogram (EMG), which is referred to as cross-modality fusion. The second type was the fusion of information provided by different types of labels, such as sleep stage (6 categories) and sleep disorder (8 categories). It is referred to as cross-label fusion. This thesis proposes a new Multimodal- and Multilabel Decision-Making System (MML-DMS). Two versions of the MML-DMS were tested: Multimodal- and Multilabel Decision-Making System, version 1 (MML-DMS1) and Multimodal- and Multilabel Decision-Making System, version 2 (MML-DMS2). The MML-DMS1 offered a cross-modality fusion of information, whereas the MML-DMS2 offered both cross-modality and cross-label fusion. Validation experiments conducted on the CAP Sleep Database showed that both methods, MMLDMS1 and MML-DMS2, outperform baseline single-modality CNN classifiers. It was also demonstrated that the MML-DMS2 outperformed the MML-DMS1 in terms of testing accuracy for both multimodal and multilabel classification, indicating that the combined cross-modality and cross-label fusion is more powerful than the cross-modality fusion only. A comparison with related sleep classification studies showed that the results outperformed existing state-of-the-art techniques. Future research will investigate how the proposed approach can be adapted to solve various problems in stock market predictions, pilot training, decision-making for autonomous vehicles, and medical diagnostics.