Multimedia Analysis and Fusion via Wasserstein Barycenter

Many multimedia analysis algorithms rely on probability distributions that characterize audio or image features as generally high dimensions.For example, music analysis methods, such as automatic music transcription (AMT) [1] and music classification [2], in these applications, having sufficient similarity (or equivalent difference) between distributions becomes crucial.The classical distance or difference of probability density includes Kullback Leibler divergence, Kolmogorov distance, Bhattacharyya distance (also known as Hellinger distance), etc.Recently, the framework of optimal transportation and Wasserstein distance [3] are also called earth mover's distance (EMD) [4], which has aroused great interest in computer vision [5], machine learning [6] and data fusion.Wasserstein distance calculates the best warped starter to map the measure m to the second n for a given input probability.Optimality corresponds to a loss function that measures the predicted value of the displacement in the warped starter.Generally, considering the accumulation of m and n, Wasserstein distance calculates the definition of the displacement of every particle from traces of its mass to the displacement of m to n.In this paper, the applications of Wasserstein Barycenter [7] algorithm in music transcription and classification are discussed.The first application described in this paper is music transcription.In this study, we use non-negative matrix factorization (NMF) as the method of converting audio signal to musical instrument digital interface (MIDI) format and Wasserstein Barycenter as the algorithm of data fusion.The second is music classification, first we used jSymbolic software to extract the key features, and then input these features into our traditional machine learning model including eXtreme gradient boosting (XGB), back propagation neural network (BPNN), support vector machine (SVM).Finally, we propose Wasserstein Barycenter as the ensemble method of these models.Wasserstein distance, also known as EMD, is used to measure the distance between two distributions.Compared with Kullback Leibler (KL) and Jenson-Shannon (JS)-divergence, Wasserstein distance has the advantage that even if the support sets of two distributions do not overlap or overlap very little, it can still reflect the distance between the two distributions.The remaining sections are organized as follows: In Section 2, we first review some of the theorems in the literature and mainly present the theorems of Wasserstein Barycenter, methodology of music transcription and model of music classification.Section 3 conducts experiment result of music transcription and music classification.Section 4 introduces the evaluation methods and the comparison of different models.In Section 5, we summarize our work and present future research directions in the field. RELATED WORKMaterials show that the Wasserstein distance presents a useful methodology for quantifying geometric differences between the different distributions.In particular, they are mostly applied as variables in content-based image retrieval [8], modeling and visualization of image intensity value [9-12], estimated average probability metrics (i.e.Barycenter of gravity) [13,14], cancer detection [15,16], super resolution [17] and other applications.Recent advances in variation minimization [18,19], particle approximation [20], multi-scale schemes [21,22], and entropy regularization [5,6,23], transmission metrics can be effectively utilized to pattern recognition, machine learning and signal processing issues.Moreover, Wang et al. [10] describes a theory for computing the transport distance (expressed as