Probabilistic Linear Discriminant Analysis (PLDA) Explained
By: Prachi Singh
Tags: Getting Started, Gmm, Probabilistic Lda, Generative Model

What is Probabilistic LDA ?

Let x={x₁,x₂,…,xₙ} be the D-dimensional observations or data samples. Probabilistic LDA or PLDA is a generative model which assumes that given data samples are generated from a distribution. We need to find the parameters of model which best describe the training data. The choice of distribution from which data is assumed to be generated is based on two factors: (1) It should represent different type of data (2) Computation of parameters is simple and fast. The most popular distribution which satisfies these conditions is Gaussian. Below figure shows the probability distribution function (pdf) of Gaussian, contour and samples generated from it.

Source: Plots by author

show code

In order to cluster data into classes, we need to represent each class with separate Gaussian distribution hence we can use Gaussian Mixture Model (GMM). GMM is a weighted mixture of Gaussians with different means and covariances for each Gaussian where each mixture can represent each class.

Probability distribution function (pdf) of GMM , is given as :

Source:Plots by author. pdf of GMM with πₖ, μₖ, Φₖ are weight, mean and covariance of k-th Gaussian

show code

Let y is a latent (hidden) class variable which represents the mean of a class/mixture in GMM. Now given this class variable y , probability of generating data sample x is given as:

Where Φw represents within class covariance of the given class. This states that once we know the class parameters of Gaussian, we can generate samples of this class. Here class variable y itself is assumed to be generated from separate distribution. The probability of generating a particular instance of y which represents a class from an assumed distribution is called prior probability.

LDA is also modeled as GMM, where mean of Gaussian of each mixture is the sample mean of training data belonging to respective class and prior probability of y is discrete, where y can take only discrete values given as

This will generate GMM which is discussed above. Maximising likelihood of this model with respect to parameters {πₖ, μₖ, Φw} recovers the standard LDA projections. But, in order to handle classes not seen during training we need to modify prior and make it continuous where y can take any real values generated from the distribution given as

This shows that latent variable y for each class can be generated using the Gaussian distribution with mean m and between-class covariance Φb . Hence it is called as probabilistic LDA. This is better explained using the figures shown below:

Source: Image by author. Small ellipses represent contours of Gaussian distribution for each class with mean at the center. Data points are represented by cross marks. The grey ellipse generates mean of each class from Gaussian distribution.

Source: Images of people are from LFW dataset, representation by author. Plots are for illustration.

Samples y1 and y2 are generated from Gaussian distribution eq (2), which represent the person identity. We sample examples x1 and x2 of each class (person) which represent different orientation of the person using eq (1) with y1 and y2 as the mean.

Latent Space

Goal of PLDA is to project data samples to a latent space such that samples from same class are modeled using same distribution. These projections are represented using latent variables which will be discussed in this section.

As discussed earlier, is Φb is between-class covariance positive semi-definite matrix and Φw is within-class covariance positive definite matrix. We obtain a transformation matrix V which converts Φw and Φb to diagonal matrices simultaneously, given as:

where I is identity matrix and Ψ is diagonal matrix. Thus we decorrelated each dimension of the data samples. Now parameters of PLDA model are {m, A, Ψ}.

You can refer to below derivations to obtain above equations. It requires knowledge of Linear algebra concepts like eigen values and eigen vectors of a matrix and eigen value decomposition.

show derivation

Let u, v are Gaussian random variables in latent space defined as,

We can find relation between data samples x, class variable y and these latent variables as follows:

Thus u represents example of class and v represents class variable in the projected space. The relationships in eq(5) and eq(6) are represented in the form of flow chart as shown below:

Source: Image by author. PLDA models the class center v and examples u1,u2 in latent space where variables are independent. Examples x1,x2 in the original feature space is related to its latent representation u via an invertible transformation A

If interested, you can refer to the derivation below:

Applications

PLDA allows to make inference about the classes not present during training. One example is speaker recognition. Model parameters are learned from training data but the model should handle examples from speakers not present during training. Some of the tasks in which PLDA can be used are discussed here.

Classification:

We have a set of examples xg ∈(x₁,x₂,…,xM) one from each M classes. Now given a probe example xp , task is find it belongs to which of the classes. This is determined by maximizing the likelihood. First, project the examples into latent space using eq(6) as

This de-correlates the data as the covariance of u is I. P(up│ug ) gives probability of probe example coming from same class as example from known set.

Therefore, class C assigned to probe example is given as

Computation of P(up|ug)

Class Inference:

One of the advantages of PLDA is that, we can find class variable y from a single example of the class also. For an example x, we compute posterior probability of y given x denoted as p(y|x) which will again be a Gaussian. The estimate of y is obtained by maximising p(y|x) with respect to y. It is nothing but the mean of Gaussian p(y|x). It can be written as follows:

Hypothesis testing:

Given two examples (u1,u2 in latent space) from unseen class(es), if we need to find whether they belong to same class or not , then we compute likelihood ratio R based on two hypothesis.

Clustering:

PLDA is also used to cluster examples into groups. Based on the log likelihood ratio R or PLDA scores, we compare each example with all the other examples. This will create a PLDA scores matrix like a similarity scores matrix which can used to perform clustering using available algorithms like k-means, agglomerative clustering, etc.

Example

Speaker diarization: The task of partitioning the input audio stream ​ into segments based on speaker sources.

The procedure is as follows

Divide audio into short segments such that each segment contains only one speaker. Extract features for each segment Compute PLDA scores matrix using pre-trained PLDA Perform clustering using scores matrix

Source: Image by author. Speaker Diarization pipeline.

I used Speech recognition toolkit Kaldi to complete step 1 and 2 and train PLDA as given below.

Following code involve : Read features -> Apply PCA -> Project in PLDA latent space-> Computes PLDA scores matrix. Complete code with features and pre-trained model can be found in GitHub.

show code

Analysis

PLDA latent representations(u): Below plot shows the effect of projecting data into PLDA latent space. We can see that, when we project the data into PLDA latent, it becomes separable into speakers even though PLDA model has never seen audio from these speakers.

2d projections of 128-d features. Left side plot is PCA transformed embeddings. Left side plot is PLDA latent representations. Each colour represents one speaker. Plot by author.

show code

2. Log likelihood ratio or PLDA scores matrix: We can compute a similarity score matrix S by finding similarity scores S(i,j) between xᵢ and xⱼ for all i and j segments extracted from the audio. Below plot shows Normalized cosine scores matrix and PLDA scores matrix, normalized by dividing with highest score in the matrix. It depicts higher contrast in PLDA scores showing higher confidence in making decision of same and different speakers. Lighter colours (shades of yellow) indicates high scores whereas dark colours (shades of blue) indicates low scores. We can see blocks of light and colour which helps to identify same speaker and different speaker regions easily.

Comparison of Cosine affinity scores matrix and PLDA affinity scores matrix. Plot by author.

show code

3. Histogram: Below plot shows distribution of PLDA scores. Higher count is present on the extreme which helps in better clustering.

x-axis: Normalized PLDA scores, y-axis: Count. Plot by author.

show code

Summary