Demystifying Generative Models by Generating Passwords — Part 1
By: Andreas Pogiatzis
Tags: Generative Model, Artificial Intelligence, Machine Learning, Data Science, Cybersecurity

Interestingly, a generative model would first learn how to draw a doodle of humans and animals and when an unseen doodle is given, it will draw an animal and a human and compare it with what’s given.

Alternatively, a discriminative model would learn the nuances of each class, i.e. animals have tails, the posture of humans, shapes etc… and use these to discriminate between the two classes when a new doodle is given without learning anything about drawing. Simple right?

Although discriminative modelling is traditionally preferred due to its simplicity [2], generative models come with their own advantages as well and are becoming increasingly popular. They behave particularly well when there are missing data, or even for detecting outliers. Even, after all, we may want to generate new data which is something that discriminative models are unable to do. In fact, it has been only recently when generative models have reached realistic levels of generated content (See figure below).

Face generation with generative modelling. Source: David Foster. “Generative Deep Learning.” [3]

Probabilities

I try to avoid the mathematical jargon as much as possible but it’s essential to introduce some mathematical concepts in order to formulate discriminative and generative models more clearly.

Suppose that we have some observations data X tagged with a label Y. From the probabilistic perspective, a discriminative classifier will attempt to model the conditional probability of P(Y|X). That is, given observations X, what is the probability that I have label Y? (Don’t confuse the conditional probability with the joint probability P(A, B) which means the probability of A and B happening at the same time). On the other hand, generative modelling, attempts to directly model the probability P(X), and in simple terms, what is the probability of getting observations X in the first place? Note that the labels are not necessary for this model, however, if we want to perform classification, the labels can be used to define P(X|Y).

Simplified Generative Modelling

In order to practically comprehend what a generative model is, let’s imagine an extremely simplified scenario with a generative model in only two dimensions. The example is inspired by David Foster’s “Generative Deep Learning” book which I highly recommend!

Consider some points generated by a rule called Pdata on a two-dimensional space below:

The purpose of a generative model is to generate a new point X = (x, y) that looks like it has been generated by Pdata. Thus, let’s construct an estimation of Pdata called Pmodel. A possible estimation could be a uniform probability distribution of generating a point anywhere within the orange bounding box and zero chance of generating a point outside of that as illustrated below.

Coming back to the generative model description from previous sections, we have just created a model that defines how the data points are created in the first place! Although overly simple, you have just developed a generative model!

Now let’s dive a little bit deeper and observe how our model performs with respect to the true Pdata distribution. Note that the original Pdata represents the probability distribution of trees growing in the land (see map below). There is roughly an equal chance of a tree being grown on a spot where there there is soil but zero chance in the water.

This underlines another very desirable trait of Data Scientist.The ability to have high level awareness of what the prior data distribution should be is directly related to the domain expertise of the Data Scientist. For this example, we knew that from natural experience that trees do not grow on water! Similarly, in a real world scenario, most of the times it is helpful for the Data Scientist to take a step back and analyse the problem domain, such that in the future it becomes possible to reason about outcomes of data models.

Notably, although Pmodel is an oversimplification of Pdata it manages to grasp the main mechanics of the original distribution. By sampling 3 points A, B, C from Pmodel, it is obvious that our model isn’t perfect. Point C is cannot be generated from Pdata, yet artificial points A and B are indistinguishable from real ones. This is the primary purpose of generative models. Generating new data, which are different from existing ones, yet look like they have been created by the same rules.

Probabilistic vs Deep Learning Approaches

In this section, we shall explore why Deep Learning has been a major contribution to the rapid evolution of generative models with practical examples. But first, it is essential that probabilistic methods are presented so that there is a benchmark to compare with.

Maximum Likelihood Estimation

Maximum Likelihood Estimation is a statistical technique that can be used to calculate parameters such that the likelihood of our model generating the given data is maximised. This may sound a little bit confusing initially so let’s put it into the context of the example above.

Imagine that the orange box above is a probability density function, in simple terms a function that given a point in our sample space (x,y on the map) returns a value from 0 to 1, a probability. Intuitively, the sum of all points on the probability density function (the integral) should always equal to 1. Coming back to our example, since the box represents a uniform distribution, the probability outside of the bounding box is 0 and inside is a constant. In mathematical notation, it can be represented as follows:

Moving on, let θ be a set of 4 parameters {θ1, θ2, θ3, θ4}, a likelihood function L(θ| x) attempts to answer the following question:

How likely are some concrete values of the parameters given the point x?

But you probably wondering what are these parameters for? These come from a well-known area of statistics called parametric modelling. In other words, it’s a way to represent a probability distribution with a finite set of parameters. In this context, our probability distribution (the box!) can be modelled with four parameters, top-left corner point (θ1, θ2) and right-bottom corner point (θ3, θ4). A more realistic example would be a gaussian(normal) distribution which has two parameters, the mean μ, and standard deviation σ.

More precisely, the likelihood function is defined as follows:

Which is the probability at x returned by the probability density function parameterised by θ.

Now if instead of just one you have n data points Xn = {x1, x2, …, xn} then likelihood can be evaluated as:

Nevertheless, because this product can be quite cumbersome to work with, it can be as well be represented in a logarithm form where the product becomes a sum and is thus easier to differentiate:

Just as a sidenote, transforming to the logarithm form is only possible because the natural logarithm is a monotonically increasing function. In plain English as x increases, y increases as well so the maximum does not change.

Since having all these, it is possible to come back to the initial concept, Maximum Likelihood Estimation. Or simply, finding some optimal parameters θ that explain best the dataset X. Which it is equivalent as saying these parameters are the most likely to have been used to model the probability distribution that generated X. Formally, this is defined as:

All these should give you a clearer picture of how Maximum Likelihood Estimation can be used to create a generative model but still, this is all theory and the example used above was simply to have a visualisation reference for easier understanding. In the next section, we will apply what has been described here to develop a somewhat useful generative model.

Naive Bayes for Password Generation

Once again let’s construct a narrative and suppose that you have a really hard time coming up with passwords so you came up with the brilliant idea of creating a generative model that creates passwords for you!

Source: Giphy.com

Keep in mind that this is a hypothetical application to apply the models described here, in practice and get hands-on experience rather than just explaining the theory behind it. More importantly, there are also real-world applications for the methodologies and techniques showed here such as domain-specific password brute-forcing, replacing the random password generation process to something more easier to remember but with equivalent security etc…. These are not described here but are provided for the reader as an insight for further exploration.

You have created a list of elements that your passwords would like to contain:

Words: pass, cafe, bingo, hyper, leet, hackerman, ninja, babe

Numeric: 1234, 111, 777, 000, 0101, 2019, 2018, 1, 2, 10, 123

Special: !, @, $, %, *, _

Capital: A, E, T, P

And defined a standard format that your password would like to follow:

Capital + Word + Numeric + Special

Fundamentally, each of your passwords can be defined using these 4 features which are always in the same order: A capital letter, A word, A numeric value and a special character. All taken from the list provided above.

In total, this equals to 8 * 11 * 6 * 4 = 2112 combinations! Of course, you could simply choose a random element for each feature in your password and be done with it BUT you would like to avoid a random password every time as it will become harder to remember. On the contrary, it would be ideal to use some passwords more often than others based on the observations of your previous passwords. In more formal terms, there should be a distribution Ppass that favours some elements over others.

Initially, you gathered 30 passwords that you have used in the past following the same format:

Here is a peek from the first 10 passwords of the dataset:

Let’s ask ourselves a question. How is the probability of having password x defined?

Recall that password x is defined by 4 features. This is equivalent to the map point we described in the simple example above but instead of x,y it now has 4 dimensions — x1,x2,x3,x4.

Intuitively, we can say that the answer is the probability of having x1, x2, x3, x4 happening together at the same time. So:

This can be expanded further to conditional probabilities using the chain rule. Consequently, the expression above becomes:

To move on with the Naive Bayes model, we have to make a strong assumption. That each feature is independent of the others. That is, the capital letter has nothing to do with the choice of the word or numeric value. This is a quite naive assumption, hence the name. More explicitly:

With this in mind, the expression derived from the chain rule earlier is simplified to:

But wait for a second, when we were describing the Maximum Likelihood Estimation, the probability density function was parameterised by the distribution parameter set θ which had 4 parameters since we were aware that the distribution can be described as a box. But, in this case, how is the probability parameterised then?

Because our dataset consists of discrete values, or more abstractly if you will, elements from a finite set of values we can use the multinomial distribution and as a result Multinomial Naive Bayes. In such a case, we can simply assign a parameter to each value of every feature of the model which results in a total of 8+11+6+4-4=25 parameters.

The -4 is to compensate for the last value of each feature because it doesn’t have to be calculated as it is forced so that the sum equal to 1.

Moreover, to find the Maximum Likelihood Estimations in a Multinomial distribution we simply divide the number of occurrences of each feature value with the total number of observations, as shown below:

Maximum Likelihood Estimation of feature parameters.

Now that we have calculated the Maximum Likelihood Estimates for each feature, it is possible to sample a value from each feature and concatenate them together to get a new password! Here is my list of 10 new passwords generated by the model:

Note that our model has generated passwords there were never in the dataset initially (Ahyper777!, Apass10$ …. )!

You can find the relevant code for the generative Naive Bayes model in the Google Colab link below:

Sidenotes — Optional Reading

Of course, there are some details that have been omitted in order to keep the post as concise as possible but I will mention them here for a reference in case you desire to look them up by yourself.

For instance: what would happen if an element does not appear in the initial dataset. Let’s say the word “hyper” is not used in any of the previous passwords? Then the MLE would return a 0 probability and therefore there will be no chance to be generated by the model. To alleviate this issue there are smoothing techniques such as Laplace or Lidstone’s smoothing.

Furthermore, we have only covered generating data and not classification since the concept of these posts is generative modelling. Naive Bayes can also be used as a classifier with the use of Bayes Theorem, but for that, it is required that the data points have labels.

Moving to Part 2

Congratulations! You have just created a real probabilistic generative model from scratch! Of course, this was a very simple approach but hopefully, you got the idea!

Recall though that this worked only because we have been relying on the Naive Bayes independence assumption, and that’s fine! For some cases, it works exceptionally well, but in the next part, we will explore what happens when this assumption collapses, and how Deep Learning can come to the rescue!

References

[1] B. Hitaj, P. Gasti, G. Ateniese, and F. Perez-Cruz, “PassGAN: A Deep Learning Approach for Password Guessing,” Applied Cryptography and Network Security Lecture Notes in Computer Science, pp. 217–237, 2019.

[2] Ng, Andrew Y., and Michael I. Jordan. “On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes.” Advances in neural information processing systems. 2002.

[3] Foster D. Generative Deep Learning: Teaching Machines To Paint, Write, Compose, And Play. 1st ed. O’Reilly; 2019.