# Strength Analysis of Real-Life Passwords Using Markov Models
## 5 Results
### Main Results
The results for all 66 pairs can be seen in Table 11. Based on the results we reject the null hypothesis (the median of the differences between two models is 0) for all 66 pairs and accept the alternative, stating that the median of the differences between two models is not equal to 0. This was, because the sample size and nature of data, somewhat expected. For this reason, we also included effect size [33]. Effect size is a measure of magnitude or size of an effect (the observed phenomenon). Effect size is calculated with the following formula:  
\[r=Z/\sqrt{N}, \tag{7}\]  
where \(r\) is the effect size, \(Z\) is the z-value produced in the sign test, and \(N\) is the number of observations. Bigger \(r\) value means a more substantial difference between the results of the two models. Using Cohen criteria, we interpret the size of the effect. Effect size between 0.1 and 0.3 is considered to be small, but definitely not trivial or something to be ignored. Bigger values that are smaller than 0.5 are said to have a medium effect, while r values over 0.5 represent a large effect. To simplify and make the effect size more understandable we also calculate the common language effect size (CL), also called probability of superiority. CL gives the probability that a random score from one model will be higher than its matching pair in the second model. CL is calculated by dividing the positive differences by the number of comparisons. The number of comparisons excludes ties (pairs where both models produce the same value). Effect size, its effect classification and CL can all be found in Table 11.  
Each row in the table represents a pair of models (Model 1 and Model 2). Next two columns are the Z and the \(p\)-value results of the sign test. Following are the metrics for the effect size. Let us for example take the first pair in Table 11. For the pair _10K_most_common_ and _10_million_passwords_ the effect size was calculated to be 0.1092. This result is higher than 0.1 and is therefore not insignificant, however, it is not especially high, so we mark the effect as small. CL is calculated by subtracting the model 2 results from model 1. The  
Figure 6: Four examples for the distribution of differences between two models. end percentage, therefore, tells us the probability of model 1 producing a higher strength evaluation than model 2 for a random password. Looking back at the previous example, the CL is 57.73%. This means _10K_most_common_ produced higher strength estimations in 57.73% of the cases, while _10_million_passwords_ only had the higher value in 42.27% of the results (excluding the ties). This brings us back to the definition of a more successful model. Generally, the model consistently producing smaller password strength estimations would be better. From the example, we can see that model _10_million_passwords_ more rarely generates the higher strength value of the two and for that reason, we would consider this model to be the better one. Because of the way the models are sorted into model 1 and 2 columns, the ones in the second column are always considered better. As a result, to find the best models we can simply just count the number of times a single model is written in the second column (higher the count, better the model). Following this rule, we find that _10_million_passwords_ is the best model, followed by _rockyou_, _plppb_, _tuscl_, _unipass_, _hotmail_, _singles_org_, _10k_most_common_, _myspace_, _faithwriters_, _passwords_, and the worst of the tested models _scribd_. We summarise this and order the models in Table 6.  
Ordering the models from the best to the worst is not very difficult, however, it should be noted that the differences between models are not even remotely the same. For this reason, we join models with similar results into groups. As stated before _10_million_passwords_ produces the best results, but _rockyou_ does not lag far behind. The effect size shows that they do in fact produce very similar results.