Final comment I promise. There's an interesting question of what to do about ties in the keys. If all of my predictions are 0/1 for example, then the ranking of 0/1 true labels within those predictions ends up being pretty arbitrary, and so does the AUC. I don't think this formula deals with that case, and you could argue it's a degenerate case or argue it's actually somewhat common. To make it at least deterministic, maybe a secondary sort by true label? I don't know whether to be optimistic or pessimistic, and sort the positives first or last. Tough one.