this comes from a relationship between AUC & Mann Whitney test.  https://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U#Area-under-curve_.28AUC.29_statistic_for_ROC_curves

What do you mean by "should never go below 0.5"?? It certainly is possible for it to go below 0.5, it just means you have a classifier that is worse than random.  So ideally you shouldn't ever learn such a classifier, but no reason the formula shouldn't allow it.