In fact, two expressions seem to have become blended here.
'to hold one's head up' meant to behave arrogantly or proudly, while 'to hold up one's head' referred to keeping one's dignity or self-respect.