Fix desynched user modes on new client connect by reworking how we track them.