Only switch user if required