Fix being teleported when changing difficulty