[Travel] Only register permission if the home/warp is public

Reduces the startuptime quite much