Avoid set_controller_port unless we use something other than JOYPAD.

Some cores do not properly range check. Avoid needlessly breaking.