Revert "Revert "Revert "usb/usb3503: add back the gpio direction setup"""

This reverts commit 21f9fd7de46694cfc521021bc02062361110b477.