usb: gadget: Fix race condition in connect/disconnect