When we fail to get a USB reference we should just return, because
there are no more references held.

MFC after:	3 days