Move the socket import to the not-win32 code path

Otherwise, we'd have to ship the socket module in the windows installer,
but it's not used on windows.
Remove an unused import of defaultdict