Simplify organization of non-socket stuff

Next up: socket stuff