Correctly close unused worker sockets