Debugging fd/client removal.

src/dest reversed in here caused issues, but this is now
correct. Adding an assert as to why we're still not
getting the right fd back from the LIST_FOREACH over
client_list, because that is where the problem now
lies.