Unref file structure on socket closing.