directory: collect all directories when invalidating

Otherwise we risk modifying the directories hash table from within the
iteration callback, which will trigger a critical warning.

https://bugzilla.gnome.org/show_bug.cgi?id=697890