Make directory listener ignore destination dir.

This fixes putting watcher into the loop.