Wait until all listeners are removed before finishing test. Closes GH-71.
Revert "Dont even try watching non-existent dirs. Ref GH-56."

This reverts commit 6648d017156f5a7575ae63ca9e0136d187db1bfe.