Only subscribe to directory once

Previously this was done on each expansion causing duplicates to
appear since extra events were being received.