Protect against unused accesses in Directory.(un)bind_directory

When a transport directory comes in or goes away, only peers using its kind of access must be updated.