Apply ignore list to remote dirs

Make sure that we don't want the user about remote_dirs in directories
which we are ignoring anyway.