Adjusted the order in which filters are added/removed to tolerate slave
crashes while adding/removing filters.

Review: https://reviews.apache.org/r/24020