Revert "Be smarter about the initial filter state"

This reverts commit 19ba34c001152977d4fc011c75e1e08bb260aacf.