keep `remove_filter` from hitting the database

Now any calls to it are cached until render time :heart:
Merge pull request #2014 from Daxter/bugfix/1876-lazy-filter-removal

[1876] Make `remove_filter` lazy