[1876] Make `remove_filter` lazy

Resolves #1876

Now any calls to it are cached until render time :heart: