Remove overly-clever optimizations

I was trying to be overly clever with the caching and pre-calculation of
filter functions, but it's obvious after trying to build something that
this micro-optimization didn't actually matter.

There will be more optimization later, but for now let's get the API and
tests stable.