Use state to manage stream invalidations instead of cache

cc @stefanpenner