Simplify caller filter, remove circular dependency.

For the common cases I can think of (particularly `rspec-fire`), this is a more desirable behaviour.