Add caching for source fetching

Speeds things up especially when you have lots of matched fields that
share the same indexed field.