added basic caching model for deduplication
all sources now provide a consistent return array (without non-required information)
deduplication algorithm added