add memory dict deduper support which can be configured in the config file