added support for deduplication
small doc changes