datamade/dedupe — pull request #234 by chancyk (2014-04-10 18:19:16+00:00)
Title: replace pair enumeration logic with dedupe.core.randomPairs

For large datasets enumerating all pairs with...

`distinct_pairs.update(itertools.combinations(sorted(data.keys()), 2))`

...requires a significant amount of memory. This logic was replaced with `dedupe.core.randomPairs`, which was already being utilized elsewhere, to calculate the pair indices.

`convenience.trainingDataLink` will still have a similar issue but its logic doesn't quite map to `dedupe.core.randomPairs`