Patent ID: 9223829
Filing Date: 2015-12-29
Classification: G06F

Abstract:
1. A computer-implemented method comprising: executing a distinct operation on each table of a plurality of tables shared by a plurality of processing units in a parallel computing environment, each table comprising a plurality of rows and a plurality of columns, each column comprising values of a corresponding attribute of a plurality of attributes, each distinct operation eliminating duplicate data from the plurality of columns in each table, an order in which a column of each table is selected for removal of the duplicate data being based on a discrimination value of an attribute specific to the column and on a dependency metric characterizing dependency between the attribute specific to the column and one or more attributes specific to other one or more columns from which duplicate data has previously been removed, the executing creating a hierarchy of table pairs and distinct results, the distinct results comprising a reduced row set for each table, at least one table pair of the table pairs having a maximum row number difference between two corresponding tables of the at least one table pair; and detecting duplicates on each reduced row set to complete the distinct operation on the plurality of tables, the executing and detecting being performed by one or more processors.