Repository: jtablesaw/tablesaw
Pull request #1297 (2025-06-26 17:53:24)

Fix drop duplicate rows
Thanks for contributing.

- [x] Tick to sign-off your agreement to the [Developer Certificate of Origin (DCO) 1.1](https://developercertificate.org)

## Description

Fixed issue with `Table.dropDuplicateRows` described in #1296 by reverting to using a custom map instead of a set of rows.

## Testing

Reported use case implemented as unit test. Added back testing of hash collision.


sonarqubecloud[bot]: ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=jtablesaw_tablesaw&pullRequest=1297) **Quality Gate passed**  
Issues  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [1 New issue](https://sonarcloud.io/project/issues?id=jtablesaw_tablesaw&pullRequest=1297&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=jtablesaw_tablesaw&pullRequest=1297&issueStatuses=ACCEPTED)

Measures  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=jtablesaw_tablesaw&pullRequest=1297&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=jtablesaw_tablesaw&pullRequest=1297&metric=new_coverage&view=list)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=jtablesaw_tablesaw&pullRequest=1297&metric=new_duplicated_lines_density&view=list)  
  
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=jtablesaw_tablesaw&pullRequest=1297)


sonarqubecloud[bot]: ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=jtablesaw_tablesaw&pullRequest=1297) **Quality Gate passed**  
Issues  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 New issues](https://sonarcloud.io/project/issues?id=jtablesaw_tablesaw&pullRequest=1297&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=jtablesaw_tablesaw&pullRequest=1297&issueStatuses=ACCEPTED)

Measures  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=jtablesaw_tablesaw&pullRequest=1297&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=jtablesaw_tablesaw&pullRequest=1297&metric=new_coverage&view=list)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=jtablesaw_tablesaw&pullRequest=1297&metric=new_duplicated_lines_density&view=list)  
  
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=jtablesaw_tablesaw&pullRequest=1297)


benmccann: > reverting to using a custom map instead of a set of rows.
Added back testing of hash collision.

Maybe it was broken in https://github.com/jtablesaw/tablesaw/pull/1267 ?
sonarqubecloud[bot]: ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=jtablesaw_tablesaw&pullRequest=1297) **Quality Gate passed**  
Issues  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 New issues](https://sonarcloud.io/project/issues?id=jtablesaw_tablesaw&pullRequest=1297&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=jtablesaw_tablesaw&pullRequest=1297&issueStatuses=ACCEPTED)

Measures  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=jtablesaw_tablesaw&pullRequest=1297&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=jtablesaw_tablesaw&pullRequest=1297&metric=new_coverage&view=list)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=jtablesaw_tablesaw&pullRequest=1297&metric=new_duplicated_lines_density&view=list)  
  
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=jtablesaw_tablesaw&pullRequest=1297)


sonarqubecloud[bot]: ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=jtablesaw_tablesaw&pullRequest=1297) **Quality Gate passed**  
Issues  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 New issues](https://sonarcloud.io/project/issues?id=jtablesaw_tablesaw&pullRequest=1297&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=jtablesaw_tablesaw&pullRequest=1297&issueStatuses=ACCEPTED)

Measures  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=jtablesaw_tablesaw&pullRequest=1297&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=jtablesaw_tablesaw&pullRequest=1297&metric=new_coverage&view=list)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=jtablesaw_tablesaw&pullRequest=1297&metric=new_duplicated_lines_density&view=list)  
  
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=jtablesaw_tablesaw&pullRequest=1297)



Patch:
diff --git a/core/src/main/java/tech/tablesaw/api/Table.java b/core/src/main/java/tech/tablesaw/api/Table.java
index 556b41aa..b2c6bd8f 100644
--- a/core/src/main/java/tech/tablesaw/api/Table.java
+++ b/core/src/main/java/tech/tablesaw/api/Table.java
@@ -20,11 +20,14 @@ import static tech.tablesaw.aggregate.AggregateFunctions.countMissing;
 import static tech.tablesaw.api.QuerySupport.not;
 import static tech.tablesaw.selection.Selection.selectNRowsAtRandom;
 
+import com.google.common.annotations.VisibleForTesting;
 import com.google.common.base.Preconditions;
 import com.google.common.collect.Streams;
 import com.google.common.primitives.Ints;
 import io.github.classgraph.ClassGraph;
 import io.github.classgraph.ScanResult;
+import it.unimi.dsi.fastutil.ints.Int2ObjectMap;
+import it.unimi.dsi.fastutil.ints.Int2ObjectOpenHashMap;
 import it.unimi.dsi.fastutil.ints.IntArrayList;
 import it.unimi.dsi.fastutil.ints.IntArrays;
 import it.unimi.dsi.fastutil.ints.IntComparator;
@@ -32,11 +35,9 @@ import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collection;
 import java.util.Comparator;
-import java.util.HashSet;
 import java.util.Iterator;
 import java.util.List;
 import java.util.NoSuchElementException;
-import java.util.Set;
 import java.util.function.Function;
 import java.util.function.IntFunction;
 import java.util.stream.Collectors;
@@ -926,16 +927,48 @@ public class Table extends Relation implements Iterable<Row> {
     return StandardTableSliceGroup.create(this, columns);
   }
 
+  /**
+   * Returns true if all the values in row are identical to those in another row previously seen and
+   * recorded in the list.
+   *
+   * @param row the row to evaluate
+   * @param uniqueHashes a map of row hashes to the id of an exemplar row that produces that hash.
+   *     If two different rows produce the same hash, then the row number for each is placed in the
+   *     list, so that there are exemplars for both
+   * @return true if the row's values exactly match a row that was previously seen
+   */
+  @VisibleForTesting
+  protected boolean isDuplicate(Row row, Int2ObjectMap<IntArrayList> uniqueHashes) {
+    int hash = row.hashCode();
+    if (!uniqueHashes.containsKey(hash)) {
+      IntArrayList rowNumbers = new IntArrayList();
+      rowNumbers.add(row.getRowNumber());
+      uniqueHashes.put(hash, rowNumbers);
+      return false;
+    }
+
+    // the hashmap contains the hash, make sure the actual row values match
+    IntArrayList matchingKeys = uniqueHashes.get(hash);
+
+    for (int key : matchingKeys) {
+      Row oldRow = this.row(key);
+      if (row.equals(oldRow)) {
+        return true;
+      }
+    }
+    uniqueHashes.get(hash).add(row.getRowNumber());
+    return false;
+  }
+
   /**
    * Returns the unique records in this table, such that any record that appears more than once in
    * this table, appears only once in the returned table.
    */
   public Table dropDuplicateRows() {
     Table temp = emptyCopy();
-    Set<Row> uniqueRows = new HashSet<>();
+    Int2ObjectMap<IntArrayList> uniqueHashes = new Int2ObjectOpenHashMap<>();
     for (Row row : this) {
-      if (!uniqueRows.contains(row)) {
-        uniqueRows.add(row);
+      if (!isDuplicate(row, uniqueHashes)) {
         temp.append(row);
       }
     }


Test patch:
diff --git a/core/src/test/java/tech/tablesaw/api/TableTest.java b/core/src/test/java/tech/tablesaw/api/TableTest.java
index 9a8f3585..9188ee3e 100644
--- a/core/src/test/java/tech/tablesaw/api/TableTest.java
+++ b/core/src/test/java/tech/tablesaw/api/TableTest.java
@@ -22,6 +22,9 @@ import static tech.tablesaw.api.ColumnType.DOUBLE;
 
 import com.google.common.collect.ImmutableList;
 import com.google.common.collect.Lists;
+import it.unimi.dsi.fastutil.ints.Int2ObjectMap;
+import it.unimi.dsi.fastutil.ints.Int2ObjectOpenHashMap;
+import it.unimi.dsi.fastutil.ints.IntArrayList;
 import java.io.File;
 import java.io.StringReader;
 import java.io.StringWriter;
@@ -335,6 +338,16 @@ public class TableTest {
     assertEquals(rowCount, t1.rowCount());
   }
 
+  @Test
+  void testDropDuplicateRows3() {
+    Table testTable = Table.create("test").addColumns(
+        IntColumn.create("part_id", new int[] {1, 1, 1, 2, 2, 2}),
+        StringColumn.create(
+            "nsequence", new String[] {"N40", "N50", "N60", "N40", "N50", "N60"}));
+    Table testUnique = testTable.selectColumns("part_id", "nsequence").dropDuplicateRows();
+    assertEquals(testTable.rowCount(), testUnique.rowCount());
+  }
+
   @Test
   void dropDuplicateRowsWithMissingValue() {
     // Add 4 rows to the table, two of which are duplicates and have missing values.
@@ -959,4 +972,17 @@ public class TableTest {
         () -> Table.compareRows(lastRowNumber, missingValues, differentTable),
         "Row outside range does not throw exception");
   }
+
+  @Test
+  void testDropDuplicateWithHashCollision() {
+    Table testTable = Table.read().usingOptions(CsvReadOptions
+      .builder(new File("../data/missing_values.csv"))
+      .missingValueIndicator("-"));
+    Row row0 = testTable.row(0);
+    Int2ObjectMap<IntArrayList> uniqueHashes = new Int2ObjectOpenHashMap<>();
+    IntArrayList value = new IntArrayList(new int[] {1, 0});
+    uniqueHashes.put(row0.hashCode(), value);
+    boolean isDuplicate = testTable.isDuplicate(row0, uniqueHashes);
+    assertTrue(isDuplicate, "Duplicate row not found");
+  }
 }