Repository: UCLALibrary/ftva-mams-data
Pull request #14: SYS-1784: script to extract duplicate rows
Author: login: ztucker4
Created: 2025-04-08T20:46:38Z | Merged: 2025-04-09T22:46:37Z

Implements [SYS-1784](https://uclalibrary.atlassian.net/browse/SYS-1784)

Adds a new script, `extract_duplicate_rows.py`, which can identify and optionally remove duplicate rows from the FTVA "Copy of DL data" spreadsheet, "Tapes(row 4560-24712)" tab. Duplicates are identified based only on the "Legacy Path" column. Also includes tests.

### Script arguments
* `--data_file`: path to the input data file, assumed to be a .xlsx spreadsheet in the format of [Copy of DL Sheet](https://docs.google.com/spreadsheets/d/1awie6qyi-Xs0boxbcnSkX-tslIctV5T63WahSFS1IOg/edit?gid=1358905874#gid=1358905874) as downloaded to Excel from Google Sheets.
* `--tapes_tab_name`: optional, the name of the tab containing "Tapes" data. Defaults to `"Tapes(row 4560-24712)"`, which matches the current Google sheet.
* `--output_duplicate_path`: optional, specifies the path to the output file of duplicate rows. Defaults to `"duplicate_rows.xlsx"`.
* `--remove_duplicates`: optional. If set, will remove all duplicate rows from the Tapes tab in-place.

### Example usage
* To identify duplicates without updating the sheet: `python extract_duplicate_rows.py --data_file Sheet_10_18_2024.xlsx`
* To remove duplicates from the sheet: `python extract_duplicate_rows.py --data_file Sheet_10_18_2024_test.xlsx --remove_duplicates`
* To run tests: `python -m unittest discover tests`

### Limitations
If the `--remove_duplicates` option is used, the Tapes tab in the Excel spreadsheet will lose all custom formatting (colors, fixed header row, text wrapping, etc.)


[SYS-1784]: https://uclalibrary.atlassian.net/browse/SYS-1784?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ


Diff:
diff --git a/README.md b/README.md
index 9ac28ca..3d64d38 100644
--- a/README.md
+++ b/README.md
@@ -72,3 +72,13 @@ user="YOUR_NAME"
 ### Proof of concept for Filemaker API (find and display data)
 
 ```python filemaker_api_test.py [-h] --config_file CONFIG_FILE```
+
+### Extract duplicate rows from FTVA spreadsheet
+
+```python extract_duplicate_rows.py --data_file SPREADSHEET.xlsx [--tapes_tab_name TAB_NAME] [--output_duplicate_path OUTPUT_PATH.xlsx] [--remove_duplicates]```
+
+This script extracts duplicate rows from the Tapes tab of the FTVA spreadsheet based on the "Legacy Path" column. Duplicate rows will be saved to the file specified by `--output_duplicate_path` (defaults to `duplicate_rows.xlsx` in the current directory).
+
+If `--tapes_tab_name` is specified, the script will look for the tab with that name within the spreadsheet. Otherwise, it will look for a tab named "Tapes(row 4560-24712)".
+
+If `--remove_duplicates` is specified, the script will remove the duplicate rows from the original spreadsheet and save it in place. Otherwise the original spreadsheet will not be modified.
diff --git a/extract_duplicate_rows.py b/extract_duplicate_rows.py
new file mode 100644
index 0000000..71aaee3
--- /dev/null
+++ b/extract_duplicate_rows.py
@@ -0,0 +1,117 @@
+import argparse
+import pandas as pd
+from pathlib import Path
+
+
+def _get_args() -> argparse.Namespace:
+    """Returns the command-line arguments for this program."""
+    parser = argparse.ArgumentParser()
+    parser.add_argument(
+        "--data_file",
+        help="Path to XLSX spreadsheet provided by FTVA",
+        required=True,
+    )
+    parser.add_argument(
+        "--tapes_tab_name",
+        help="Name of the spreadsheet tab containing tape data",
+        default="Tapes(row 4560-24712)",
+    )
+    parser.add_argument(
+        "--output_duplicate_path",
+        help="Path to the output spreadsheet for duplicate rows",
+        default="duplicate_rows.xlsx",
+    )
+    parser.add_argument(
+        "--remove_duplicates",
+        help="If specified, remove duplicate rows from the spreadsheet in place",
+        action="store_true",
+    )
+    args = parser.parse_args()
+    return args
+
+
+def _find_duplicate_rows(df: pd.DataFrame) -> pd.DataFrame:
+    """Find duplicate rows in the DataFrame based on the 'Legacy Path' column."""
+    # Check if 'Legacy Path' column exists
+    if "Legacy Path" not in df.columns:
+        raise ValueError("The DataFrame does not contain a 'Legacy Path' column.")
+
+    # Find duplicate rows based on the 'Legacy Path' column
+    duplicate_rows = df[df.duplicated(subset=["Legacy Path"], keep=False)]
+
+    return duplicate_rows
+
+
+def _format_duplicate_rows(duplicate_rows: pd.DataFrame) -> pd.DataFrame:
+    """Format duplicate rows for better readability in an output spreadsheet."""
+    # Create a copy of the DataFrame to avoid modifying the original slice,
+    # which causes pandas to raise a SettingWithCopyWarning
+    duplicate_rows = duplicate_rows.copy()
+    # Add a new column with the original row index
+    duplicate_rows.reset_index(inplace=True, names="Original Row Number")
+    # increment each index by 2 to match the original spreadsheet
+    # (1 based index, plus header row)
+    duplicate_rows["Original Row Number"] += 2
+
+    return duplicate_rows
+
+
+def _remove_duplicates_from_df(df: pd.DataFrame) -> pd.DataFrame:
+    """Remove duplicate rows from the DataFrame."""
+    # Remove duplicates based on 'Legacy Path' column, keeping the first occurrence
+    # and dropping the rest
+    df.drop_duplicates(subset=["Legacy Path"], keep="first", inplace=True)
+    return df
+
+
+def _remove_duplicates_from_spreadsheet(
+    df: pd.DataFrame, data_file_path: Path, tapes_tab_name: str
+) -> None:
+    """Remove duplicate rows from the DataFrame and save it back to the spreadsheet."""
+    # Remove duplicates from the DataFrame
+    df = _remove_duplicates_from_df(df)
+    # Save the cleaned DataFrame back to the spreadsheet;
+    # only overwrite the specified tab, and keep the rest of the spreadsheet intact
+    with pd.ExcelWriter(
+        data_file_path, engine="openpyxl", mode="a", if_sheet_exists="replace"
+    ) as writer:
+        df.to_excel(writer, sheet_name=tapes_tab_name, index=False)
+    print(f"Duplicate rows removed. Updated spreadsheet saved to {data_file_path}.")
+
+
+def main():
+    args = _get_args()
+    data_file_path = Path(args.data_file)
+
+    if not data_file_path.exists():
+        raise FileExistsError("Data file does not exist")
+
+    # check that tab exists within the spreadsheet
+    xls = pd.ExcelFile(data_file_path)
+    if args.tapes_tab_name not in xls.sheet_names:
+        raise ValueError(
+            f"Tab '{args.tapes_tab_name}' does not exist in the spreadsheet"
+        )
+
+    # Read the spreadsheet into a DataFrame
+    df = pd.read_excel(data_file_path, sheet_name=args.tapes_tab_name)
+    print(f"Data loaded from {data_file_path}, tab: {args.tapes_tab_name}.")
+
+    # Find duplicate rows
+    duplicate_rows = _find_duplicate_rows(df)
+    if duplicate_rows.empty:
+        print("No duplicate rows found.")
+    else:
+        print(f"{len(duplicate_rows)} duplicate rows found.")
+        duplicate_rows = _format_duplicate_rows(duplicate_rows)
+        duplicate_rows_path = Path(args.output_duplicate_path)
+        duplicate_rows.to_excel(duplicate_rows_path, index=False)
+        print(f"Duplicate rows saved to {duplicate_rows_path}.")
+
+        # Optionally, remove duplicates
+        if args.remove_duplicates:
+            _remove_duplicates_from_spreadsheet(df, data_file_path, args.tapes_tab_name)
+
+
+if __name__ == "__main__":
+    main()
diff --git a/tests/test_extract_duplicate_rows.py b/tests/test_extract_duplicate_rows.py
new file mode 100644
index 0000000..7b1412c
--- /dev/null
+++ b/tests/test_extract_duplicate_rows.py
@@ -0,0 +1,101 @@
+import unittest
+from extract_duplicate_rows import (
+    _find_duplicate_rows,
+    _format_duplicate_rows,
+    _remove_duplicates_from_df,
+)
+import pandas as pd
+
+
+class TestFormatDuplicateRows(unittest.TestCase):
+
+    def setUp(self):
+        self.df = pd.DataFrame(
+            {
+                "Legacy Path": [
+                    "path/to/file1",
+                    "path/to/file1",  # Duplicate
+                    "path/to/file2",
+                    "path/to/file3",
+                    "path/to/file3",  # Duplicate
+                ],
+                "Other Column": [1, 2, 3, 4, 5],
+            }
+        )
+
+    def test_format_duplicate_rows(self):
+        duplicate_rows = _find_duplicate_rows(self.df)
+        formatted_rows = _format_duplicate_rows(duplicate_rows)
+        expected_output = pd.DataFrame(
+            {
+                # "Original Row Number" is the index of the original DataFrame + 2
+                # to account for 1-based indexing and the header row
+                "Original Row Number": [2, 3, 5, 6],
+                "Legacy Path": [
+                    "path/to/file1",
+                    "path/to/file1",
+                    "path/to/file3",
+                    "path/to/file3",
+                ],
+                "Other Column": [1, 2, 4, 5],
+            }
+        )
+        pd.testing.assert_frame_equal(formatted_rows, expected_output)
+
+    def test_format_duplicate_rows_no_duplicates(self):
+        # Test with a DataFrame that has no duplicates
+        no_duplicates_df = pd.DataFrame(
+            {
+                "Legacy Path": ["path/to/file1", "path/to/file2", "path/to/file3"],
+                "Other Column": [1, 2, 3],
+            }
+        )
+        no_duplicates_df = _find_duplicate_rows(no_duplicates_df)
+        formatted_rows = _format_duplicate_rows(no_duplicates_df)
+        self.assertTrue(formatted_rows.empty)
+        self.assertIn("Original Row Number", formatted_rows.columns)
+
+
+class TestRemoveDuplicatesFromDf(unittest.TestCase):
+    def setUp(self):
+        self.df_with_duplicates = pd.DataFrame(
+            {
+                "Legacy Path": [
+                    "path/to/file1",
+                    "path/to/file2",
+                    "path/to/file1",  # Duplicate
+                    "path/to/file3",
+                    "path/to/file2",  # Duplicate
+                ],
+                "Other Column": [1, 2, 3, 4, 5],
+            }
+        )
+        self.df_without_duplicates = pd.DataFrame(
+            {
+                "Legacy Path": [
+                    "path/to/file1",
+                    "path/to/file2",
+                    "path/to/file3",
+                ],
+                "Other Column": [1, 2, 4],
+            }
+        )
+
+    def test_remove_duplicates_from_df(self):
+        cleaned_df = _remove_duplicates_from_df(self.df_with_duplicates)
+        expected_df = pd.DataFrame(
+            {
+                "Legacy Path": [
+                    "path/to/file1",
+                    "path/to/file2",
+                    "path/to/file3",
+                ],
+                "Other Column": [1, 2, 4],
+            },
+            index=[0, 1, 3],
+        )
+        pd.testing.assert_frame_equal(cleaned_df, expected_df)
+
+    def test_remove_duplicates_from_df_no_duplicates(self):
+        cleaned_df = _remove_duplicates_from_df(self.df_without_duplicates)
+        pd.testing.assert_frame_equal(cleaned_df, self.df_without_duplicates)


Reviews:
user: login: akohler
body: @ztucker4 This seems to work correctly, finding 1266 total duplicate rows.  It removed the duplicates when I added the flag, then correctly found no duplicates when I ran it again with the updated file.  All 4 tests pass, and I learned about pandas testing utility methods - good to know.

There's some (apparently) unneeded code to clean up, and a possible minor simplification to how `reset_index` is used - though I did not experiment with that locally, so might not do what I think it does.

Thanks --Andy
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-04-08T22:26:57Z
user: login: ztucker4
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-09T22:31:05Z
user: login: akohler
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-09T22:42:38Z
user: login: akohler
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-09T22:44:35Z

Review comments:
diff_hunk: @@ -0,0 +1,119 @@
+import argparse
+import pandas as pd
+from pathlib import Path
+
+
+def _get_args() -> argparse.Namespace:
+    """Returns the command-line arguments for this program."""
+    parser = argparse.ArgumentParser()
+    parser.add_argument(
+        "--data_file",
+        help="Path to XLSX spreadsheet provided by FTVA",
+        required=True,
+    )
+    parser.add_argument(
+        "--tapes_tab_name",
+        help="Name of the spreadsheet tab containing tape data",
+        default="Tapes(row 4560-24712)",
+    )
+    parser.add_argument(
+        "--output_duplicate_path",
+        help="Path to the output spreadsheet for duplicate rows",
+        default="duplicate_rows.xlsx",
+    )
+    parser.add_argument(
+        "--remove_duplicates",
+        help="If specified, remove duplicate rows from the spreadsheet in place",
+        action="store_true",
+    )
+    args = parser.parse_args()
+    return args
+
+
+def _find_duplicate_rows(df: pd.DataFrame) -> pd.DataFrame:
+    """Find duplicate rows in the DataFrame based on the 'Legacy Path' column."""
+    # Check if 'Legacy Path' column exists
+    if "Legacy Path" not in df.columns:
+        raise ValueError("The DataFrame does not contain a 'Legacy Path' column.")
+
+    # Find duplicate rows based on the 'Legacy Path' column
+    duplicate_rows = df[df.duplicated(subset=["Legacy Path"], keep=False)]
+
+    return duplicate_rows
+
+
+def _format_duplicate_rows(duplicate_rows: pd.DataFrame) -> pd.DataFrame:
+    """Format duplicate rows for better readability in an output spreadsheet."""
+    # Create a copy of the DataFrame to avoid modifying the original slice,
+    # which causes pandas to raise a SettingWithCopyWarning
+    duplicate_rows = duplicate_rows.copy()
+    # Add a new column with the original row index
+    duplicate_rows.reset_index(inplace=True)
path: extract_duplicate_rows.py
user: login: akohler
body: The help for `reset_index` says there's a `names` parameter:
```
names : int, str or 1-dimensional list, default None
Using the given string, rename the DataFrame column which contains the index data. If the DataFrame has a MultiIndex, this has to be a list or tuple with length equal to the number of levels.
```
I think this could allow assigning the "Original Row Number" name in this `reset_index` step, removing the need for the explicit `rename` later.  If so, the index increment column reference might need to change.
created_at: 2025-04-08T21:51:50Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,119 @@
+import argparse
+import pandas as pd
+from pathlib import Path
+
+
+def _get_args() -> argparse.Namespace:
+    """Returns the command-line arguments for this program."""
+    parser = argparse.ArgumentParser()
+    parser.add_argument(
+        "--data_file",
+        help="Path to XLSX spreadsheet provided by FTVA",
+        required=True,
+    )
+    parser.add_argument(
+        "--tapes_tab_name",
+        help="Name of the spreadsheet tab containing tape data",
+        default="Tapes(row 4560-24712)",
+    )
+    parser.add_argument(
+        "--output_duplicate_path",
+        help="Path to the output spreadsheet for duplicate rows",
+        default="duplicate_rows.xlsx",
+    )
+    parser.add_argument(
+        "--remove_duplicates",
+        help="If specified, remove duplicate rows from the spreadsheet in place",
+        action="store_true",
+    )
+    args = parser.parse_args()
+    return args
+
+
+def _find_duplicate_rows(df: pd.DataFrame) -> pd.DataFrame:
+    """Find duplicate rows in the DataFrame based on the 'Legacy Path' column."""
+    # Check if 'Legacy Path' column exists
+    if "Legacy Path" not in df.columns:
+        raise ValueError("The DataFrame does not contain a 'Legacy Path' column.")
+
+    # Find duplicate rows based on the 'Legacy Path' column
+    duplicate_rows = df[df.duplicated(subset=["Legacy Path"], keep=False)]
+
+    return duplicate_rows
+
+
+def _format_duplicate_rows(duplicate_rows: pd.DataFrame) -> pd.DataFrame:
+    """Format duplicate rows for better readability in an output spreadsheet."""
+    # Create a copy of the DataFrame to avoid modifying the original slice,
+    # which causes pandas to raise a SettingWithCopyWarning
+    duplicate_rows = duplicate_rows.copy()
+    # Add a new column with the original row index
+    duplicate_rows.reset_index(inplace=True)
+    # increment each index by 2 to match the original spreadsheet
+    # (1 based index, plus header row)
+    duplicate_rows["index"] += 2
+    # rename the index column
+    duplicate_rows.rename(columns={"index": "Original Row Number"}, inplace=True)
path: extract_duplicate_rows.py
user: login: akohler
body: See earlier comment about possibly incorporating this into the `reset_index` step.
created_at: 2025-04-08T21:52:20Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,119 @@
+import argparse
+import pandas as pd
+from pathlib import Path
+
+
+def _get_args() -> argparse.Namespace:
+    """Returns the command-line arguments for this program."""
+    parser = argparse.ArgumentParser()
+    parser.add_argument(
+        "--data_file",
+        help="Path to XLSX spreadsheet provided by FTVA",
+        required=True,
+    )
+    parser.add_argument(
+        "--tapes_tab_name",
+        help="Name of the spreadsheet tab containing tape data",
+        default="Tapes(row 4560-24712)",
+    )
+    parser.add_argument(
+        "--output_duplicate_path",
+        help="Path to the output spreadsheet for duplicate rows",
+        default="duplicate_rows.xlsx",
+    )
+    parser.add_argument(
+        "--remove_duplicates",
+        help="If specified, remove duplicate rows from the spreadsheet in place",
+        action="store_true",
+    )
+    args = parser.parse_args()
+    return args
+
+
+def _find_duplicate_rows(df: pd.DataFrame) -> pd.DataFrame:
+    """Find duplicate rows in the DataFrame based on the 'Legacy Path' column."""
+    # Check if 'Legacy Path' column exists
+    if "Legacy Path" not in df.columns:
+        raise ValueError("The DataFrame does not contain a 'Legacy Path' column.")
+
+    # Find duplicate rows based on the 'Legacy Path' column
+    duplicate_rows = df[df.duplicated(subset=["Legacy Path"], keep=False)]
+
+    return duplicate_rows
+
+
+def _format_duplicate_rows(duplicate_rows: pd.DataFrame) -> pd.DataFrame:
+    """Format duplicate rows for better readability in an output spreadsheet."""
+    # Create a copy of the DataFrame to avoid modifying the original slice,
+    # which causes pandas to raise a SettingWithCopyWarning
+    duplicate_rows = duplicate_rows.copy()
+    # Add a new column with the original row index
+    duplicate_rows.reset_index(inplace=True)
+    # increment each index by 2 to match the original spreadsheet
+    # (1 based index, plus header row)
+    duplicate_rows["index"] += 2
+    # rename the index column
+    duplicate_rows.rename(columns={"index": "Original Row Number"}, inplace=True)
+
+    return duplicate_rows
+
+
+def _remove_duplicates_from_df(df: pd.DataFrame, tapes_tab_name: str) -> pd.DataFrame:
path: extract_duplicate_rows.py
user: login: akohler
body: The `tapes_tab_name` parameter is not used by this method.  And since this method only runs one command, and is only called from one place, I think it's not necessary: there's no repeated code, and no flexibility from changing parameter values.  Maybe it did more initially?  As-is, the method can be removed and the one command called directly in `_remove_duplicates_from_spreadsheet()`.
created_at: 2025-04-08T21:56:32Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,119 @@
+import argparse
+import pandas as pd
+from pathlib import Path
+
+
+def _get_args() -> argparse.Namespace:
+    """Returns the command-line arguments for this program."""
+    parser = argparse.ArgumentParser()
+    parser.add_argument(
+        "--data_file",
+        help="Path to XLSX spreadsheet provided by FTVA",
+        required=True,
+    )
+    parser.add_argument(
+        "--tapes_tab_name",
+        help="Name of the spreadsheet tab containing tape data",
+        default="Tapes(row 4560-24712)",
+    )
+    parser.add_argument(
+        "--output_duplicate_path",
+        help="Path to the output spreadsheet for duplicate rows",
+        default="duplicate_rows.xlsx",
+    )
+    parser.add_argument(
+        "--remove_duplicates",
+        help="If specified, remove duplicate rows from the spreadsheet in place",
+        action="store_true",
+    )
+    args = parser.parse_args()
+    return args
+
+
+def _find_duplicate_rows(df: pd.DataFrame) -> pd.DataFrame:
+    """Find duplicate rows in the DataFrame based on the 'Legacy Path' column."""
+    # Check if 'Legacy Path' column exists
+    if "Legacy Path" not in df.columns:
+        raise ValueError("The DataFrame does not contain a 'Legacy Path' column.")
+
+    # Find duplicate rows based on the 'Legacy Path' column
+    duplicate_rows = df[df.duplicated(subset=["Legacy Path"], keep=False)]
+
+    return duplicate_rows
+
+
+def _format_duplicate_rows(duplicate_rows: pd.DataFrame) -> pd.DataFrame:
+    """Format duplicate rows for better readability in an output spreadsheet."""
+    # Create a copy of the DataFrame to avoid modifying the original slice,
+    # which causes pandas to raise a SettingWithCopyWarning
+    duplicate_rows = duplicate_rows.copy()
+    # Add a new column with the original row index
+    duplicate_rows.reset_index(inplace=True)
+    # increment each index by 2 to match the original spreadsheet
+    # (1 based index, plus header row)
+    duplicate_rows["index"] += 2
+    # rename the index column
+    duplicate_rows.rename(columns={"index": "Original Row Number"}, inplace=True)
+
+    return duplicate_rows
+
+
+def _remove_duplicates_from_df(df: pd.DataFrame, tapes_tab_name: str) -> pd.DataFrame:
+    """Remove duplicate rows from the DataFrame."""
+    # Remove duplicates based on 'Legacy Path' column, keeping the first occurrence
+    # and dropping the rest
+    df.drop_duplicates(subset=["Legacy Path"], keep="first", inplace=True)
+    return df
+
+
+def _remove_duplicates_from_spreadsheet(
+    df: pd.DataFrame, data_file_path: Path, tapes_tab_name: str
+) -> None:
+    """Remove duplicate rows from the DataFrame and save it back to the spreadsheet."""
+    # Remove duplicates from the DataFrame
+    df = _remove_duplicates_from_df(df, tapes_tab_name)
+    # Save the cleaned DataFrame back to the spreadsheet
+    # only overwrite the specified tab, and keep the rest of the spreadsheet intact
path: extract_duplicate_rows.py
user: login: akohler
body: Nitpick:  The 2 lines of this comment seem to belong together, since no punctuation at end of 1st line and the 2nd line starts with lower case, but it's really 2 statements.  Either end line 1 with semicolon, or start line 2 with `Only` instead of `only`. :pedant_face:
created_at: 2025-04-08T22:02:45Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,101 @@
+import unittest
+from extract_duplicate_rows import (
+    _find_duplicate_rows,
+    _format_duplicate_rows,
+    _remove_duplicates_from_df,
+)
+import pandas as pd
+
+
+class TestFormatDuplicateRows(unittest.TestCase):
+
+    def setUp(self):
+        self.df = pd.DataFrame(
+            {
+                "Legacy Path": [
+                    "path/to/file1",
+                    "path/to/file1",  # Duplicate
+                    "path/to/file2",
+                    "path/to/file3",
+                    "path/to/file3",  # Duplicate
+                ],
+                "Other Column": [1, 2, 3, 4, 5],
+            }
+        )
+
+    def test_format_duplicate_rows(self):
+        duplicate_rows = _find_duplicate_rows(self.df)
+        formatted_rows = _format_duplicate_rows(duplicate_rows)
+        expected_output = pd.DataFrame(
+            {
+                # "Original Row Number" is the index of the original DataFrame + 2
+                # to account for 1-based indexing and the header row
+                "Original Row Number": [2, 3, 5, 6],
+                "Legacy Path": [
+                    "path/to/file1",
+                    "path/to/file1",
+                    "path/to/file3",
+                    "path/to/file3",
+                ],
+                "Other Column": [1, 2, 4, 5],
+            }
+        )
+        pd.testing.assert_frame_equal(formatted_rows, expected_output)
+
+    def test_format_duplicate_rows_no_duplicates(self):
+        # Test with a DataFrame that has no duplicates
+        no_duplicates_df = pd.DataFrame(
+            {
+                "Legacy Path": ["path/to/file1", "path/to/file2", "path/to/file3"],
+                "Other Column": [1, 2, 3],
+            }
+        )
+        no_duplicates_df = _find_duplicate_rows(no_duplicates_df)
+        formatted_rows = _format_duplicate_rows(no_duplicates_df)
+        self.assertTrue(formatted_rows.empty)
+        self.assertIn("Original Row Number", formatted_rows.columns)
+
+
+class TestRemoveDuplicatesFromDf(unittest.TestCase):
+    def setUp(self):
+        self.df_with_duplicates = pd.DataFrame(
+            {
+                "Legacy Path": [
+                    "path/to/file1",
+                    "path/to/file2",
+                    "path/to/file1",  # Duplicate
+                    "path/to/file3",
+                    "path/to/file2",  # Duplicate
+                ],
+                "Other Column": [1, 2, 3, 4, 5],
+            }
+        )
+        self.df_without_duplicates = pd.DataFrame(
+            {
+                "Legacy Path": [
+                    "path/to/file1",
+                    "path/to/file2",
+                    "path/to/file3",
+                ],
+                "Other Column": [1, 2, 4],
+            }
+        )
+
+    def test_remove_duplicates_from_df(self):
+        cleaned_df = _remove_duplicates_from_df(self.df_with_duplicates, "Tapes")
path: tests/test_extract_duplicate_rows.py
user: login: akohler
body: The 2nd parameter, "Tapes", is never defined in the test data (and not used by the method under test - see previous comments).  Applies also to line 100.
created_at: 2025-04-08T22:15:34Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,119 @@
+import argparse
+import pandas as pd
+from pathlib import Path
+
+
+def _get_args() -> argparse.Namespace:
+    """Returns the command-line arguments for this program."""
+    parser = argparse.ArgumentParser()
+    parser.add_argument(
+        "--data_file",
+        help="Path to XLSX spreadsheet provided by FTVA",
+        required=True,
+    )
+    parser.add_argument(
+        "--tapes_tab_name",
+        help="Name of the spreadsheet tab containing tape data",
+        default="Tapes(row 4560-24712)",
+    )
+    parser.add_argument(
+        "--output_duplicate_path",
+        help="Path to the output spreadsheet for duplicate rows",
+        default="duplicate_rows.xlsx",
+    )
+    parser.add_argument(
+        "--remove_duplicates",
+        help="If specified, remove duplicate rows from the spreadsheet in place",
+        action="store_true",
+    )
+    args = parser.parse_args()
+    return args
+
+
+def _find_duplicate_rows(df: pd.DataFrame) -> pd.DataFrame:
+    """Find duplicate rows in the DataFrame based on the 'Legacy Path' column."""
+    # Check if 'Legacy Path' column exists
+    if "Legacy Path" not in df.columns:
+        raise ValueError("The DataFrame does not contain a 'Legacy Path' column.")
+
+    # Find duplicate rows based on the 'Legacy Path' column
+    duplicate_rows = df[df.duplicated(subset=["Legacy Path"], keep=False)]
+
+    return duplicate_rows
+
+
+def _format_duplicate_rows(duplicate_rows: pd.DataFrame) -> pd.DataFrame:
+    """Format duplicate rows for better readability in an output spreadsheet."""
+    # Create a copy of the DataFrame to avoid modifying the original slice,
+    # which causes pandas to raise a SettingWithCopyWarning
+    duplicate_rows = duplicate_rows.copy()
+    # Add a new column with the original row index
+    duplicate_rows.reset_index(inplace=True)
+    # increment each index by 2 to match the original spreadsheet
+    # (1 based index, plus header row)
+    duplicate_rows["index"] += 2
+    # rename the index column
+    duplicate_rows.rename(columns={"index": "Original Row Number"}, inplace=True)
+
+    return duplicate_rows
+
+
+def _remove_duplicates_from_df(df: pd.DataFrame, tapes_tab_name: str) -> pd.DataFrame:
path: extract_duplicate_rows.py
user: login: ztucker4
body: I updated this method to remove the reference to `tapes_tab_name`. I do think the method is worth keeping around to make testing easier - without it, tests for data deletion would need to actually manipulate excel files instead of just dataframes. But maybe we don't need tests to confirm that pandas is working as expected?
created_at: 2025-04-09T22:31:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,119 @@
+import argparse
+import pandas as pd
+from pathlib import Path
+
+
+def _get_args() -> argparse.Namespace:
+    """Returns the command-line arguments for this program."""
+    parser = argparse.ArgumentParser()
+    parser.add_argument(
+        "--data_file",
+        help="Path to XLSX spreadsheet provided by FTVA",
+        required=True,
+    )
+    parser.add_argument(
+        "--tapes_tab_name",
+        help="Name of the spreadsheet tab containing tape data",
+        default="Tapes(row 4560-24712)",
+    )
+    parser.add_argument(
+        "--output_duplicate_path",
+        help="Path to the output spreadsheet for duplicate rows",
+        default="duplicate_rows.xlsx",
+    )
+    parser.add_argument(
+        "--remove_duplicates",
+        help="If specified, remove duplicate rows from the spreadsheet in place",
+        action="store_true",
+    )
+    args = parser.parse_args()
+    return args
+
+
+def _find_duplicate_rows(df: pd.DataFrame) -> pd.DataFrame:
+    """Find duplicate rows in the DataFrame based on the 'Legacy Path' column."""
+    # Check if 'Legacy Path' column exists
+    if "Legacy Path" not in df.columns:
+        raise ValueError("The DataFrame does not contain a 'Legacy Path' column.")
+
+    # Find duplicate rows based on the 'Legacy Path' column
+    duplicate_rows = df[df.duplicated(subset=["Legacy Path"], keep=False)]
+
+    return duplicate_rows
+
+
+def _format_duplicate_rows(duplicate_rows: pd.DataFrame) -> pd.DataFrame:
+    """Format duplicate rows for better readability in an output spreadsheet."""
+    # Create a copy of the DataFrame to avoid modifying the original slice,
+    # which causes pandas to raise a SettingWithCopyWarning
+    duplicate_rows = duplicate_rows.copy()
+    # Add a new column with the original row index
+    duplicate_rows.reset_index(inplace=True)
+    # increment each index by 2 to match the original spreadsheet
+    # (1 based index, plus header row)
+    duplicate_rows["index"] += 2
+    # rename the index column
+    duplicate_rows.rename(columns={"index": "Original Row Number"}, inplace=True)
+
+    return duplicate_rows
+
+
+def _remove_duplicates_from_df(df: pd.DataFrame, tapes_tab_name: str) -> pd.DataFrame:
path: extract_duplicate_rows.py
user: login: akohler
body: Good point about the testing.  I don't generally like tests which just confirm that a 3rd-party package is working as intended.  But since there are some nuances added by our custom script, and we've got tests and test data to cover those, I'm fine with leaving the extra method in place.  Well-argued, I'm convinced!

created_at: 2025-04-09T22:42:38Z
author_association: MEMBER

PR comments:
user: login: akohler
created_at: 2025-04-08T22:28:54Z
body: @ztucker4 One thing I forgot: Please add brief usage notes to the Scripts section of `README.md`.
author_association: MEMBER
user: login: ztucker4
created_at: 2025-04-09T22:33:27Z
body: Hi @akohler, this is ready for re-review. All requested changes are addressed, other than removing `_remove_duplicates_from_df()` - see other comment, and let me know if you still think I should refactor. Thanks!
author_association: CONTRIBUTOR
user: login: akohler
created_at: 2025-04-09T22:46:33Z
body: @ztucker4 This all looks good to me now, merging.
@henry-r18 FYI, please rebase, and you may have merge conflicts with the `tests` directory.

author_association: MEMBER