Repository: astronomy-commons/lsdb
Pull request #602: Crossmatch pandas dataframe with an lsdb catalog
Author: login: olivialynn
Created: 2025-03-04T19:53:38Z | Merged: 2025-03-06T03:13:51Z

Closes #578 

Note:
- The function signatures between `crossmatch` and `crossmatch_dataframe` were almost completely preserved, ~~except the omission of `require_right_margin`, which doesn't make sense for a dataframe where we can (and unless otherwise specified in kwargs, will) generate the margin on the spot.~~  Mar. 11: Turns out I lied, and this is still in here. I'll keep it pending the API questions I'm asking about catalog margin options.
- I took my best shot at allowing the user to pass kwargs to `from_dataframe`, but I'm not thrilled at the need to keep this list up to date with the possible args `from_dataframe` will accept. However, without filtering the overall kwargs at this point, `self.crossmatch` on L312 will crash when encountering unexpected kwargs. Open to alternate strategies here.

Linked issue #578: Crossmatch pandas dataframe with an lsdb catalog
Implement a simple wrapper function which would enable user to crossmatch directly their pandas dataframe with an lsdb catalog. For the moment, in order to determine the ra and dec columns of the pandas dataframe request user to provide those. We will make it more robust and user-friendly in the future tickets. 

Diff:
diff --git a/src/lsdb/catalog/catalog.py b/src/lsdb/catalog/catalog.py
index 167c8f78a..a5099a3be 100644
--- a/src/lsdb/catalog/catalog.py
+++ b/src/lsdb/catalog/catalog.py
@@ -1,5 +1,6 @@
 from __future__ import annotations
 
+import inspect
 from typing import Callable, Type
 
 import hats as hc
@@ -225,6 +226,61 @@ def crossmatch(
         )
         return self.__class__(ddf, ddf_map, hc_catalog)
 
+    def crossmatch_dataframe(
+        self,
+        other: npd.NestedFrame,
+        suffixes: tuple[str, str] | None = None,
+        algorithm: (
+            Type[AbstractCrossmatchAlgorithm] | BuiltInCrossmatchAlgorithm
+        ) = BuiltInCrossmatchAlgorithm.KD_TREE,
+        output_catalog_name: str | None = None,
+        require_right_margin: bool = False,
+        **kwargs,
+    ) -> Catalog:
+        """Perform a cross-match between a catalog and a DataFrame.
+
+        This function converts the given DataFrame into a `Catalog` using `from_dataframe`
+        before calling `crossmatch`. It allows additional keyword arguments to be passed
+        to `from_dataframe` for customization.
+
+        See `crossmatch` for details on the cross-matching process.
+
+        Args:
+            other (npd.NestedFrame): The frame to cross-match against the catalog.
+            suffixes (Tuple[str, str], optional): A pair of suffixes to be appended to each column
+                name when they are joined. Defaults to using the catalog name as the suffix.
+            algorithm (BuiltInCrossmatchAlgorithm | Type[AbstractCrossmatchAlgorithm], optional): The
+                algorithm used for cross-matching. Can be either a string to specify one of the
+                built-in cross-matching methods, or a custom method defined by subclassing
+                AbstractCrossmatchAlgorithm. Defaults to `BuiltInCrossmatchAlgorithm.KD_TREE`.
+            output_catalog_name (str, optional): The name of the resulting catalog.
+                Default: `{left_name}_x_{right_name}`
+            require_right_margin (bool, optional): If `True`, raises an error if the right margin is missing,
+                which could lead to incomplete crossmatches. Default: `False`.
+            **kwargs: Additional keyword arguments for both `from_dataframe` and `crossmatch`.
+                Any arguments recognized by `from_dataframe` will be passed accordingly.
+
+        Returns:
+            Catalog: A `Catalog` with data from the input catalog and DataFrame, cross-matched
+            according to the specified algorithm.
+        """
+        # Lazy import to avoid circular dependencies.
+        # pylint: disable=C0415
+        from lsdb.loaders.dataframe.from_dataframe import from_dataframe
+
+        # Separate kwargs for from_dataframe and crossmatch
+        sig = inspect.signature(from_dataframe)
+        from_dataframe_arg_names = list(sig.parameters.keys())
+        from_dataframe_kwargs = {k: kwargs.pop(k) for k in from_dataframe_arg_names if k in kwargs}
+
+        # Convert the given DataFrame to a Catalog.
+        other_catalog = from_dataframe(other, **from_dataframe_kwargs)
+
+        # Call the crossmatch method with the newly generated Catalog.
+        return self.crossmatch(
+            other_catalog, suffixes, algorithm, output_catalog_name, require_right_margin, **kwargs
+        )
+
     def merge_map(
         self,
         map_catalog: MapCatalog,
diff --git a/tests/lsdb/catalog/test_crossmatch.py b/tests/lsdb/catalog/test_crossmatch.py
index b14ec271f..6fce2e503 100644
--- a/tests/lsdb/catalog/test_crossmatch.py
+++ b/tests/lsdb/catalog/test_crossmatch.py
@@ -153,6 +153,20 @@ def test_right_margin_missing(algo, small_sky_catalog, small_sky_xmatch_catalog)
         with pytest.raises(ValueError, match="Right catalog margin"):
             small_sky_catalog.crossmatch(small_sky_xmatch_catalog, algorithm=algo, require_right_margin=True)
 
+    @staticmethod
+    def test_dataframe_crossmatch(algo, small_sky_catalog, small_sky_xmatch_catalog, xmatch_correct):
+        small_sky_xmatch_dataframe = small_sky_xmatch_catalog.compute()
+        xmatched = small_sky_catalog.crossmatch_dataframe(
+            small_sky_xmatch_dataframe, algorithm=algo, radius_arcsec=0.01 * 3600, margin_threshold=100
+        ).compute()
+        assert isinstance(xmatched, npd.NestedFrame)
+        assert len(xmatched) == len(xmatch_correct)
+        for _, correct_row in xmatch_correct.iterrows():
+            assert correct_row["ss_id"] in xmatched["id_small_sky"].to_numpy()
+            xmatch_row = xmatched[xmatched["id_small_sky"] == correct_row["ss_id"]]
+            assert xmatch_row["id_from_lsdb_dataframe"].to_numpy() == correct_row["xmatch_id"]
+            assert xmatch_row["_dist_arcsec"].to_numpy() == pytest.approx(correct_row["dist"] * 3600)
+
 
 @pytest.mark.parametrize("algo", [BoundedKdTreeCrossmatch])
 class TestBoundedCrossmatch:


Reviews:
user: login: camposandro
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-05T14:57:22Z
user: login: olivialynn
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-05T17:15:41Z
user: login: olivialynn
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-05T17:33:02Z
user: login: camposandro
body: Looks good to me!
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-03-05T19:18:56Z
user: login: camposandro
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-05T19:19:45Z
user: login: olivialynn
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-06T02:56:08Z

Review comments:
diff_hunk: @@ -242,6 +242,77 @@ def crossmatch(
         )
         return Catalog(ddf, ddf_map, hc_catalog)
 
+    def crossmatch_dataframe(
+        self,
+        other: pd.DataFrame,
path: src/lsdb/catalog/catalog.py
user: login: camposandro
body: Super minor but I would probably rename this argument to `frame`
created_at: 2025-03-05T14:49:20Z
author_association: COLLABORATOR
diff_hunk: @@ -242,6 +242,77 @@ def crossmatch(
         )
         return Catalog(ddf, ddf_map, hc_catalog)
 
+    def crossmatch_dataframe(
+        self,
+        other: pd.DataFrame,
+        suffixes: tuple[str, str] | None = None,
+        algorithm: (
+            Type[AbstractCrossmatchAlgorithm] | BuiltInCrossmatchAlgorithm
+        ) = BuiltInCrossmatchAlgorithm.KD_TREE,
+        output_catalog_name: str | None = None,
+        require_right_margin: bool = False,
+        **kwargs,
+    ) -> Catalog:
+        """Perform a cross-match between a catalog and a DataFrame.
+
+        This function converts the given DataFrame into a `Catalog` using `from_dataframe`
+        before calling `crossmatch`. It allows additional keyword arguments to be passed
+        to `from_dataframe` for customization.
+
+        See `crossmatch` for details on the cross-matching process.
+
+        Args:
+            other (pd.DataFrame): The DataFrame to cross-match against the catalog.
+            suffixes (Tuple[str, str], optional): A pair of suffixes to be appended to each column
+                name when they are joined. Defaults to using the catalog name as the suffix.
+            algorithm (BuiltInCrossmatchAlgorithm | Type[AbstractCrossmatchAlgorithm], optional): The
+                algorithm used for cross-matching. Can be either a string to specify one of the
+                built-in cross-matching methods, or a custom method defined by subclassing
+                AbstractCrossmatchAlgorithm. Defaults to `BuiltInCrossmatchAlgorithm.KD_TREE`.
+            output_catalog_name (str, optional): The name of the resulting catalog.
+                Default: `{left_name}_x_{right_name}`
+            require_right_margin (bool, optional): If `True`, raises an error if the right margin is missing,
+                which could lead to incomplete crossmatches. Default: `False`.
+            **kwargs: Additional keyword arguments for both `from_dataframe` and `crossmatch`.
+                Any arguments recognized by `from_dataframe` will be passed accordingly.
+
+        Returns:
+            Catalog: A `Catalog` with data from the input catalog and DataFrame, cross-matched
+            according to the specified algorithm.
+        """
+        # Lazy import to avoid circular dependencies.
+        # pylint: disable=C0415
+        from lsdb.loaders.dataframe.from_dataframe import from_dataframe
+
+        # Separate kwargs for from_dataframe and crossmatch
+        from_dataframe_kwargs = {
path: src/lsdb/catalog/catalog.py
user: login: camposandro
body: We could pass these `from_dataframe` kwargs as a dictionary argument to the function:
```
 def crossmatch_dataframe(..., from_dataframe_kwargs: dict = {}, **kwargs)
```

But I don't love it, it would definitely be less user-friendly.

I just tried this for the first time - apparently we can dynamically inspect a method to get its argument names:
```
import inspect
sig = inspect.signature(lsdb.from_dataframe)
from_dataframe_arg_names = list(sig.parameters.keys())
```

This way we would not have to keep track of changes on the `from_dataframe` signature. 

What do you think of this solution?
created_at: 2025-03-05T14:50:56Z
author_association: COLLABORATOR
diff_hunk: @@ -242,6 +242,77 @@ def crossmatch(
         )
         return Catalog(ddf, ddf_map, hc_catalog)
 
+    def crossmatch_dataframe(
+        self,
+        other: pd.DataFrame,
path: src/lsdb/catalog/catalog.py
user: login: olivialynn
body: ```suggestion
        other: npd.NestedFrame,
```
created_at: 2025-03-05T17:15:41Z
author_association: MEMBER
diff_hunk: @@ -242,6 +242,77 @@ def crossmatch(
         )
         return Catalog(ddf, ddf_map, hc_catalog)
 
+    def crossmatch_dataframe(
+        self,
+        other: pd.DataFrame,
+        suffixes: tuple[str, str] | None = None,
+        algorithm: (
+            Type[AbstractCrossmatchAlgorithm] | BuiltInCrossmatchAlgorithm
+        ) = BuiltInCrossmatchAlgorithm.KD_TREE,
+        output_catalog_name: str | None = None,
+        require_right_margin: bool = False,
+        **kwargs,
+    ) -> Catalog:
+        """Perform a cross-match between a catalog and a DataFrame.
+
+        This function converts the given DataFrame into a `Catalog` using `from_dataframe`
+        before calling `crossmatch`. It allows additional keyword arguments to be passed
+        to `from_dataframe` for customization.
+
+        See `crossmatch` for details on the cross-matching process.
+
+        Args:
+            other (pd.DataFrame): The DataFrame to cross-match against the catalog.
+            suffixes (Tuple[str, str], optional): A pair of suffixes to be appended to each column
+                name when they are joined. Defaults to using the catalog name as the suffix.
+            algorithm (BuiltInCrossmatchAlgorithm | Type[AbstractCrossmatchAlgorithm], optional): The
+                algorithm used for cross-matching. Can be either a string to specify one of the
+                built-in cross-matching methods, or a custom method defined by subclassing
+                AbstractCrossmatchAlgorithm. Defaults to `BuiltInCrossmatchAlgorithm.KD_TREE`.
+            output_catalog_name (str, optional): The name of the resulting catalog.
+                Default: `{left_name}_x_{right_name}`
+            require_right_margin (bool, optional): If `True`, raises an error if the right margin is missing,
+                which could lead to incomplete crossmatches. Default: `False`.
+            **kwargs: Additional keyword arguments for both `from_dataframe` and `crossmatch`.
+                Any arguments recognized by `from_dataframe` will be passed accordingly.
+
+        Returns:
+            Catalog: A `Catalog` with data from the input catalog and DataFrame, cross-matched
+            according to the specified algorithm.
+        """
+        # Lazy import to avoid circular dependencies.
+        # pylint: disable=C0415
+        from lsdb.loaders.dataframe.from_dataframe import from_dataframe
+
+        # Separate kwargs for from_dataframe and crossmatch
+        from_dataframe_kwargs = {
path: src/lsdb/catalog/catalog.py
user: login: olivialynn
body: Yeah! I was thinking about the dictionary argument, but it feels a little counter to the idea of adding this method for increased usability, so I admit I also wasn't too keen.

I love the dynamic inspection—nice find, thanks!
created_at: 2025-03-05T17:33:02Z
author_association: MEMBER
diff_hunk: @@ -225,6 +226,61 @@ def crossmatch(
         )
         return self.__class__(ddf, ddf_map, hc_catalog)
 
+    def crossmatch_dataframe(
+        self,
+        other: npd.NestedFrame,
+        suffixes: tuple[str, str] | None = None,
+        algorithm: (
+            Type[AbstractCrossmatchAlgorithm] | BuiltInCrossmatchAlgorithm
+        ) = BuiltInCrossmatchAlgorithm.KD_TREE,
+        output_catalog_name: str | None = None,
+        require_right_margin: bool = False,
+        **kwargs,
+    ) -> Catalog:
+        """Perform a cross-match between a catalog and a DataFrame.
+
+        This function converts the given DataFrame into a `Catalog` using `from_dataframe`
+        before calling `crossmatch`. It allows additional keyword arguments to be passed
+        to `from_dataframe` for customization.
+
+        See `crossmatch` for details on the cross-matching process.
+
+        Args:
+            other (pd.DataFrame): The DataFrame to cross-match against the catalog.
path: src/lsdb/catalog/catalog.py
user: login: camposandro
body: Oops sorry, if you could still update this argument type in the docstring before you merge that'd be great!
created_at: 2025-03-05T19:19:45Z
author_association: COLLABORATOR
diff_hunk: @@ -225,6 +226,61 @@ def crossmatch(
         )
         return self.__class__(ddf, ddf_map, hc_catalog)
 
+    def crossmatch_dataframe(
+        self,
+        other: npd.NestedFrame,
+        suffixes: tuple[str, str] | None = None,
+        algorithm: (
+            Type[AbstractCrossmatchAlgorithm] | BuiltInCrossmatchAlgorithm
+        ) = BuiltInCrossmatchAlgorithm.KD_TREE,
+        output_catalog_name: str | None = None,
+        require_right_margin: bool = False,
+        **kwargs,
+    ) -> Catalog:
+        """Perform a cross-match between a catalog and a DataFrame.
+
+        This function converts the given DataFrame into a `Catalog` using `from_dataframe`
+        before calling `crossmatch`. It allows additional keyword arguments to be passed
+        to `from_dataframe` for customization.
+
+        See `crossmatch` for details on the cross-matching process.
+
+        Args:
+            other (pd.DataFrame): The DataFrame to cross-match against the catalog.
path: src/lsdb/catalog/catalog.py
user: login: olivialynn
body: ```suggestion
            other (npd.NestedFrame): The frame to cross-match against the catalog.
```
created_at: 2025-03-06T02:56:08Z
author_association: MEMBER

PR comments:
user: login: codecov[bot]
created_at: 2025-03-04T19:57:58Z
body: ## [Codecov](https://app.codecov.io/gh/astronomy-commons/lsdb/pull/602?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=astronomy-commons) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 97.37%. Comparing base [(`c2cdcf0`)](https://app.codecov.io/gh/astronomy-commons/lsdb/commit/c2cdcf0bad00ca246d8716079a4d5e53e11cbf6d?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=astronomy-commons) to head [(`6be4287`)](https://app.codecov.io/gh/astronomy-commons/lsdb/commit/6be42879bda463f7d3f6b9fb057f6e45ffcdd41f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=astronomy-commons).
> Report is 196 commits behind head on main.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main     #602      +/-   ##
==========================================
+ Coverage   97.36%   97.37%   +0.01%     
==========================================
  Files          43       43              
  Lines        1785     1793       +8     
==========================================
+ Hits         1738     1746       +8     
  Misses         47       47              
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/astronomy-commons/lsdb/pull/602?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=astronomy-commons).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=astronomy-commons).

<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
author_association: NONE
user: login: github-actions[bot]
created_at: 2025-03-04T19:59:34Z
body: | Before [c2cdcf0b]    | After [daa3d8ec]    |   Ratio | Benchmark (Parameter)                   |
|----------------------|---------------------|---------|-----------------------------------------|
| 12.8±0.01s           | 12.9±0.06s          |    1.01 | benchmarks.time_create_large_catalog    |
| 88.9±1ms             | 88.5±0.9ms          |    1    | benchmarks.time_kdtree_crossmatch       |
| 1.91±0.02s           | 1.89±0.02s          |    0.99 | benchmarks.time_create_midsize_catalog  |
| 15.7±0.3ms           | 15.2±0.1ms          |    0.97 | benchmarks.time_box_filter_on_partition |
| 29.9±0.5ms           | 28.9±0.5ms          |    0.97 | benchmarks.time_polygon_search          |


Click [here](https://github.com/astronomy-commons/lsdb/actions/runs/13690242486/job/38282070639#step:10:1) to view all benchmarks.
author_association: NONE