Repository: cleanlab/cleanlab

Issue:
Datalab.get_issues() for cleanvision issue types fails
<!-- Briefly summarize the issue. INCLUDE the exact code you are trying to run! Ideally in a self-contained script so we can reproduce your bug. -->

`Datalab.get_issues()` looks at the list of possible issue types as of #1025, this does not consider cases when an image column name is specified, as this includes Imagelab that looks for image-related issue types.

This led to an error in the CI (see [stack trace](#user-content-stack-trace)), where the image tutorial notebook cannot call `lab.get_issues("dark")` for an image dataset, while "dark" is a valid issue type in the cleanvision package. Datalab can already find "dark" issues, but the validation step in the `get_issues` method considers cleanvision-specific issue types "invalid".

This issue is resolved when:

- `Datalab` can list the issue types it supports in Imagelab when calling `find_issues`(there are a few that are not currently supported).
- A test is added where `lab.get_issues("dark")` should return the expected type of DataFrame (or rather that the expected columns score and issue columns are there).

<h1 id="stack-trace">Stack trace</h1>

<!-- If applicable, please include a full stack trace here. If you need to omit
the bottom of the stack trace (e.g. it includes stack frames from your private
code), that is okay. Try to include all cleanlab stack frames. -->

```shell
Notebook error:
CellExecutionError in tutorials/image.ipynb:
------------------
dark_issues = lab.get_issues("dark")
dark_issues_df = dark_issues.query("is_dark_issue").sort_values("dark_score")
dark_issues_df.head()
------------------


---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Input In [26], in <module>
----> 1 dark_issues = lab.get_issues("dark")
      2 dark_issues_df = dark_issues.query("is_dark_issue").sort_values("dark_score")
      3 dark_issues_df.head()

File ~/work/cleanlab/cleanlab/cleanlab/datalab/datalab.py:519, in Datalab.get_issues(self, issue_name)
    517 # Validate issue_name
    518 if issue_name is not None and issue_name not in self.list_possible_issue_types():
--> 519     raise ValueError(
    520         f"""Invalid issue_name: {issue_name}. Please specify a valid issue_name from the list of possible issue types.
    521     Either, specify one of the following: {self.list_possible_issue_types()}
    522     or set issue_name as None to get all issue types.
    523     """
    524     )
    525 return self.data_issues.get_issues(issue_name=issue_name)

ValueError: Invalid issue_name: dark. Please specify a valid issue_name from the list of possible issue types.
            Either, specify one of the following: ['outlier', 'label', 'near_duplicate', 'non_iid', 'class_imbalance', 'underperforming_group', 'data_valuation', 'null']
            or set issue_name as None to get all issue types.
            

You can ignore this error by setting the following in conf.py:

    nbsphinx_allow_errors = True

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.11.8/x64/bin/sphinx-multiversion", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.8/x64/lib/python3.11/site-packages/sphinx_multiversion/main.py", line [338](https://github.com/cleanlab/cleanlab/actions/runs/8028055736/job/21932780898#step:18:339), in main
    subprocess.check_call(cmd, cwd=current_cwd)
  File "/opt/hostedtoolcache/Python/3.11.8/x64/lib/python3.11/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '('/opt/hostedtoolcache/Python/3.11.8/x64/bin/python3', '-R', '-m', 'sphinx', '-D', 'smv_metadata_path=/tmp/tmpnljvr_tm/versions.json', '-D', 'smv_branch_whitelist=master', '-D', 'smv_tag_whitelist=None', '-D', 'smv_current_version=master', '-c', '/home/runner/work/cleanlab/cleanlab/docs/source', '/tmp/tmpnljvr_tm/22aa91c29e81ea930cff74d93df083548fcf1f8a/docs/source', '/home/runner/work/cleanlab/cleanlab/cleanlab-docs/master')' returned non-zero exit status 2.
reading sources... [ 88%] tutorials/image
Error: Process completed with exit code 1.
```
<!-- Please include any other information that could be helpful for debugging. -->



Discussion:

Resolving patch:
diff --git a/cleanlab/datalab/datalab.py b/cleanlab/datalab/datalab.py
--- a/cleanlab/datalab/datalab.py
+++ b/cleanlab/datalab/datalab.py
@@ -28,6 +28,7 @@
 import pandas as pd
 
 import cleanlab
+from cleanlab.datalab.internal.adapter.constants import DEFAULT_CLEANVISION_ISSUES
 from cleanlab.datalab.internal.adapter.imagelab import create_imagelab
 from cleanlab.datalab.internal.data import Data
 from cleanlab.datalab.internal.display import _Displayer
@@ -571,7 +572,10 @@ def list_possible_issue_types(self) -> List[str]:
         --------
         :py:class:`REGISTRY <cleanlab.datalab.internal.issue_manager_factory.REGISTRY>` : All available issue types and their corresponding issue managers can be found here.
         """
-        return _list_possible_issue_types(task=self.task)
+        possible_issue_types = _list_possible_issue_types(task=self.task)
+        if self._imagelab is not None:
+            possible_issue_types.extend(DEFAULT_CLEANVISION_ISSUES.keys())
+        return possible_issue_types
 
     def list_default_issue_types(self) -> List[str]:
         """Returns a list of the issue types that are run by default
@@ -581,7 +585,10 @@ def list_default_issue_types(self) -> List[str]:
         --------
         :py:class:`REGISTRY <cleanlab.datalab.internal.issue_manager_factory.REGISTRY>` : All available issue types and their corresponding issue managers can be found here.
         """
-        return _list_default_issue_types(task=self.task)
+        default_issue_types = _list_default_issue_types(task=self.task)
+        if self._imagelab is not None:
+            default_issue_types.extend(DEFAULT_CLEANVISION_ISSUES.keys())
+        return default_issue_types
 
     def save(self, path: str, force: bool = False) -> None:
         """Saves this Datalab object to file (all files are in folder at `path/`).


Test patch:
diff --git a/tests/datalab/test_cleanvision_integration.py b/tests/datalab/test_cleanvision_integration.py
--- a/tests/datalab/test_cleanvision_integration.py
+++ b/tests/datalab/test_cleanvision_integration.py
@@ -250,3 +250,53 @@ def test_labels_not_required_for_imagelab_issues(
 
         for issue_type in IMAGELAB_ISSUE_TYPES:
             assert issue_type in captured.out
+
+    @pytest.fixture
+    def lab(self, image_dataset):
+        lab = Datalab(data=image_dataset, label_name=LABEL_NAME, image_key=IMAGE_NAME)
+        lab.find_issues()
+        return lab
+
+    def test_get_summary(self, lab):
+        summary = lab.get_issue_summary("dark")
+        assert len(summary) == 1
+        num_issues = summary["num_issues"].values[0]
+        assert num_issues == 1
+
+    @pytest.mark.parametrize(
+        "list_method", ["list_possible_issue_types", "list_default_issue_types"]
+    )
+    def test_list_issue_type_method(self, image_dataset, lab, list_method):
+        method = getattr(lab, list_method)
+        issue_types = method()
+
+        # Check that Datalab without Imagelab injected has just a subset of possible/default issue types
+        minimal_lab = Datalab(data=image_dataset)
+        minimal_method = getattr(minimal_lab, list_method)
+        datalab_issue_types = minimal_method()
+        assert set(datalab_issue_types).issubset(set(issue_types))
+
+        # The additional issue types found by method should be the same as IMAGELAB_ISSUE_TYPES
+        assert set(issue_types).difference(datalab_issue_types) == set(IMAGELAB_ISSUE_TYPES)
+
+    @pytest.mark.issue1027
+    def test_get_issues(self, lab):
+        """
+        Test the `get_issues` method of the `lab` object.
+
+        This method checks if the columns returned by the `get_issues` method
+        match the expected columns for each issue type defined in `IMAGELAB_ISSUE_TYPES`.
+
+        Raises:
+            AssertionError: If the columns returned by `get_issues` do not match the expected columns.
+
+        """
+        test_condition = lambda s: set(lab.get_issues(s).columns) == set(
+            [f"{s}_score", f"is_{s}_issue"]
+        )
+        failed_assertions = [
+            issue_type for issue_type in IMAGELAB_ISSUE_TYPES if not test_condition(issue_type)
+        ]
+        assert (
+            len(failed_assertions) == 0
+        ), f"Tests for `get_issues` with these `issue_types` failed: {failed_assertions}"