Repository: laminlabs/lamindb
Pull request #2742: 🐛 Fix curating DataFrame index & 🚸 Automatically pass validation of ensembl gene IDs  & ✨ add curator __repr__
Author: login: Zethson
Created: 2025-05-05T10:05:17Z | Merged: 2025-05-06T16:12:19Z

1. Fixes the curation of DataFrame indices and ensures correct behavior with a new test.

```python
ensembl_ids = [
    "ENSG00000139618", # BRCA2
    "ENSG00009000000",
]

df = pd.DataFrame(index=pd.Index(ensembl_ids, name="ensembl"))
df

schema = ln.Schema(
    index=ln.Feature(name="ensembl", dtype=bt.Gene.ensembl_gene_id).save()
).save()

curator = ln.curators.DataFrameCurator(df, schema)
```

led to

```
---------------------------------------------------------------------------
UnboundLocalError                         Traceback (most recent call last)
Cell In[19], line 1
----> 1 curator = ln.curators.DataFrameCurator(df, schema)
File ~/code/lamindb/lamindb/curators/core.py:430, in DataFrameCurator.init(self, dataset, schema, slot)
    425             categoricals.append(feature)
    426 if schema._index_feature_uid is not None:
    427     # in almost no case, an index should have a pandas.CategoricalDtype in a DataFrame
    428     # so, we're typing it as str here
    429     index = pandera.Index(
--> 430         schema.index.dtype if not feature.dtype.startswith("cat") else str
    431     )
    432 else:
    433     index = None
UnboundLocalError: cannot access local variable 'feature' where it is not associated with a value
```

2. Currently, we consider additional columns in `obs` DataFrames that are not a part of the `minimal_set` as just additional columns that we do not validate and simply ignore. Analogously to this behavior, we now consider additional features in `var.T` such as (for now) ensembl gene IDs as fine and do not raise a ValidationError. This is an easy improvement for cases where additional gene IDs are found because of different ensembl versions.

| Before | After |
|--------|-------|
| ![image](https://github.com/user-attachments/assets/66a6fcf1-5aae-464b-a66e-84d1641e950e) | ![image](https://github.com/user-attachments/assets/01e3d162-9834-4fe3-99ab-5aaf4ed5da66) |

With `maximal_set=True`, additional ensembl IDs are not allowed and it will raise an error accordingly.

![image](https://github.com/user-attachments/assets/446febab-9e6f-4a9e-b51e-afc23d04df53)

3. Introduces new `__repr__` for `CatCurate` and more importantly, `Curator`

| Before | After |
|--------|-------|
| ![image](https://github.com/user-attachments/assets/a3b725b7-47c3-49aa-ae0b-0c3034e75ded) | ![image](https://github.com/user-attachments/assets/6099b9b3-616a-4df3-ab7a-9786c2e26d55) |

4. Improves the Schema docstrings

Fixes https://github.com/laminlabs/lamindb/issues/2735


Diff:
diff --git a/docs/curate.ipynb b/docs/curate.ipynb
index 71c944758a..9990262ca0 100644
--- a/docs/curate.ipynb
+++ b/docs/curate.ipynb
@@ -821,7 +821,7 @@
    "id": "70",
    "metadata": {},
    "source": [
-    "## SpatialData"
+    "## MuData"
    ]
   },
   {
@@ -830,24 +830,17 @@
    "metadata": {},
    "source": [
     "```{eval-rst}\n",
-    ".. literalinclude:: scripts/define_schema_spatialdata.py\n",
+    ".. literalinclude:: scripts/curate-mudata.py\n",
     "   :language: python\n",
-    "   :caption: define_schema_spatialdata.py\n",
     "```"
    ]
   },
   {
-   "cell_type": "code",
-   "execution_count": null,
+   "cell_type": "markdown",
    "id": "72",
-   "metadata": {
-    "tags": [
-     "hide-output"
-    ]
-   },
-   "outputs": [],
+   "metadata": {},
    "source": [
-    "!python scripts/define_schema_spatialdata.py"
+    "## SpatialData"
    ]
   },
   {
@@ -856,9 +849,9 @@
    "metadata": {},
    "source": [
     "```{eval-rst}\n",
-    ".. literalinclude:: scripts/curate_spatialdata.py\n",
+    ".. literalinclude:: scripts/define_schema_spatialdata.py\n",
     "   :language: python\n",
-    "   :caption: curate_spatialdata.py\n",
+    "   :caption: define_schema_spatialdata.py\n",
     "```"
    ]
   },
@@ -873,7 +866,7 @@
    },
    "outputs": [],
    "source": [
-    "!python scripts/curate_spatialdata.py"
+    "!python scripts/define_schema_spatialdata.py"
    ]
   },
   {
@@ -881,18 +874,25 @@
    "id": "75",
    "metadata": {},
    "source": [
-    "## MuData"
+    "```{eval-rst}\n",
+    ".. literalinclude:: scripts/curate_spatialdata.py\n",
+    "   :language: python\n",
+    "   :caption: curate_spatialdata.py\n",
+    "```"
    ]
   },
   {
-   "cell_type": "markdown",
+   "cell_type": "code",
+   "execution_count": null,
    "id": "76",
-   "metadata": {},
+   "metadata": {
+    "tags": [
+     "hide-output"
+    ]
+   },
+   "outputs": [],
    "source": [
-    "```{eval-rst}\n",
-    ".. literalinclude:: scripts/curate-mudata.py\n",
-    "   :language: python\n",
-    "```"
+    "!python scripts/curate_spatialdata.py"
    ]
   },
   {
diff --git a/docs/scripts/define_schema_spatialdata.py b/docs/scripts/define_schema_spatialdata.py
index 91b42bb908..7c1d50cd88 100644
--- a/docs/scripts/define_schema_spatialdata.py
+++ b/docs/scripts/define_schema_spatialdata.py
@@ -32,7 +32,8 @@
     ],
 ).save()
 
-varT_schema = ln.Schema(itype=bt.Gene.ensembl_gene_id).save()
+# Schema enforces only registered Ensembl Gene IDs are valid (maximal_set=True)
+varT_schema = ln.Schema(itype=bt.Gene.ensembl_gene_id, maximal_set=True).save()
 
 sdata_schema = ln.Schema(
     name="spatialdata_blobs_schema",
diff --git a/lamindb/core/datasets/mini_immuno.py b/lamindb/core/datasets/mini_immuno.py
index f476976879..7ff9fa6bd4 100644
--- a/lamindb/core/datasets/mini_immuno.py
+++ b/lamindb/core/datasets/mini_immuno.py
@@ -64,7 +64,9 @@ def get_dataset1(
     with_cell_type_synonym: bool = False,
     with_cell_type_typo: bool = False,
     with_gene_typo: bool = False,
+    with_outdated_gene: bool = False,
     with_wrong_subtype: bool = False,
+    with_index_type_mismatch: bool = False,
 ) -> pd.DataFrame | ad.AnnData:
     """A small tabular dataset measuring expression & metadata."""
     # define the data in the dataset
@@ -77,7 +79,11 @@ def get_dataset1(
         var_ids = [
             "ENSG00000153563",
             "ENSG00000010610",
-            "ENSG00000170458" if not with_gene_typo else "GeneTypo",
+            "ENSG00000170458"
+            if not with_gene_typo
+            else "GeneTypo"
+            if not with_outdated_gene
+            else "ENSG00000278198",
         ]
     abt_cell = (
         "CD8-pos alpha-beta T cell"
@@ -107,7 +113,12 @@ def get_dataset1(
         "study_note": "We had a great time performing this study and the results look compelling.",
     }
     # the dataset as DataFrame
-    dataset_df = pd.DataFrame(dataset_dict, index=["sample1", "sample2", "sample3"])
+    dataset_df = pd.DataFrame(
+        dataset_dict,
+        index=["sample1", "sample2", 0]  # type: ignore
+        if with_index_type_mismatch
+        else ["sample1", "sample2", "sample3"],
+    )
     if otype == "DataFrame":
         for key, value in metadata.items():
             dataset_df.attrs[key] = value
diff --git a/lamindb/curators/core.py b/lamindb/curators/core.py
index b00eb4cc63..1f6f878fc1 100644
--- a/lamindb/curators/core.py
+++ b/lamindb/curators/core.py
@@ -205,6 +205,69 @@ def save_artifact(
         # constructor signature
         pass  # pragma: no cover
 
+    def __repr__(self) -> str:
+        from lamin_utils import colors
+
+        if self._schema is not None:
+            # Schema might have different attributes
+            if hasattr(self._schema, "name") and self._schema.name:
+                schema_str = colors.italic(self._schema.name)
+            elif hasattr(self._schema, "uid"):
+                schema_str = colors.italic(f"uid={self._schema.uid}")
+            elif hasattr(self._schema, "id"):
+                schema_str = colors.italic(f"id={self._schema.id}")
+            else:
+                schema_str = colors.italic("unnamed")
+
+            # Add schema type info if available
+            if hasattr(self._schema, "otype") and self._schema.otype:
+                schema_str += f" ({self._schema.otype})"
+        else:
+            schema_str = colors.warning("None")
+
+        status_str = ""
+        if self._is_validated:
+            status_str = f", {colors.green('validated')}"
+        else:
+            status_str = f", {colors.yellow('unvalidated')}"
+
+        cls_name = colors.green(self.__class__.__name__)
+
+        # Get additional info based on curator type
+        extra_info = ""
+        if hasattr(self, "_slots") and self._slots:
+            # For SlotsCurator and its subclasses
+            slots_count = len(self._slots)
+            if slots_count > 0:
+                slot_names = list(self._slots.keys())
+                if len(slot_names) <= 3:
+                    extra_info = f", slots: {slot_names}"
+                else:
+                    extra_info = f", slots: [{', '.join(slot_names[:3])}... +{len(slot_names) - 3} more]"
+        elif (
+            cls_name == "DataFrameCurator"
+            and hasattr(self, "cat")
+            and hasattr(self.cat, "_categoricals")
+        ):
+            # For DataFrameCurator
+            cat_count = len(getattr(self.cat, "_categoricals", []))
+            if cat_count > 0:
+                extra_info = f", categorical_features={cat_count}"
+
+        artifact_info = ""
+        if self._artifact is not None:
+            artifact_uid = getattr(self._artifact, "uid", str(self._artifact))
+            short_uid = (
+                str(artifact_uid)[:8] + "..."
+                if len(str(artifact_uid)) > 8
+                else str(artifact_uid)
+            )
+            artifact_info = f", artifact: {colors.italic(short_uid)}"
+
+        return (
+            f"{cls_name}{artifact_info}(Schema: {schema_str}{extra_info}{status_str})"
+        )
+
 
 # default implementation for AnnDataCurator, MuDataCurator, and SpatialDataCurator
 class SlotsCurator(Curator):
@@ -427,7 +490,9 @@ def __init__(
                 # in almost no case, an index should have a pandas.CategoricalDtype in a DataFrame
                 # so, we're typing it as `str` here
                 index = pandera.Index(
-                    schema.index.dtype if not feature.dtype.startswith("cat") else str
+                    schema.index.dtype
+                    if not schema.index.dtype.startswith("cat")
+                    else str
                 )
             else:
                 index = None
@@ -445,6 +510,7 @@ def __init__(
             categoricals=categoricals,
             index=schema.index,
             slot=slot,
+            schema_maximal_set=schema.maximal_set,
         )
 
     @property
@@ -498,6 +564,7 @@ def standardize(self) -> None:
 
     def _cat_manager_validate(self) -> None:
         self.cat.validate()
+
         if self.cat._is_validated:
             self._is_validated = True
         else:
@@ -783,7 +850,7 @@ def __init__(
 
 
 class CatVector:
-    """Categorical column for `DataFrame`.
+    """Categorical vector for `DataFrame`.
 
     Args:
         values_getter: A callable or iterable that returns the values to validate.
@@ -803,6 +870,7 @@ def __init__(
         feature: Feature | None = None,
         cat_manager: DataFrameCatManager | None = None,
         subtype_str: str = "",
+        maximal_set: bool = False,  # Passed during validation. Whether unvalidated categoricals cause validation failure.
     ) -> None:
         self._values_getter = values_getter
         self._values_setter = values_setter
@@ -818,6 +886,7 @@ def __init__(
         self._cat_manager = cat_manager
         self.feature = feature
         self.records = None
+        self._maximal_set = maximal_set
         if hasattr(field.field.model, "_name_field"):
             label_ref_is_name = field.field.name == field.field.model._name_field
         else:
@@ -842,11 +911,22 @@ def values(self, new_values):
 
     @property
     def is_validated(self) -> bool:
-        """Return whether the column is validated."""
-        return len(self._non_validated) == 0
+        """Whether the vector is validated."""
+        # ensembl gene IDs pass even if they were not validated
+        # this is a simple solution to the ensembl gene version problem
+        if self._field.field.attname == "ensembl_gene_id":
+            # if none of the ensembl gene ids were validated, we are probably not looking at ensembl gene IDs
+            if len(self.values) == len(self._non_validated):
+                return False
+            # if maximal set, we do not allow additional unvalidated genes
+            elif len(self._non_validated) != 0 and self._maximal_set:
+                return False
+            return True
+        else:
+            return len(self._non_validated) == 0
 
     def _replace_synonyms(self) -> list[str]:
-        """Replace synonyms in the column with standardized values."""
+        """Replace synonyms in the vector with standardized values."""
         syn_mapper = self._synonyms
         # replace the values in df
         std_values = self.values.map(
@@ -870,6 +950,20 @@ def _replace_synonyms(self) -> list[str]:
             )
         return std_values
 
+    def __repr__(self) -> str:
+        if self._non_validated is None:
+            status = "unvalidated"
+        else:
+            status = (
+                "validated"
+                if len(self._non_validated) == 0
+                else f"non-validated ({len(self._non_validated)})"
+            )
+
+        field_name = getattr(self._field, "name", str(self._field))
+        values_count = len(self.values) if hasattr(self.values, "__len__") else "?"
+        return f"CatVector(key='{self._key}', field='{field_name}', values={values_count}, {status})"
+
     def _add_validated(self) -> tuple[list, list]:
         """Save features or labels records in the default instance."""
         from lamindb.models.save import save as ln_save
@@ -1060,16 +1154,17 @@ def _log_mapping_info():
             return non_validated, syn_mapper
 
     def validate(self) -> None:
-        """Validate the column."""
+        """Validate the vector."""
         # add source-validated values to the registry
         self._validated, self._non_validated = self._add_validated()
         self._non_validated, self._synonyms = self._validate(values=self._non_validated)
+
         # always register new Features if they are columns
         if self._key == "columns" and self._field == Feature.name:
             self.add_new()
 
     def standardize(self) -> None:
-        """Standardize the column."""
+        """Standardize the vector."""
         registry = self._field.field.model
         if not hasattr(registry, "standardize"):
             return self.values
@@ -1123,6 +1218,7 @@ def __init__(
         sources: dict[str, Record] | None = None,
         index: Feature | None = None,
         slot: str | None = None,
+        schema_maximal_set: bool = False,
     ) -> None:
         self._non_validated = None
         self._index = index
@@ -1139,6 +1235,8 @@ def __init__(
         self._validate_category_error_messages: str = ""
         self._cat_vectors: dict[str, CatVector] = {}
         self._slot = slot
+        self._maximal_set = schema_maximal_set
+
         if columns_names is None:
             columns_names = []
         if columns_field == Feature.name:
@@ -1148,6 +1246,7 @@ def __init__(
                 key="columns" if isinstance(self._dataset, pd.DataFrame) else "keys",
                 source=self._sources.get("columns"),
                 cat_manager=self,
+                maximal_set=self._maximal_set,
             )
         else:
             self._cat_vectors["columns"] = CatVector(
@@ -1159,6 +1258,7 @@ def __init__(
                 key="columns",
                 source=self._sources.get("columns"),
                 cat_manager=self,
+                maximal_set=self._maximal_set,
             )
         for feature in self._categoricals:
             result = parse_dtype(feature.dtype)[
@@ -1180,6 +1280,7 @@ def __init__(
                 feature=feature,
                 cat_manager=self,
                 subtype_str=subtype_str,
+                maximal_set=self._maximal_set,
             )
         if index is not None and index.dtype.startswith("cat"):
             result = parse_dtype(index.dtype)[0]
@@ -1191,6 +1292,7 @@ def __init__(
                 key=key,
                 feature=index,
                 cat_manager=self,
+                maximal_set=self._maximal_set,
             )
 
     @property
@@ -1237,7 +1339,10 @@ def validate(self) -> bool:
         if self._index is not None:
             # cat_vector.validate() populates validated labels
             # the index should become part of the feature set corresponding to the dataframe
-            self._cat_vectors["columns"].records.insert(0, self._index)  # type: ignore
+            if self._cat_vectors["columns"].records is not None:
+                self._cat_vectors["columns"].records.insert(0, self._index)  # type: ignore
+            else:
+                self._cat_vectors["columns"].records = [self._index]  # type: ignore
 
         return self._is_validated
 
diff --git a/lamindb/models/schema.py b/lamindb/models/schema.py
index b0ab05e5a6..b6ad02142a 100644
--- a/lamindb/models/schema.py
+++ b/lamindb/models/schema.py
@@ -252,36 +252,31 @@ def add(self, features: Feature | list[Feature]) -> None:
 
 
 class Schema(Record, CanCurate, TracksRun):
-    """Schemas of datasets such as the set of columns of a `DataFrame`.
+    """Schemas of a dataset such as the set of columns of a `DataFrame`.
 
-    A simple schema is a feature set such as the set of columns of a `DataFrame`.
-
-    A composite schema has multiple slots, e.g., for an `AnnData`, one schema for slot `obs` and another one for `var`.
-
-    A schema can also merely define abstract constraints or instructions for dataset validation & annotation.
+    Composite schemas can have multiple slots, e.g., for an `AnnData`, one schema for slot `obs` and another one for `var`.
 
     Args:
-        features: `list[Record] | list[tuple[Feature, dict]] | None = None` A list of feature
-            records, e.g., `[Feature(...), Feature(...)]` or features with their config, e.g., `[Feature(...).with_config(optional=True)]`.
-        index: `Feature | None = None` A :class:`~lamindb.Feature` record to validate an index of a `DataFrame`.
-        slots: `dict[str, Schema] | None = None` A dictionary mapping slot names to
-            :class:`~lamindb.Schema` objects.
-        name: `str | None = None` A name.
-        description: `str | None = None` A description.
-        itype: `str | None = None` The feature identifier type (e.g. :class:`~lamindb.Feature`, :class:`~bionty.Gene`, ...).
+        features: `list[Record] | list[tuple[Feature, dict]] | None = None` Feature
+            records, e.g., `[Feature(...), Feature(...)]` or Features with their config, e.g., `[Feature(...).with_config(optional=True)]`.
+        index: `Feature | None = None` A :class:`~lamindb.Feature` record to validate an index of a `DataFrame` and therefore also, e.g., `AnnData` obs and var indices.
+        slots: `dict[str, Schema] | None = None` A dictionary mapping slot names to :class:`~lamindb.Schema` objects.
+        name: `str | None = None` Name of the Schema.
+        description: `str | None = None` Description of the Schema.
         flexible: `bool | None = None` Whether to include any feature of the same `itype` in validation
-            and annotation. If no features are passed, defaults to `True`, otherwise to `False`.
-            This means that if you explicitly pass features, any additional features will be disregarded during validation & annotation.
-        type: `Schema | None = None` A type.
-        is_type: `bool = False` Distinguish types from instances of the type.
-        otype: `str | None = None` An object type to define the structure of a composite schema.
-        dtype: `str | None = None` The simple type. Defaults to
-            `None` for sets of :class:`~lamindb.Feature` records.
-            Otherwise defaults to `"num"` (e.g., for sets of :class:`~bionty.Gene`).
-        minimal_set: `bool = True` Whether all passed features are to be considered required by default.
+            and annotation. If no Features are passed, defaults to `True`, otherwise to `False`.
+            This means that if you explicitly pass Features, any additional Features will be disregarded during validation & annotation.
+        type: `Schema | None = None` Type of Schema to group measurements by.
+            Define types like `ln.Schema(name="ProteinPanel", is_type=True)`.
+        is_type: `bool = False` Whether the Schema is a Type.
+        itype: `str | None = None` The feature identifier type (e.g. :class:`~lamindb.Feature`, :class:`~bionty.Gene`, ...).
+        otype: `str | None = None` An object type to define the structure of a composite schema (e.g., DataFrame, AnnData).
+        dtype: `str | None = None` The simple type (e.g., "num", "float", "int").
+            Defaults to `None` for sets of :class:`~lamindb.Feature` records and to `"num"` (e.g., for sets of :class:`~bionty.Gene`) otherwise.
+        minimal_set: `bool = True` Whether all passed Features are required by default.
             See :attr:`~lamindb.Schema.optionals` for more-fine-grained control.
-        ordered_set: `bool = False` Whether features are required to be ordered.
-        maximal_set: `bool = False` If `True`, no additional features are allowed.
+        maximal_set: `bool = False` Whether additional Features are allowed.
+        ordered_set: `bool = False` Whether Features are required to be ordered.
         coerce_dtype: `bool = False` When True, attempts to coerce values to the specified dtype
             during validation, see :attr:`~lamindb.Schema.coerce_dtype`.
 
@@ -379,13 +374,6 @@ class Meta(Record.Meta, TracksRun.Meta, TracksUpdates.Meta):
     """A description."""
     n: int = IntegerField()
     """Number of features in the schema."""
-    itype: str | None = CharField(
-        max_length=120, db_index=True, null=True, editable=False
-    )
-    """A registry that stores feature identifiers used in this schema, e.g., `'Feature'` or `'bionty.Gene'`.
-
-    Depending on `itype`, `.members` stores, e.g., `Feature` or `bionty.Gene` records.
-    """
     type: Schema | None = ForeignKey("self", PROTECT, null=True, related_name="records")
     """Type of schema.
 
@@ -399,6 +387,13 @@ class Meta(Record.Meta, TracksRun.Meta, TracksUpdates.Meta):
     """Records of this type."""
     is_type: bool = BooleanField(default=False, db_index=True, null=True)
     """Distinguish types from instances of the type."""
+    itype: str | None = CharField(
+        max_length=120, db_index=True, null=True, editable=False
+    )
+    """A registry that stores feature identifier types used in this schema, e.g., `'Feature'` or `'bionty.Gene'`.
+
+    Depending on `itype`, `.members` stores, e.g., `Feature` or `bionty.Gene` records.
+    """
     otype: str | None = CharField(max_length=64, db_index=True, null=True)
     """Default Python object type, e.g., DataFrame, AnnData."""
     dtype: str | None = CharField(max_length=64, null=True, editable=False)
@@ -485,6 +480,7 @@ def __init__(
         otype: str | None = None,
         dtype: str | Type[int | float | str] | None = None,  # noqa
         ordered_set: bool = False,
+        minimal_set: bool = True,
         maximal_set: bool = False,
         coerce_dtype: bool = False,
         n: int | None = None,
diff --git a/tests/curators/test_curators_examples.py b/tests/curators/test_curators_examples.py
index fe1b3f1ea0..28d04d4d8f 100644
--- a/tests/curators/test_curators_examples.py
+++ b/tests/curators/test_curators_examples.py
@@ -157,6 +157,7 @@ def test_dataframe_curator(small_dataset1_schema: ln.Schema, ccaplog):
         error.exconly()
         == "lamindb.errors.ValidationError: Column 'treatment_time_h' failed series or dataframe validator 0: <Check check_function: Column 'treatment_time_h' failed dtype check for 'float': got int64>"
     )
+
     schema.delete()
     feature_to_fail.delete()
 
@@ -214,9 +215,26 @@ def test_dataframe_curator(small_dataset1_schema: ln.Schema, ccaplog):
         assert str(error).startswith("column 'sample_note' not in dataframe")
     curator.standardize()
     curator.validate()
+
     artifact.delete(permanent=True)
 
 
+def test_dataframe_curator_index():
+    """Test validating a DataFrame index."""
+    df = datasets.small_dataset1(otype="DataFrame", with_index_type_mismatch=True)
+    feature = ln.Feature(name="test", dtype="str").save()
+    schema = ln.Schema(index=feature).save()
+    curator = ln.curators.DataFrameCurator(df, schema)
+    with pytest.raises(ln.errors.ValidationError) as error:
+        curator.validate()
+    assert error.exconly().startswith(
+        "lamindb.errors.ValidationError: expected series 'None' to have type str"
+    )
+
+    schema.delete()
+    feature.delete()
+
+
 def test_dataframe_curator_validate_all_annotate_cat(small_dataset1_schema):
     """Do not pass any features."""
 
@@ -270,6 +288,90 @@ def test_dataframe_curator_validate_all_annotate_cat2(small_dataset1_schema):
     schema.delete()
 
 
+def test_schema_new_genes(ccaplog):
+    df = pd.DataFrame(
+        index=pd.Index(
+            [
+                "ENSG00000139618",  # BRCA2
+                "ENSG00000141510",  # TP53
+                "ENSG00999000001",  # Invalid ID
+                "ENSG00999000002",  # Invalid ID
+            ],
+            name="ensembl",
+        )
+    )
+    feature = ln.Feature(name="ensembl", dtype=bt.Gene.ensembl_gene_id).save()
+    schema = ln.Schema(
+        index=feature,
+    ).save()
+    curator = ln.curators.DataFrameCurator(df, schema)
+    curator.validate()
+    assert curator._is_validated
+
+    assert (
+        "2 terms not validated in feature 'index': 'ENSG00999000001', 'ENSG00999000002'"
+        in ccaplog.text
+    )
+
+    schema.delete()
+    feature.delete()
+
+
+def test_schema_no_match_ensembl():
+    df = pd.DataFrame(
+        index=pd.Index(
+            [
+                "ENSG99999999998",  # Invalid ID
+                "ENSG99999999999",  # Invalid ID
+            ],
+            name="ensembl",
+        )
+    )
+    schema = ln.Schema(
+        index=ln.Feature(name="ensembl", dtype=bt.Gene.ensembl_gene_id).save()
+    ).save()
+    curator = ln.curators.DataFrameCurator(df, schema)
+    with pytest.raises(ln.errors.ValidationError) as error:
+        curator.validate()
+    assert (
+        error.exconly()
+        == """lamindb.errors.ValidationError: 2 terms not validated in feature 'index': 'ENSG99999999998', 'ENSG99999999999'
+    → fix typos, remove non-existent values, or save terms via: curator.cat.add_new_from('index')"""
+    )
+
+    schema.delete()
+
+
+def test_schema_mixed_ensembl_symbols(ccaplog):
+    """Quite some datasets have mixed ensembl gene IDs and symbols.
+
+    The expected behavior is that an error is raised when such a dataset is encountered because
+    currently lamindb does not support validating values against a union of Fields.
+
+    The current behavior is that these cases automatically pass.
+    """
+    df = pd.DataFrame(
+        index=pd.Index(
+            [
+                "ENSG00000139618",
+                "ENSG00000141510",
+                "BRCA2",  # symbol
+                "TP53",  # symbol
+            ],
+            name="ensembl",
+        )
+    )
+    schema = ln.Schema(
+        index=ln.Feature(name="ensembl", dtype=bt.Gene.ensembl_gene_id).save()
+    ).save()
+    curator = ln.curators.DataFrameCurator(df, schema)
+    curator.validate()
+
+    assert "2 terms not validated in feature 'index': 'BRCA2', 'TP53'" in ccaplog.text
+
+    schema.delete()
+
+
 def test_anndata_curator_different_components(small_dataset1_schema: ln.Schema):
     obs_schema = small_dataset1_schema
 
@@ -349,9 +451,7 @@ def test_anndata_curator_different_components(small_dataset1_schema: ln.Schema):
 
 def test_anndata_curator_varT_curation():
     ln.Schema.filter(itype="bionty.Gene.ensembl_gene_id").delete()
-    varT_schema = ln.Schema(
-        itype=bt.Gene.ensembl_gene_id,
-    ).save()
+    varT_schema = ln.Schema(itype=bt.Gene.ensembl_gene_id, maximal_set=True).save()
     slot = "var.T"
     components = {slot: varT_schema}
     anndata_schema = ln.Schema(
@@ -389,13 +489,15 @@ def test_anndata_curator_varT_curation():
                         "ENSG00000010610",
                         "ENSG00000170458",
                     ]
+
                 artifact.delete(permanent=True)
+
             anndata_schema.delete()
             varT_schema.delete()
 
 
 def test_anndata_curator_varT_curation_legacy(ccaplog):
-    varT_schema = ln.Schema(itype=bt.Gene.ensembl_gene_id).save()
+    varT_schema = ln.Schema(itype=bt.Gene.ensembl_gene_id, maximal_set=True).save()
     slot = "var"
     components = {slot: varT_schema}
     anndata_schema = ln.Schema(
@@ -429,7 +531,9 @@ def test_anndata_curator_varT_curation_legacy(ccaplog):
                 "ENSG00000010610",
                 "ENSG00000170458",
             }
+
             artifact.delete(permanent=True)
+
             anndata_schema.delete()
             varT_schema.delete()
 
@@ -461,6 +565,7 @@ def test_soma_curator(
     }
 
     assert artifact._key_is_virtual
+
     artifact.delete(permanent=True)
     shutil.rmtree("./small_dataset1.tiledbsoma")
 
@@ -476,6 +581,7 @@ def test_anndata_curator_no_var(small_dataset1_schema: ln.Schema):
     assert small_dataset1_schema.id is not None, small_dataset1_schema
     adata = datasets.small_dataset1(otype="AnnData")
     curator = ln.curators.AnnDataCurator(adata, anndata_schema_no_var)
+
     artifact = curator.save_artifact(key="examples/dataset1_no_var.h5ad")
     artifact.delete(permanent=True)
     anndata_schema_no_var.delete()
@@ -637,4 +743,5 @@ def test_spatialdata_curator(
         BRCA2               num
         BRAF                num"""
     )
+
     artifact.delete(permanent=True)
diff --git a/tests/curators/test_curators_general.py b/tests/curators/test_curators_general.py
index ed50012c49..0c967fe39c 100644
--- a/tests/curators/test_curators_general.py
+++ b/tests/curators/test_curators_general.py
@@ -1,14 +1,21 @@
 import re
+import textwrap
 
+import bionty as bt
 import lamindb as ln
 import pandas as pd
 import pytest
 from lamindb.core.exceptions import ValidationError
 
 
+def _strip_ansi(text: str) -> str:
+    """Remove ANSI escape sequences from a string."""
+    ansi_escape = re.compile(r"\x1B(?:[@-Z\\-_]|\[[0-?]*[ -/]*[@-~])")
+    return ansi_escape.sub("", text)
+
+
 @pytest.fixture
 def df():
-    # DataFrames
     return pd.DataFrame(
         {
             "sample_id": ["sample1", "sample2"],
@@ -20,7 +27,6 @@ def df():
 
 @pytest.fixture
 def df_missing_sample_type_column():
-    # missing a column
     return pd.DataFrame(
         {
             "sample_id": ["sample1", "sample2"],
@@ -40,8 +46,7 @@ def df_missing_sample_name_column():
 
 
 @pytest.fixture
-def df_changed_order():
-    # changed columns order
+def df_changed_col_order():
     return pd.DataFrame(
         {
             "sample_name": ["Sample 1", "Sample 2"],
@@ -53,7 +58,6 @@ def df_changed_order():
 
 @pytest.fixture
 def df_extra_column():
-    # additional column
     return pd.DataFrame(
         {
             "sample_id": ["sample1", "sample2"],
@@ -64,6 +68,24 @@ def df_extra_column():
     )
 
 
+def test_curator__repr__(df):
+    schema = ln.Schema(
+        name="sample schema",
+        features=[ln.Feature(name="sample_id", dtype="str").save()],
+    ).save()
+    curator = ln.curators.DataFrameCurator(df, schema)
+
+    expected_repr = textwrap.dedent("""\
+    DataFrameCurator(Schema: sample schema, unvalidated)
+    """).strip()
+
+    actual_repr = _strip_ansi(repr(curator))
+    print(actual_repr)
+    assert actual_repr.strip() == expected_repr.strip()
+
+    schema.delete()
+
+
 def test_nullable():
     disease = ln.Feature(name="disease", dtype=ln.ULabel, nullable=False).save()
     schema = ln.Schema(features=[disease]).save()
@@ -84,6 +106,7 @@ def test_nullable():
         # match=re.escape("1 term is not validated: 'asthma'"),  # TODO: need the message
     ):
         curator.validate()
+
     schema.delete()
     disease.delete()
 
@@ -91,7 +114,7 @@ def test_nullable():
 def test_pandera_dataframe_schema(
     df,
     df_missing_sample_type_column,
-    df_changed_order,
+    df_changed_col_order,
     df_extra_column,
     df_missing_sample_name_column,
 ):
@@ -133,7 +156,7 @@ def test_pandera_dataframe_schema(
         ).validate()
     # doesn't care about order
     ln.curators.DataFrameCurator(
-        df_changed_order, schema=schema_all_required
+        df_changed_col_order, schema=schema_all_required
     ).validate()
     # extra column is fine
     ln.curators.DataFrameCurator(df_extra_column, schema=schema_all_required).validate()
@@ -152,7 +175,7 @@ def test_pandera_dataframe_schema(
     # ordered_set=True, order matters
     with pytest.raises(ValidationError):
         ln.curators.DataFrameCurator(
-            df_changed_order, schema=schema_ordered_set
+            df_changed_col_order, schema=schema_ordered_set
         ).validate()
 
     # a feature is optional
@@ -189,7 +212,9 @@ def test_schema_optionals():
             ln.Feature(name="sample_type", dtype=str).save(),
         ],
     ).save()
-    assert schema.optionals.get().list("name") == ["sample_name"]
+    assert schema.optionals.get().list("name") == [
+        "sample_name",
+    ]
 
     # set sample_type to optional
     with pytest.raises(
@@ -235,3 +260,41 @@ def test_schema_ordered_set(df):
     # clean up
     ln.Schema.filter().delete()
     ln.Feature.filter().delete()
+
+
+@pytest.mark.parametrize("minimal_set", [True, False])
+def test_schema_minimal_set_var_allowed(minimal_set):
+    """Independent of the value of minimal_set, invalid ensembl gene IDs are allowed."""
+    adata = ln.core.datasets.mini_immuno.get_dataset1(otype="AnnData")
+    adata.var_names = [adata.var_names[0], adata.var_names[1], "NOT_VALID_ENSEMBL"]
+
+    var_schema = ln.Schema(
+        itype=bt.Gene.ensembl_gene_id,
+        minimal_set=minimal_set,
+    ).save()
+    schema = ln.Schema(otype="AnnData", slots={"var.T": var_schema}).save()
+    curator = ln.curators.AnnDataCurator(adata, schema)
+    curator.validate()
+
+    # clean up
+    schema.delete()
+
+
+def test_schema_maximal_set_var():
+    """If maximal_set is True, invalid ensembl gene IDs are not allowed."""
+    adata = ln.core.datasets.mini_immuno.get_dataset1(otype="AnnData")
+    adata.var_names = [adata.var_names[0], adata.var_names[1], "NOT_VALID_ENSEMBL"]
+
+    var_schema = ln.Schema(itype=bt.Gene.ensembl_gene_id, maximal_set=True).save()
+    schema = ln.Schema(otype="AnnData", slots={"var.T": var_schema}).save()
+
+    curator = ln.curators.AnnDataCurator(adata, schema)
+    with pytest.raises(ValidationError) as error:
+        curator.validate()
+    assert error.exconly() == (
+        "lamindb.errors.ValidationError: 1 term not validated in feature 'columns' in slot 'var.T': 'NOT_VALID_ENSEMBL'\n"
+        "    → fix typos, remove non-existent values, or save terms via: curator.slots['var.T'].cat.add_new_from('columns')"
+    )
+
+    # clean up
+    schema.delete()


Reviews:
user: login: falexwolf
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-06T11:05:42Z
user: login: falexwolf
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-06T11:06:01Z
user: login: falexwolf
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-06T11:06:06Z
user: login: falexwolf
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-06T11:06:30Z
user: login: falexwolf
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-06T11:07:06Z
user: login: falexwolf
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-06T11:11:25Z
user: login: falexwolf
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-06T11:13:34Z
user: login: falexwolf
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-06T11:15:06Z
user: login: falexwolf
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-06T11:23:21Z
user: login: falexwolf
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-06T11:23:43Z
user: login: falexwolf
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-06T11:28:12Z
user: login: falexwolf
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-06T11:29:05Z
user: login: falexwolf
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-06T11:29:44Z
user: login: Zethson
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T11:30:56Z
user: login: Zethson
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T11:54:14Z
user: login: falexwolf
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-06T12:25:48Z
user: login: falexwolf
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-06T12:26:15Z
user: login: falexwolf
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-06T12:27:22Z
user: login: falexwolf
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-06T12:30:50Z
user: login: falexwolf
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-06T12:31:48Z
user: login: falexwolf
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-06T16:05:31Z

Review comments:
diff_hunk: @@ -32,7 +32,7 @@
     ],
 ).save()
 
-varT_schema = ln.Schema(itype=bt.Gene.ensembl_gene_id).save()
+varT_schema = ln.Schema(itype=bt.Gene.ensembl_gene_id, maximal_set=True).save()
path: docs/scripts/define_schema_spatialdata.py
user: login: falexwolf
body: Can you add a comment explaining that this means that no non-registered Ensembl Gene ids are valid?
created_at: 2025-05-06T11:05:42Z
author_association: MEMBER
diff_hunk: @@ -64,7 +64,9 @@ def get_dataset1(
     with_cell_type_synonym: bool = False,
     with_cell_type_typo: bool = False,
     with_gene_typo: bool = False,
+    with_outdated_gene: bool = False,
path: lamindb/core/datasets/mini_immuno.py
user: login: falexwolf
body: Thank you! Great!
created_at: 2025-05-06T11:06:01Z
author_association: MEMBER
diff_hunk: @@ -64,7 +64,9 @@ def get_dataset1(
     with_cell_type_synonym: bool = False,
     with_cell_type_typo: bool = False,
     with_gene_typo: bool = False,
+    with_outdated_gene: bool = False,
     with_wrong_subtype: bool = False,
+    with_index_type_mismatch: bool = False,
path: lamindb/core/datasets/mini_immuno.py
user: login: falexwolf
body: Thank you!
created_at: 2025-05-06T11:06:05Z
author_association: MEMBER
diff_hunk: @@ -107,7 +113,12 @@ def get_dataset1(
         "study_note": "We had a great time performing this study and the results look compelling.",
     }
     # the dataset as DataFrame
-    dataset_df = pd.DataFrame(dataset_dict, index=["sample1", "sample2", "sample3"])
+    dataset_df = pd.DataFrame(
+        dataset_dict,
+        index=["sample1", "sample2", 0]  # type: ignore
path: lamindb/core/datasets/mini_immuno.py
user: login: falexwolf
body: Great catch!
created_at: 2025-05-06T11:06:30Z
author_association: MEMBER
diff_hunk: @@ -205,6 +205,69 @@ def save_artifact(
         # constructor signature
         pass  # pragma: no cover
 
+    def __repr__(self) -> str:
+        from lamin_utils import colors
path: lamindb/curators/core.py
user: login: falexwolf
body: The whole thing isn't called anywhere. So, need a test.
created_at: 2025-05-06T11:07:06Z
author_association: MEMBER
diff_hunk: @@ -427,7 +490,9 @@ def __init__(
                 # in almost no case, an index should have a pandas.CategoricalDtype in a DataFrame
                 # so, we're typing it as `str` here
                 index = pandera.Index(
-                    schema.index.dtype if not feature.dtype.startswith("cat") else str
path: lamindb/curators/core.py
user: login: falexwolf
body: Oh, this was the bug, I guess. And there was no test for an index with a categorical dtype. Got it.
created_at: 2025-05-06T11:11:25Z
author_association: MEMBER
diff_hunk: @@ -803,6 +870,7 @@ def __init__(
         feature: Feature | None = None,
         cat_manager: DataFrameCatManager | None = None,
         subtype_str: str = "",
+        maximal_set: bool = False,
path: lamindb/curators/core.py
user: login: falexwolf
body: A comment would be good here because it's subtly different from what it means on `Schema`. But because the arg is so generically named it's still a good name also here.
created_at: 2025-05-06T11:13:34Z
author_association: MEMBER
diff_hunk: @@ -843,7 +912,18 @@ def values(self, new_values):
     @property
     def is_validated(self) -> bool:
         """Return whether the column is validated."""
path: lamindb/curators/core.py
user: login: falexwolf
body: There are all these occurances of the word `column` which should be replaced with `vector` because we're using this for any vector, not just columns of DataFrame (e.g. `.index` and `.columns` and presumably soon other things).
created_at: 2025-05-06T11:15:06Z
author_association: MEMBER
diff_hunk: @@ -843,7 +912,18 @@ def values(self, new_values):
     @property
     def is_validated(self) -> bool:
         """Return whether the column is validated."""
-        return len(self._non_validated) == 0
+        # ensembl gene IDs pass even if they were not validated
+        # this is a simple solution to the ensembl gene version problem
+        if self._field.field.attname == "ensembl_gene_id":
+            # if none of the ensembl gene ids were validated, we are probably not looking at ensembl gene IDs
+            if len(self.values) == len(self._non_validated):
path: lamindb/curators/core.py
user: login: falexwolf
body: I think we should rather make this the general behavior than only having it for ensembl_gene_id.

If there is not a single valid category, likely the dtype is wrong. We could introduce a setting `n_minimal_valid_values=1` or something like this on `CatVector` with _rough_ analogy to `minimal_set` in `Schema` (where we require defined features to be present, not just any valid values).

For now, we don't need to introduce this setting explicitly, but we could simply assume it. I think that if you validate a categorical and there is not a single valid thing in there, you probably have the wrong dtype.

Of course there will be edge cases with this but doing something only for Ensembl Gene IDs will also produce edge cases.
created_at: 2025-05-06T11:23:21Z
author_association: MEMBER
diff_hunk: @@ -843,7 +912,18 @@ def values(self, new_values):
     @property
     def is_validated(self) -> bool:
         """Return whether the column is validated."""
-        return len(self._non_validated) == 0
+        # ensembl gene IDs pass even if they were not validated
+        # this is a simple solution to the ensembl gene version problem
+        if self._field.field.attname == "ensembl_gene_id":
+            # if none of the ensembl gene ids were validated, we are probably not looking at ensembl gene IDs
+            if len(self.values) == len(self._non_validated):
+                return False
+            # if maximal set, we do not allow additional unvalidated genes
+            elif len(self._non_validated) != 0 and self._maximal_set:
path: lamindb/curators/core.py
user: login: falexwolf
body: Also this should run for any categorical, not just for Ensembl Gene Ids.
created_at: 2025-05-06T11:23:43Z
author_association: MEMBER
diff_hunk: @@ -245,36 +245,31 @@ def add(self, features: Feature | list[Feature]) -> None:
 
 
 class Schema(Record, CanCurate, TracksRun):
-    """Schemas of datasets such as the set of columns of a `DataFrame`.
+    """A schema is a feature set, such as the set of columns of a `DataFrame`.
path: lamindb/models/schema.py
user: login: falexwolf
body: That's wrong, isn't it? A schema is typically more than "just a set of columns". All of the close relatives like `pyarrow.lib.Schema` or `pandera.Schema` etc. are more than a set of columns, and so it `lamindb.Schema`. In lamindb the departure from the feature set it particularly striking because there are composite schemas and schemas that just enforce an `itype`.


created_at: 2025-05-06T11:28:12Z
author_association: MEMBER
diff_hunk: @@ -245,36 +245,31 @@ def add(self, features: Feature | list[Feature]) -> None:
 
 
 class Schema(Record, CanCurate, TracksRun):
-    """Schemas of datasets such as the set of columns of a `DataFrame`.
+    """A schema is a feature set, such as the set of columns of a `DataFrame`.
 
-    A simple schema is a feature set such as the set of columns of a `DataFrame`.
-
-    A composite schema has multiple slots, e.g., for an `AnnData`, one schema for slot `obs` and another one for `var`.
-
-    A schema can also merely define abstract constraints or instructions for dataset validation & annotation.
path: lamindb/models/schema.py
user: login: falexwolf
body: Why remove this last one (I get why you remove the first sentence).
created_at: 2025-05-06T11:29:05Z
author_association: MEMBER
diff_hunk: @@ -372,13 +367,6 @@ class Meta(Record.Meta, TracksRun.Meta, TracksUpdates.Meta):
     """A description."""
     n: int = IntegerField()
     """Number of features in the schema."""
-    itype: str | None = CharField(
-        max_length=120, db_index=True, null=True, editable=False
-    )
-    """A registry that stores feature identifiers used in this schema, e.g., `'Feature'` or `'bionty.Gene'`.
path: lamindb/models/schema.py
user: login: falexwolf
body: Can we call out "feature identifier type"? So that people understand why it's called `itype`?
created_at: 2025-05-06T11:29:43Z
author_association: MEMBER
diff_hunk: @@ -245,36 +245,31 @@ def add(self, features: Feature | list[Feature]) -> None:
 
 
 class Schema(Record, CanCurate, TracksRun):
-    """Schemas of datasets such as the set of columns of a `DataFrame`.
+    """A schema is a feature set, such as the set of columns of a `DataFrame`.
 
-    A simple schema is a feature set such as the set of columns of a `DataFrame`.
-
-    A composite schema has multiple slots, e.g., for an `AnnData`, one schema for slot `obs` and another one for `var`.
-
-    A schema can also merely define abstract constraints or instructions for dataset validation & annotation.
path: lamindb/models/schema.py
user: login: Zethson
body: I was thinking about this several times. I ended up following my initial impression which was: This is a difficult to understand sentence and doesn't have enough practical value.

If you think that the message is important, I am happy to try to phrase it more simply so that it's more easily understood.
created_at: 2025-05-06T11:30:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -245,36 +245,31 @@ def add(self, features: Feature | list[Feature]) -> None:
 
 
 class Schema(Record, CanCurate, TracksRun):
-    """Schemas of datasets such as the set of columns of a `DataFrame`.
+    """A schema is a feature set, such as the set of columns of a `DataFrame`.
path: lamindb/models/schema.py
user: login: Zethson
body: Sorry, I omitted something that I should have kept. Rephrased!
created_at: 2025-05-06T11:54:14Z
author_association: CONTRIBUTOR
diff_hunk: @@ -214,9 +215,21 @@ def test_dataframe_curator(small_dataset1_schema: ln.Schema, ccaplog):
         assert str(error).startswith("column 'sample_note' not in dataframe")
     curator.standardize()
     curator.validate()
+
     artifact.delete(permanent=True)
 
 
+def test_dataframe_curator_index():
+    """Test validating a DataFrame index."""
+    df = datasets.small_dataset1(otype="DataFrame", with_index_type_mismatch=True)
+    schema = ln.Schema(index=ln.Feature(name="test", dtype="str").save()).save()
+    curator = ln.curators.DataFrameCurator(df, schema)
+    try:
path: tests/curators/test_curators_examples.py
user: login: falexwolf
body: Sorry, was dragged into another meeting before.

We need `pytest.raises()` here so that we're sure the error is actually raised.
created_at: 2025-05-06T12:25:48Z
author_association: MEMBER
diff_hunk: @@ -214,9 +215,21 @@ def test_dataframe_curator(small_dataset1_schema: ln.Schema, ccaplog):
         assert str(error).startswith("column 'sample_note' not in dataframe")
     curator.standardize()
     curator.validate()
+
     artifact.delete(permanent=True)
 
 
+def test_dataframe_curator_index():
+    """Test validating a DataFrame index."""
+    df = datasets.small_dataset1(otype="DataFrame", with_index_type_mismatch=True)
+    schema = ln.Schema(index=ln.Feature(name="test", dtype="str").save()).save()
path: tests/curators/test_curators_examples.py
user: login: falexwolf
body: Please delete the schema and the feature at the end of this test so that it doesn't interact with other tests.
created_at: 2025-05-06T12:26:15Z
author_association: MEMBER
diff_hunk: @@ -270,6 +283,88 @@ def test_dataframe_curator_validate_all_annotate_cat2(small_dataset1_schema):
     schema.delete()
 
 
+def test_schema_new_genes(ccaplog):
+    df = pd.DataFrame(
+        index=pd.Index(
+            [
+                "ENSG00000139618",  # BRCA2
+                "ENSG00000141510",  # TP53
+                "ENSG00999000001",  # Invalid ID
+                "ENSG00999000002",  # Invalid ID
+            ],
+            name="ensembl",
+        )
+    )
+    schema = ln.Schema(
+        index=ln.Feature(name="ensembl", dtype=bt.Gene.ensembl_gene_id).save()
path: tests/curators/test_curators_examples.py
user: login: falexwolf
body: Please also delete the feature here at the end of the test.
created_at: 2025-05-06T12:27:21Z
author_association: MEMBER
diff_hunk: @@ -235,3 +239,40 @@ def test_schema_ordered_set(df):
     # clean up
     ln.Schema.filter().delete()
     ln.Feature.filter().delete()
+
+
+def test_schema_no_minimal_set_var():
+    """If minimal_set is False, invalid ensembl gene IDs are allowed."""
+    adata = ln.core.datasets.mini_immuno.get_dataset1(otype="AnnData")
+    adata.var_names = [adata.var_names[0], adata.var_names[1], "NOT_VALID_ENSEMBL"]
+
+    var_schema = ln.Schema(
+        itype=bt.Gene.ensembl_gene_id,
+        minimal_set=False,
+    ).save()
+    schema = ln.Schema(otype="AnnData", slots={"var": var_schema}).save()
path: tests/curators/test_curators_general.py
user: login: falexwolf
body: This should be `var.T`, shouldn't it? I think this is the default behavior we typically want, don't we? So that people don't have to deal with messing with different Ensembl versions.

Can you add `minimal_set=True` to `ln.schemas.anndata....`?
created_at: 2025-05-06T12:30:50Z
author_association: MEMBER
diff_hunk: @@ -235,3 +239,40 @@ def test_schema_ordered_set(df):
     # clean up
     ln.Schema.filter().delete()
     ln.Feature.filter().delete()
+
+
+def test_schema_no_minimal_set_var():
+    """If minimal_set is False, invalid ensembl gene IDs are allowed."""
path: tests/curators/test_curators_general.py
user: login: falexwolf
body: Wait, now I'm confused. Why does this test refer to to minimal set and not to maximal set?
created_at: 2025-05-06T12:31:48Z
author_association: MEMBER

PR comments:
user: login: codecov[bot]
created_at: 2025-05-05T10:14:02Z
body: ## [Codecov](https://app.codecov.io/gh/laminlabs/lamindb/pull/2742?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=laminlabs) Report
Attention: Patch coverage is `54.38596%` with `26 lines` in your changes missing coverage. Please review.
> Project coverage is 91.98%. Comparing base [(`3b3372c`)](https://app.codecov.io/gh/laminlabs/lamindb/commit/3b3372c1c6d30d34652f22157868f03dc1f941e5?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=laminlabs) to head [(`eaa36d1`)](https://app.codecov.io/gh/laminlabs/lamindb/commit/eaa36d15320d5e179d5f712a35512d9773e39378?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=laminlabs).
> Report is 47 commits behind head on main.

| [Files with missing lines](https://app.codecov.io/gh/laminlabs/lamindb/pull/2742?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=laminlabs) | Patch % | Lines |
|---|---|---|
| [lamindb/curators/core.py](https://app.codecov.io/gh/laminlabs/lamindb/pull/2742?src=pr&el=tree&filepath=lamindb%2Fcurators%2Fcore.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=laminlabs#diff-bGFtaW5kYi9jdXJhdG9ycy9jb3JlLnB5) | 52.72% | [26 Missing :warning: ](https://app.codecov.io/gh/laminlabs/lamindb/pull/2742?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=laminlabs) |

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #2742      +/-   ##
==========================================
- Coverage   92.09%   91.98%   -0.11%     
==========================================
  Files          60       69       +9     
  Lines       10100    10673     +573     
==========================================
+ Hits         9302     9818     +516     
- Misses        798      855      +57     
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/laminlabs/lamindb/pull/2742?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=laminlabs).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=laminlabs).

<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
author_association: NONE
user: login: github-actions[bot]
created_at: 2025-05-05T10:18:36Z
body: Deployment URL: https://3b6be30f.lamindb.pages.dev
author_association: NONE