Repository: unionai-oss/pandera

Issue:
SchemaError on index name for single-index dataframe when using SchemaModel
Thanks for the awesome `SchemaModel` interface! That really improves readability and usability.

**Describe the bug**
I am not able to validate a pandas DataFrame with a single index column when that index is named. The code on line 304 in `pandera/model.py` seems to explicitly force the schema to have a `None`-named index when the index only has one column. How can I get around this?

- [x] I have checked that this issue has not already been reported.
- [x] I have confirmed this bug exists on the latest version of pandas.
- [] (optional) I have confirmed this bug exists on the master branch of pandas.

**Note**: Please read [this guide](https://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports) detailing how to provide the necessary information for us to reproduce your bug.

#### Code Sample, a copy-pastable example

```python
import pandera as pa
import pandas as pd

df_invalid = pd.DataFrame(
    {"year": [2019, 2020], "value_1": [1, 2], "value_2": [2, 3]}
).set_index("year")
df_valid = pd.DataFrame({"value_1": [1, 2], "value_2": [2, 3]}, index=[2019, 2020])


class Schema(pa.SchemaModel):
    year: pa.typing.Index[int]
    value_1: pa.typing.Series[int]
    value_2: pa.typing.Series[int]


Schema.validate(df_valid)
Schema.validate(df_invalid)

```

#### Expected behavior
I expect this to read the index name for the column, and accept the index. Alternatively that I can specify in the `Config` class that I want the schema to accept named indices.

#### Desktop (please complete the following information):

 - OS: MacOS
 - Browser Chrome
 - Version 86




Discussion:
thanks for submitting this bug @kristomi, I think the rationale for this behavior is was that we didn't want users to have to name their indexes to specify valid dataframes (@jeffzi am I getting that correct?), although I think this behavior needs to be amended to support your use case.

### Potential Solutions

1. Skip the name check for single-indexed dataframes during validation altogether.
2. Add `Config` option to accept named indices

```python
class Schema(pa.SchemaModel):
    ...

    class Config:
        named_index: True  # default False
```

3. Don't override index name with `None` in https://github.com/pandera-dev/pandera/blob/master/pandera/model.py#L295 and provide a special name for un-named single indexes, for example:

```python
class Schema(pa.SchemaModel):
    __index__: pa.typing.Index[int]  # un-named index
    value_1: pa.typing.Series[int]
    value_2: pa.typing.Series[int]
```

I'm sort of leaning toward 2 or 3... the nice thing about 2 is that it opens up support for un-named multiindex.

Thoughts @kristomi and @jeffzi?
Thanks for the quick response.

The way I constructed the invalid dataframe is a very typical workflow for me, where dataframes are passed around and indices are set and reset all the time. On that background, I would prefer solution nr 3, because indices would then be validated as named by default, unless you construct the schema with your special notation.

> I think the rationale for this behavior is was that we didn't want users to have to name their indexes to specify valid dataframes (@jeffzi am I getting that correct?)

Yes, that's right.

I also like the fact that 2. addresses unnamed multiindex and aligns the model api with the standard api. My issue with 3. is that it increases the complexity for new-comers. It would also be confusing in inherited models:

```py
class Schema(pa.SchemaModel):
    __index__: pa.typing.Index[int]  # un-named index
    value_1: pa.typing.Series[int]
    value_2: pa.typing.Series[int]


class SubSchema(Schema):
    # With current implementation, we would create a MultiIndex !
    #  What if we want to name the unnamed __index__?
    year: pa.typing.Index[int]  
```

>  On that background, I would prefer solution nr 3, because indices would then be validated as named by default, unless you construct the schema with your special notation.

If 3. is selected, I agree it would be reasonable, and perhaps more natural, to validate index name by default. After all, you do name the index. 

Edit: I meant "If 2. is selected".
You guys have way more insight than I do, and I clearly see the problem with inheritance now that you mention it. Perhaps nr 2 is a better choice, then. Anyway, I'm not in a position to see all the implications of solving this one way or the other.
@kristomi np! It's already a great help to report bugs and feedback 🙂
  
@cosmicBboy Once a decision has been reached, I'd be happy to take care of the changes. I saw you already have a lot on your plate.
thanks for the feedback! After thinking about it for a little bit, I'd like to go for solution # 2, with a slight addition:

#### Introduce a `verify_name` kwarg to the `SeriesSchemaBase` class constructor and `verify_names` kwarg in `MultiIndex` constructor

- `verify_name(s)` should be an instance property.
- If True, verify the name of the Series/Index/MultiIndex, otherwise don't.
- This would be True by default in `SeriesSchemaBase` and `SeriesSchema`, False by default in the `Index`, and False (I think?) by default in `MultiIndex`
- The `Field` field should also add this new kwarg.
- The SchemaModel Config should have `multi_index__verify_names` kwarg
- For `MultiIndex`, if `verify_names==False`, modify the checked dataframe [here](https://github.com/pandera-dev/pandera/blob/master/pandera/schema_components.py#L478) so that the column names are 0-indexed integers. (**Note:** `MultiIndex` inherits from `DataFrameSchema` and uses its underlying validation logic by casting the `pd.MultiIndex` into a `DataFrame`... a questionable design decision perhaps, but a vestigial part of the early days of this library when I was lazy and didn't want to re-implement validation for multiindexes 🙃)

My reasoning for this is:
- I agree with @jeffzi that I'd prefer not to introduce a special concept for un-named indexes, as it degrades usability
- The MultiIndex object already handles un-named MultiIndexes provided by the user in the validated object by falling back to [0-indexed integer column names](https://github.com/pandera-dev/pandera/blob/master/pandera/schema_components.py#L399-L409). `verify_names` just adds granular control on whether the user wants to explicitly check these names on the `MultiIindex.validate` call.
- This keeps the API tight in the sense that users can control the name-checking behavior of both single indexes and multiindexes, while providing reasonable defaults... typically, I don't really care whether a single index has the correct name, but if I do I just specify `pa.Index(verify_name=True)` or `Field(verify_name=True)` in the class-based API. This setup should also support the `df_invalid` in your example code @kristomi _and_ gracefully handle the inheritance case:

```python
class Schema(pa.SchemaModel):
    year: pa.typing.Index[int]  # by default, pa.Index(verify_name=False) so pandera won't check the "year" name here
    value_1: pa.typing.Series[int]
    value_2: pa.typing.Series[int]

class SubSchema(Schema):
    month: pa.typing.Index[int]  # by default, pa.MultiIndex(verify_names=True) so pandera will construct multiindex here and check index names
```
> Once a decision has been reached, I'd be happy to take care of the changes.

Thanks @jeffzi! Let me know what you think about the above proposal and if you have any questions/concerns about it
I would suggest `check_name` instead of `verify_name` to keep the same vocabulary used elsewhere in pandera.

Name validation is mandatory for Series because we need the name to get the column to validate from the DataFrame:
https://github.com/pandera-dev/pandera/blob/31fe07b30b267527ecea7f6b1435f49b6b963abf/pandera/schemas.py#L937-L941

That would not be an issue if pandera supported columns order. It is something that I actually wanted to suggest for the machine learning use case. Many ML libraries ignore the column names and rely on the order (possibly casting to a numpy array). 

> False by default in the Index, and False (I think?) by default in MultiIndex

Your example says it should be True by default :question:    
Personally I only give a name to a single index when I set it explicitly, which is unnecessary most of the time. I always give a name to a MultiIndex though. True by default for MultiIndex is also closer to pandera's standard API.

I agree with your other points.
> I would suggest `check_name` instead of `verify_name`

sounds good 👍

> Your example says it should be True by default

woops, yes I meant True by default :) 

Resolving patch:
diff --git a/pandera/model.py b/pandera/model.py
--- a/pandera/model.py
+++ b/pandera/model.py
@@ -48,6 +48,7 @@ class BaseConfig:  # pylint:disable=R0903
     name: Optional[str] = None  #: name of schema
     coerce: bool = False  #: coerce types of all schema components
     strict: bool = False  #: make sure all specified columns are in dataframe
+    check_index_name: bool = False
     multiindex_name: Optional[str] = None  #: name of multiindex
 
     #: coerce types of all MultiIndex components
@@ -145,7 +146,7 @@ def validate(
         )
 
     @classmethod
-    def _build_columns_index(
+    def _build_columns_index(  # pylint:disable=too-many-locals
         cls,
         checks: Dict[str, List[Check]],
         annotations: Dict[str, Any],
@@ -154,39 +155,59 @@ def _build_columns_index(
         Dict[str, schema_components.Column],
         Optional[Union[schema_components.Index, schema_components.MultiIndex]],
     ]:
+        annotations = {
+            field_name: (parse_annotation(raw_annotation), raw_annotation)
+            for field_name, raw_annotation in annotations.items()
+        }
+        index_count = sum(
+            annotation.origin is Index
+            for annotation, _ in annotations.values()
+        )
+
         columns: Dict[str, schema_components.Column] = {}
         indices: List[schema_components.Index] = []
-        for field_name, raw_annotation in annotations.items():
-            annotation_info = parse_annotation(raw_annotation)
+        for field_name, (annotation, raw_annotation) in annotations.items():
 
-            field = getattr(cls, field_name, None)
-            if field is not None and not isinstance(field, FieldInfo):
-                raise SchemaInitError(
-                    f"'{field_name}' can only be assigned a 'Field', "
-                    + f"not a '{field.__class__}.'"
-                )
+            field: FieldInfo = getattr(cls, field_name, None)
+            _check_fieldinfo(field, field_name)
 
             field_checks = checks.get(field_name, [])
-            if annotation_info.origin is Series:
+            check_name = getattr(field, "check_name", None)
+
+            if annotation.origin is Series:
                 col_constructor = (
                     field.to_column if field else schema_components.Column
                 )
+
+                if check_name is False:
+                    raise SchemaInitError(
+                        f"'check_name' is not supported for {field_name}."
+                    )
+
                 columns[field_name] = col_constructor(  # type: ignore
-                    annotation_info.arg,
-                    required=not annotation_info.optional,
+                    annotation.arg,
+                    required=not annotation.optional,
                     checks=field_checks,
                     name=field_name,
                 )
-            elif annotation_info.origin is Index:
-                if annotation_info.optional:
+            elif annotation.origin is Index:
+                if annotation.optional:
                     raise SchemaInitError(
                         f"Index '{field_name}' cannot be Optional."
                     )
+
+                if check_name is False or (
+                    # default single index
+                    check_name is None
+                    and index_count == 1
+                ):
+                    field_name = None  # type:ignore
+
                 index_constructor = (
                     field.to_index if field else schema_components.Index
                 )
                 index = index_constructor(  # type: ignore
-                    annotation_info.arg, checks=field_checks, name=field_name
+                    annotation.arg, checks=field_checks, name=field_name
                 )
                 indices.append(index)
             else:
@@ -293,15 +314,12 @@ def _not_routine(member: Any) -> bool:
 
 
 def _build_schema_index(
-    indices: List[schema_components.Index],
-    **multiindex_kwargs: Any,
+    indices: List[schema_components.Index], **multiindex_kwargs: Any
 ) -> Optional[SchemaIndex]:
     index: Optional[SchemaIndex] = None
     if indices:
         if len(indices) == 1:
             index = indices[0]
-            # don't force name on single index
-            index._name = None  # pylint:disable=W0212
         else:
             index = schema_components.MultiIndex(indices, **multiindex_kwargs)
     return index
@@ -314,3 +332,11 @@ def _regex_filter(seq: Iterable, regexps: Iterable[str]) -> Set[str]:
         pattern = re.compile(regex)
         matched.update(filter(pattern.match, seq))
     return matched
+
+
+def _check_fieldinfo(field: FieldInfo, field_name: str) -> None:
+    if field is not None and not isinstance(field, FieldInfo):
+        raise SchemaInitError(
+            f"'{field_name}' can only be assigned a 'Field', "
+            + f"not a '{field.__class__}.'"
+        )
diff --git a/pandera/model_components.py b/pandera/model_components.py
--- a/pandera/model_components.py
+++ b/pandera/model_components.py
@@ -44,7 +44,14 @@ class FieldInfo:
     *new in 0.5.0*
     """
 
-    __slots__ = ("checks", "nullable", "allow_duplicates", "coerce", "regex")
+    __slots__ = (
+        "checks",
+        "nullable",
+        "allow_duplicates",
+        "coerce",
+        "regex",
+        "check_name",
+    )
 
     def __init__(
         self,
@@ -53,12 +60,14 @@ def __init__(
         allow_duplicates: bool = True,
         coerce: bool = False,
         regex: bool = False,
+        check_name: bool = None,
     ) -> None:
         self.checks = _to_checklist(checks)
         self.nullable = nullable
         self.allow_duplicates = allow_duplicates
         self.coerce = coerce
         self.regex = regex
+        self.check_name = check_name
 
     def _to_schema_component(
         self,
@@ -131,6 +140,7 @@ def Field(
     ignore_na: bool = True,
     raise_warning: bool = False,
     n_failure_cases: int = 10,
+    check_name: bool = None,
 ) -> Any:
     """Used to provide extra information about a field of a SchemaModel.
 
@@ -138,6 +148,10 @@ def Field(
 
     Some arguments apply only to number dtypes and some apply only to ``str``.
     See the :ref:`User Guide <schema_models>` for more.
+
+    :param check_name: Whether to check the name of the column/index during validation.
+        `None` is the default behavior, which translates to `True` for columns and
+        multi-index, and to `False` for a single index.
     """
     # pylint:disable=C0103,W0613,R0914
     check_kwargs = {
@@ -163,6 +177,7 @@ def Field(
         allow_duplicates=allow_duplicates,
         coerce=coerce,
         regex=regex,
+        check_name=check_name,
     )
 
 
diff --git a/pandera/schemas.py b/pandera/schemas.py
--- a/pandera/schemas.py
+++ b/pandera/schemas.py
@@ -943,7 +943,7 @@ def validate(
         if not inplace:
             series = series.copy()
 
-        if series.name != self._name:
+        if self.name is not None and series.name != self._name:
             msg = "Expected %s to have name '%s', found '%s'" % (
                 type(self),
                 self._name,


Test patch:
diff --git a/tests/test_model.py b/tests/test_model.py
--- a/tests/test_model.py
+++ b/tests/test_model.py
@@ -135,6 +135,74 @@ class Schema(pa.SchemaModel):
     assert expected == Schema.to_schema()
 
 
+def test_column_check_name():
+    """Test that column name is mandatory."""
+
+    class Schema(pa.SchemaModel):
+        a: Series[int] = pa.Field(check_name=False)
+
+    with pytest.raises(pa.errors.SchemaInitError):
+        Schema.to_schema()
+
+
+def test_single_index_check_name():
+    """Test single index name."""
+    df = pd.DataFrame(index=pd.Index(["cat", "dog"], name="animal"))
+
+    class DefaultSchema(pa.SchemaModel):
+        a: Index[str]
+
+    assert isinstance(DefaultSchema.validate(df), pd.DataFrame)
+
+    class DefaultFieldSchema(pa.SchemaModel):
+        a: Index[str] = pa.Field(check_name=None)
+
+    assert isinstance(DefaultFieldSchema.validate(df), pd.DataFrame)
+
+    class NotCheckNameSchema(pa.SchemaModel):
+        a: Index[str] = pa.Field(check_name=False)
+
+    assert isinstance(NotCheckNameSchema.validate(df), pd.DataFrame)
+
+    class SchemaNamedIndex(pa.SchemaModel):
+        a: Index[str] = pa.Field(check_name=True)
+
+    err_msg = "name 'a', found 'animal'"
+    with pytest.raises(pa.errors.SchemaError, match=err_msg):
+        SchemaNamedIndex.validate(df)
+
+
+def test_multiindex_check_name():
+    """Test a MultiIndex name."""
+
+    df = pd.DataFrame(
+        index=pd.MultiIndex.from_arrays(
+            [["foo", "bar"], [0, 1]], names=["a", "b"]
+        )
+    )
+
+    class DefaultSchema(pa.SchemaModel):
+        a: Index[str]
+        b: Index[int]
+
+    assert isinstance(DefaultSchema.validate(df), pd.DataFrame)
+
+    class CheckNameSchema(pa.SchemaModel):
+        a: Index[str] = pa.Field(check_name=True)
+        b: Index[int] = pa.Field(check_name=True)
+
+    assert isinstance(CheckNameSchema.validate(df), pd.DataFrame)
+
+    class NotCheckNameSchema(pa.SchemaModel):
+        a: Index[str] = pa.Field(check_name=False)
+        b: Index[int] = pa.Field(check_name=False)
+
+    df = pd.DataFrame(
+        index=pd.MultiIndex.from_arrays([["foo", "bar"], [0, 1]])
+    )
+    assert isinstance(NotCheckNameSchema.validate(df), pd.DataFrame)
+
+
 def test_check_validate_method():
     """Test validate method on valid data."""