Repository: unionai-oss/pandera (Python)

Issue:
Parser is executed after validation, not before
I am using this class:

```python
import pandas as pd
import pandera as pa
from pandera.typing import Index
from pandera.errors import SchemaError

class Schema(pa.DataFrameModel):
    """Schema."""

    index: Index[int]

    class Config:
        """Schema config."""

        strict = False
        coerce = True
        add_missing_columns = True

    a: Series[str]
    b: Series[int]
    c: Series[int]

    @pa.dataframe_parser
    @classmethod
    def preprocess(cls, df: pd.DataFrame) -> pd.DataFrame:
        """Preprocessing."""
        if 'b' not in df.columns and 'c' not in df.columns:
            raise SchemaError(schema=cls, data=df, message=f"No `b` or `c` in {df.columns}")
        
        # raise Exception()
        
        if `b' not in df.columns:
            df['b'] = df['c']
        if `c' not in df.columns:
            df['c'] = df['b']
        return df

Schema.validate(pd.DataFrame({'a': ['xxx'], 'b': 0}))
```

and it complains that `c` is not in the dataframe...

It should not, I add `c` in the parser.

Plus, if I uncomment that `raise Exception()` line, this exception is never raised as the error I get about missing column is happening before.

`pandera == 0.19.3`


Discussion: Hello,

I've checked and it seems like if the inner Config class is provided, then those checks (core parsers) will run before the custom parsers. 

When it tries to add column 'C' to the dataframe due to `add_missing_columns = True`, it fails, because there are no default values provided, therefore it never gets to your custom parsers.

If the order of execution in /pandera/backends/pandas/container.py:

```python
        # Collect status of columns against schema
        column_info = self.collect_column_info(check_obj, schema)
        core_parsers: List[Tuple[Callable[..., Any], Tuple[Any, ...]]] = [
            (self.add_missing_columns, (schema, column_info)),
            (self.strict_filter_columns, (schema, column_info)),
            (self.coerce_dtype, (schema,)),
        ]

        for parser, args in core_parsers:
            try:
                check_obj = parser(check_obj, *args)
            except SchemaError as exc:
                error_handler.collect_error(
                    validation_type(exc.reason_code), exc.reason_code, exc
                )
            except SchemaErrors as exc:
                error_handler.collect_errors(exc.schema_errors)

        # run custom parsers
        check_obj = self.run_parsers(schema, check_obj)
```

was changed to this, then it would work:

```python
        # run custom parsers
        check_obj = self.run_parsers(schema, check_obj)
        
        # Collect status of columns against schema
        column_info = self.collect_column_info(check_obj, schema)

        core_parsers: List[Tuple[Callable[..., Any], Tuple[Any, ...]]] = [
            (self.add_missing_columns, (schema, column_info)),
            (self.strict_filter_columns, (schema, column_info)),
            (self.coerce_dtype, (schema,)),
        ]

        for parser, args in core_parsers:
            try:
                check_obj = parser(check_obj, *args)
            except SchemaError as exc:
                error_handler.collect_error(
                    validation_type(exc.reason_code), exc.reason_code, exc
                )
            except SchemaErrors as exc:
                error_handler.collect_errors(exc.schema_errors)
```



I have no idea if this is an intended behavior or not and I haven't checked this any deeper. If this is indeed a bug, I'd be happy to pick this issue up.
Thanks a lot for the clarification.

> When it tries to add column 'C' to the dataframe due to add_missing_columns = True, it fails, because there are no default values provided, therefore it never gets to your custom parsers.

Indeed, but for my use-case, where the column `c` needs to be populated with values from column `b`, I cannot provide a default value.

In my opinion, this cannot be an intended behavior, especially if this can be fixed (perhaps with your suggestion) without breaking anything else.

Or something even cooler is to let users provide a `mode` argument to `@pa.dataframe_parser`, a la `pydantic`.
I agree, I expected the parsers to run before any other operation.

In my use case, I have a column with integers and nan so it gets interpreted as a float. My parsers `parsers=[pa.Parser(lambda series: series.dropna().astype(int))]` intend to drop all nans and convert the column dtype to int. However because of the current ordering of operations, the output type is still float AND I cannot use `coerce=True` without it erroring out.
looking into this



Fix (patch):
diff --git a/pandera/backends/pandas/array.py b/pandera/backends/pandas/array.py
index f7a5c1cc8..968ea3444 100644
--- a/pandera/backends/pandas/array.py
+++ b/pandera/backends/pandas/array.py
@@ -54,10 +54,7 @@ def validate(
             check_obj = self.set_default(check_obj, schema)
 
         # run custom parsers
-        check_obj = self.run_parsers(
-            schema,
-            check_obj,
-        )
+        check_obj = self.run_parsers(schema, check_obj)
 
         try:
             if is_field(check_obj) and schema.coerce:
diff --git a/pandera/backends/pandas/container.py b/pandera/backends/pandas/container.py
index 117a9e689..5b3b71088 100644
--- a/pandera/backends/pandas/container.py
+++ b/pandera/backends/pandas/container.py
@@ -68,6 +68,9 @@ def validate(
         if hasattr(check_obj, "pandera"):
             check_obj = check_obj.pandera.add_schema(schema)
 
+        # run custom parsers
+        check_obj = self.run_parsers(schema, check_obj)
+
         # Collect status of columns against schema
         column_info = self.collect_column_info(check_obj, schema)
 
@@ -78,9 +81,6 @@ def validate(
             (self.coerce_dtype, (schema,)),
         ]
 
-        # run custom parsers
-        check_obj = self.run_parsers(schema, check_obj)
-
         for parser, args in core_parsers:
             try:
                 check_obj = parser(check_obj, *args)