Repository: apache/airflow

Issue:
Fix POST /taskInstances/list with wildcards returns unhelpful error
This PR closes #26424
Problem Statement: The POST Request to the endpoint  `/api/v1/dags/~/dagRuns/~/taskInstances/list` returns an unhelpful error -> "detail": "None is not of type 'object'", when the JSON Body is empty. Upon further inspection, it seems that this error is raised by vaildation.py and is not handled properly leading to a generic error message and a Python Exception inside the logs.
Fix: Allowing the request payload to be nullable and then checking if the data in the request is None before parsing it into JSON -> allows us to handle this error gracefully and even check for other unhandled cases.

New Response: 

```json
{
    "detail": "['POST Body must not be None']",
    "status": 400,
    "title": "Bad Request",
    "type": "http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/stable-rest-api-ref.html#section/Errors/BadRequest"
}
```

Old Response:

```json
{
    "detail": "None is not of type 'object'",
    "status": 400,
    "title": "Bad Request",
    "type": "http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/stable-rest-api-ref.html#section/Errors/BadRequest"
}
```

Old Logs:

```
2023-03-28T17:16:17.637+0000 validation.py[200] None is not type object
```
`POST /taskInstances/list` with wildcards returns unhelpful error
### Apache Airflow version

2.3.4

### What happened

https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/get_task_instances_batch
fails with an error with wildcards while
https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/get_task_instances 
succeeds with wildcards

Error:
```
400
"None is not of type 'object'"
```


### What you think should happen instead

_No response_

### How to reproduce

1) `astro dev init`
2) `astro dev start`
3) `test1.py` and `python test1.py`
```
import requests
host = "http://localhost:8080/api/v1"
kwargs = {
    'auth': ('admin', 'admin'),
    'headers': {'content-type': 'application/json'}
}
r = requests.post(f'{host}/dags/~/dagRuns/~/taskInstances/list', **kwargs, timeout=10)
print(r.url, r.text)
```

output
```
http://localhost:8080/api/v1/dags/~/dagRuns/~/taskInstances/list 
{
  "detail": "None is not of type 'object'",
  "status": 400,
  "title": "Bad Request",
  "type": "http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/stable-rest-api-ref.html#section/Errors/BadRequest"
}
```

3) `test2.py` and `python test2.py`
```
import requests
host = "http://localhost:8080/api/v1"
kwargs = {
    'auth': ('admin', 'admin'),
    'headers': {'content-type': 'application/json'}
}
r = requests.get(f'{host}/dags/~/dagRuns/~/taskInstances', **kwargs, timeout=10)  # change here
print(r.url, r.text)
```
```
<correct output>
```

### Operating System

Debian

### Versions of Apache Airflow Providers

_No response_

### Deployment

Astronomer

### Deployment details

_No response_

### Anything else

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)



Discussion:
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (ruff, mypy and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/BREEZE.rst) for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#coding-style-and-best-practices).
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack

I prefer this hack:
```diff
diff --git a/airflow/api_connexion/exceptions.py b/airflow/api_connexion/exceptions.py
index 11468e1506..8789ef8f39 100644
--- a/airflow/api_connexion/exceptions.py
+++ b/airflow/api_connexion/exceptions.py
@@ -41,7 +41,8 @@ EXCEPTIONS_LINK_MAP = {
 def common_error_handler(exception: BaseException) -> flask.Response:
     """Used to capture connexion exceptions and add link to the type field."""
     if isinstance(exception, ProblemException):
-
+        if exception.detail == "None is not of type 'object'":
+            exception.detail = "POST Body must not be empty"
         link = EXCEPTIONS_LINK_MAP.get(exception.status)
         if link:
             response = problem(
```
it probably does not make sense to support wildcards on the batch (i.e. POST) endpoint, because wildcards can be handled by the normal, non-batch version.  batch's stated pupose is for when you have too many dag_ids for the GET request.  so, since it doesn't work now anyway, probably best solution here is to raise 404 / some helpful message suggesting to use the get endpoint.   or... could just fix it and support it 🤷 
@fritz-astronomer,  The POST parameter data cannot be None

```python
import requests
host = "http://localhost:8080/api/v1"
kwargs = {
    'auth': ('admin', 'admin'),
    'data':'{}',  
    'headers': {'content-type': 'application/json'}
}
r = requests.post(f'{host}/dags/~/dagRuns/~/taskInstances/list', **kwargs, timeout=10)
print(r.url, r.text)
```
Oh, @chenglongyan - thanks. That makes a lot of sense 😞 
Maybe this can just be a more helpful error message on that route saying that a data payload is required
current message
```
  "detail": "None is not of type 'object'",
```
vs 
```
  "detail": "POST Body must not be None",
```
> Oh, @chenglongyan - thanks. That makes a lot of sense 😞 Maybe this can just be a more helpful error message on that route saying that a data payload is required current message
> 
> ```
>   "detail": "None is not of type 'object'",
> ```
> 
> vs
> 
> ```
>   "detail": "POST Body must not be None",
> ```

Good idea :). Maybe you would like to contribute a fix for that  @fritz-astronomer :) ?
or maybe the API can be made to handle None?
i will work on this issue , can you please assign it to me
@maheshsv are you still working on this issue?
@eladkal I'd like to work on this issue, please /assign.
@eladkal @potiuk I've allowed the Request Payload to be nullable and then checked it later before JSON is parsed to allow custom error. Let me know if it seems fine.

Resolving patch:
diff --git a/airflow/www/extensions/init_views.py b/airflow/www/extensions/init_views.py
--- a/airflow/www/extensions/init_views.py
+++ b/airflow/www/extensions/init_views.py
@@ -21,6 +21,8 @@
 from os import path
 
 from connexion import FlaskApi, ProblemException, Resolver
+from connexion.decorators.validation import RequestBodyValidator
+from connexion.exceptions import BadRequestProblem
 from flask import Flask, request
 
 from airflow.api_connexion.exceptions import common_error_handler
@@ -209,6 +211,20 @@ def resolve(self, operation):
         return _LazyResolution(self.resolve_function_from_operation_id, operation_id)
 
 
+class _CustomErrorRequestBodyValidator(RequestBodyValidator):
+    """Custom request body validator that overrides error messages.
+
+    By default, Connextion emits a very generic *None is not of type 'object'*
+    error when receiving an empty request body (with the view specifying the
+    body as non-nullable). We overrides it to provide a more useful message.
+    """
+
+    def validate_schema(self, data, url):
+        if not self.is_null_value_valid and data is None:
+            raise BadRequestProblem(detail="Request body must not be empty")
+        return super().validate_schema(data, url)
+
+
 def init_api_connexion(app: Flask) -> None:
     """Initialize Stable API"""
     base_path = "/api/v1"
@@ -245,6 +261,7 @@ def _handle_method_not_allowed(ex):
         },
         strict_validation=True,
         validate_responses=True,
+        validator_map={"body": _CustomErrorRequestBodyValidator},
     ).blueprint
     api_bp.after_request(set_cors_headers_on_response)
 


Test patch:
diff --git a/tests/api_connexion/endpoints/test_task_instance_endpoint.py b/tests/api_connexion/endpoints/test_task_instance_endpoint.py
--- a/tests/api_connexion/endpoints/test_task_instance_endpoint.py
+++ b/tests/api_connexion/endpoints/test_task_instance_endpoint.py
@@ -875,6 +875,14 @@ def test_should_raise_403_forbidden(self):
         )
         assert response.status_code == 403
 
+    def test_should_raise_400_for_no_json(self):
+        response = self.client.post(
+            "/api/v1/dags/~/dagRuns/~/taskInstances/list",
+            environ_overrides={"REMOTE_USER": "test"},
+        )
+        assert response.status_code == 400
+        assert response.json["detail"] == "Request body must not be empty"
+
     @pytest.mark.parametrize(
         "payload, expected",
         [