Repository: hyperledger/aries-cloudagent-python

Issue:
Empty ServiceDecorator in OobRecord Response causes 422 Unprocessable Entity Error
The ACA swagger spec provides the following spec for `OobRecord`:

```py
class OobRecord(BaseModel):
    """NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech).
...
    invi_msg_id: str
    invitation: InvitationMessage
    oob_id: str
    state: Literal["initial","prepare-response","await-response","reuse-not-accepted","reuse-accepted","done","deleted"]
    attach_thread_id: Optional[str] = None
    connection_id: Optional[str] = None
    created_at: Optional[str] = None
    our_recipient_key: Optional[str] = None
    role: Optional[Literal["sender", "receiver"]] = None
    their_service: Optional[ServiceDecorator] = None
    trace: Optional[bool] = None
    updated_at: Optional[str] = None
```

I draw your attention to: `their_service: Optional[ServiceDecorator]`

For `ServiceDecorator` we have the following:

```py
class ServiceDecorator(BaseModel):
    """NOTE: This class is auto generated by OpenAPI Generator (https://openapi-generator.tech).
...
    recipient_keys: List[str] = Field(..., alias="recipientKeys")
    service_endpoint: str = Field(..., alias="serviceEndpoint")
    routing_keys: Optional[List[str]] = Field(None, alias="routingKeys")
```

What you'll notice is that `recipientKeys` and `serviceEndpoint` are required when providing a `ServiceDecorator`.

However, when receiving an `OobRecord` response from `receive_invitation`, we've encountered `422 Unprocessable Entity` due to the response providing an empty `their_service` field.

We could not yet intercept the body of the message before parsing into the response object, but the error message indicates: 
```
b'{"detail":[{"loc":["their_service","recipientKeys"],"msg":"none is not an allowed value","type":"type_error.none.not_allowed"},{"loc":["their_service","serviceEndpoint"],"msg":"none is not an allowed value","type":"type_error.none.not_allowed"}]}'
```

In a nutshell: we're receiving a `their_service`, but with None for its required fields.

This occurs when creating a connectionless invitation in a multitenant environment, and so perhaps this particular case was missed.

A simple workaround for us is to manually make the `ServiceDecorator` fields optional:
```py
    recipient_keys: Optional[List[str]] = Field(None, alias="recipientKeys")
    service_endpoint: Optional[str] = Field(None, alias="serviceEndpoint")
    routing_keys: Optional[List[str]] = Field(None, alias="routingKeys")
```

But of course, this is malforming the `ServiceDecorator` class. It would be more appropriate for `their_service` to be None, when its required fields are empty.

I'd be happy to look into making a PR for this if someone can direct me to the relevant code. Thanks!


Discussion:
Thanks for the detail.  I’m not sure where this change would be made. Perhaps @ianco or @jcourt562 could provide a pointer?
I've looked at this again, and I believe I've traced the source of the problem.

To recap, the `ServiceDecorator` has required fields for `endpoint` and `recipient_keys`.
I presume there is good reason for this. Because it's possible to just make those optional, but let's rather not.

I've looked at all the instances where a `ServiceDecorator` is referenced in the codebase, and I notice the following in `_service_decorator_from_service`, [in the OutOfBand manager](https://github.com/hyperledger/aries-cloudagent-python/blob/8b6c87fba2a6dc6cce4ced92d4db554fb4cad37a/aries_cloudagent/protocols/out_of_band/v1_0/manager.py#L591):
```py
    async def _service_decorator_from_service(
        self, service: Union[Service, str]
    ) -> ServiceDecorator:
        if isinstance(service, str):
            (
                endpoint,
                recipient_keys,
                routing_keys,
            ) = await self.resolve_invitation(service)

            return ServiceDecorator(
                endpoint=endpoint,
                recipient_keys=recipient_keys,
                routing_keys=routing_keys,
            )
        else:
            # Create ~service decorator from the oob service
            recipient_keys = [
                DIDKey.from_did(did_key).public_key_b58
                for did_key in service.recipient_keys
            ]
            routing_keys = [
                DIDKey.from_did(did_key).public_key_b58
                for did_key in service.routing_keys
            ]

            return ServiceDecorator(
                endpoint=service.service_endpoint,
                recipient_keys=recipient_keys,
                routing_keys=routing_keys,
            )
```
The else block seems to have the problem.

The `self.resolve_invitation(service)` in the if block seems to be safe - it interacts with the Service definitions from `pydid`:
```py
class Service(Resource):
    """Representation of DID Document Services."""

    id: DIDUrl
    type: str
    service_endpoint: Union[DIDUrl, AnyUrl, Literal[""], List[ServiceEndpoint]]
```
which will always return a string for service_endpoint - at worst an empty string `""`.

The else block, however, seems that it can potentially introduce nulls into the ServiceDecorator object, because `service.service_endpoint` can be None (that's the default value):
```py
class Service(BaseModel):
...
    def __init__(
...
        service_endpoint: str = None,
    ):
```

As an aside, it's a bit of a code smell to cast None to a str ... it should actually be a Optional[str]. Because the compiler indicates it's a string, when its default is actually None.

My proposal is that the following should only be attempted if service.service_endpoint is not None:
```py
            return ServiceDecorator(
                endpoint=service.service_endpoint,
                recipient_keys=recipient_keys,
                routing_keys=routing_keys,
            )
```

It seems reasonable to me that if the endpoint is None, then we should return None, instead of a ServiceDecorator. Because the only place the result is used, is to call `handle_message` in the OobMessageProcessor, which expects `their_service` to be an Optional ServiceDecorator:
```py
    async def handle_message(
        self,
        profile: Profile,
        messages: List[Dict[str, Any]],
        oob_record: OobRecord,
        their_service: Optional[ServiceDecorator] = None,
    ):
```

If their_service has an empty endpoint, then a ServiceDecorator object shouldn't be necessary. I feel it should be the same if the endpoint is an empty string. 

@dbluhm - it seems you helped with the Service implementation. 
I know this looks like a lot of info to ask about, but it's just to show my working. 
What do you think would be reasonable solution for how empty (none or "") endpoints should be handled? 

Crux of the issue is that, at the moment, the OobRecord response made in `OutOfBandManager.receive_invitation` can provide a malformed response (trying to define a ServiceDecorator with a None for required fields). Seems to me like `their_service` should just be None in those cases.
> As an aside, it's a bit of a code smell to cast None to a str ... it should actually be a Optional[str]. Because the compiler indicates it's a string, when its default is actually None.

Yes, unfortunately, this is all over ACA-Py. Before the type checking tools really matured to where they are today, it was a common convention (at least in ACA-Py) for Optional values to be expressed as `my_value: str = None`. Pyright/Pylance, Mypy, etc. now complain about this. I correct it as I come across these but it's still quite prevalent in the code base.

> What do you think would be reasonable solution for how empty (none or "") endpoints should be handled?

The empty string `serviceEndpoint` was an early mechanism for indicating to the receiver that the sender did not have a publicly accessible endpoint. This should eventually be phased out in favor of more explicit mechanisms such as `didcomm:transport/queue` as [recommended in the DIDComm v2 extension on return routing](https://github.com/decentralized-identity/didcomm-messaging/blob/main/extensions/return_route/main.md#queue-transport).

I'm curious, how is the connectionless invitation constructed that is resulting in these issues? If there isn't a valid service endpoint to message, it seems like it will ultimately result in an error still since the receiver wouldn't be able to respond to it.

I agree that the types in these methods needs to be tightened up and perhaps returning a `None` is the right answer when there is no valid service endpoint passed to that `_service_decorator_from_service` helper.

Resolving patch:
diff --git a/aries_cloudagent/protocols/out_of_band/v1_0/manager.py b/aries_cloudagent/protocols/out_of_band/v1_0/manager.py
--- a/aries_cloudagent/protocols/out_of_band/v1_0/manager.py
+++ b/aries_cloudagent/protocols/out_of_band/v1_0/manager.py
@@ -582,7 +582,10 @@ async def _process_request_attach(self, oob_record: OobRecord):
         if not oob_record.connection_id:
             service = oob_record.invitation.services[0]
             their_service = await self._service_decorator_from_service(service)
-            LOGGER.debug("Found service for oob record %s", their_service)
+            if their_service:
+                LOGGER.debug("Found service for oob record %s", their_service)
+            else:
+                LOGGER.debug("No service decorator obtained from %s", service)
 
         await message_processor.handle_message(
             self.profile, messages, oob_record=oob_record, their_service=their_service
@@ -590,7 +593,7 @@ async def _process_request_attach(self, oob_record: OobRecord):
 
     async def _service_decorator_from_service(
         self, service: Union[Service, str]
-    ) -> ServiceDecorator:
+    ) -> Optional[ServiceDecorator]:
         if isinstance(service, str):
             (
                 endpoint,
@@ -598,13 +601,20 @@ async def _service_decorator_from_service(
                 routing_keys,
             ) = await self.resolve_invitation(service)
 
+            if not endpoint:
+                return None
+
             return ServiceDecorator(
                 endpoint=endpoint,
                 recipient_keys=recipient_keys,
                 routing_keys=routing_keys,
             )
-        else:
-            # Create ~service decorator from the oob service
+        elif isinstance(service, Service):
+            endpoint = service.service_endpoint
+
+            if not endpoint:
+                return None
+
             recipient_keys = [
                 DIDKey.from_did(did_key).public_key_b58
                 for did_key in service.recipient_keys
@@ -615,10 +625,16 @@ async def _service_decorator_from_service(
             ]
 
             return ServiceDecorator(
-                endpoint=service.service_endpoint,
+                endpoint=endpoint,
                 recipient_keys=recipient_keys,
                 routing_keys=routing_keys,
             )
+        else:
+            LOGGER.warning(
+                "Unexpected type `%s` passed to `_service_decorator_from_service`",
+                type(service),
+            )
+            return None
 
     async def _wait_for_reuse_response(
         self, oob_id: str, timeout: int = 15


Test patch:
diff --git a/aries_cloudagent/protocols/out_of_band/v1_0/tests/test_manager.py b/aries_cloudagent/protocols/out_of_band/v1_0/tests/test_manager.py
--- a/aries_cloudagent/protocols/out_of_band/v1_0/tests/test_manager.py
+++ b/aries_cloudagent/protocols/out_of_band/v1_0/tests/test_manager.py
@@ -1717,6 +1717,71 @@ async def test_service_decorator_from_service_object(self):
         assert service.recipient_keys == [TestConfig.test_verkey]
         assert service.routing_keys == self.test_mediator_routing_keys
 
+    async def test_service_decorator_from_service_str_empty_endpoint(self):
+        did = "did:sov:something"
+
+        self.manager.resolve_invitation = async_mock.CoroutineMock()
+        self.manager.resolve_invitation.return_value = (
+            "",  # empty endpoint
+            [TestConfig.test_verkey],
+            self.test_mediator_routing_keys,
+        )
+
+        service = await self.manager._service_decorator_from_service(did)
+
+        assert service is None
+
+    async def test_service_decorator_from_service_str_none_endpoint(self):
+        did = "did:sov:something"
+
+        self.manager.resolve_invitation = async_mock.CoroutineMock()
+        self.manager.resolve_invitation.return_value = (
+            None,  # None endpoint
+            [TestConfig.test_verkey],
+            self.test_mediator_routing_keys,
+        )
+
+        service = await self.manager._service_decorator_from_service(did)
+
+        assert service is None
+
+    async def test_service_decorator_from_service_object_empty_endpoint(self):
+        oob_service = OobService(
+            service_endpoint="",  # empty endpoint
+            recipient_keys=[
+                DIDKey.from_public_key_b58(TestConfig.test_verkey, ED25519).did
+            ],
+            routing_keys=[
+                DIDKey.from_public_key_b58(verkey, ED25519).did
+                for verkey in self.test_mediator_routing_keys
+            ],
+        )
+        service = await self.manager._service_decorator_from_service(oob_service)
+
+        assert service is None
+
+    async def test_service_decorator_from_service_object_none_endpoint(self):
+        oob_service = OobService(
+            service_endpoint=None,  # None endpoint
+            recipient_keys=[
+                DIDKey.from_public_key_b58(TestConfig.test_verkey, ED25519).did
+            ],
+            routing_keys=[
+                DIDKey.from_public_key_b58(verkey, ED25519).did
+                for verkey in self.test_mediator_routing_keys
+            ],
+        )
+        service = await self.manager._service_decorator_from_service(oob_service)
+
+        assert service is None
+
+    async def test_service_decorator_from_service_invalid_type(self):
+        invalid_type = 123  # integer, for example
+
+        service = await self.manager._service_decorator_from_service(invalid_type)
+
+        assert service is None
+
     async def test_delete_stale_connection_by_invitation(self):
         current_datetime = datetime_now()
         older_datetime = current_datetime - timedelta(hours=4)