Repository: hyperledger/aries-cloudagent-python

Issue:
Connection Errors in Regression Testing with Bifold based wallet 
Last nights BC Wallet regression tests failed connecting to the test agents which are pulled from acapy main. The agent gets to response-sent and nothing else happens, the wallet is spinning on the connection screen. Using acapy main from the night before is fine, along with the official release of ACA-Py.
The error in my agent at the time of the problem is,
```
2023-08-24 15:34:37,113 aries_cloudagent.core.dispatcher ERROR Handler error: BaseConnectionManager.get_connection_targets
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/pydid/validation.py", line 13, in wrap_validation_error
    yield
  File "/usr/local/lib/python3.9/site-packages/pydid/resource.py", line 68, in deserialize
    return parse_obj_as(cls, value)
  File "pydantic/tools.py", line 38, in pydantic.tools.parse_obj_as
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for ParsingModel[DIDDocumentRoot]
__root__ -> service -> 0 -> id
  did:sov:PkWfCgY4SSAYeSoaWx3RFP;PkWfCgY4SSAYeSoaWx3RFP#IndyAgentService is not a valid absolute or relative DID URL (type=value_error.invaliddidurl)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/asyncio/tasks.py", line 256, in __step
    result = coro.send(None)
  File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/connections/base_manager.py", line 603, in get_connection_targets
    targets = await self.fetch_connection_targets(connection)
  File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/connections/base_manager.py", line 564, in fetch_connection_targets
    return await self.resolve_connection_targets(
  File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/connections/base_manager.py", line 356, in resolve_connection_targets
    doc, didcomm_services = await self.resolve_didcomm_services(did)
  File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/connections/base_manager.py", line 263, in resolve_didcomm_services
    doc: ResolvedDocument = pydid.deserialize_document(doc_dict, strict=True)
  File "/usr/local/lib/python3.9/site-packages/pydid/__init__.py", line 63, in deserialize_document
    return cls.deserialize(value)
  File "/usr/local/lib/python3.9/site-packages/pydid/doc/doc.py", line 171, in deserialize
    DIDDocumentRoot.deserialize(value)
  File "/usr/local/lib/python3.9/site-packages/pydid/resource.py", line 68, in deserialize
    return parse_obj_as(cls, value)
  File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.9/site-packages/pydid/validation.py", line 15, in wrap_validation_error
    raise error_to_raise(
ValueError: Failed to deserialize DIDDocumentRoot:
1 validation error for ParsingModel[DIDDocumentRoot]
__root__ -> service -> 0 -> id
  did:sov:PkWfCgY4SSAYeSoaWx3RFP;PkWfCgY4SSAYeSoaWx3RFP#IndyAgentService is not a valid absolute or relative DID URL (type=value_error.invaliddidurl)
2023-08-24 15:34:37,185 aries_cloudagent.core.conductor ERROR Exception in message handler:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/pydid/validation.py", line 13, in wrap_validation_error
    yield
  File "/usr/local/lib/python3.9/site-packages/pydid/resource.py", line 68, in deserialize
    return parse_obj_as(cls, value)
  File "pydantic/tools.py", line 38, in pydantic.tools.parse_obj_as
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for ParsingModel[DIDDocumentRoot]
__root__ -> service -> 0 -> id
  did:sov:PkWfCgY4SSAYeSoaWx3RFP;PkWfCgY4SSAYeSoaWx3RFP#IndyAgentService is not a valid absolute or relative DID URL (type=value_error.invaliddidurl)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/asyncio/tasks.py", line 258, in __step
    result = coro.throw(exc)
  File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/core/dispatcher.py", line 261, in handle_message
    await handler(context, responder)
  File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/protocols/connections/v1_0/handlers/connection_request_handler.py", line 47, in handle
    await responder.send_reply(
  File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/messaging/responder.py", line 142, in send_reply
    return await self.send_outbound(
  File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/core/dispatcher.py", line 406, in send_outbound
    return await self._send(context.profile, message, self._inbound_message)
  File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/core/conductor.py", line 637, in outbound_message_router
    status: OutboundSendStatus = await self._outbound_message_router(
  File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/core/conductor.py", line 665, in _outbound_message_router
    return await self.queue_outbound(profile, outbound, inbound)
  File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/core/conductor.py", line 697, in queue_outbound
    outbound.target_list = await self.dispatcher.run_task(
  File "/usr/local/lib/python3.9/asyncio/futures.py", line 284, in __await__
    yield self  # This tells Task to wait for completion.
  File "/usr/local/lib/python3.9/asyncio/tasks.py", line 328, in __wakeup
    future.result()
  File "/usr/local/lib/python3.9/asyncio/futures.py", line 201, in result
    raise self._exception
  File "/usr/local/lib/python3.9/asyncio/tasks.py", line 256, in __step
    result = coro.send(None)
  File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/connections/base_manager.py", line 603, in get_connection_targets
    targets = await self.fetch_connection_targets(connection)
  File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/connections/base_manager.py", line 564, in fetch_connection_targets
    return await self.resolve_connection_targets(
  File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/connections/base_manager.py", line 356, in resolve_connection_targets
    doc, didcomm_services = await self.resolve_didcomm_services(did)
  File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/connections/base_manager.py", line 263, in resolve_didcomm_services
    doc: ResolvedDocument = pydid.deserialize_document(doc_dict, strict=True)
  File "/usr/local/lib/python3.9/site-packages/pydid/__init__.py", line 63, in deserialize_document
    return cls.deserialize(value)
  File "/usr/local/lib/python3.9/site-packages/pydid/doc/doc.py", line 171, in deserialize
    DIDDocumentRoot.deserialize(value)
  File "/usr/local/lib/python3.9/site-packages/pydid/resource.py", line 68, in deserialize
    return parse_obj_as(cls, value)
  File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.9/site-packages/pydid/validation.py", line 15, in wrap_validation_error
    raise error_to_raise(
ValueError: Failed to deserialize DIDDocumentRoot:
1 validation error for ParsingModel[DIDDocumentRoot]
__root__ -> service -> 0 -> id
  did:sov:PkWfCgY4SSAYeSoaWx3RFP;PkWfCgY4SSAYeSoaWx3RFP#IndyAgentService is not a valid absolute or relative DID URL (type=value_error.invaliddidurl)
2023-08-24 15:34:37,203 aries_cloudagent.core.conductor ERROR DON'T shutdown on ValueError Failed to deserialize DIDDocumentRoot:
1 validation error for ParsingModel[DIDDocumentRoot]
__root__ -> service -> 0 -> id
  did:sov:PkWfCgY4SSAYeSoaWx3RFP;PkWfCgY4SSAYeSoaWx3RFP#IndyAgentService is not a valid absolute or relative DID URL (type=value_error.invaliddidurl)
2023-08-24 15:34:37,204 aries_cloudagent.core.dispatcher ERROR Handler error: Dispatcher.handle_message
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/pydid/validation.py", line 13, in wrap_validation_error
    yield
  File "/usr/local/lib/python3.9/site-packages/pydid/resource.py", line 68, in deserialize
    return parse_obj_as(cls, value)
  File "pydantic/tools.py", line 38, in pydantic.tools.parse_obj_as
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for ParsingModel[DIDDocumentRoot]
__root__ -> service -> 0 -> id
  did:sov:PkWfCgY4SSAYeSoaWx3RFP;PkWfCgY4SSAYeSoaWx3RFP#IndyAgentService is not a valid absolute or relative DID URL (type=value_error.invaliddidurl)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/asyncio/tasks.py", line 258, in __step
    result = coro.throw(exc)
  File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/core/dispatcher.py", line 261, in handle_message
    await handler(context, responder)
  File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/protocols/connections/v1_0/handlers/connection_request_handler.py", line 47, in handle
    await responder.send_reply(
  File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/messaging/responder.py", line 142, in send_reply
    return await self.send_outbound(
  File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/core/dispatcher.py", line 406, in send_outbound
    return await self._send(context.profile, message, self._inbound_message)
  File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/core/conductor.py", line 637, in outbound_message_router
    status: OutboundSendStatus = await self._outbound_message_router(
  File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/core/conductor.py", line 665, in _outbound_message_router
    return await self.queue_outbound(profile, outbound, inbound)
  File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/core/conductor.py", line 697, in queue_outbound
    outbound.target_list = await self.dispatcher.run_task(
  File "/usr/local/lib/python3.9/asyncio/futures.py", line 284, in __await__
    yield self  # This tells Task to wait for completion.
  File "/usr/local/lib/python3.9/asyncio/tasks.py", line 328, in __wakeup
    future.result()
  File "/usr/local/lib/python3.9/asyncio/futures.py", line 201, in result
    raise self._exception
  File "/usr/local/lib/python3.9/asyncio/tasks.py", line 256, in __step
    result = coro.send(None)
  File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/connections/base_manager.py", line 603, in get_connection_targets
    targets = await self.fetch_connection_targets(connection)
  File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/connections/base_manager.py", line 564, in fetch_connection_targets
    return await self.resolve_connection_targets(
  File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/connections/base_manager.py", line 356, in resolve_connection_targets
    doc, didcomm_services = await self.resolve_didcomm_services(did)
  File "/usr/local/lib/python3.9/site-packages/aries_cloudagent/connections/base_manager.py", line 263, in resolve_didcomm_services
    doc: ResolvedDocument = pydid.deserialize_document(doc_dict, strict=True)
  File "/usr/local/lib/python3.9/site-packages/pydid/__init__.py", line 63, in deserialize_document
    return cls.deserialize(value)
  File "/usr/local/lib/python3.9/site-packages/pydid/doc/doc.py", line 171, in deserialize
    DIDDocumentRoot.deserialize(value)
  File "/usr/local/lib/python3.9/site-packages/pydid/resource.py", line 68, in deserialize
    return parse_obj_as(cls, value)
  File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.9/site-packages/pydid/validation.py", line 15, in wrap_validation_error
    raise error_to_raise(
ValueError: Failed to deserialize DIDDocumentRoot:
1 validation error for ParsingModel[DIDDocumentRoot]
__root__ -> service -> 0 -> id
  did:sov:PkWfCgY4SSAYeSoaWx3RFP;PkWfCgY4SSAYeSoaWx3RFP#IndyAgentService is not a valid absolute or relative DID URL (type=value_error.invaliddidurl)
```


Discussion:
Do you have a sample DID Doc attachment for what is getting sent from the BC Wallet to ACA-Py?
Attached is the entire output from the agent. Hope it helps. 
[acapy-main.log](https://github.com/hyperledger/aries-cloudagent-python/files/12432072/acapy-main.log)

That helps, thanks! Indeed, the service ID being sent is `<nym>#IndyAgentService`. I'll make sure we're accounting for values like that in the LegacyDocCorrections.

Resolving patch:
diff --git a/aries_cloudagent/resolver/default/legacy_peer.py b/aries_cloudagent/resolver/default/legacy_peer.py
--- a/aries_cloudagent/resolver/default/legacy_peer.py
+++ b/aries_cloudagent/resolver/default/legacy_peer.py
@@ -6,7 +6,7 @@
 from copy import deepcopy
 from dataclasses import asdict, dataclass
 import logging
-from typing import Optional, Sequence, Text, Union
+from typing import List, Optional, Sequence, Text, Union
 
 from pydid import DID
 
@@ -118,7 +118,8 @@ def didcomm_services_use_updated_conventions(value: dict) -> dict:
             for service in value["service"]:
                 if "type" in service and service["type"] == "IndyAgent":
                     service["type"] = "did-communication"
-                    service["id"] = service["id"].replace(";indy", "#didcomm")
+                    if ";" in service["id"]:
+                        service["id"] = value["id"] + "#didcomm"
         return value
 
     @staticmethod
@@ -139,6 +140,74 @@ def didcomm_services_recip_keys_are_refs_routing_keys_are_did_key(
                     ]
         return value
 
+    @staticmethod
+    def qualified(did_or_did_url: str) -> str:
+        """Make sure DID or DID URL is fully qualified."""
+        if not did_or_did_url.startswith("did:"):
+            return f"did:sov:{did_or_did_url}"
+        return did_or_did_url
+
+    @classmethod
+    def fully_qualified_ids_and_controllers(cls, value: dict) -> dict:
+        """Make sure IDs and controllers are fully qualified."""
+
+        def _make_qualified(value: dict) -> dict:
+            if "id" in value:
+                ident = value["id"]
+                value["id"] = cls.qualified(ident)
+            if "controller" in value:
+                controller = value["controller"]
+                value["controller"] = cls.qualified(controller)
+            return value
+
+        value = _make_qualified(value)
+        vms = []
+        for verification_method in value.get("verificationMethod", []):
+            vms.append(_make_qualified(verification_method))
+
+        services = []
+        for service in value.get("service", []):
+            services.append(_make_qualified(service))
+
+        auths = []
+        for authn in value.get("authentication", []):
+            if isinstance(authn, dict):
+                auths.append(_make_qualified(authn))
+            elif isinstance(authn, str):
+                auths.append(cls.qualified(authn))
+            else:
+                raise ValueError("Unexpected authentication value type")
+
+        value["authentication"] = auths
+        value["verificationMethod"] = vms
+        value["service"] = services
+        return value
+
+    @staticmethod
+    def remove_verification_method(
+        vms: List[dict], public_key_base58: str
+    ) -> List[dict]:
+        """Remove the verification method with the given key."""
+        return [vm for vm in vms if vm["publicKeyBase58"] != public_key_base58]
+
+    @classmethod
+    def remove_routing_keys_from_verification_method(cls, value: dict) -> dict:
+        """Remove routing keys from verification methods.
+
+        This was an old convention; routing keys were added to the public keys
+        of the doc even though they're usually not owned by the doc sender.
+
+        This correction should be applied before turning the routing keys into
+        did keys.
+        """
+        vms = value.get("verificationMethod", [])
+        for service in value.get("service", []):
+            if "routingKeys" in service:
+                for routing_key in service["routingKeys"]:
+                    vms = cls.remove_verification_method(vms, routing_key)
+        value["verificationMethod"] = vms
+        return value
+
     @classmethod
     def apply(cls, value: dict) -> dict:
         """Apply all corrections to the given DID document."""
@@ -146,7 +215,9 @@ def apply(cls, value: dict) -> dict:
         for correction in (
             cls.public_key_is_verification_method,
             cls.authentication_is_list_of_verification_methods_and_refs,
+            cls.fully_qualified_ids_and_controllers,
             cls.didcomm_services_use_updated_conventions,
+            cls.remove_routing_keys_from_verification_method,
             cls.didcomm_services_recip_keys_are_refs_routing_keys_are_did_key,
         ):
             value = correction(value)


Test patch:
diff --git a/aries_cloudagent/resolver/default/tests/test_legacy_peer.py b/aries_cloudagent/resolver/default/tests/test_legacy_peer.py
--- a/aries_cloudagent/resolver/default/tests/test_legacy_peer.py
+++ b/aries_cloudagent/resolver/default/tests/test_legacy_peer.py
@@ -179,3 +179,125 @@ def test_corrections_examples(self):
         actual = test_module.LegacyDocCorrections.apply(input_doc)
         assert actual == expected
         assert expected == test_module.LegacyDocCorrections.apply(expected)
+
+    def test_corrections_bifold(self):
+        input_doc = {
+            "@context": "https://w3id.org/did/v1",
+            "publicKey": [
+                {
+                    "id": "PkWfCgY4SSAYeSoaWx3RFP#1",
+                    "controller": "PkWfCgY4SSAYeSoaWx3RFP",
+                    "type": "Ed25519VerificationKey2018",
+                    "publicKeyBase58": "DQBMbzLmAK5iyiuPnqNvfkx3CQ2iTJ2sKz98utvdET4K",
+                }
+            ],
+            "service": [
+                {
+                    "id": "PkWfCgY4SSAYeSoaWx3RFP#IndyAgentService",
+                    "serviceEndpoint": "https://aries-mediator-agent.vonx.io",
+                    "type": "IndyAgent",
+                    "priority": 0,
+                    "recipientKeys": ["DQBMbzLmAK5iyiuPnqNvfkx3CQ2iTJ2sKz98utvdET4K"],
+                    "routingKeys": ["cK7fwfjpakMuv8QKVv2y6qouZddVw4TxZNQPUs2fFTd"],
+                }
+            ],
+            "authentication": [
+                {
+                    "publicKey": "PkWfCgY4SSAYeSoaWx3RFP#1",
+                    "type": "Ed25519SignatureAuthentication2018",
+                }
+            ],
+            "id": "PkWfCgY4SSAYeSoaWx3RFP",
+        }
+        expected = {
+            "@context": "https://w3id.org/did/v1",
+            "id": "did:sov:PkWfCgY4SSAYeSoaWx3RFP",
+            "verificationMethod": [
+                {
+                    "id": "did:sov:PkWfCgY4SSAYeSoaWx3RFP#1",
+                    "type": "Ed25519VerificationKey2018",
+                    "controller": "did:sov:PkWfCgY4SSAYeSoaWx3RFP",
+                    "publicKeyBase58": "DQBMbzLmAK5iyiuPnqNvfkx3CQ2iTJ2sKz98utvdET4K",
+                }
+            ],
+            "authentication": ["did:sov:PkWfCgY4SSAYeSoaWx3RFP#1"],
+            "service": [
+                {
+                    "id": "did:sov:PkWfCgY4SSAYeSoaWx3RFP#IndyAgentService",
+                    "type": "did-communication",
+                    "priority": 0,
+                    "recipientKeys": ["did:sov:PkWfCgY4SSAYeSoaWx3RFP#1"],
+                    "routingKeys": [
+                        "did:key:z6Mkf4aAGBvBA8Eq2Qy714sspCPoj8uUupJpeaHLDkq3aUF1#z6Mkf4aAGBvBA8Eq2Qy714sspCPoj8uUupJpeaHLDkq3aUF1"
+                    ],
+                    "serviceEndpoint": "https://aries-mediator-agent.vonx.io",
+                }
+            ],
+        }
+        actual = test_module.LegacyDocCorrections.apply(input_doc)
+        assert actual == expected
+        assert expected == test_module.LegacyDocCorrections.apply(expected)
+
+    def test_corrections_stored(self):
+        input_doc = {
+            "@context": "https://w3id.org/did/v1",
+            "id": "did:sov:PkWfCgY4SSAYeSoaWx3RFP",
+            "publicKey": [
+                {
+                    "id": "did:sov:PkWfCgY4SSAYeSoaWx3RFP#1",
+                    "type": "Ed25519VerificationKey2018",
+                    "controller": "did:sov:PkWfCgY4SSAYeSoaWx3RFP",
+                    "publicKeyBase58": "DQBMbzLmAK5iyiuPnqNvfkx3CQ2iTJ2sKz98utvdET4K",
+                },
+                {
+                    "id": "did:sov:PkWfCgY4SSAYeSoaWx3RFP#QPUs2fFT",
+                    "type": "Ed25519VerificationKey2018",
+                    "controller": "did:sov:PkWfCgY4SSAYeSoaWx3RFP",
+                    "publicKeyBase58": "cK7fwfjpakMuv8QKVv2y6qouZddVw4TxZNQPUs2fFTd",
+                },
+            ],
+            "authentication": [
+                {
+                    "type": "Ed25519SignatureAuthentication2018",
+                    "publicKey": "did:sov:PkWfCgY4SSAYeSoaWx3RFP#1",
+                }
+            ],
+            "service": [
+                {
+                    "id": "did:sov:PkWfCgY4SSAYeSoaWx3RFP;PkWfCgY4SSAYeSoaWx3RFP#IndyAgentService",
+                    "type": "IndyAgent",
+                    "priority": 0,
+                    "recipientKeys": ["DQBMbzLmAK5iyiuPnqNvfkx3CQ2iTJ2sKz98utvdET4K"],
+                    "routingKeys": ["cK7fwfjpakMuv8QKVv2y6qouZddVw4TxZNQPUs2fFTd"],
+                    "serviceEndpoint": "https://aries-mediator-agent.vonx.io",
+                }
+            ],
+        }
+        expected = {
+            "@context": "https://w3id.org/did/v1",
+            "id": "did:sov:PkWfCgY4SSAYeSoaWx3RFP",
+            "verificationMethod": [
+                {
+                    "id": "did:sov:PkWfCgY4SSAYeSoaWx3RFP#1",
+                    "type": "Ed25519VerificationKey2018",
+                    "controller": "did:sov:PkWfCgY4SSAYeSoaWx3RFP",
+                    "publicKeyBase58": "DQBMbzLmAK5iyiuPnqNvfkx3CQ2iTJ2sKz98utvdET4K",
+                }
+            ],
+            "authentication": ["did:sov:PkWfCgY4SSAYeSoaWx3RFP#1"],
+            "service": [
+                {
+                    "id": "did:sov:PkWfCgY4SSAYeSoaWx3RFP#didcomm",
+                    "type": "did-communication",
+                    "priority": 0,
+                    "recipientKeys": ["did:sov:PkWfCgY4SSAYeSoaWx3RFP#1"],
+                    "routingKeys": [
+                        "did:key:z6Mkf4aAGBvBA8Eq2Qy714sspCPoj8uUupJpeaHLDkq3aUF1#z6Mkf4aAGBvBA8Eq2Qy714sspCPoj8uUupJpeaHLDkq3aUF1"
+                    ],
+                    "serviceEndpoint": "https://aries-mediator-agent.vonx.io",
+                }
+            ],
+        }
+        actual = test_module.LegacyDocCorrections.apply(input_doc)
+        assert actual == expected
+        assert expected == test_module.LegacyDocCorrections.apply(expected)