Repository: hyperledger/aries-cloudagent-python

Issue:
present-proof v2 puts records to the storage it can't read back any more
I ran into this issue and I think it might be possible at the moment that someone sends data to running 3rd party acapy instances to break their usage of reading their own received presentation exchanges from acapy - and it wouldn't be straight forward to recover from it (deleting such records via the API).

Here are the steps to reproduce the situation:
I'm trying to send a W3C presentation. I'm using
`/present-proof-2.0/send-proposal` for this and I'm probably using some wrong format I guess, but this is not the problem here.
Here is the (probably wrong) data that I send to the endpoint:

```

{
    "auto_present": true,
    "comment": "test",
    "connection_id": "7a5d96b0-98fa-4208-9cec-8779ebb55c21",
    "presentation_proposal": {
      "dif": {

        "input_descriptors": [
            {
                "id": "XXX",
                "name": "CO2 Confirmation",
                "schema": [
                    {
                        "uri": "https://www.w3.org/2018/credentials#VerifiableCredential"
                    }
                ],
                "constraints": {
                    "fields": [
                        {
                            "path": [
                                "$.type"
                            ],
                            "filter": {
                                "type": "string",
                                "pattern": "Co2ConfirmationCredential"
                            }
                        }
                    ]
                }
            }
        ]
      }
    },
    "trace": false
  }
```

Above data is just to make the problem reproducible.

I'm using sub-wallets with askar profile: `"--multitenancy-config", "{\"wallet_type\":\"askar-profile\"}"`

Now, what happens on the other side (receiver of the proposal, the acapy behind the given connection_id) is (a lot of auto-flags ON... but probably not relevant here) acapy stores a record that it can't read back any more.
When trying to read via `/present-proof-2.0/records`
```
marshmallow.exceptions.ValidationError: {'presentation_definition': ['Missing data for required field.'], 'input_descriptors': ['Unknown field.']}
```
It breaks in a way that the endpoint doesn't deliver any other item from this endpoint any more.
Debugging the pres_ex_id and trying to delete from the according endpoint `DELETE /present-proof-2.0/records/{pres_ex_id}` also doesn't work because it tries to read the item from storage before it removes it.

So my question - if I should try to fix this - what would be the expected behavior in such cases?

1. we should never get such wrong data items into the storage in the first place and it's ok to not being able to read this AND other data back from storage in such rare cases (current situation)
2. `DELETE` should still work and command line should print out the problematic `pres_ex_id`
3. Listing records should still work for non-problematic records
4. Listing records should still work and add a "dummy" record that just contains the `pres_ex_id` for the problematic record (being able to use the DELETE endpoint with it...) 
5. any other options?




Discussion:
Good questions -- and that is an ugly bug.  Needs to be fixed pretty quickly.  @shaangill025 and @andrewwhitehead please review and weigh in on this.

My $0.02CDN (but I would question if I'm the right person here...):

The first one is the most important. To me, the data validation needs to be beefed up in this case.  My question would be whether that is all we need to do?

Item 3 should still work, but I suspect that would create an ugly performance issue -- e.g. having to read record by record from the database somehow vs. getting the set.  If it is possible, could it be that if the current query be tried and on failure, go to some clean up routine to iterate through the records to list the good ones.

After that, I'm over my head.  I hope option 1 is sufficient.

Resolving patch:
diff --git a/aries_cloudagent/protocols/present_proof/dif/pres_proposal_schema.py b/aries_cloudagent/protocols/present_proof/dif/pres_proposal_schema.py
--- a/aries_cloudagent/protocols/present_proof/dif/pres_proposal_schema.py
+++ b/aries_cloudagent/protocols/present_proof/dif/pres_proposal_schema.py
@@ -3,7 +3,7 @@
 
 from ....messaging.models.openapi import OpenAPISchema
 
-from .pres_exch import InputDescriptorsSchema
+from .pres_exch import InputDescriptorsSchema, DIFOptionsSchema
 
 
 class DIFProofProposalSchema(OpenAPISchema):
@@ -16,3 +16,7 @@ class DIFProofProposalSchema(OpenAPISchema):
         ),
         required=False,
     )
+    options = fields.Nested(
+        DIFOptionsSchema(),
+        required=False,
+    )
diff --git a/aries_cloudagent/protocols/present_proof/v2_0/formats/dif/handler.py b/aries_cloudagent/protocols/present_proof/v2_0/formats/dif/handler.py
--- a/aries_cloudagent/protocols/present_proof/v2_0/formats/dif/handler.py
+++ b/aries_cloudagent/protocols/present_proof/v2_0/formats/dif/handler.py
@@ -151,9 +151,18 @@ async def create_bound_request(
             A tuple (updated presentation exchange record, presentation request message)
 
         """
-        dif_proof_request = pres_ex_record.pres_proposal.attachment(
+        dif_proof_request = {}
+        pres_proposal_dict = pres_ex_record.pres_proposal.attachment(
             DIFPresFormatHandler.format
         )
+        if "options" not in pres_proposal_dict:
+            dif_proof_request["options"] = {"challenge": str(uuid4())}
+        else:
+            dif_proof_request["options"] = pres_proposal_dict["options"]
+            del pres_proposal_dict["options"]
+            if "challenge" not in dif_proof_request.get("options"):
+                dif_proof_request["options"]["challenge"] = str(uuid4())
+        dif_proof_request["presentation_definition"] = pres_proposal_dict
 
         return self.get_format_data(PRES_20_REQUEST, dif_proof_request)
 


Test patch:
diff --git a/aries_cloudagent/protocols/present_proof/v2_0/formats/dif/tests/test_handler.py b/aries_cloudagent/protocols/present_proof/v2_0/formats/dif/tests/test_handler.py
--- a/aries_cloudagent/protocols/present_proof/v2_0/formats/dif/tests/test_handler.py
+++ b/aries_cloudagent/protocols/present_proof/v2_0/formats/dif/tests/test_handler.py
@@ -284,7 +284,7 @@ async def test_get_all_suites(self):
         for suite in suites:
             assert type(suite) in types
 
-    async def test_create_bound_request(self):
+    async def test_create_bound_request_a(self):
         dif_proposal_dict = {
             "input_descriptors": [
                 {
@@ -339,6 +339,118 @@ async def test_create_bound_request(self):
             output[1], AttachDecorator
         )
 
+    async def test_create_bound_request_b(self):
+        dif_proposal_dict = {
+            "options": {"challenge": "test123"},
+            "input_descriptors": [
+                {
+                    "id": "citizenship_input_1",
+                    "name": "EU Driver's License",
+                    "group": ["A"],
+                    "schema": [
+                        {
+                            "uri": "https://www.w3.org/2018/credentials#VerifiableCredential"
+                        }
+                    ],
+                    "constraints": {
+                        "limit_disclosure": "required",
+                        "fields": [
+                            {
+                                "path": ["$.credentialSubject.givenName"],
+                                "purpose": "The claim must be from one of the specified issuers",
+                                "filter": {"type": "string", "enum": ["JOHN", "CAI"]},
+                            }
+                        ],
+                    },
+                }
+            ],
+        }
+        dif_pres_proposal = V20PresProposal(
+            formats=[
+                V20PresFormat(
+                    attach_id="dif",
+                    format_=ATTACHMENT_FORMAT[PRES_20_PROPOSAL][
+                        V20PresFormat.Format.DIF.api
+                    ],
+                )
+            ],
+            proposals_attach=[
+                AttachDecorator.data_json(dif_proposal_dict, ident="dif")
+            ],
+        )
+        record = V20PresExRecord(
+            pres_ex_id="pxid",
+            thread_id="thid",
+            connection_id="conn_id",
+            initiator="init",
+            role="role",
+            state="state",
+            pres_proposal=dif_pres_proposal,
+            verified="false",
+            auto_present=True,
+            error_msg="error",
+        )
+        output = await self.handler.create_bound_request(pres_ex_record=record)
+        assert isinstance(output[0], V20PresFormat) and isinstance(
+            output[1], AttachDecorator
+        )
+
+    async def test_create_bound_request_c(self):
+        dif_proposal_dict = {
+            "options": {"domain": "test123"},
+            "input_descriptors": [
+                {
+                    "id": "citizenship_input_1",
+                    "name": "EU Driver's License",
+                    "group": ["A"],
+                    "schema": [
+                        {
+                            "uri": "https://www.w3.org/2018/credentials#VerifiableCredential"
+                        }
+                    ],
+                    "constraints": {
+                        "limit_disclosure": "required",
+                        "fields": [
+                            {
+                                "path": ["$.credentialSubject.givenName"],
+                                "purpose": "The claim must be from one of the specified issuers",
+                                "filter": {"type": "string", "enum": ["JOHN", "CAI"]},
+                            }
+                        ],
+                    },
+                }
+            ],
+        }
+        dif_pres_proposal = V20PresProposal(
+            formats=[
+                V20PresFormat(
+                    attach_id="dif",
+                    format_=ATTACHMENT_FORMAT[PRES_20_PROPOSAL][
+                        V20PresFormat.Format.DIF.api
+                    ],
+                )
+            ],
+            proposals_attach=[
+                AttachDecorator.data_json(dif_proposal_dict, ident="dif")
+            ],
+        )
+        record = V20PresExRecord(
+            pres_ex_id="pxid",
+            thread_id="thid",
+            connection_id="conn_id",
+            initiator="init",
+            role="role",
+            state="state",
+            pres_proposal=dif_pres_proposal,
+            verified="false",
+            auto_present=True,
+            error_msg="error",
+        )
+        output = await self.handler.create_bound_request(pres_ex_record=record)
+        assert isinstance(output[0], V20PresFormat) and isinstance(
+            output[1], AttachDecorator
+        )
+
     async def test_create_pres(self):
         dif_pres_request = V20PresRequest(
             formats=[