Repository: great-expectations/great_expectations

Issue:
ERROR: TypeError: Keys in TupleS3StoreBackend must be instances of <class 'tuple'>, not <class 'great_expectations.data_context.types.resource_identifiers.ValidationResultIdentifier'>
A few users reported this error, but we do not have precise steps to reproduce this - in progress...
TODO: fill in the details


**Describe the bug**
TODO

**To Reproduce**
TODO
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

**Expected behavior**
Error should not happen

**Environment (please complete the following information):**
 - OS: [e.g. iOS]
 - GE Version: [e.g. 0.10.0]

**Additional context**
Add any other context about the problem here.



Discussion:
Got this same issue when trying to build data docs to our company site.

Issue goes away when you delete the existing contents of the site from s3 and then run validations, but then of course reappears the next time.

Happening on 0.12.0 of GE on Windows and 0.12.1 on Ubuntu.
Another example here: https://greatexpectationstalk.slack.com/archives/CUUMYDJAX/p1597954314013700

See this Slack message for the same with `ExpectationSuite` instead of `ValidationResultIdentifier`: https://greatexpectationstalk.slack.com/archives/CUTCNHN82/p1600718063029700

Triaged - this is a real bug, the core team is working on reproducing and fixing it.
I have the same problem. Getting GE and Prefect running on Azure Container Instances - a week in, after having fought 20 other things :D

As this worked earlier, I tried to track down which dependencies might be involved - but doesn't seem to be a dependency. These are versions of dependencies that changed in september, here with the original version from sept 1st:

```
toolz==0.10.0
numpy==1.19.1
pandas==1.1.1
attrs==20.1.0
pyrsistent==0.16.0
zipp==3.1.0
importlib-metadata==1.7.0
traitlets==5.0.0
pygments==2.6.1
bleach==3.1.5
nbconvert==5.6.1
cffi==1.14.2
terminado==0.8.3
notebook==6.1.3
ruamel.yaml.clib==0.2.0
ruamel.yaml==0.16.10
great-expectations==0.12.0
marshmallow==3.7.1
dask==2.25.0
distributed==2.25.0
prefect==0.13.5
botocore==1.17.53
boto3==1.14.53
```
Hi all - we have been trying many different configurations over the past few days to try to reproduce this bug. 

The closest we have come is to remove validations from our validation store (.json), but leave them in our data docs validations directory (.html). The error we reproduced was not exactly the same as what is mentioned as this issue (and deserves attention):
```
ParamValidationError: Parameter validation failed:
Invalid type for parameter Prefix, value: None, type: <class 'NoneType'>, valid types: <class 'str'>
```
However - can anyone help with a specific set of steps to reproduce? Perhaps with a simple Great Expectations setup using just s3 and local filesystem (and no sensitive info so we can share). Please share your great_expectations.yml, the version of GE you are using, your OS, python version, any other configuration details (e.g. output of `pip freeze`) and what was / wasn't in your metadata store bucket(s) when you encounter the error (actually a `tree` of your buckets would be super helpful!).
What about a debugging zoom call on monday? And if we can’t fix it, I can give access to my repo or share all files

_
Lars Corneliussen
________________________________
Von: anthonyburdi <notifications@github.com>
Gesendet: Friday, September 25, 2020 9:26:10 PM
An: great-expectations/great_expectations <great_expectations@noreply.github.com>
Cc: Lars Corneliussen <lars@corneliussen.de>; Comment <comment@noreply.github.com>
Betreff: Re: [great-expectations/great_expectations] ERROR: TypeError: Keys in TupleS3StoreBackend must be instances of <class 'tuple'>, not <class 'great_expectations.data_context.types.resource_identifiers.ValidationResultIdentifier'> (#1907)


Hi all - we have been trying many different configurations over the past few days to try to reproduce this bug.

The closest we have come is to remove validations from our validation store (.json), but leave them in our data docs validations directory (.html). The error we reproduced was not exactly the same as what is mentioned as this issue (and deserves attention):

ParamValidationError: Parameter validation failed:
Invalid type for parameter Prefix, value: None, type: <class 'NoneType'>, valid types: <class 'str'>


However - can anyone help with a specific set of steps to reproduce? Perhaps with a simple Great Expectations setup using just s3 and local filesystem (and no sensitive info so we can share). Please share your great_expectations.yml, the version of GE you are using, your OS, python version, any other configuration details (e.g. output of pip freeze) and what was / wasn't in your metadata store bucket(s) when you encounter the error (actually a tree of your buckets would be super helpful!).

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub<https://github.com/great-expectations/great_expectations/issues/1907#issuecomment-699112070>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAA6NQEW276FRKPSPJJSK3DSHTVFFANCNFSM4RRHBBSQ>.

This error crops up weekly for me so if you've not tracked it down next time it occurs I will do some debugging.
Hi, I was able to get a stacktrace of when the "must be tuple not ValidationResultIdentifier" error is happening. For my case, the issue starts at [site_builder.py#L769](https://github.com/great-expectations/great_expectations/blame/29190428e345c2a89671d94054478693211a36da/great_expectations/render/renderer/site_builder.py#L769).

Below is a screenshot of the stacktrace:
![stacktrace](https://user-images.githubusercontent.com/9522388/94506600-c173f600-01db-11eb-8457-2d3e359e291a.png)

**Environment:**
Running inside a Centos dev container, Python 3.7.7
```
great-expectations==0.12.1
boto3==1.13.22
botocore==1.16.22
numpy==1.18.4
pandas==0.25.3
```

**Metadata store yaml configs sample:**
```
stores:
  expectations_store:
    class_name: ExpectationsStore
    store_backend:
      class_name: TupleS3StoreBackend
      bucket: myBucket
      prefix: stores/expectations/
 
  validations_store:
    class_name: ValidationsStore
    store_backend:
      class_name: TupleS3StoreBackend
      bucket: myBucket
      prefix: stores/validations/

data_docs_sites:
  s3_site:
    class_name: SiteBuilder
    show_how_to_buttons: false
    store_backend:
      class_name: TupleS3StoreBackend
      bucket: myBucket
      prefix: docs/
    site_index_builder:
      class_name: DefaultSiteIndexBuilder
```
**Other notes:**
I couldn't get a tree of the bucket content due to my S3 permission restrictions. I am in the process of migrating from GE 0.10.x to 0.12.1, the bucket has files/directories with the old naming convention (pre 0.11.x) that I have yet to move over. 
However, I'm not sure if those files are the cause of issue. In https://greatexpectationstalk.slack.com/archives/CUTCNHN82/p1599650889160100, it was mentioned that the ExpectationSuiteIdentifier version of the exception occurs whenever there is previous validation in the store.

Please let me know if there's more info I could provide!
We did some debugging with @anthonyburdi yesterday. It seems that GE is getting the current contents of the S3 bucket, then running some kind of validation (maybe determining if the file should be kept in S3 or deleted) and storing that in a `ValidationResultIdentifier` which is created using `from_tuple` [here](https://github.com/great-expectations/great_expectations/blob/29190428e345c2a89671d94054478693211a36da/great_expectations/render/renderer/site_builder.py#L710)

Then it figures out which of theses to remove, and runs the deletion using ValidationResultIdentifier instead of the corresponding tuple that specifies the path.

A naive fix in [site_builder.py](https://github.com/great-expectations/great_expectations/blob/29190428e345c2a89671d94054478693211a36da/great_expectations/render/renderer/site_builder.py#L721) replacing `.remove_key(expectation_suite_site_key)` with `.remove_key(expectation_suite_site_key.to_tuple())` yielded a different error. But at least it is a starting point.

```
validation_and_profiling_result_source_keys = self.data_context.stores[
    self.site_section_builders_config[source_store].get("source_store_name")
].list_keys()
validation_and_profiling_result_site_keys = [
>>  ValidationResultIdentifier.from_tuple(validation_result_tuple)
    for validation_result_tuple in self.target_store.store_backends[
        ValidationResultIdentifier
    ].list_keys()
]

if skip_and_clean_missing:
    cleaned_keys = []
    for (
        validation_result_site_key
    ) in validation_and_profiling_result_site_keys:
        if (
            validation_result_site_key
            not in validation_and_profiling_result_source_keys
        ):
            self.target_store.store_backends[
                ValidationResultIdentifier
>>          ].remove_key(validation_result_site_key)
        else:
            cleaned_keys.append(validation_result_site_key)
    validation_and_profiling_result_site_keys = cleaned_keys
```


Seems like the code that has problems was introduced here:
https://github.com/great-expectations/great_expectations/commit/3216afe1d755d0d6327c8358fafeb8e4952cb4b0

Currently in my case it is failing everytime I redeploy my container, as this also resets the validation results. I guess I'd have to use some kind of store to keep validations around, so they are rebuilt as part of the data docs generation?
It would be nice if one could set skip_and_clean_missing to false from yaml. In my case where I launch fresh containers this would be useful anyway. And it would, for now, most probably give a workaround for the bug.

https://github.com/great-expectations/great_expectations/blob/29190428e345c2a89671d94054478693211a36da/great_expectations/render/renderer/site_builder.py#L292
Just a heads up - I was able to replicate this issue locally and am working on a fix. Thank you for all the help and I will keep you posted!

Resolving patch:
diff --git a/great_expectations/data_context/store/tuple_store_backend.py b/great_expectations/data_context/store/tuple_store_backend.py
--- a/great_expectations/data_context/store/tuple_store_backend.py
+++ b/great_expectations/data_context/store/tuple_store_backend.py
@@ -375,7 +375,7 @@ class TupleS3StoreBackend(TupleStoreBackend):
     def __init__(
         self,
         bucket,
-        prefix=None,
+        prefix="",
         filepath_template=None,
         filepath_prefix=None,
         filepath_suffix=None,


Test patch: