Repository: great-expectations/great_expectations

Issue:
[FEATURE] Support expect_multicolumn_values_to_be_unique on Spark
Hello, it is my feature implementation time for Spark.

- [X] Test your changes - Total `4` cases about this feature
- [X] Update the documentation.
- [X] Sign the Contributor License Agreement.
- [X] Rebase your changes - Currently latest on `develop` branch

Please feel free to review.
[FEATURE] Support expect_multicolumn_values_to_be_unique on Spark
Hello, it is my feature implementation time for Spark.

- [X] Test your changes - Total `4` cases about this feature
- [X] Update the documentation.
- [X] Sign the Contributor License Agreement.
- [X] Rebase your changes - Currently latest on `develop` branch

Please feel free to review.


Discussion:
Hello @WilliamWsyHK! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

* In the file [`great_expectations/dataset/sparkdf_dataset.py`](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py):

> [Line 353:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L353): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (110 > 79 characters)
> [Line 354:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L354): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (91 > 79 characters)
> [Line 360:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L360): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (97 > 79 characters)
> [Line 367:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L367): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (108 > 79 characters)
> [Line 368:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L368): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (115 > 79 characters)
> [Line 373:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L373): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (82 > 79 characters)
> [Line 375:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L375): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (81 > 79 characters)
> [Line 384:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L384): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (106 > 79 characters)
> [Line 389:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L389): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (106 > 79 characters)
> [Line 400:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L400): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (80 > 79 characters)
> [Line 413:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L413): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (108 > 79 characters)
> [Line 431:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L431): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (80 > 79 characters)
> [Line 432:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L432): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (80 > 79 characters)
> [Line 433:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L433): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (151 > 79 characters)
> [Line 434:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L434): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (88 > 79 characters)
> [Line 439:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L439): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (98 > 79 characters)
> [Line 440:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L440): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (102 > 79 characters)
> [Line 441:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L441): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (128 > 79 characters)
> [Line 442:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L442): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (125 > 79 characters)
> [Line 443:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L443): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (122 > 79 characters)
> [Line 444:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L444): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (130 > 79 characters)
> [Line 445:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L445): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (127 > 79 characters)
> [Line 446:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L446): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (123 > 79 characters)
> [Line 447:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L447): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (126 > 79 characters)
> [Line 448:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L448): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (130 > 79 characters)
> [Line 1125:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L1125): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (90 > 79 characters)





##### Comment last updated at 2020-05-12 16:38:14 UTC
# [Codecov](https://codecov.io/gh/great-expectations/great_expectations/pull/1294?src=pr&el=h1) Report
> Merging [#1294](https://codecov.io/gh/great-expectations/great_expectations/pull/1294?src=pr&el=desc) into [develop](https://codecov.io/gh/great-expectations/great_expectations/commit/afd653bb1838b0230a54ddd5024c6ce04b985399&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `75.00%`.

[![Impacted file tree graph](https://codecov.io/gh/great-expectations/great_expectations/pull/1294/graphs/tree.svg?width=650&height=150&src=pr&token=rbHxgTxYTs)](https://codecov.io/gh/great-expectations/great_expectations/pull/1294?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           develop    #1294      +/-   ##
===========================================
- Coverage    78.17%   78.16%   -0.02%     
===========================================
  Files          126      126              
  Lines        13816    13875      +59     
===========================================
+ Hits         10801    10845      +44     
- Misses        3015     3030      +15     
```


| [Impacted Files](https://codecov.io/gh/great-expectations/great_expectations/pull/1294?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [great\_expectations/dataset/sparkdf\_dataset.py](https://codecov.io/gh/great-expectations/great_expectations/pull/1294/diff?src=pr&el=tree#diff-Z3JlYXRfZXhwZWN0YXRpb25zL2RhdGFzZXQvc3BhcmtkZl9kYXRhc2V0LnB5) | `84.36% <75.00%> (-1.42%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/great-expectations/great_expectations/pull/1294?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/great-expectations/great_expectations/pull/1294?src=pr&el=footer). Last update [afd653b...a229d1a](https://codecov.io/gh/great-expectations/great_expectations/pull/1294?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


[![Coverage Status](https://coveralls.io/builds/30871600/badge)](https://coveralls.io/builds/30871600)

Coverage decreased (-0.2%) to 73.564% when pulling **a229d1aa3e34e87bc6c0ffa7f0bab50c0d9f2346 on WilliamWsyHK:feature/spark-expect_multicolumn_values_to_be_unique** into **afd653bb1838b0230a54ddd5024c6ce04b985399 on great-expectations:develop**.

Hi, Thanks for the review.

1. Where can I find the change log?
2. Please review
3. Talking about caching, I have a gut guess that the "persist = false" logic was wrong (currently implemented within `SparkDFDataset`), due to the fact that all current "real" caching logic is implemented within `MetaSparkDFDataset`; and for other discussion I was not aware about, please guide me thru those.
> Hi, Thanks for the review.
> 
> 1. Where can I find the change log?
> 2. Please review
> 3. Talking about caching, I have a gut guess that the "persist = false" logic was wrong (currently implemented within `SparkDFDataset`), due to the fact that all current "real" caching logic is implemented within `MetaSparkDFDataset`; and for other discussion I was not aware about, please guide me thru those.

For 1. `docs/changelog.rst`: https://github.com/great-expectations/great_expectations/blob/develop/docs/changelog.rst
Hello @WilliamWsyHK! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

* In the file [`great_expectations/dataset/sparkdf_dataset.py`](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py):

> [Line 353:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L353): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (110 > 79 characters)
> [Line 354:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L354): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (91 > 79 characters)
> [Line 360:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L360): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (97 > 79 characters)
> [Line 367:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L367): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (108 > 79 characters)
> [Line 368:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L368): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (115 > 79 characters)
> [Line 373:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L373): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (82 > 79 characters)
> [Line 375:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L375): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (81 > 79 characters)
> [Line 384:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L384): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (106 > 79 characters)
> [Line 389:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L389): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (106 > 79 characters)
> [Line 400:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L400): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (80 > 79 characters)
> [Line 413:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L413): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (108 > 79 characters)
> [Line 431:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L431): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (80 > 79 characters)
> [Line 432:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L432): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (80 > 79 characters)
> [Line 433:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L433): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (151 > 79 characters)
> [Line 434:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L434): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (88 > 79 characters)
> [Line 439:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L439): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (98 > 79 characters)
> [Line 440:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L440): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (102 > 79 characters)
> [Line 441:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L441): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (128 > 79 characters)
> [Line 442:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L442): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (125 > 79 characters)
> [Line 443:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L443): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (122 > 79 characters)
> [Line 444:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L444): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (130 > 79 characters)
> [Line 445:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L445): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (127 > 79 characters)
> [Line 446:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L446): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (123 > 79 characters)
> [Line 447:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L447): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (126 > 79 characters)
> [Line 448:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L448): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (130 > 79 characters)
> [Line 1125:80](https://github.com/great-expectations/great_expectations/blob/1dd82f02f5757c5e8aaaf704879756b849c576a3/great_expectations/dataset/sparkdf_dataset.py#L1125): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (90 > 79 characters)





##### Comment last updated at 2020-05-12 16:38:14 UTC
# [Codecov](https://codecov.io/gh/great-expectations/great_expectations/pull/1294?src=pr&el=h1) Report
> Merging [#1294](https://codecov.io/gh/great-expectations/great_expectations/pull/1294?src=pr&el=desc) into [develop](https://codecov.io/gh/great-expectations/great_expectations/commit/afd653bb1838b0230a54ddd5024c6ce04b985399&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `75.00%`.

[![Impacted file tree graph](https://codecov.io/gh/great-expectations/great_expectations/pull/1294/graphs/tree.svg?width=650&height=150&src=pr&token=rbHxgTxYTs)](https://codecov.io/gh/great-expectations/great_expectations/pull/1294?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           develop    #1294      +/-   ##
===========================================
- Coverage    78.17%   78.16%   -0.02%     
===========================================
  Files          126      126              
  Lines        13816    13875      +59     
===========================================
+ Hits         10801    10845      +44     
- Misses        3015     3030      +15     
```


| [Impacted Files](https://codecov.io/gh/great-expectations/great_expectations/pull/1294?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [great\_expectations/dataset/sparkdf\_dataset.py](https://codecov.io/gh/great-expectations/great_expectations/pull/1294/diff?src=pr&el=tree#diff-Z3JlYXRfZXhwZWN0YXRpb25zL2RhdGFzZXQvc3BhcmtkZl9kYXRhc2V0LnB5) | `84.36% <75.00%> (-1.42%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/great-expectations/great_expectations/pull/1294?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/great-expectations/great_expectations/pull/1294?src=pr&el=footer). Last update [afd653b...a229d1a](https://codecov.io/gh/great-expectations/great_expectations/pull/1294?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


[![Coverage Status](https://coveralls.io/builds/30871600/badge)](https://coveralls.io/builds/30871600)

Coverage decreased (-0.2%) to 73.564% when pulling **a229d1aa3e34e87bc6c0ffa7f0bab50c0d9f2346 on WilliamWsyHK:feature/spark-expect_multicolumn_values_to_be_unique** into **afd653bb1838b0230a54ddd5024c6ce04b985399 on great-expectations:develop**.

Hi, Thanks for the review.

1. Where can I find the change log?
2. Please review
3. Talking about caching, I have a gut guess that the "persist = false" logic was wrong (currently implemented within `SparkDFDataset`), due to the fact that all current "real" caching logic is implemented within `MetaSparkDFDataset`; and for other discussion I was not aware about, please guide me thru those.
> Hi, Thanks for the review.
> 
> 1. Where can I find the change log?
> 2. Please review
> 3. Talking about caching, I have a gut guess that the "persist = false" logic was wrong (currently implemented within `SparkDFDataset`), due to the fact that all current "real" caching logic is implemented within `MetaSparkDFDataset`; and for other discussion I was not aware about, please guide me thru those.

For 1. `docs/changelog.rst`: https://github.com/great-expectations/great_expectations/blob/develop/docs/changelog.rst

Resolving patch:
diff --git a/great_expectations/dataset/sparkdf_dataset.py b/great_expectations/dataset/sparkdf_dataset.py
--- a/great_expectations/dataset/sparkdf_dataset.py
+++ b/great_expectations/dataset/sparkdf_dataset.py
@@ -438,7 +438,7 @@ def inner_wrapper(
                         if val is None or not all(v for k, v in val):
                             parsed_maybe_limited_unexpected_list.append(val)
                         else:
-                            if all(isinstance(v, string_types) for k, v in val):
+                            if all(isinstance(v, str) for k, v in val):
                                 val = OrderedDict((k, parse(v)) for k, v in val)
                             parsed_maybe_limited_unexpected_list.append(
                                 OrderedDict(
@@ -1121,6 +1121,59 @@ def expect_column_pair_values_to_be_equal(
             when(col(column_A_name) == col(column_B_name), True).otherwise(False),
         )
 
+    @DocInherit
+    @MetaSparkDFDataset.column_pair_map_expectation
+    def expect_column_pair_values_A_to_be_greater_than_B(
+        self,
+        column_A,
+        column_B,
+        or_equal=None,
+        parse_strings_as_datetimes=None,
+        allow_cross_type_comparisons=None,
+        ignore_row_if="both_values_are_missing",
+        result_format=None,
+        include_config=True,
+        catch_exceptions=None,
+        meta=None,
+    ):
+        # FIXME
+        if allow_cross_type_comparisons:
+            raise NotImplementedError
+
+        column_A_name = column_A.schema.names[1]
+        column_B_name = column_B.schema.names[1]
+
+        if parse_strings_as_datetimes:
+            _udf = udf(parse, sparktypes.TimestampType())
+            # Create new columns for comparison without replacing original values.
+            (timestamp_column_A, timestamp_column_B) = (
+                "__ts_{0}".format(column_A_name),
+                "__ts_{0}".format(column_B_name),
+            )
+            temp_column_A = column_A.withColumn(timestamp_column_A, _udf(column_A_name))
+            temp_column_B = column_B.withColumn(timestamp_column_B, _udf(column_B_name))
+            # Use the new columns to compare instead of original columns.
+            (column_A_name, column_B_name) = (timestamp_column_A, timestamp_column_B)
+
+        else:
+            temp_column_A = column_A
+            temp_column_B = column_B
+
+        join_df = temp_column_A.join(
+            temp_column_B, temp_column_A["__row"] == temp_column_B["__row"], how="inner"
+        )
+
+        if or_equal:
+            return join_df.withColumn(
+                "__success",
+                when(col(column_A_name) >= col(column_B_name), True).otherwise(False),
+            )
+        else:
+            return join_df.withColumn(
+                "__success",
+                when(col(column_A_name) > col(column_B_name), True).otherwise(False),
+            )
+
     @DocInherit
     @MetaSparkDFDataset.multicolumn_map_expectation
     def expect_multicolumn_values_to_be_unique(


Test patch:
diff --git a/tests/test_definitions/column_pair_map_expectations/expect_column_pair_values_A_to_be_greater_than_B.json b/tests/test_definitions/column_pair_map_expectations/expect_column_pair_values_A_to_be_greater_than_B.json
--- a/tests/test_definitions/column_pair_map_expectations/expect_column_pair_values_A_to_be_greater_than_B.json
+++ b/tests/test_definitions/column_pair_map_expectations/expect_column_pair_values_A_to_be_greater_than_B.json
@@ -9,6 +9,16 @@
       "a" : ["1/1/2016", "2/2/2016", "3/3/2016", "4/4/2016", "5/5/2016", null, null, null, null, null],
       "b" : ["1/1/2016", "2/1/2016", "3/1/2016", "4/1/2016", "5/1/2020", null, null, null, null, null]
     },
+    "schemas": {
+      "spark": {
+        "w": "IntegerType",
+        "x": "IntegerType",
+        "y": "IntegerType",
+        "z": "IntegerType",
+        "a": "StringType",
+        "b": "StringType"
+      }
+    },
     "tests" : [{
       "title" : "Basic positive test",
       "exact_match_out" : false,
diff --git a/tests/test_utils.py b/tests/test_utils.py
--- a/tests/test_utils.py
+++ b/tests/test_utils.py
@@ -475,7 +475,7 @@ def candidate_test_is_on_temporary_notimplemented_list(context, expectation_type
             # "expect_column_kl_divergence_to_be_less_than",
             "expect_column_parameterized_distribution_ks_test_p_value_to_be_greater_than",
             # "expect_column_pair_values_to_be_equal",
-            "expect_column_pair_values_A_to_be_greater_than_B",
+            # "expect_column_pair_values_A_to_be_greater_than_B",
             "expect_column_pair_values_to_be_in_set",
             # "expect_multicolumn_values_to_be_unique"
         ]