Repository: dask/dask

Issue:
read_parquet equality (=) filter returns incorrect result
<!-- Please include a self-contained copy-pastable example that generates the issue if possible.

Please be concise with code posted. See guidelines below on how to provide a good bug report:

- Craft Minimal Bug Reports http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports
- Minimal Complete Verifiable Examples https://stackoverflow.com/help/mcve

Bug reports that follow these guidelines are easier to diagnose, and so are often handled much more quickly.
-->

**What happened**:
When using a `filter` with `read_parquet` consisting of the equality op `=` , results returned are incorrect. Using `==` however returns the correct result.

**What you expected to happen**:
Same behavior with `=` and `==`.

**Minimal Complete Verifiable Example**:

```python
import pandas as pd
from dask import dataframe as dd

df = pd.DataFrame({'col0':['a','b','c'], 'col1':[1,2,3]})
dd.from_pandas(df,1).to_parquet("filter.parquet")

dd.read_parquet("filter.parquet",filters=[('col1', '=', 2)]).compute()
```
returns
```python
Empty DataFrame
Columns: [col0, col1]
Index: []
```

while `==` returns the right result:
```python
dd.read_parquet("filter.parquet",filters=[('col1', '==', 2)]).compute()
```
```python
  col0  col1
1    b     2
```

The same issue doesn't exist in pandas: 
```python
pd.read_parquet("filter.parquet",filters=[('col1', '=', 2)])
```
```python
                    col0  col1
__null_dask_index__           
1                      b     2
```


The same issue exists with `dask_cudf.read_parquet` as well.

**Anything else we need to know?**:

When pointing to the list of parquet files instead of the folder the filter works as expected
```python
dd.read_parquet("filter.parquet/*.parquet",filters=[('col1', '=', 2)]).compute()
``` 
```python
  col0  col1
1    b     2
```

**Environment**:

- Dask version: from main @ dask@9147ca54
- Python version: 3.8.10
- Operating System: Ubuntu 18.04
- Install method (conda, pip, source): `pip install git+https://github.com/dask/dask.git@main`


cc: @randerzander


Discussion:
I noticed that in the docs for `dd.read_parquet` it mentions an example with "==" see filters section in this docstring. https://github.com/dask/dask/blob/898afc04208aba91dc52acb5e04cc7a76f9c1013/dask/dataframe/io/parquet/core.py#L133-L138
However, the function `apply_filters` mentions "=" see docstrings https://github.com/dask/dask/blob/main/dask/dataframe/io/parquet/core.py#L1006-L1071

So there is definitely an inconsistency on documentation at least. I wasn't able to find the pandas source code where they apply the filters and compare how it's done there. But it looks like the logic for "==" on dask.dataframe lives in this function 
https://github.com/dask/dask/blob/898afc04208aba91dc52acb5e04cc7a76f9c1013/dask/dataframe/io/parquet/core.py#L1006
and the tests here https://github.com/dask/dask/blob/898afc04208aba91dc52acb5e04cc7a76f9c1013/dask/dataframe/io/tests/test_parquet.py#L1350

@ayushdg would you like to take a crack at this, and open a PR?

cc: @jrbourbeau in case I got something wrong, on the comments above. 
Happy to open a PR for this and look into it a bit more if needed. Based on the discussion above is the recommendation to standardize documentation around `=` vs `==` or is it worth looking into adding the `=` case as well in this bit:
https://github.com/dask/dask/blob/898afc04208aba91dc52acb5e04cc7a76f9c1013/dask/dataframe/io/parquet/core.py#L100


Also cc @rjzamora In case you have thoughts.

I think we do need to support both `"="` and `"=="` since they are both supported by pyarrow and fastparquet independently.  I think that `"="` being left out of `apply_conjunction` was just an oversight. The fix should be pretty trivial if I understand correctly. 

**Side note**: The "correct" behavior for "fastparquet" and "pyarrow-legacy" should be to filter nothing for the example above. Fitering everything is clearly a bug, but only "pyarrow-dataset" should result in full row-wise filtering within a row-group.
Thanks for the details @rjzamora. Happy to look into this and open a PR for the fix soon.

Resolving patch:
diff --git a/dask/dataframe/io/parquet/core.py b/dask/dataframe/io/parquet/core.py
--- a/dask/dataframe/io/parquet/core.py
+++ b/dask/dataframe/io/parquet/core.py
@@ -1050,7 +1050,7 @@ def apply_conjunction(parts, statistics, conjunction):
                     out_statistics.append(stats)
                 else:
                     if (
-                        operator == "=="
+                        operator in ("==", "=")
                         and min <= value <= max
                         or operator == "<"
                         and min < value


Test patch:
diff --git a/dask/dataframe/io/tests/test_parquet.py b/dask/dataframe/io/tests/test_parquet.py
--- a/dask/dataframe/io/tests/test_parquet.py
+++ b/dask/dataframe/io/tests/test_parquet.py
@@ -1386,6 +1386,11 @@ def test_filters(tmpdir, write_engine, read_engine):
     assert e.npartitions == 2
     assert ((e.x < 2) | (e.x > 7)).all().compute()
 
+    f = dd.read_parquet(tmp_path, engine=read_engine, filters=[("y", "=", "c")])
+    assert f.npartitions == 1
+    assert len(f)
+    assert (f.y == "c").all().compute()
+
 
 @write_read_engines()
 def test_filters_v0(tmpdir, write_engine, read_engine):
@@ -1407,10 +1412,15 @@ def test_filters_v0(tmpdir, write_engine, read_engine):
     ddf2 = dd.read_parquet(
         fn, index=False, engine=read_engine, filters=[("at", "==", "aa")]
     ).compute()
+    ddf3 = dd.read_parquet(
+        fn, index=False, engine=read_engine, filters=[("at", "=", "aa")]
+    ).compute()
     if pyarrow_row_filtering:
         assert_eq(ddf2, ddf[ddf["at"] == "aa"], check_index=False)
+        assert_eq(ddf3, ddf[ddf["at"] == "aa"], check_index=False)
     else:
         assert_eq(ddf2, ddf)
+        assert_eq(ddf3, ddf)
 
     # with >1 partition and no filters
     ddf.repartition(npartitions=2, force=True).to_parquet(fn, engine=write_engine)
@@ -1421,14 +1431,21 @@ def test_filters_v0(tmpdir, write_engine, read_engine):
     if read_engine == "fastparquet":
         ddf.repartition(npartitions=2, force=True).to_parquet(fn, engine=write_engine)
         df2 = fastparquet.ParquetFile(fn).to_pandas(filters=[("at", "==", "aa")])
+        df3 = fastparquet.ParquetFile(fn).to_pandas(filters=[("at", "=", "aa")])
         assert len(df2) > 0
+        assert len(df3) > 0
 
     # with >1 partition and filters
     ddf.repartition(npartitions=2, force=True).to_parquet(fn, engine=write_engine)
     ddf2 = dd.read_parquet(
         fn, engine=read_engine, filters=[("at", "==", "aa")]
     ).compute()
+    ddf3 = dd.read_parquet(
+        fn, engine=read_engine, filters=[("at", "=", "aa")]
+    ).compute()
     assert len(ddf2) > 0
+    assert len(ddf3) > 0
+    assert_eq(ddf2, ddf3)
 
 
 def test_filtering_pyarrow_dataset(tmpdir, engine):