Repository: jorgecarleitao/arrow2
Pull request #1102 (2022-06-25 07:56:49)

Enabled more integration tests with pyarrow

codecov[bot]: # [Codecov](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1102?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao) Report
> Merging [#1102](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1102?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao) (2d3f0c5) into [main](https://codecov.io/gh/jorgecarleitao/arrow2/commit/88f05bb1ea37c05688c19d062e10dd35e00c0987?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao) (88f05bb) will **increase** coverage by `0.01%`.
> The diff coverage is `0.00%`.

> :exclamation: Current head 2d3f0c5 differs from pull request most recent head 16d2278. Consider uploading reports for the commit 16d2278 to get more accurate results

```diff
@@            Coverage Diff             @@
##             main    #1102      +/-   ##
==========================================
+ Coverage   81.24%   81.25%   +0.01%     
==========================================
  Files         366      366              
  Lines       35336    35336              
==========================================
+ Hits        28707    28713       +6     
+ Misses       6629     6623       -6     
```


| [Impacted Files](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1102?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao) | Coverage Δ | |
|---|---|---|
| [src/io/parquet/write/pages.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1102/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2lvL3BhcnF1ZXQvd3JpdGUvcGFnZXMucnM=) | `92.19% <0.00%> (ø)` | |
| [src/io/ipc/read/reader.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1102/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2lvL2lwYy9yZWFkL3JlYWRlci5ycw==) | `96.18% <0.00%> (ø)` | |
| [src/array/utf8/mod.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1102/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2FycmF5L3V0ZjgvbW9kLnJz) | `82.74% <0.00%> (+0.95%)` | :arrow_up: |
| [src/chunk.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1102/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2NodW5rLnJz) | `90.47% <0.00%> (+7.14%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1102?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1102?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao). Last update [88f05bb...16d2278](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1102?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao).


Patch:
diff --git a/.github/workflows/integration-parquet.yml b/.github/workflows/integration-parquet.yml
index 9e253a7491..53a76d5b52 100644
--- a/.github/workflows/integration-parquet.yml
+++ b/.github/workflows/integration-parquet.yml
@@ -46,7 +46,7 @@ jobs:
           python -m venv venv
           source venv/bin/activate
           pip install --upgrade pip
-          pip install pyarrow==6 pyspark==3
+          pip install pyarrow==8 pyspark==3
           python main.py
           # test against spark
           python main_spark.py
diff --git a/src/io/parquet/write/pages.rs b/src/io/parquet/write/pages.rs
index 4c0c0c2342..28f45a06fd 100644
--- a/src/io/parquet/write/pages.rs
+++ b/src/io/parquet/write/pages.rs
@@ -172,7 +172,7 @@ fn to_leafs_recursive<'a>(array: &'a dyn Array, leafs: &mut Vec<&'a dyn Array>)
         }
         Null | Boolean | Primitive(_) | Binary | FixedSizeBinary | LargeBinary | Utf8
         | LargeUtf8 | Dictionary(_) => leafs.push(array),
-        _ => todo!(),
+        other => todo!("Writing {:?} to parquet not yet implemented", other),
     }
 }
 


Test patch:
diff --git a/arrow-parquet-integration-testing/main.py b/arrow-parquet-integration-testing/main.py
index 545a6e08b5..42e19e81b8 100644
--- a/arrow-parquet-integration-testing/main.py
+++ b/arrow-parquet-integration-testing/main.py
@@ -49,11 +49,8 @@ non_native_types = [
     pyarrow.date64(),
     pyarrow.time32("s"),
     pyarrow.timestamp("s"),
+    # the issue here is the second, not the tz
     pyarrow.timestamp("s", tz="UTC"),
-    pyarrow.duration("s"),
-    pyarrow.duration("ms"),
-    pyarrow.duration("us"),
-    pyarrow.duration("ns"),
 ]
 
 
diff --git a/arrow-parquet-integration-testing/src/main.rs b/arrow-parquet-integration-testing/src/main.rs
index c7e9a03431..f62fba5e09 100644
--- a/arrow-parquet-integration-testing/src/main.rs
+++ b/arrow-parquet-integration-testing/src/main.rs
@@ -11,8 +11,8 @@ use arrow2::{
         json_integration::read,
         json_integration::ArrowJson,
         parquet::write::{
-            CompressionOptions as ParquetCompression, Encoding, FileWriter, RowGroupIterator,
-            Version as ParquetVersion, WriteOptions,
+            transverse, CompressionOptions as ParquetCompression, Encoding, FileWriter,
+            RowGroupIterator, Version as ParquetVersion, WriteOptions,
         },
     },
 };
@@ -174,16 +174,18 @@ fn main() -> Result<()> {
     let encodings = schema
         .fields
         .iter()
-        .map(|x| match x.data_type() {
-            DataType::Dictionary(..) => vec![Encoding::RleDictionary],
-            DataType::Utf8 | DataType::LargeUtf8 => {
-                vec![if args.encoding_utf8 == EncodingScheme::Delta {
-                    Encoding::DeltaLengthByteArray
-                } else {
-                    Encoding::Plain
-                }]
-            }
-            _ => vec![Encoding::Plain],
+        .map(|f| {
+            transverse(&f.data_type, |dt| match dt {
+                DataType::Dictionary(..) => Encoding::RleDictionary,
+                DataType::Utf8 | DataType::LargeUtf8 => {
+                    if args.encoding_utf8 == EncodingScheme::Delta {
+                        Encoding::DeltaLengthByteArray
+                    } else {
+                        Encoding::Plain
+                    }
+                }
+                _ => Encoding::Plain,
+            })
         })
         .collect();