Repository: jorgecarleitao/arrow2
Pull request #1103 (2022-06-25 08:23:10)

Added more tests

codecov[bot]: # [Codecov](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1103?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao) Report
> Merging [#1103](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1103?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao) (13edf5a) into [main](https://codecov.io/gh/jorgecarleitao/arrow2/commit/88f05bb1ea37c05688c19d062e10dd35e00c0987?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao) (88f05bb) will **increase** coverage by `0.09%`.
> The diff coverage is `n/a`.

```diff
@@            Coverage Diff             @@
##             main    #1103      +/-   ##
==========================================
+ Coverage   81.24%   81.33%   +0.09%     
==========================================
  Files         366      366              
  Lines       35336    35336              
==========================================
+ Hits        28707    28739      +32     
+ Misses       6629     6597      -32     
```


| [Impacted Files](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1103?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao) | Coverage Δ | |
|---|---|---|
| [src/io/ipc/read/reader.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1103/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2lvL2lwYy9yZWFkL3JlYWRlci5ycw==) | `95.41% <0.00%> (-0.77%)` | :arrow_down: |
| [src/chunk.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1103/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2NodW5rLnJz) | `90.47% <0.00%> (+7.14%)` | :arrow_up: |
| [src/array/utf8/mod.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1103/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2FycmF5L3V0ZjgvbW9kLnJz) | `91.69% <0.00%> (+9.90%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1103?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1103?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao). Last update [88f05bb...13edf5a](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1103?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao).


Patch:
diff --git a/.github/workflows/integration-parquet.yml b/.github/workflows/integration-parquet.yml
index 9e253a7491..53a76d5b52 100644
--- a/.github/workflows/integration-parquet.yml
+++ b/.github/workflows/integration-parquet.yml
@@ -46,7 +46,7 @@ jobs:
           python -m venv venv
           source venv/bin/activate
           pip install --upgrade pip
-          pip install pyarrow==6 pyspark==3
+          pip install pyarrow==8 pyspark==3
           python main.py
           # test against spark
           python main_spark.py
diff --git a/src/io/parquet/write/pages.rs b/src/io/parquet/write/pages.rs
index 4c0c0c2342..28f45a06fd 100644
--- a/src/io/parquet/write/pages.rs
+++ b/src/io/parquet/write/pages.rs
@@ -172,7 +172,7 @@ fn to_leafs_recursive<'a>(array: &'a dyn Array, leafs: &mut Vec<&'a dyn Array>)
         }
         Null | Boolean | Primitive(_) | Binary | FixedSizeBinary | LargeBinary | Utf8
         | LargeUtf8 | Dictionary(_) => leafs.push(array),
-        _ => todo!(),
+        other => todo!("Writing {:?} to parquet not yet implemented", other),
     }
 }
 


Test patch:
diff --git a/arrow-parquet-integration-testing/main.py b/arrow-parquet-integration-testing/main.py
index 545a6e08b5..42e19e81b8 100644
--- a/arrow-parquet-integration-testing/main.py
+++ b/arrow-parquet-integration-testing/main.py
@@ -49,11 +49,8 @@ non_native_types = [
     pyarrow.date64(),
     pyarrow.time32("s"),
     pyarrow.timestamp("s"),
+    # the issue here is the second, not the tz
     pyarrow.timestamp("s", tz="UTC"),
-    pyarrow.duration("s"),
-    pyarrow.duration("ms"),
-    pyarrow.duration("us"),
-    pyarrow.duration("ns"),
 ]
 
 
diff --git a/arrow-parquet-integration-testing/src/main.rs b/arrow-parquet-integration-testing/src/main.rs
index c7e9a03431..f62fba5e09 100644
--- a/arrow-parquet-integration-testing/src/main.rs
+++ b/arrow-parquet-integration-testing/src/main.rs
@@ -11,8 +11,8 @@ use arrow2::{
         json_integration::read,
         json_integration::ArrowJson,
         parquet::write::{
-            CompressionOptions as ParquetCompression, Encoding, FileWriter, RowGroupIterator,
-            Version as ParquetVersion, WriteOptions,
+            transverse, CompressionOptions as ParquetCompression, Encoding, FileWriter,
+            RowGroupIterator, Version as ParquetVersion, WriteOptions,
         },
     },
 };
@@ -174,16 +174,18 @@ fn main() -> Result<()> {
     let encodings = schema
         .fields
         .iter()
-        .map(|x| match x.data_type() {
-            DataType::Dictionary(..) => vec![Encoding::RleDictionary],
-            DataType::Utf8 | DataType::LargeUtf8 => {
-                vec![if args.encoding_utf8 == EncodingScheme::Delta {
-                    Encoding::DeltaLengthByteArray
-                } else {
-                    Encoding::Plain
-                }]
-            }
-            _ => vec![Encoding::Plain],
+        .map(|f| {
+            transverse(&f.data_type, |dt| match dt {
+                DataType::Dictionary(..) => Encoding::RleDictionary,
+                DataType::Utf8 | DataType::LargeUtf8 => {
+                    if args.encoding_utf8 == EncodingScheme::Delta {
+                        Encoding::DeltaLengthByteArray
+                    } else {
+                        Encoding::Plain
+                    }
+                }
+                _ => Encoding::Plain,
+            })
         })
         .collect();
 
diff --git a/tests/it/array/utf8/to_mutable.rs b/tests/it/array/utf8/to_mutable.rs
index 12f52151c6..e9956feffd 100644
--- a/tests/it/array/utf8/to_mutable.rs
+++ b/tests/it/array/utf8/to_mutable.rs
@@ -1,14 +1,67 @@
-use arrow2::array::Utf8Array;
+use arrow2::{array::Utf8Array, bitmap::Bitmap, buffer::Buffer, datatypes::DataType};
 
 #[test]
-#[allow(clippy::redundant_clone)]
-fn array_to_mutable() {
+fn not_shared() {
     let array = Utf8Array::<i32>::from(&[Some("hello"), Some(" "), None]);
-    let mutable = array.into_mut().unwrap_right();
+    assert!(array.into_mut().is_right());
+}
+
+#[test]
+#[allow(clippy::redundant_clone)]
+fn shared_validity() {
+    let validity = Bitmap::from([true]);
+    let array = Utf8Array::<i32>::new(
+        DataType::Utf8,
+        vec![0, 1].into(),
+        b"a".to_vec().into(),
+        Some(validity.clone()),
+    );
+    assert!(array.into_mut().is_left())
+}
 
-    let array: Utf8Array<i32> = mutable.into();
-    let array2 = array.clone();
-    let maybe_mut = array2.into_mut();
-    // the ref count is 2 we should not get a mutable.
-    assert!(maybe_mut.is_left())
+#[test]
+#[allow(clippy::redundant_clone)]
+fn shared_values() {
+    let values: Buffer<u8> = b"a".to_vec().into();
+    let array = Utf8Array::<i32>::new(
+        DataType::Utf8,
+        vec![0, 1].into(),
+        values.clone(),
+        Some(Bitmap::from([true])),
+    );
+    assert!(array.into_mut().is_left())
+}
+
+#[test]
+#[allow(clippy::redundant_clone)]
+fn shared_offsets_values() {
+    let offsets: Buffer<i32> = vec![0, 1].into();
+    let values: Buffer<u8> = b"a".to_vec().into();
+    let array = Utf8Array::<i32>::new(
+        DataType::Utf8,
+        offsets.clone(),
+        values.clone(),
+        Some(Bitmap::from([true])),
+    );
+    assert!(array.into_mut().is_left())
+}
+
+#[test]
+#[allow(clippy::redundant_clone)]
+fn shared_offsets() {
+    let offsets: Buffer<i32> = vec![0, 1].into();
+    let array = Utf8Array::<i32>::new(
+        DataType::Utf8,
+        offsets.clone(),
+        b"a".to_vec().into(),
+        Some(Bitmap::from([true])),
+    );
+    assert!(array.into_mut().is_left())
+}
+
+#[test]
+#[allow(clippy::redundant_clone)]
+fn shared_all() {
+    let array = Utf8Array::<i32>::from(&[Some("hello"), Some(" "), None]);
+    assert!(array.clone().into_mut().is_left())
 }