Repository: jorgecarleitao/arrow2
Pull request #1220 (2022-08-11 04:44:06)

Added utf8 <> binary cast

codecov[bot]: # [Codecov](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1220?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao) Report
> Merging [#1220](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1220?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao) (563d0bb) into [main](https://codecov.io/gh/jorgecarleitao/arrow2/commit/2a12d172a7a852e78ee25de09ebf68f7356a116c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao) (2a12d17) will **increase** coverage by `0.01%`.
> The diff coverage is `93.15%`.

```diff
@@            Coverage Diff             @@
##             main    #1220      +/-   ##
==========================================
+ Coverage   83.21%   83.22%   +0.01%     
==========================================
  Files         358      358              
  Lines       37224    37260      +36     
==========================================
+ Hits        30976    31010      +34     
- Misses       6248     6250       +2     
```


| [Impacted Files](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1220?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao) | Coverage Δ | |
|---|---|---|
| [src/compute/cast/mod.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1220/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2NvbXB1dGUvY2FzdC9tb2QucnM=) | `94.05% <86.84%> (-0.61%)` | :arrow_down: |
| [src/compute/cast/binary\_to.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1220/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2NvbXB1dGUvY2FzdC9iaW5hcnlfdG8ucnM=) | `87.00% <100.00%> (+4.33%)` | :arrow_up: |
| [src/compute/cast/utf8\_to.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1220/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2NvbXB1dGUvY2FzdC91dGY4X3RvLnJz) | `100.00% <100.00%> (ø)` | |
| [src/io/ipc/read/reader.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1220/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2lvL2lwYy9yZWFkL3JlYWRlci5ycw==) | `94.66% <0.00%> (-1.34%)` | :arrow_down: |
| [src/io/ipc/read/file.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1220/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2lvL2lwYy9yZWFkL2ZpbGUucnM=) | `96.87% <0.00%> (-0.45%)` | :arrow_down: |
| [src/io/ipc/read/common.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1220/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2lvL2lwYy9yZWFkL2NvbW1vbi5ycw==) | `94.90% <0.00%> (-0.40%)` | :arrow_down: |
| [src/io/ipc/read/schema.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1220/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2lvL2lwYy9yZWFkL3NjaGVtYS5ycw==) | `95.17% <0.00%> (-0.33%)` | :arrow_down: |
| [src/array/utf8/mod.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1220/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2FycmF5L3V0ZjgvbW9kLnJz) | `83.64% <0.00%> (+0.92%)` | :arrow_up: |
| [src/array/binary/mod.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1220/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2FycmF5L2JpbmFyeS9tb2QucnM=) | `90.12% <0.00%> (+1.23%)` | :arrow_up: |

Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao)


Patch:
diff --git a/src/compute/cast/binary_to.rs b/src/compute/cast/binary_to.rs
index aee7c8fcab..44224321b5 100644
--- a/src/compute/cast/binary_to.rs
+++ b/src/compute/cast/binary_to.rs
@@ -36,6 +36,37 @@ pub fn binary_large_to_binary(
     ))
 }
 
+/// Conversion to utf8
+pub fn binary_to_utf8<O: Offset>(
+    from: &BinaryArray<O>,
+    to_data_type: DataType,
+) -> Result<Utf8Array<O>> {
+    Utf8Array::<O>::try_new(
+        to_data_type,
+        from.offsets().clone(),
+        from.values().clone(),
+        from.validity().cloned(),
+    )
+}
+
+/// Conversion to utf8
+/// # Errors
+/// This function errors if the values are not valid utf8
+pub fn binary_to_large_utf8(
+    from: &BinaryArray<i32>,
+    to_data_type: DataType,
+) -> Result<Utf8Array<i64>> {
+    let values = from.values().clone();
+    let offsets = from
+        .offsets()
+        .iter()
+        .map(|x| *x as i64)
+        .collect::<Vec<_>>()
+        .into();
+
+    Utf8Array::<i64>::try_new(to_data_type, offsets, values, from.validity().cloned())
+}
+
 /// Casts a [`BinaryArray`] to a [`PrimitiveArray`] at best-effort using `lexical_core::parse_partial`, making any uncastable value as zero.
 pub fn partial_binary_to_primitive<O: Offset, T>(
     from: &BinaryArray<O>,
diff --git a/src/compute/cast/mod.rs b/src/compute/cast/mod.rs
index d3c37cd73f..d58210fb9c 100644
--- a/src/compute/cast/mod.rs
+++ b/src/compute/cast/mod.rs
@@ -106,23 +106,31 @@ pub fn can_cast_types(from_type: &DataType, to_type: &DataType) -> bool {
                 || to_type == &LargeBinary
         }
 
-        (Utf8, Date32) => true,
-        (Utf8, Date64) => true,
-        (Utf8, Timestamp(TimeUnit::Nanosecond, _)) => true,
-        (Utf8, LargeUtf8) => true,
-        (Utf8, _) => is_numeric(to_type),
-        (LargeUtf8, Date32) => true,
-        (LargeUtf8, Date64) => true,
-        (LargeUtf8, Timestamp(TimeUnit::Nanosecond, _)) => true,
-        (LargeUtf8, Utf8) => true,
-        (LargeUtf8, _) => is_numeric(to_type),
+        (Utf8, to_type) => {
+            is_numeric(to_type)
+                || matches!(
+                    to_type,
+                    LargeUtf8 | Binary | Date32 | Date64 | Timestamp(TimeUnit::Nanosecond, _)
+                )
+        }
+        (LargeUtf8, to_type) => {
+            is_numeric(to_type)
+                || matches!(
+                    to_type,
+                    Utf8 | LargeBinary | Date32 | Date64 | Timestamp(TimeUnit::Nanosecond, _)
+                )
+        }
+
+        (Binary, to_type) => {
+            is_numeric(to_type) || matches!(to_type, LargeBinary | Utf8 | LargeUtf8)
+        }
+        (LargeBinary, to_type) => is_numeric(to_type) || matches!(to_type, Binary | LargeUtf8),
+
         (Timestamp(_, _), Utf8) => true,
         (Timestamp(_, _), LargeUtf8) => true,
         (_, Utf8) => is_numeric(from_type) || from_type == &Binary,
-        (_, LargeUtf8) => is_numeric(from_type) || from_type == &Binary,
+        (_, LargeUtf8) => is_numeric(from_type) || from_type == &LargeBinary,
 
-        (Binary, _) => is_numeric(to_type) || to_type == &LargeBinary,
-        (LargeBinary, _) => is_numeric(to_type) || to_type == &Binary,
         (_, Binary) => is_numeric(from_type),
         (_, LargeBinary) => is_numeric(from_type),
 
@@ -380,22 +388,18 @@ pub fn cast(array: &dyn Array, to_type: &DataType, options: CastOptions) -> Resu
         )),
         (List(_), List(_)) => {
             cast_list::<i32>(array.as_any().downcast_ref().unwrap(), to_type, options)
-                .map(|x| Box::new(x) as Box<dyn Array>)
+                .map(|x| x.boxed())
         }
         (LargeList(_), LargeList(_)) => {
             cast_list::<i64>(array.as_any().downcast_ref().unwrap(), to_type, options)
-                .map(|x| Box::new(x) as Box<dyn Array>)
+                .map(|x| x.boxed())
+        }
+        (List(lhs), LargeList(rhs)) if lhs == rhs => {
+            Ok(cast_list_to_large_list(array.as_any().downcast_ref().unwrap(), to_type).boxed())
+        }
+        (LargeList(lhs), List(rhs)) if lhs == rhs => {
+            Ok(cast_large_to_list(array.as_any().downcast_ref().unwrap(), to_type).boxed())
         }
-        (List(lhs), LargeList(rhs)) if lhs == rhs => Ok(cast_list_to_large_list(
-            array.as_any().downcast_ref().unwrap(),
-            to_type,
-        ))
-        .map(|x| Box::new(x) as Box<dyn Array>),
-        (LargeList(lhs), List(rhs)) if lhs == rhs => Ok(cast_large_to_list(
-            array.as_any().downcast_ref().unwrap(),
-            to_type,
-        ))
-        .map(|x| Box::new(x) as Box<dyn Array>),
 
         (_, List(to)) => {
             // cast primitive to list's primitive
@@ -467,6 +471,11 @@ pub fn cast(array: &dyn Array, to_type: &DataType, options: CastOptions) -> Resu
             LargeUtf8 => Ok(Box::new(utf8_to_large_utf8(
                 array.as_any().downcast_ref().unwrap(),
             ))),
+            Binary => Ok(utf8_to_binary::<i32>(
+                array.as_any().downcast_ref().unwrap(),
+                to_type.clone(),
+            )
+            .boxed()),
             Timestamp(TimeUnit::Nanosecond, None) => utf8_to_naive_timestamp_ns_dyn::<i32>(array),
             Timestamp(TimeUnit::Nanosecond, Some(tz)) => {
                 utf8_to_timestamp_ns_dyn::<i32>(array, tz.clone())
@@ -489,8 +498,12 @@ pub fn cast(array: &dyn Array, to_type: &DataType, options: CastOptions) -> Resu
             Float64 => utf8_to_primitive_dyn::<i64, f64>(array, to_type, options),
             Date32 => utf8_to_date32_dyn::<i64>(array),
             Date64 => utf8_to_date64_dyn::<i64>(array),
-            Utf8 => utf8_large_to_utf8(array.as_any().downcast_ref().unwrap())
-                .map(|x| Box::new(x) as Box<dyn Array>),
+            Utf8 => utf8_large_to_utf8(array.as_any().downcast_ref().unwrap()).map(|x| x.boxed()),
+            LargeBinary => Ok(utf8_to_binary::<i64>(
+                array.as_any().downcast_ref().unwrap(),
+                to_type.clone(),
+            )
+            .boxed()),
             Timestamp(TimeUnit::Nanosecond, None) => utf8_to_naive_timestamp_ns_dyn::<i64>(array),
             Timestamp(TimeUnit::Nanosecond, Some(tz)) => {
                 utf8_to_timestamp_ns_dyn::<i64>(array, tz.clone())
@@ -548,16 +561,11 @@ pub fn cast(array: &dyn Array, to_type: &DataType, options: CastOptions) -> Resu
             Int64 => primitive_to_utf8_dyn::<i64, i64>(array),
             Float32 => primitive_to_utf8_dyn::<f32, i64>(array),
             Float64 => primitive_to_utf8_dyn::<f64, i64>(array),
-            Binary => {
-                let array = array.as_any().downcast_ref::<BinaryArray<i32>>().unwrap();
-
-                // perf todo: the offsets are equal; we can speed-up this
-                let iter = array
-                    .iter()
-                    .map(|x| x.and_then(|x| simdutf8::basic::from_utf8(x).ok()));
-
-                let array = Utf8Array::<i64>::from_trusted_len_iter(iter);
-                Ok(Box::new(array))
+            Binary => binary_to_large_utf8(array.as_any().downcast_ref().unwrap(), to_type.clone())
+                .map(|x| x.boxed()),
+            LargeBinary => {
+                binary_to_utf8::<i64>(array.as_any().downcast_ref().unwrap(), to_type.clone())
+                    .map(|x| x.boxed())
             }
             Timestamp(from_unit, Some(tz)) => {
                 let from = array.as_any().downcast_ref().unwrap();
@@ -588,6 +596,9 @@ pub fn cast(array: &dyn Array, to_type: &DataType, options: CastOptions) -> Resu
                 array.as_any().downcast_ref().unwrap(),
                 to_type.clone(),
             ))),
+            Utf8 => binary_to_utf8::<i32>(array.as_any().downcast_ref().unwrap(), to_type.clone())
+                .map(|x| x.boxed()),
+
             _ => Err(Error::NotYetImplemented(format!(
                 "Casting from {:?} to {:?} not supported",
                 from_type, to_type,
@@ -607,7 +618,11 @@ pub fn cast(array: &dyn Array, to_type: &DataType, options: CastOptions) -> Resu
             Float64 => binary_to_primitive_dyn::<i64, f64>(array, to_type, options),
             Binary => {
                 binary_large_to_binary(array.as_any().downcast_ref().unwrap(), to_type.clone())
-                    .map(|x| Box::new(x) as Box<dyn Array>)
+                    .map(|x| x.boxed())
+            }
+            LargeUtf8 => {
+                binary_to_utf8::<i64>(array.as_any().downcast_ref().unwrap(), to_type.clone())
+                    .map(|x| x.boxed())
             }
             _ => Err(Error::NotYetImplemented(format!(
                 "Casting from {:?} to {:?} not supported",
diff --git a/src/compute/cast/utf8_to.rs b/src/compute/cast/utf8_to.rs
index d6a235500e..996889174a 100644
--- a/src/compute/cast/utf8_to.rs
+++ b/src/compute/cast/utf8_to.rs
@@ -175,3 +175,15 @@ pub fn utf8_large_to_utf8(from: &Utf8Array<i64>) -> Result<Utf8Array<i32>> {
     // Safety: sound because `offsets` fulfills the same invariants as `from.offsets()`
     Ok(unsafe { Utf8Array::<i32>::from_data_unchecked(data_type, offsets, values, validity) })
 }
+
+/// Conversion to binary
+pub fn utf8_to_binary<O: Offset>(from: &Utf8Array<O>, to_data_type: DataType) -> BinaryArray<O> {
+    unsafe {
+        BinaryArray::<O>::new_unchecked(
+            to_data_type,
+            from.offsets().clone(),
+            from.values().clone(),
+            from.validity().cloned(),
+        )
+    }
+}


Test patch:
diff --git a/tests/it/compute/cast.rs b/tests/it/compute/cast.rs
index 9e9888926d..a9a7c45e34 100644
--- a/tests/it/compute/cast.rs
+++ b/tests/it/compute/cast.rs
@@ -481,7 +481,10 @@ fn consistency() {
                 if let Ok(result) = result {
                     assert_eq!(result.data_type(), d2, "type not equal: {:?} {:?}", d1, d2);
                 } else {
-                    panic!("Cast should have not failed {:?} {:?}", d1, d2);
+                    panic!(
+                        "Cast should have not have failed {:?} {:?}: {:?}",
+                        d1, d2, result
+                    );
                 }
             } else if cast(array.as_ref(), d2, CastOptions::default()).is_ok() {
                 panic!("Cast should have failed {:?} {:?}", d1, d2);