Repository: jorgecarleitao/arrow2
Pull request #337 (2021-08-24 22:52:18)

Added support for reading binary from CSV
Useful when the CSV has fields with non-utf8 entries that are not to be discarded.

codecov[bot]: # [Codecov](https://codecov.io/gh/jorgecarleitao/arrow2/pull/337?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao) Report
> Merging [#337](https://codecov.io/gh/jorgecarleitao/arrow2/pull/337?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao) (6c3aa4b) into [main](https://codecov.io/gh/jorgecarleitao/arrow2/commit/f79ae3e61879121774f7fdd6c07c2cab17309a6b?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao) (f79ae3e) will **increase** coverage by `0.01%`.
> The diff coverage is `91.66%`.

[![Impacted file tree graph](https://codecov.io/gh/jorgecarleitao/arrow2/pull/337/graphs/tree.svg?width=650&height=150&src=pr&token=AgyTF60R3D&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao)](https://codecov.io/gh/jorgecarleitao/arrow2/pull/337?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao)

```diff
@@            Coverage Diff             @@
##             main     #337      +/-   ##
==========================================
+ Coverage   80.97%   80.99%   +0.01%     
==========================================
  Files         326      326              
  Lines       21167    21179      +12     
==========================================
+ Hits        17141    17154      +13     
+ Misses       4026     4025       -1     
```


| [Impacted Files](https://codecov.io/gh/jorgecarleitao/arrow2/pull/337?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao) | Coverage Δ | |
|---|---|---|
| [src/io/csv/read/deserialize.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/337/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2lvL2Nzdi9yZWFkL2Rlc2VyaWFsaXplLnJz) | `57.42% <80.00%> (+1.17%)` | :arrow_up: |
| [tests/it/io/csv/read.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/337/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-dGVzdHMvaXQvaW8vY3N2L3JlYWQucnM=) | `100.00% <100.00%> (ø)` | |
| [src/array/equal/mod.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/337/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2FycmF5L2VxdWFsL21vZC5ycw==) | `91.26% <0.00%> (+1.58%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/jorgecarleitao/arrow2/pull/337?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/jorgecarleitao/arrow2/pull/337?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao). Last update [f79ae3e...6c3aa4b](https://codecov.io/gh/jorgecarleitao/arrow2/pull/337?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao).


Patch:
diff --git a/src/io/csv/read/deserialize.rs b/src/io/csv/read/deserialize.rs
index 272b042122..57e43f29a6 100644
--- a/src/io/csv/read/deserialize.rs
+++ b/src/io/csv/read/deserialize.rs
@@ -58,6 +58,11 @@ fn deserialize_utf8<O: Offset>(rows: &[ByteRecord], column: usize) -> Arc<dyn Ar
     Arc::new(Utf8Array::<O>::from_trusted_len_iter(iter))
 }
 
+fn deserialize_binary<O: Offset>(rows: &[ByteRecord], column: usize) -> Arc<dyn Array> {
+    let iter = rows.iter().map(|row| row.get(column));
+    Arc::new(BinaryArray::<O>::from_trusted_len_iter(iter))
+}
+
 pub fn deserialize_column(
     rows: &[ByteRecord],
     column: usize,
@@ -151,6 +156,8 @@ pub fn deserialize_column(
         }
         Utf8 => deserialize_utf8::<i32>(rows, column),
         LargeUtf8 => deserialize_utf8::<i64>(rows, column),
+        Binary => deserialize_binary::<i32>(rows, column),
+        LargeBinary => deserialize_binary::<i64>(rows, column),
         other => {
             return Err(ArrowError::NotYetImplemented(format!(
                 "Deserializing type \"{:?}\" is not implemented",


Test patch:
diff --git a/tests/it/io/csv/read.rs b/tests/it/io/csv/read.rs
index bdba648e03..e0c13a28f5 100644
--- a/tests/it/io/csv/read.rs
+++ b/tests/it/io/csv/read.rs
@@ -173,3 +173,15 @@ fn float32() -> Result<()> {
     assert_eq!(expected, result.as_ref());
     Ok(())
 }
+
+#[test]
+fn binary() -> Result<()> {
+    let input = vec!["aa", "bb"];
+    let input = input.join("\n");
+
+    let expected = BinaryArray::<i32>::from([Some(b"aa"), Some(b"bb")]);
+
+    let result = test_deserialize(&input, DataType::Binary)?;
+    assert_eq!(expected, result.as_ref());
+    Ok(())
+}