Repository: jorgecarleitao/arrow2
Pull request #204 (2021-07-12 05:28:47)

Feature gate the compute module
There are many use-cases that do not require the compute module (e.g. converting CSV to parquet). Yet, the compute module is an expensive unit of compilation and binary size. 

This PR adds a new feature "compute" that disables the `compute` mod from the crate. This significantly improves compile times when that feature is not needed.

```
cargo clean && cargo build --no-default-features --release
Finished release [optimized] target(s) in 36.50s

6.1M libarrow2.rlib

cargo clean && cargo build --no-default-features --features compute --release
Finished release [optimized] target(s) in 1m 15s

14M libarrow2.rlib
```


codecov[bot]: # [Codecov](https://codecov.io/gh/jorgecarleitao/arrow2/pull/204?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao) Report
> Merging [#204](https://codecov.io/gh/jorgecarleitao/arrow2/pull/204?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao) (2c0c2f4) into [main](https://codecov.io/gh/jorgecarleitao/arrow2/commit/bca2973fe3f49fb2a731786d1f238910ed2f66c2?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao) (bca2973) will **decrease** coverage by `0.01%`.
> The diff coverage is `75.60%`.

> :exclamation: Current head 2c0c2f4 differs from pull request most recent head b0e4ffb. Consider uploading reports for the commit b0e4ffb to get more accurate results
[![Impacted file tree graph](https://codecov.io/gh/jorgecarleitao/arrow2/pull/204/graphs/tree.svg?width=650&height=150&src=pr&token=AgyTF60R3D&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao)](https://codecov.io/gh/jorgecarleitao/arrow2/pull/204?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao)

```diff
@@            Coverage Diff             @@
##             main     #204      +/-   ##
==========================================
- Coverage   76.61%   76.59%   -0.02%     
==========================================
  Files         221      221              
  Lines       18972    18981       +9     
==========================================
+ Hits        14535    14539       +4     
- Misses       4437     4442       +5     
```


| [Impacted Files](https://codecov.io/gh/jorgecarleitao/arrow2/pull/204?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao) | Coverage Δ | |
|---|---|---|
| [src/io/parquet/read/mod.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/204/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2lvL3BhcnF1ZXQvcmVhZC9tb2QucnM=) | `88.65% <60.00%> (-2.32%)` | :arrow_down: |
| [src/io/parquet/mod.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/204/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2lvL3BhcnF1ZXQvbW9kLnJz) | `94.21% <100.00%> (+0.46%)` | :arrow_up: |
| [src/io/parquet/read/record\_batch.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/204/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2lvL3BhcnF1ZXQvcmVhZC9yZWNvcmRfYmF0Y2gucnM=) | `88.40% <100.00%> (-0.33%)` | :arrow_down: |
| [src/io/parquet/read/schema/convert.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/204/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2lvL3BhcnF1ZXQvcmVhZC9zY2hlbWEvY29udmVydC5ycw==) | `90.81% <0.00%> (-0.52%)` | :arrow_down: |
| [src/array/mod.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/204/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2FycmF5L21vZC5ycw==) | `50.90% <0.00%> (-0.46%)` | :arrow_down: |
| [src/io/parquet/write/mod.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/204/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2lvL3BhcnF1ZXQvd3JpdGUvbW9kLnJz) | `65.08% <0.00%> (+0.59%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/jorgecarleitao/arrow2/pull/204?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/jorgecarleitao/arrow2/pull/204?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao). Last update [bca2973...b0e4ffb](https://codecov.io/gh/jorgecarleitao/arrow2/pull/204?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao).


Patch:
diff --git a/Cargo.toml b/Cargo.toml
index 742b70d767..3ecba6a8d5 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -72,7 +72,7 @@ doc-comment = "0.3"
 crossbeam-channel = "0.5.1"
 
 [features]
-default = ["io_csv", "io_json", "io_ipc", "io_ipc_compression", "io_json_integration", "io_print", "io_parquet", "regex", "merge_sort", "ahash", "benchmarks"]
+default = ["io_csv", "io_json", "io_ipc", "io_ipc_compression", "io_json_integration", "io_print", "io_parquet", "regex", "merge_sort", "ahash", "benchmarks", "compute"]
 merge_sort = ["itertools"]
 io_csv = ["csv", "lazy_static", "regex"]
 io_json = ["serde", "serde_derive", "serde_json", "indexmap"]
@@ -80,6 +80,8 @@ io_ipc = ["flatbuffers"]
 io_ipc_compression = ["lz4", "zstd"]
 io_json_integration = ["io_json", "hex"]
 io_print = ["prettytable-rs"]
+# the compute kernels. Disabling this significantly reduces compile time.
+compute = []
 # base64 + io_ipc because arrow schemas are stored as base64-encoded ipc format.
 io_parquet = ["parquet2", "io_ipc", "base64", "futures"]
 benchmarks = ["rand"]
diff --git a/src/io/ipc/compression.rs b/src/io/ipc/compression.rs
index fc3e6ab506..2b25fdd41f 100644
--- a/src/io/ipc/compression.rs
+++ b/src/io/ipc/compression.rs
@@ -1,27 +1,27 @@
-use std::io::Read;
-
 use crate::error::Result;
 
 #[cfg(feature = "io_ipc_compression")]
 pub fn decompress_lz4(input_buf: &[u8], output_buf: &mut [u8]) -> Result<()> {
+    use std::io::Read;
     let mut decoder = lz4::Decoder::new(input_buf)?;
     decoder.read_exact(output_buf).map_err(|e| e.into())
 }
 
 #[cfg(feature = "io_ipc_compression")]
 pub fn decompress_zstd(input_buf: &[u8], output_buf: &mut [u8]) -> Result<()> {
+    use std::io::Read;
     let mut decoder = zstd::Decoder::new(input_buf)?;
     decoder.read_exact(output_buf).map_err(|e| e.into())
 }
 
 #[cfg(not(feature = "io_ipc_compression"))]
-pub fn decompress_lz4(input_buf: &[u8], output_buf: &mut [u8]) -> Result<()> {
+pub fn decompress_lz4(_input_buf: &[u8], _output_buf: &mut [u8]) -> Result<()> {
     use crate::error::ArrowError;
     Err(ArrowError::Ipc("The crate was compiled without IPC compression. Use `io_ipc_compression` to read compressed IPC.".to_string()))
 }
 
 #[cfg(not(feature = "io_ipc_compression"))]
-pub fn decompress_zstd(input_buf: &[u8], output_buf: &mut [u8]) -> Result<()> {
+pub fn decompress_zstd(_input_buf: &[u8], _output_buf: &mut [u8]) -> Result<()> {
     use crate::error::ArrowError;
     Err(ArrowError::Ipc("The crate was compiled without IPC compression. Use `io_ipc_compression` to read compressed IPC.".to_string()))
 }
diff --git a/src/lib.rs b/src/lib.rs
index 3b81d66a5b..060b627888 100644
--- a/src/lib.rs
+++ b/src/lib.rs
@@ -7,6 +7,7 @@ pub mod error;
 pub mod trusted_len;
 pub mod types;
 
+#[cfg(feature = "compute")]
 pub mod compute;
 pub mod io;
 pub mod record_batch;


Test patch:
diff --git a/arrow-pyarrow-integration-testing/Cargo.toml b/arrow-pyarrow-integration-testing/Cargo.toml
index c81f38a84d..97a64a9c50 100644
--- a/arrow-pyarrow-integration-testing/Cargo.toml
+++ b/arrow-pyarrow-integration-testing/Cargo.toml
@@ -31,7 +31,7 @@ name = "arrow_pyarrow_integration_testing"
 crate-type = ["cdylib"]
 
 [dependencies]
-arrow2 = { path = "../", default-features = false, features = [] }
+arrow2 = { path = "../", default-features = false, features = ["compute"] }
 pyo3 = { version = "0.12.1", features = ["extension-module"] }
 
 [package.metadata.maturin]