Repository: jorgecarleitao/arrow2
Pull request #1094 (2022-06-23 03:40:30)

Exposed utilities in `io::flight`
Exposed some utilities from `io::ipc`  used in `io::flight` at `io::flight`.

codecov[bot]: # [Codecov](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1094?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao) Report
> Merging [#1094](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1094?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao) (1dad93f) into [main](https://codecov.io/gh/jorgecarleitao/arrow2/commit/2f1a223f09c792f14109a101a4e635673cff64ff?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao) (2f1a223) will **increase** coverage by `0.19%`.
> The diff coverage is `n/a`.

```diff
@@            Coverage Diff             @@
##             main    #1094      +/-   ##
==========================================
+ Coverage   81.02%   81.21%   +0.19%     
==========================================
  Files         367      367              
  Lines       35350    35350              
==========================================
+ Hits        28641    28711      +70     
+ Misses       6709     6639      -70     
```


| [Impacted Files](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1094?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao) | Coverage Δ | |
|---|---|---|
| [src/io/flight/mod.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1094/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2lvL2ZsaWdodC9tb2QucnM=) | `66.66% <ø> (+66.66%)` | :arrow_up: |
| [src/bitmap/utils/slice\_iterator.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1094/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2JpdG1hcC91dGlscy9zbGljZV9pdGVyYXRvci5ycw==) | `97.56% <0.00%> (-1.22%)` | :arrow_down: |
| [src/io/ipc/read/reader.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1094/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2lvL2lwYy9yZWFkL3JlYWRlci5ycw==) | `96.56% <0.00%> (+0.76%)` | :arrow_up: |
| [src/array/binary/mod.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1094/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2FycmF5L2JpbmFyeS9tb2QucnM=) | `90.04% <0.00%> (+1.24%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1094?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1094?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao). Last update [2f1a223...1dad93f](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1094?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao).


Patch:
diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md
index cc9211ef99..d81e645583 100644
--- a/DEVELOPMENT.md
+++ b/DEVELOPMENT.md
@@ -4,6 +4,13 @@ This crate follows the standard for developing a Rust library via `cargo`.
 The CI is our "ground truth" over the state of the library. Check out the different parts of
 the CI to understand how to test the different parts of this library locally.
 
+## Git clone with submodules
+The crate comes with additional submodules to aid with testing, to ensure you have them if you plan on testing, using `--recurse-submodules` will clone the submodules alongside the repository.
+
+```bash
+git clone --recurse-submodules https://github.com/jorgecarleitao/arrow2
+```
+
 ## Testing
 
 The simplest way to test the crate is to run 
@@ -18,6 +25,41 @@ This runs the tests of the crate without features. To run all features, use
 cargo test --features full
 ```
 
+If you get warnings about parquet files not existing, you can generate the test files by using Python:
+
+```bash
+# Ubuntu: sudo apt install python3-pip python3-venv
+# Mac: brew install python3
+# Archlinux: sudo pacman -Syu python
+
+# Create a virtual environment for python, to keep dependencies contained
+python3 -m venv venv
+
+# Activate the virtual environment
+source venv/bin/activate
+
+# Make sure pip is up to date
+pip install pip --upgrade
+
+# Install pyarrow, version 6
+pip install pyarrow==6
+
+# Generate the parquet files (this might take some time, depending on your computer setup)
+python parquet_integration/write_parquet.py
+
+# Get out of venv, back to normal terminal
+deactivate
+```
+
+If you receive warnings about other files not found (IPC), ensure you have all submodules:
+```bash
+# If you didn't clone with `git clone --recurse-submodules https://github.com/jorgecarleitao/arrow2`
+git submodule update --init --recursive
+
+# Update to the latest submodules
+git submodule update --recursive --remote
+```
+
 during development of particular parts of the crate, it is usually faster
 to reduce the feature set - the tests are gated to only the relevant tests
 of that feature set. For example, if improving JSON, you can use
diff --git a/src/io/flight/mod.rs b/src/io/flight/mod.rs
index b439ced2d9..8aa4a1de0e 100644
--- a/src/io/flight/mod.rs
+++ b/src/io/flight/mod.rs
@@ -11,12 +11,14 @@ use crate::{
     error::{Error, Result},
     io::ipc::read,
     io::ipc::write,
-    io::ipc::write::common::{encode_chunk, DictionaryTracker, EncodedData, WriteOptions},
+    io::ipc::write::common::{encode_chunk, DictionaryTracker, EncodedData},
 };
 
-pub use super::ipc::write::default_ipc_fields;
 use super::ipc::{IpcField, IpcSchema};
 
+pub use super::ipc::write::default_ipc_fields;
+pub use crate::io::ipc::write::common::WriteOptions;
+
 /// Serializes [`Chunk`] to a vector of [`FlightData`] representing the serialized dictionaries
 /// and a [`FlightData`] representing the batch.
 /// # Errors


Test patch:
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
index 58abc32ef9..40015100d0 100644
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -88,7 +88,7 @@ jobs:
 
       - name: Run
         # --skip io: miri can't handle opening of files, so we skip those
-        run: cargo miri test --features full -- --skip io::parquet --skip io::ipc
+        run: cargo miri test --features full -- --skip io::parquet --skip io::ipc --skip io::flight
 
   miri-checks-io:
     name: MIRI on IO IPC
diff --git a/tests/it/io/flight/mod.rs b/tests/it/io/flight/mod.rs
new file mode 100644
index 0000000000..9718b74902
--- /dev/null
+++ b/tests/it/io/flight/mod.rs
@@ -0,0 +1,32 @@
+use arrow2::array::Array;
+use arrow2::chunk::Chunk;
+use arrow2::datatypes::Schema;
+use arrow2::error::Error;
+
+use arrow2::io::flight::*;
+use arrow2::io::ipc::write::{default_ipc_fields, WriteOptions};
+
+use super::ipc::read_gzip_json;
+
+fn round_trip(schema: Schema, chunk: Chunk<Box<dyn Array>>) -> Result<(), Error> {
+    let fields = default_ipc_fields(&schema.fields);
+    let serialized = serialize_schema(&schema, Some(&fields));
+    let (result, ipc_schema) = deserialize_schemas(&serialized.data_header)?;
+    assert_eq!(schema, result);
+
+    let (_, batch) = serialize_batch(&chunk, &fields, &WriteOptions { compression: None })?;
+
+    let result = deserialize_batch(&batch, &result.fields, &ipc_schema, &Default::default())?;
+    assert_eq!(result, chunk);
+    Ok(())
+}
+
+#[test]
+fn generated_nested_dictionary() -> Result<(), Error> {
+    let (schema, _, mut batches) =
+        read_gzip_json("1.0.0-littleendian", "generated_nested").unwrap();
+
+    round_trip(schema, batches.pop().unwrap())?;
+
+    Ok(())
+}
diff --git a/tests/it/io/mod.rs b/tests/it/io/mod.rs
index d192b806ee..3fc0c2123e 100644
--- a/tests/it/io/mod.rs
+++ b/tests/it/io/mod.rs
@@ -22,3 +22,6 @@ mod avro;
     feature = "io_csv_read_async"
 ))]
 mod csv;
+
+#[cfg(feature = "io_flight")]
+mod flight;