Repository: jorgecarleitao/arrow2
Pull request #336 (2021-08-24 19:07:44)

Added support for remaining non-nested datatypes
This PR adds support to dictionary-encoding (parquet) to the remaining non-nested datatypes.

This is demonstrated by:

1. a roundtrip test of all columns in the `generated_dictionary` IPC file pair (containing dictionaries with different types and validities)
2. a `rust -> pyspark` test (equality over all values)

Together, they demonstrate that
* spark reads dictionary-encoded arrays written by this crate
* this crate reads dictionary-encoded arrays written by this crate

where "read" is in the sense that data integrity is preserved over all values (nulls or not).

Tests against pyarrow currently fail, likely due to [ARROW-13486](https://issues.apache.org/jira/browse/ARROW-13486) and [ARROW-13487](https://issues.apache.org/jira/browse/ARROW-13487).

codecov[bot]: # [Codecov](https://codecov.io/gh/jorgecarleitao/arrow2/pull/336?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao) Report
> Merging [#336](https://codecov.io/gh/jorgecarleitao/arrow2/pull/336?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao) (b7e2352) into [main](https://codecov.io/gh/jorgecarleitao/arrow2/commit/749d798ffc11a46be55fb1807f3a05b84339d32d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao) (749d798) will **increase** coverage by `0.01%`.
> The diff coverage is `69.23%`.

[![Impacted file tree graph](https://codecov.io/gh/jorgecarleitao/arrow2/pull/336/graphs/tree.svg?width=650&height=150&src=pr&token=AgyTF60R3D&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao)](https://codecov.io/gh/jorgecarleitao/arrow2/pull/336?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao)

```diff
@@            Coverage Diff             @@
##             main     #336      +/-   ##
==========================================
+ Coverage   80.96%   80.97%   +0.01%     
==========================================
  Files         325      326       +1     
  Lines       21081    21167      +86     
==========================================
+ Hits        17069    17141      +72     
- Misses       4012     4026      +14     
```


| [Impacted Files](https://codecov.io/gh/jorgecarleitao/arrow2/pull/336?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao) | Coverage Δ | |
|---|---|---|
| [src/array/dictionary/iterator.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/336/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2FycmF5L2RpY3Rpb25hcnkvaXRlcmF0b3IucnM=) | `50.00% <ø> (ø)` | |
| [src/io/parquet/read/mod.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/336/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2lvL3BhcnF1ZXQvcmVhZC9tb2QucnM=) | `51.21% <30.55%> (-18.79%)` | :arrow_down: |
| [src/io/parquet/read/binary/dictionary.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/336/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2lvL3BhcnF1ZXQvcmVhZC9iaW5hcnkvZGljdGlvbmFyeS5ycw==) | `80.00% <80.00%> (ø)` | |
| [src/array/dictionary/mod.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/336/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2FycmF5L2RpY3Rpb25hcnkvbW9kLnJz) | `72.50% <100.00%> (ø)` | |
| [src/array/equal/dictionary.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/336/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2FycmF5L2VxdWFsL2RpY3Rpb25hcnkucnM=) | `100.00% <100.00%> (ø)` | |
| [src/io/parquet/write/dictionary.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/336/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2lvL3BhcnF1ZXQvd3JpdGUvZGljdGlvbmFyeS5ycw==) | `72.72% <100.00%> (+29.48%)` | :arrow_up: |
| [src/io/parquet/write/mod.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/336/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2lvL3BhcnF1ZXQvd3JpdGUvbW9kLnJz) | `51.28% <100.00%> (ø)` | |
| [src/io/parquet/write/record\_batch.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/336/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2lvL3BhcnF1ZXQvd3JpdGUvcmVjb3JkX2JhdGNoLnJz) | `91.30% <100.00%> (ø)` | |
| [tests/it/array/equal/dictionary.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/336/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-dGVzdHMvaXQvYXJyYXkvZXF1YWwvZGljdGlvbmFyeS5ycw==) | `100.00% <100.00%> (ø)` | |
| [tests/it/io/parquet/mod.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/336/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-dGVzdHMvaXQvaW8vcGFycXVldC9tb2QucnM=) | `94.93% <100.00%> (ø)` | |
| ... and [5 more](https://codecov.io/gh/jorgecarleitao/arrow2/pull/336/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/jorgecarleitao/arrow2/pull/336?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/jorgecarleitao/arrow2/pull/336?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao). Last update [8e96ec4...b7e2352](https://codecov.io/gh/jorgecarleitao/arrow2/pull/336?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao).


Patch:
diff --git a/src/io/parquet/read/binary/dictionary.rs b/src/io/parquet/read/binary/dictionary.rs
new file mode 100644
index 0000000000..d55fc81364
--- /dev/null
+++ b/src/io/parquet/read/binary/dictionary.rs
@@ -0,0 +1,158 @@
+use std::sync::Arc;
+
+use parquet2::{
+    encoding::{bitpacking, hybrid_rle, uleb128, Encoding},
+    metadata::{ColumnChunkMetaData, ColumnDescriptor},
+    page::{BinaryPageDict, DataPage},
+    read::StreamingIterator,
+};
+
+use super::super::utils as other_utils;
+use crate::{
+    array::{Array, DictionaryArray, DictionaryKey, Offset, PrimitiveArray, Utf8Array},
+    bitmap::{utils::BitmapIter, MutableBitmap},
+    buffer::MutableBuffer,
+    error::{ArrowError, Result},
+};
+
+fn read_dict_optional<K, O>(
+    validity_buffer: &[u8],
+    indices_buffer: &[u8],
+    additional: usize,
+    dict: &BinaryPageDict,
+    indices: &mut MutableBuffer<K>,
+    offsets: &mut MutableBuffer<O>,
+    values: &mut MutableBuffer<u8>,
+    validity: &mut MutableBitmap,
+) where
+    K: DictionaryKey,
+    O: Offset,
+{
+    values.extend_from_slice(dict.values());
+    offsets.extend(
+        dict.offsets()
+            .iter()
+            .map(|x| O::from_usize(*x as usize).unwrap()),
+    );
+
+    // SPEC: Data page format: the bit width used to encode the entry ids stored as 1 byte (max bit width = 32),
+    // SPEC: followed by the values encoded using RLE/Bit packed described above (with the given bit width).
+    let bit_width = indices_buffer[0];
+    let indices_buffer = &indices_buffer[1..];
+
+    let (_, consumed) = uleb128::decode(indices_buffer);
+    let indices_buffer = &indices_buffer[consumed..];
+
+    let non_null_indices_len = indices_buffer.len() * 8 / bit_width as usize;
+
+    let mut new_indices = bitpacking::Decoder::new(indices_buffer, bit_width, non_null_indices_len);
+
+    let validity_iterator = hybrid_rle::Decoder::new(validity_buffer, 1);
+
+    for run in validity_iterator {
+        match run {
+            hybrid_rle::HybridEncoded::Bitpacked(packed) => {
+                let remaining = additional - indices.len();
+                let len = std::cmp::min(packed.len() * 8, remaining);
+                for is_valid in BitmapIter::new(packed, 0, len) {
+                    let value = if is_valid {
+                        K::from_u32(new_indices.next().unwrap()).unwrap()
+                    } else {
+                        K::default()
+                    };
+                    indices.push(value);
+                }
+                validity.extend_from_slice(packed, 0, len);
+            }
+            hybrid_rle::HybridEncoded::Rle(value, additional) => {
+                let is_set = value[0] == 1;
+                validity.extend_constant(additional, is_set);
+                if is_set {
+                    (0..additional).for_each(|_| {
+                        let index = K::from_u32(new_indices.next().unwrap()).unwrap();
+                        indices.push(index)
+                    })
+                } else {
+                    indices.extend_constant(additional, *indices.last().unwrap())
+                }
+            }
+        }
+    }
+}
+
+fn extend_from_page<K, O>(
+    page: &DataPage,
+    descriptor: &ColumnDescriptor,
+    indices: &mut MutableBuffer<K>,
+    offsets: &mut MutableBuffer<O>,
+    values: &mut MutableBuffer<u8>,
+    validity: &mut MutableBitmap,
+) -> Result<()>
+where
+    K: DictionaryKey,
+    O: Offset,
+{
+    let additional = page.num_values();
+
+    assert_eq!(descriptor.max_rep_level(), 0);
+    let is_optional = descriptor.max_def_level() == 1;
+
+    let (validity_buffer, values_buffer, version) = other_utils::split_buffer(page, is_optional);
+
+    match (&page.encoding(), page.dictionary_page(), is_optional) {
+        (Encoding::PlainDictionary | Encoding::RleDictionary, Some(dict), true) => {
+            read_dict_optional(
+                validity_buffer,
+                values_buffer,
+                additional,
+                dict.as_any().downcast_ref().unwrap(),
+                indices,
+                offsets,
+                values,
+                validity,
+            )
+        }
+        _ => {
+            return Err(other_utils::not_implemented(
+                &page.encoding(),
+                is_optional,
+                page.dictionary_page().is_some(),
+                version,
+                "primitive",
+            ))
+        }
+    }
+    Ok(())
+}
+
+pub fn iter_to_array<K, O, I, E>(
+    mut iter: I,
+    metadata: &ColumnChunkMetaData,
+) -> Result<Box<dyn Array>>
+where
+    ArrowError: From<E>,
+    O: Offset,
+    K: DictionaryKey,
+    E: Clone,
+    I: StreamingIterator<Item = std::result::Result<DataPage, E>>,
+{
+    let capacity = metadata.num_values() as usize;
+    let mut indices = MutableBuffer::<K>::with_capacity(capacity);
+    let mut values = MutableBuffer::<u8>::with_capacity(0);
+    let mut offsets = MutableBuffer::<O>::with_capacity(1 + capacity);
+    let mut validity = MutableBitmap::with_capacity(capacity);
+    while let Some(page) = iter.next() {
+        extend_from_page(
+            page.as_ref().map_err(|x| x.clone())?,
+            metadata.descriptor(),
+            &mut indices,
+            &mut offsets,
+            &mut values,
+            &mut validity,
+        )?
+    }
+
+    let keys = PrimitiveArray::from_data(K::DATA_TYPE, indices.into(), validity.into());
+    let values = Arc::new(Utf8Array::from_data(offsets.into(), values.into(), None));
+    Ok(Box::new(DictionaryArray::<K>::from_data(keys, values)))
+}
diff --git a/src/io/parquet/read/binary/mod.rs b/src/io/parquet/read/binary/mod.rs
index 912f1bb026..54da7b3f1d 100644
--- a/src/io/parquet/read/binary/mod.rs
+++ b/src/io/parquet/read/binary/mod.rs
@@ -1,6 +1,8 @@
 mod basic;
+mod dictionary;
 mod nested;
 
 pub use basic::iter_to_array;
+pub use dictionary::iter_to_array as iter_to_dict_array;
 pub use basic::stream_to_array;
 pub use nested::iter_to_array as iter_to_array_nested;
diff --git a/src/io/parquet/read/mod.rs b/src/io/parquet/read/mod.rs
index 81797a698f..6146787a44 100644
--- a/src/io/parquet/read/mod.rs
+++ b/src/io/parquet/read/mod.rs
@@ -21,7 +21,7 @@ pub use parquet2::{
 };
 
 use crate::{
-    array::Array,
+    array::{Array, DictionaryKey},
     datatypes::{DataType, IntervalUnit, TimeUnit},
     error::{ArrowError, Result},
 };
@@ -83,6 +83,89 @@ pub async fn read_metadata_async<R: AsyncRead + AsyncSeek + Send + Unpin>(
     Ok(_read_metadata_async(reader).await?)
 }
 
+fn dict_read<
+    K: DictionaryKey,
+    I: StreamingIterator<Item = std::result::Result<DataPage, ParquetError>>,
+>(
+    iter: &mut I,
+    metadata: &ColumnChunkMetaData,
+    data_type: DataType,
+) -> Result<Box<dyn Array>> {
+    use DataType::*;
+    let values_data_type = if let Dictionary(_, v) = &data_type {
+        v.as_ref()
+    } else {
+        panic!()
+    };
+
+    match values_data_type {
+        UInt8 => primitive::iter_to_dict_array::<K, _, _, _, _, _>(
+            iter,
+            metadata,
+            data_type,
+            |x: i32| x as u8,
+        ),
+        UInt16 => primitive::iter_to_dict_array::<K, _, _, _, _, _>(
+            iter,
+            metadata,
+            data_type,
+            |x: i32| x as u16,
+        ),
+        UInt32 => primitive::iter_to_dict_array::<K, _, _, _, _, _>(
+            iter,
+            metadata,
+            data_type,
+            |x: i32| x as u32,
+        ),
+        Int8 => primitive::iter_to_dict_array::<K, _, _, _, _, _>(
+            iter,
+            metadata,
+            data_type,
+            |x: i32| x as i8,
+        ),
+        Int16 => primitive::iter_to_dict_array::<K, _, _, _, _, _>(
+            iter,
+            metadata,
+            data_type,
+            |x: i32| x as i16,
+        ),
+        Int32 | Date32 | Time32(_) | Interval(IntervalUnit::YearMonth) => {
+            primitive::iter_to_dict_array::<K, _, _, _, _, _>(
+                iter,
+                metadata,
+                data_type,
+                |x: i32| x as i32,
+            )
+        }
+        Timestamp(TimeUnit::Nanosecond, None) => match metadata.descriptor().type_() {
+            ParquetType::PrimitiveType { physical_type, .. } => match physical_type {
+                PhysicalType::Int96 => primitive::iter_to_dict_array::<K, _, _, _, _, _>(
+                    iter,
+                    metadata,
+                    DataType::Timestamp(TimeUnit::Nanosecond, None),
+                    int96_to_i64_ns,
+                ),
+                _ => primitive::iter_to_dict_array::<K, _, _, _, _, _>(
+                    iter,
+                    metadata,
+                    data_type,
+                    |x: i64| x,
+                ),
+            },
+            _ => unreachable!(),
+        },
+        Int64 | Date64 | Time64(_) | Duration(_) | Timestamp(_, _) => {
+            primitive::iter_to_dict_array::<K, _, _, _, _, _>(iter, metadata, data_type, |x: i64| x)
+        }
+        Utf8 => binary::iter_to_dict_array::<K, i32, _, _>(iter, metadata),
+        LargeUtf8 => binary::iter_to_dict_array::<K, i64, _, _>(iter, metadata),
+        other => Err(ArrowError::NotYetImplemented(format!(
+            "Reading dictionaries of type {:?}",
+            other
+        ))),
+    }
+}
+
 pub fn page_iter_to_array<
     I: StreamingIterator<Item = std::result::Result<DataPage, ParquetError>>,
 >(
@@ -174,17 +257,16 @@ pub fn page_iter_to_array<
             ))),
         },
 
-        Dictionary(ref key, ref values) => match key.as_ref() {
-            Int32 => match values.as_ref() {
-                Int32 => primitive::iter_to_dict_array::<i32, _, _, _, _, _>(
-                    iter,
-                    metadata,
-                    data_type,
-                    |x: i32| x as i32,
-                ),
-                _ => todo!(),
-            },
-            _ => todo!(),
+        Dictionary(ref key, _) => match key.as_ref() {
+            Int8 => dict_read::<i8, _>(iter, metadata, data_type),
+            Int16 => dict_read::<i16, _>(iter, metadata, data_type),
+            Int32 => dict_read::<i32, _>(iter, metadata, data_type),
+            Int64 => dict_read::<i64, _>(iter, metadata, data_type),
+            UInt8 => dict_read::<u8, _>(iter, metadata, data_type),
+            UInt16 => dict_read::<u16, _>(iter, metadata, data_type),
+            UInt32 => dict_read::<u32, _>(iter, metadata, data_type),
+            UInt64 => dict_read::<u64, _>(iter, metadata, data_type),
+            _ => unreachable!(),
         },
 
         other => Err(ArrowError::NotYetImplemented(format!(
diff --git a/src/io/parquet/write/dictionary.rs b/src/io/parquet/write/dictionary.rs
index 5a7e5ad16b..c379040d8e 100644
--- a/src/io/parquet/write/dictionary.rs
+++ b/src/io/parquet/write/dictionary.rs
@@ -26,12 +26,14 @@ fn encode_keys<K: DictionaryKey>(
 
     let mut buffer = vec![];
 
-    if let Some(validity) = validity {
-        let projected_val = array.iter().map(|x| {
+    let null_count = if let Some(validity) = validity {
+        let projected_validity = array.iter().map(|x| {
             x.map(|x| validity.get_bit(x.to_usize().unwrap()))
                 .unwrap_or(false)
         });
-        let projected_val = Bitmap::from_trusted_len_iter(projected_val);
+        let projected_val = Bitmap::from_trusted_len_iter(projected_validity);
+
+        let null_count = projected_val.null_count();
 
         utils::write_def_levels(
             &mut buffer,
@@ -40,6 +42,7 @@ fn encode_keys<K: DictionaryKey>(
             array.len(),
             options.version,
         )?;
+        null_count
     } else {
         utils::write_def_levels(
             &mut buffer,
@@ -48,7 +51,8 @@ fn encode_keys<K: DictionaryKey>(
             array.len(),
             options.version,
         )?;
-    }
+        array.null_count()
+    };
 
     let definition_levels_byte_length = buffer.len();
 
@@ -70,7 +74,7 @@ fn encode_keys<K: DictionaryKey>(
             .flatten();
         let num_bits = utils::get_bit_width(keys.clone().max().unwrap_or(0) as u64) as u8;
 
-        let keys = utils::ExactSizedIter::new(keys, array.len() - array.null_count());
+        let keys = utils::ExactSizedIter::new(keys, array.len() - null_count);
 
         // num_bits as a single byte
         buffer.push(num_bits);
@@ -104,7 +108,7 @@ fn encode_keys<K: DictionaryKey>(
         None,
         descriptor,
         options,
-        Encoding::PlainDictionary,
+        Encoding::RleDictionary,
     )
     .map(CompressedPage::Data)
 }
diff --git a/src/io/parquet/write/mod.rs b/src/io/parquet/write/mod.rs
index 19c64cbe25..febc803efc 100644
--- a/src/io/parquet/write/mod.rs
+++ b/src/io/parquet/write/mod.rs
@@ -11,8 +11,6 @@ mod utils;
 
 pub mod stream;
 
-use std::sync::Arc;
-
 use crate::array::*;
 use crate::bitmap::Bitmap;
 use crate::buffer::{Buffer, MutableBuffer};
@@ -105,7 +103,7 @@ pub fn can_encode(data_type: &DataType, encoding: Encoding) -> bool {
 
 /// Returns an iterator of compressed pages,
 pub fn array_to_pages(
-    array: Arc<dyn Array>,
+    array: &dyn Array,
     descriptor: ColumnDescriptor,
     options: WriteOptions,
     encoding: Encoding,
@@ -121,7 +119,7 @@ pub fn array_to_pages(
                 )
             })
         }
-        _ => array_to_page(array.as_ref(), descriptor, options, encoding)
+        _ => array_to_page(array, descriptor, options, encoding)
             .map(|page| DynIter::new(std::iter::once(Ok(page)))),
     }
 }
diff --git a/src/io/parquet/write/record_batch.rs b/src/io/parquet/write/record_batch.rs
index 3db125f815..1dffa13a03 100644
--- a/src/io/parquet/write/record_batch.rs
+++ b/src/io/parquet/write/record_batch.rs
@@ -59,7 +59,7 @@ impl<I: Iterator<Item = Result<RecordBatch>>> Iterator for RowGroupIterator<I> {
                     .zip(self.parquet_schema.columns().to_vec().into_iter())
                     .zip(encodings.into_iter())
                     .map(move |((array, type_), encoding)| {
-                        array_to_pages(array, type_, options, encoding)
+                        array_to_pages(array.as_ref(), type_, options, encoding)
                     }),
             ))
         })


Test patch:
diff --git a/arrow-parquet-integration-testing/main_spark.py b/arrow-parquet-integration-testing/main_spark.py
index a51dcbaf65..876a1a808c 100644
--- a/arrow-parquet-integration-testing/main_spark.py
+++ b/arrow-parquet-integration-testing/main_spark.py
@@ -43,3 +43,9 @@ test("generated_primitive", "2", ("utf8_nullable", 24), "delta")
 
 test("generated_dictionary", "1", ("dict0", 0), "")
 test("generated_dictionary", "2", ("dict0", 0), "")
+
+test("generated_dictionary", "1", ("dict1", 1), "")
+test("generated_dictionary", "2", ("dict1", 1), "")
+
+test("generated_dictionary", "1", ("dict2", 2), "")
+test("generated_dictionary", "2", ("dict2", 2), "")
diff --git a/tests/it/io/parquet/mod.rs b/tests/it/io/parquet/mod.rs
index 138538e32f..3a520e006e 100644
--- a/tests/it/io/parquet/mod.rs
+++ b/tests/it/io/parquet/mod.rs
@@ -402,12 +402,12 @@ fn integration_write(schema: &Schema, batches: &[RecordBatch]) -> Result<Vec<u8>
     let row_groups = batches.iter().map(|batch| {
         let iterator = DynIter::new(batch.columns().iter().zip(descritors.clone()).map(
             |(array, type_)| {
-                Ok(DynIter::new(std::iter::once(array_to_page(
-                    array.as_ref(),
-                    type_,
-                    options,
-                    Encoding::Plain,
-                ))))
+                let encoding = if let DataType::Dictionary(_, _) = array.data_type() {
+                    Encoding::RleDictionary
+                } else {
+                    Encoding::Plain
+                };
+                array_to_pages(array.as_ref(), type_, options, encoding)
             },
         ));
         Ok(iterator)
@@ -455,6 +455,12 @@ fn roundtrip_100_primitive() -> Result<()> {
     test_file("1.0.0-bigendian", "generated_primitive")
 }
 
+#[test]
+fn roundtrip_100_dict() -> Result<()> {
+    test_file("1.0.0-littleendian", "generated_dictionary")?;
+    test_file("1.0.0-bigendian", "generated_dictionary")
+}
+
 /// Tests that when arrow-specific types (Duration and LargeUtf8) are written to parquet, we can rountrip its
 /// logical types.
 #[test]