Repository: jorgecarleitao/arrow2
Pull request #1320 (2022-12-06 15:01:24)

feat: add typed dictionary_iter
Iterating over a `DictionaryArray` is very expensive as it allocates an opaque heap value of each iteration.

This adds a generic typed iterator that only compiles upon instantiation and thus will not add compiler bloat for all types. I want to follow up with a PR that implements writing `DictionaryArrays` to json. That was the reason I stumbled upon this.

The iterator impementation will `panic` iff the `values` contain null values. Otherwise it can become pretty awkward with nested arrays (which is something we could follow up on if needed)>

The existing iterators are flawed IMO. They will happily iterate over a `values` array with null values and return the masked out values. Shall we also add a panic check there? That's much better than return the wrong results. 

ritchie46: @jorgecarleitao could we update rustc in CI? I needed GATS here. 
codecov[bot]: # [Codecov](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1320?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao) Report
Base: **83.13**% // Head: **83.14**% // Increases project coverage by **`+0.00%`** :tada:
> Coverage data is based on head [(`2c22df4`)](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1320?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao) compared to base [(`fc212a0`)](https://codecov.io/gh/jorgecarleitao/arrow2/commit/fc212a0e090ff3b6da80636e0602c0521e6aa300?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao).
> Patch coverage: 87.67% of modified lines in pull request are covered.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #1320   +/-   ##
=======================================
  Coverage   83.13%   83.14%           
=======================================
  Files         369      370    +1     
  Lines       40241    40313   +72     
=======================================
+ Hits        33456    33517   +61     
- Misses       6785     6796   +11     
```


| [Impacted Files](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1320?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao) | Coverage Δ | |
|---|---|---|
| [src/array/dictionary/typed\_iterator.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1320/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2FycmF5L2RpY3Rpb25hcnkvdHlwZWRfaXRlcmF0b3IucnM=) | `82.35% <82.35%> (ø)` | |
| [src/array/dictionary/mod.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1320/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2FycmF5L2RpY3Rpb25hcnkvbW9kLnJz) | `92.59% <100.00%> (+0.92%)` | :arrow_up: |
| [src/array/binary/mod.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1320/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2FycmF5L2JpbmFyeS9tb2QucnM=) | `91.55% <0.00%> (-0.98%)` | :arrow_down: |
| [src/io/ipc/read/file.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1320/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2lvL2lwYy9yZWFkL2ZpbGUucnM=) | `96.87% <0.00%> (+0.44%)` | :arrow_up: |

Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao)

</details>

[:umbrella: View full report at Codecov](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1320?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao).


Patch:
diff --git a/src/array/dictionary/mod.rs b/src/array/dictionary/mod.rs
index 8dc1bbd7c9..4602320956 100644
--- a/src/array/dictionary/mod.rs
+++ b/src/array/dictionary/mod.rs
@@ -17,6 +17,9 @@ pub(super) mod fmt;
 mod iterator;
 mod mutable;
 use crate::array::specification::check_indexes_unchecked;
+mod typed_iterator;
+
+use crate::array::dictionary::typed_iterator::{DictValue, DictionaryValuesIterTyped};
 pub use iterator::*;
 pub use mutable::*;
 
@@ -237,6 +240,38 @@ impl<K: DictionaryKey> DictionaryArray<K> {
         DictionaryValuesIter::new(self)
     }
 
+    /// Returns an iterator over the the values [`V::IterValue`].
+    ///
+    /// # Panics
+    ///
+    /// Panics if the keys of this [`DictionaryArray`] have any null types.
+    /// If they do [`DictionaryArray::iter_typed`] should be called
+    pub fn values_iter_typed<V: DictValue>(
+        &self,
+    ) -> Result<DictionaryValuesIterTyped<K, V>, Error> {
+        let keys = &self.keys;
+        assert_eq!(keys.null_count(), 0);
+        let values = self.values.as_ref();
+        let values = V::downcast_values(values)?;
+        Ok(unsafe { DictionaryValuesIterTyped::new(keys, values) })
+    }
+
+    /// Returns an iterator over the the optional values of  [`Option<V::IterValue>`].
+    ///
+    /// # Panics
+    ///
+    /// This function panics if the `values` array
+    pub fn iter_typed<V: DictValue>(
+        &self,
+    ) -> Result<ZipValidity<V::IterValue<'_>, DictionaryValuesIterTyped<K, V>, BitmapIter>, Error>
+    {
+        let keys = &self.keys;
+        let values = self.values.as_ref();
+        let values = V::downcast_values(values)?;
+        let values_iter = unsafe { DictionaryValuesIterTyped::new(keys, values) };
+        Ok(ZipValidity::new_with_validity(values_iter, self.validity()))
+    }
+
     /// Returns the [`DataType`] of this [`DictionaryArray`]
     #[inline]
     pub fn data_type(&self) -> &DataType {
diff --git a/src/array/dictionary/typed_iterator.rs b/src/array/dictionary/typed_iterator.rs
new file mode 100644
index 0000000000..0e90a1cf4d
--- /dev/null
+++ b/src/array/dictionary/typed_iterator.rs
@@ -0,0 +1,111 @@
+use crate::array::{Array, PrimitiveArray, Utf8Array};
+use crate::error::{Error, Result};
+use crate::trusted_len::TrustedLen;
+use crate::types::Offset;
+
+use super::DictionaryKey;
+
+pub trait DictValue {
+    type IterValue<'this>
+    where
+        Self: 'this;
+
+    /// # Safety
+    /// Will not do any bound checks but must check validity.
+    unsafe fn get_unchecked(&self, item: usize) -> Self::IterValue<'_>;
+
+    /// Take a [`dyn Array`] an try to downcast it to the type of `DictValue`.
+    fn downcast_values(array: &dyn Array) -> Result<&Self>
+    where
+        Self: Sized;
+}
+
+impl<O: Offset> DictValue for Utf8Array<O> {
+    type IterValue<'a> = &'a str;
+
+    unsafe fn get_unchecked(&self, item: usize) -> Self::IterValue<'_> {
+        self.value_unchecked(item)
+    }
+
+    fn downcast_values(array: &dyn Array) -> Result<&Self>
+    where
+        Self: Sized,
+    {
+        array
+            .as_any()
+            .downcast_ref::<Self>()
+            .ok_or(Error::InvalidArgumentError(
+                "could not convert array to dictionary value".into(),
+            ))
+            .map(|arr| {
+                assert_eq!(
+                    arr.null_count(),
+                    0,
+                    "null values in values not supported in iteration"
+                );
+                arr
+            })
+    }
+}
+
+/// Iterator of values of an `ListArray`.
+pub struct DictionaryValuesIterTyped<'a, K: DictionaryKey, V: DictValue> {
+    keys: &'a PrimitiveArray<K>,
+    values: &'a V,
+    index: usize,
+    end: usize,
+}
+
+impl<'a, K: DictionaryKey, V: DictValue> DictionaryValuesIterTyped<'a, K, V> {
+    pub(super) unsafe fn new(keys: &'a PrimitiveArray<K>, values: &'a V) -> Self {
+        Self {
+            keys,
+            values,
+            index: 0,
+            end: keys.len(),
+        }
+    }
+}
+
+impl<'a, K: DictionaryKey, V: DictValue> Iterator for DictionaryValuesIterTyped<'a, K, V> {
+    type Item = V::IterValue<'a>;
+
+    #[inline]
+    fn next(&mut self) -> Option<Self::Item> {
+        if self.index == self.end {
+            return None;
+        }
+        let old = self.index;
+        self.index += 1;
+        unsafe {
+            let key = self.keys.value_unchecked(old);
+            let idx = key.as_usize();
+            Some(self.values.get_unchecked(idx))
+        }
+    }
+
+    #[inline]
+    fn size_hint(&self) -> (usize, Option<usize>) {
+        (self.end - self.index, Some(self.end - self.index))
+    }
+}
+
+unsafe impl<'a, K: DictionaryKey, V: DictValue> TrustedLen for DictionaryValuesIterTyped<'a, K, V> {}
+
+impl<'a, K: DictionaryKey, V: DictValue> DoubleEndedIterator
+    for DictionaryValuesIterTyped<'a, K, V>
+{
+    #[inline]
+    fn next_back(&mut self) -> Option<Self::Item> {
+        if self.index == self.end {
+            None
+        } else {
+            self.end -= 1;
+            unsafe {
+                let key = self.keys.value_unchecked(self.end);
+                let idx = key.as_usize();
+                Some(self.values.get_unchecked(idx))
+            }
+        }
+    }
+}
diff --git a/src/compute/filter.rs b/src/compute/filter.rs
index e2b19f0c7d..a293695b85 100644
--- a/src/compute/filter.rs
+++ b/src/compute/filter.rs
@@ -1,15 +1,13 @@
 //! Contains operators to filter arrays such as [`filter`].
 use crate::array::growable::{make_growable, Growable};
-use crate::bitmap::utils::{BitChunk, BitChunkIterExact, BitChunksExact};
+use crate::bitmap::utils::{BitChunkIterExact, BitChunksExact};
 use crate::bitmap::{utils::SlicesIterator, Bitmap, MutableBitmap};
 use crate::chunk::Chunk;
 use crate::datatypes::DataType;
 use crate::error::Result;
-use crate::types::simd::{NativeSimd, Simd};
+use crate::types::simd::Simd;
 use crate::types::BitChunkOnes;
 use crate::{array::*, types::NativeType};
-use num_traits::One;
-use num_traits::Zero;
 
 /// Function that can filter arbitrary arrays
 pub type Filter<'a> = Box<dyn Fn(&dyn Array) -> Box<dyn Array> + 'a + Send + Sync>;
@@ -20,29 +18,32 @@ pub type Filter<'a> = Box<dyn Fn(&dyn Array) -> Box<dyn Array> + 'a + Send + Syn
 unsafe fn nonnull_filter_impl<T, I>(values: &[T], mut mask_chunks: I, filter_count: usize) -> Vec<T>
 where
     T: NativeType + Simd,
-    I: BitChunkIterExact<<<T as Simd>::Simd as NativeSimd>::Chunk>,
+    I: BitChunkIterExact<u64>,
 {
-    let mut chunks = values.chunks_exact(T::Simd::LANES);
+    let mut chunks = values.chunks_exact(64);
     let mut new = Vec::<T>::with_capacity(filter_count);
     let mut dst = new.as_mut_ptr();
+
     chunks
         .by_ref()
         .zip(mask_chunks.by_ref())
-        .for_each(|(chunk, validity_chunk)| {
-            let ones_iter = BitChunkOnes::new(validity_chunk);
+        .for_each(|(chunk, mask_chunk)| {
+            let ones = mask_chunk.count_ones();
+            let leading_ones = mask_chunk.leading_ones();
 
-            let (size, _) = ones_iter.size_hint();
-            if size == T::Simd::LANES {
-                // Fast path: all lanes are set
+            if ones == leading_ones {
+                let size = leading_ones as usize;
                 unsafe {
                     std::ptr::copy(chunk.as_ptr(), dst, size);
                     dst = dst.add(size);
                 }
-            } else {
-                for pos in ones_iter {
-                    dst.write(chunk[pos]);
-                    dst = dst.add(1);
-                }
+                return;
+            }
+
+            let ones_iter = BitChunkOnes::from_known_count(mask_chunk, ones as usize);
+            for pos in ones_iter {
+                dst.write(*chunk.get_unchecked(pos));
+                dst = dst.add(1);
             }
         });
 
@@ -74,11 +75,11 @@ unsafe fn null_filter_impl<T, I>(
 ) -> (Vec<T>, MutableBitmap)
 where
     T: NativeType + Simd,
-    I: BitChunkIterExact<<<T as Simd>::Simd as NativeSimd>::Chunk>,
+    I: BitChunkIterExact<u64>,
 {
-    let mut chunks = values.chunks_exact(T::Simd::LANES);
+    let mut chunks = values.chunks_exact(64);
 
-    let mut validity_chunks = validity.chunks::<<T::Simd as NativeSimd>::Chunk>();
+    let mut validity_chunks = validity.chunks::<u64>();
 
     let mut new = Vec::<T>::with_capacity(filter_count);
     let mut dst = new.as_mut_ptr();
@@ -89,25 +90,26 @@ where
         .zip(validity_chunks.by_ref())
         .zip(mask_chunks.by_ref())
         .for_each(|((chunk, validity_chunk), mask_chunk)| {
-            let ones_iter = BitChunkOnes::new(mask_chunk);
-            let (size, _) = ones_iter.size_hint();
+            let ones = mask_chunk.count_ones();
+            let leading_ones = mask_chunk.leading_ones();
 
-            if size == T::Simd::LANES {
-                // Fast path: all lanes are set
+            if ones == leading_ones {
+                let size = leading_ones as usize;
                 unsafe {
                     std::ptr::copy(chunk.as_ptr(), dst, size);
                     dst = dst.add(size);
+
                     new_validity.extend_from_slice(validity_chunk.to_ne_bytes().as_ref(), 0, size);
                 }
-            } else {
-                for pos in ones_iter {
-                    dst.write(chunk[pos]);
-                    dst = dst.add(1);
-                    new_validity.push(
-                        validity_chunk & (<<<T as Simd>::Simd as NativeSimd>::Chunk>::one() << pos)
-                            > <<<T as Simd>::Simd as NativeSimd>::Chunk>::zero(),
-                    );
-                }
+                return;
+            }
+
+            // this triggers a bitcount
+            let ones_iter = BitChunkOnes::from_known_count(mask_chunk, ones as usize);
+            for pos in ones_iter {
+                dst.write(*chunk.get_unchecked(pos));
+                dst = dst.add(1);
+                new_validity.push_unchecked(validity_chunk & (1 << pos) > 0);
             }
         });
 
@@ -140,10 +142,10 @@ fn null_filter_simd<T: NativeType + Simd>(
 
     let (slice, offset, length) = mask.as_slice();
     if offset == 0 {
-        let mask_chunks = BitChunksExact::<<T::Simd as NativeSimd>::Chunk>::new(slice, length);
+        let mask_chunks = BitChunksExact::<u64>::new(slice, length);
         unsafe { null_filter_impl(values, validity, mask_chunks, filter_count) }
     } else {
-        let mask_chunks = mask.chunks::<<T::Simd as NativeSimd>::Chunk>();
+        let mask_chunks = mask.chunks::<u64>();
         unsafe { null_filter_impl(values, validity, mask_chunks, filter_count) }
     }
 }
@@ -154,10 +156,10 @@ fn nonnull_filter_simd<T: NativeType + Simd>(values: &[T], mask: &Bitmap) -> Vec
 
     let (slice, offset, length) = mask.as_slice();
     if offset == 0 {
-        let mask_chunks = BitChunksExact::<<T::Simd as NativeSimd>::Chunk>::new(slice, length);
+        let mask_chunks = BitChunksExact::<u64>::new(slice, length);
         unsafe { nonnull_filter_impl(values, mask_chunks, filter_count) }
     } else {
-        let mask_chunks = mask.chunks::<<T::Simd as NativeSimd>::Chunk>();
+        let mask_chunks = mask.chunks::<u64>();
         unsafe { nonnull_filter_impl(values, mask_chunks, filter_count) }
     }
 }
diff --git a/src/io/parquet/write/nested/rep.rs b/src/io/parquet/write/nested/rep.rs
index 68c7609aca..c7c5d15eb8 100644
--- a/src/io/parquet/write/nested/rep.rs
+++ b/src/io/parquet/write/nested/rep.rs
@@ -8,14 +8,21 @@ impl<A: Iterator<Item = usize> + std::fmt::Debug> DebugIter for A {}
 fn iter<'a>(nested: &'a [Nested]) -> Vec<Box<dyn DebugIter + 'a>> {
     nested
         .iter()
-        .filter_map(|nested| match nested {
+        .enumerate()
+        .filter_map(|(i, nested)| match nested {
             Nested::Primitive(_, _, _) => None,
             Nested::List(nested) => Some(Box::new(to_length(nested.offsets)) as Box<dyn DebugIter>),
             Nested::LargeList(nested) => {
                 Some(Box::new(to_length(nested.offsets)) as Box<dyn DebugIter>)
             }
             Nested::Struct(_, _, length) => {
-                Some(Box::new(std::iter::repeat(1usize).take(*length)) as Box<dyn DebugIter>)
+                // only return 1, 1, 1, (x len) if struct is outer structure.
+                // otherwise treat as leaf
+                if i == 0 {
+                    Some(Box::new(std::iter::repeat(1usize).take(*length)) as Box<dyn DebugIter>)
+                } else {
+                    None
+                }
             }
         })
         .collect()
@@ -223,4 +230,25 @@ mod tests {
 
         test(nested, expected)
     }
+
+    #[test]
+    fn list_struct_list() {
+        let nested = vec![
+            Nested::List(ListNested {
+                is_optional: true,
+                offsets: &[0i32, 2, 3],
+                validity: None,
+            }),
+            Nested::Struct(None, true, 3),
+            Nested::List(ListNested {
+                is_optional: true,
+                offsets: &[0i32, 3, 6, 7],
+                validity: None,
+            }),
+            Nested::Primitive(None, true, 7),
+        ];
+        let expected = vec![0, 2, 2, 1, 2, 2, 0];
+
+        test(nested, expected)
+    }
 }
diff --git a/src/types/bit_chunk.rs b/src/types/bit_chunk.rs
index 0937b50058..796a608b1b 100644
--- a/src/types/bit_chunk.rs
+++ b/src/types/bit_chunk.rs
@@ -129,6 +129,12 @@ impl<T: BitChunk> BitChunkOnes<T> {
             remaining: value.count_ones() as usize,
         }
     }
+
+    #[inline]
+    #[cfg(feature = "compute_filter")]
+    pub(crate) fn from_known_count(value: T, remaining: usize) -> Self {
+        Self { value, remaining }
+    }
 }
 
 impl<T: BitChunk> Iterator for BitChunkOnes<T> {


Test patch:
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
index 74ebe22eed..9a9e91c243 100644
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -79,7 +79,7 @@ jobs:
       - uses: actions/checkout@v2
       - uses: actions-rs/toolchain@v1
         with:
-          toolchain: nightly-2022-07-12
+          toolchain: nightly-2022-12-05
           override: true
       - uses: Swatinem/rust-cache@v1
         with:
@@ -102,7 +102,7 @@ jobs:
           submodules: true # needed to test IPC, which are located in a submodule
       - uses: actions-rs/toolchain@v1
         with:
-          toolchain: nightly-2022-07-12
+          toolchain: nightly-2022-12-05
           override: true
       - uses: Swatinem/rust-cache@v1
         with:
@@ -121,7 +121,7 @@ jobs:
       - uses: actions/checkout@v2
       - uses: actions-rs/toolchain@v1
         with:
-          toolchain: nightly-2022-07-12
+          toolchain: nightly-2022-12-05
           override: true
       - uses: Swatinem/rust-cache@v1
         with:
@@ -166,7 +166,7 @@ jobs:
           submodules: true
       - uses: actions-rs/toolchain@v1
         with:
-          toolchain: nightly-2022-03-16
+          toolchain: nightly-2022-12-05
           target: ${{ matrix.target }}
           override: true
       - uses: Swatinem/rust-cache@v1
@@ -183,7 +183,7 @@ jobs:
       - uses: actions/checkout@v2
       - uses: actions-rs/toolchain@v1
         with:
-          toolchain: nightly-2022-07-24
+          toolchain: nightly-2022-12-05
           override: true
       - uses: Swatinem/rust-cache@v1
       - name: Run
diff --git a/tests/it/array/dictionary/mod.rs b/tests/it/array/dictionary/mod.rs
index b139c94c53..3498041700 100644
--- a/tests/it/array/dictionary/mod.rs
+++ b/tests/it/array/dictionary/mod.rs
@@ -165,3 +165,48 @@ fn keys_values_iter() {
 
     assert_eq!(array.keys_values_iter().collect::<Vec<_>>(), vec![1, 0]);
 }
+
+#[test]
+fn iter_values_typed() {
+    let values = Utf8Array::<i32>::from_slice(["a", "aa"]);
+    let array =
+        DictionaryArray::try_from_keys(PrimitiveArray::from_vec(vec![1, 0, 0]), values.boxed())
+            .unwrap();
+
+    let iter = array.values_iter_typed::<Utf8Array<i32>>().unwrap();
+    assert_eq!(iter.size_hint(), (3, Some(3)));
+    assert_eq!(iter.collect::<Vec<_>>(), vec!["aa", "a", "a"]);
+
+    let iter = array.iter_typed::<Utf8Array<i32>>().unwrap();
+    assert_eq!(iter.size_hint(), (3, Some(3)));
+    assert_eq!(
+        iter.collect::<Vec<_>>(),
+        vec![Some("aa"), Some("a"), Some("a")]
+    );
+}
+
+#[test]
+#[should_panic]
+fn iter_values_typed_panic() {
+    let values = Utf8Array::<i32>::from_iter([Some("a"), Some("aa"), None]);
+    let array =
+        DictionaryArray::try_from_keys(PrimitiveArray::from_vec(vec![1, 0, 0]), values.boxed())
+            .unwrap();
+
+    // should not be iterating values
+    let iter = array.values_iter_typed::<Utf8Array<i32>>().unwrap();
+    let _ = iter.collect::<Vec<_>>();
+}
+
+#[test]
+#[should_panic]
+fn iter_values_typed_panic_2() {
+    let values = Utf8Array::<i32>::from_iter([Some("a"), Some("aa"), None]);
+    let array =
+        DictionaryArray::try_from_keys(PrimitiveArray::from_vec(vec![1, 0, 0]), values.boxed())
+            .unwrap();
+
+    // should not be iterating values
+    let iter = array.iter_typed::<Utf8Array<i32>>().unwrap();
+    let _ = iter.collect::<Vec<_>>();
+}
diff --git a/tests/it/bitmap/mutable.rs b/tests/it/bitmap/mutable.rs
index 6d4efd290d..60af81bf92 100644
--- a/tests/it/bitmap/mutable.rs
+++ b/tests/it/bitmap/mutable.rs
@@ -368,7 +368,11 @@ fn extend_bitmap() {
     assert_eq!(b.len(), 5 + 4);
 }
 
+// TODO! undo miri ignore once issue is fixed in miri
+// this test was a memory hog and lead to OOM in CI
+// given enough memory it was able to pass succesfully on a local
 #[test]
+#[cfg_attr(miri, ignore)]
 fn extend_constant1() {
     use std::iter::FromIterator;
     for i in 0..64 {