Repository: apache/arrow-rs
Pull request #7485: [Variant] Add (empty) `parquet-variant` crate, update `parquet-testing` pin
Author: login: alamb
Created: 2025-05-08T19:15:09Z | Merged: 2025-05-11T10:47:33Z

# Which issue does this PR close?

- Part of https://github.com/apache/arrow-rs/issues/6736

# Rationale for this change
 
We are adding support for Variant and thus we should add a new crate where the new code will live and that clearly explains the current status. 

# What changes are included in this PR?

1. Add a new `parquet-variant` crate
2. Add README
3. Update `parquet-testing` submodule to include variant test data added in https://github.com/apache/parquet-testing/pull/76

<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->

# Are there any user-facing changes?
Not yet

<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->

<!---
If there are any breaking changes to public APIs, please call them out.
-->



Diff:
diff --git a/Cargo.toml b/Cargo.toml
index 7e7cae206a3f..a8e8067b2b36 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -38,6 +38,7 @@ members = [
     "arrow-select",
     "arrow-string",
     "parquet",
+    "parquet-variant",
     "parquet_derive",
     "parquet_derive_test",
 ]
@@ -100,4 +101,4 @@ chrono = { version = "0.4.40", default-features = false, features = ["clock"] }
 [profile.profiling]
 inherits = "release"
 debug = true
-strip = false
\ No newline at end of file
+strip = false
diff --git a/parquet-testing b/parquet-testing
index e845e41789ec..2dc8bf140ed6 160000
--- a/parquet-testing
+++ b/parquet-testing
@@ -1 +1 @@
-Subproject commit e845e41789ec8d3aa6317e6464cdbbca987bf91d
+Subproject commit 2dc8bf140ed6e28652fc347211c7d661714c7f95
diff --git a/parquet-variant/Cargo.toml b/parquet-variant/Cargo.toml
new file mode 100644
index 000000000000..60c9b316cdf1
--- /dev/null
+++ b/parquet-variant/Cargo.toml
@@ -0,0 +1,37 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+[package]
+name = "parquet-variant"
+# This package is still in development and thus the version does
+# not follow the versions of the rest of the crates in this repo.
+version = "0.1.0"
+license = { workspace = true }
+description = "Apache Parquet Variant implementation in Rust"
+homepage = { workspace = true }
+repository = { workspace = true }
+authors = { workspace = true }
+keywords = ["arrow", "parquet", "variant"]
+readme = "README.md"
+edition = { workspace = true }
+rust-version = { workspace = true }
+
+[dependencies]
+
+[lib]
+
+
diff --git a/parquet-variant/README.md b/parquet-variant/README.md
new file mode 100644
index 000000000000..7da7b2afb650
--- /dev/null
+++ b/parquet-variant/README.md
@@ -0,0 +1,44 @@
+<!---
+  Licensed to the Apache Software Foundation (ASF) under one
+  or more contributor license agreements.  See the NOTICE file
+  distributed with this work for additional information
+  regarding copyright ownership.  The ASF licenses this file
+  to you under the Apache License, Version 2.0 (the
+  "License"); you may not use this file except in compliance
+  with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing,
+  software distributed under the License is distributed on an
+  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  KIND, either express or implied.  See the License for the
+  specific language governing permissions and limitations
+  under the License.
+-->
+
+# Apache Parquet Variant Rust Implementation
+
+[![crates.io](https://img.shields.io/crates/v/parquet-variant.svg)](https://crates.io/crates/parquet-variant)
+[![docs.rs](https://img.shields.io/docsrs/parquet-variant.svg)](https://docs.rs/parquet/latest/parquet-variant/)
+
+This crate contains an implementation of [Variant Binary Encoding] from
+[Apache Parquet]. This software is developed as part of the [Apache Arrow] project.
+
+[Variant Binary Encoding]: https://github.com/apache/parquet-format/blob/master/VariantEncoding.md
+[Apache Parquet]: https://parquet.apache.org/
+[Apache Arrow]: https://arrow.apache.org/
+
+Please see the [API documentation](https://docs.rs/parquet-variant/latest) for more details.
+
+## 🚧 Work In Progress
+
+NOTE: This crate is under active development and is not yet ready for production use. 
+If you are interested in helping, you can find more information on the GitHub [Variant issue]
+
+[Variant issue]: https://github.com/apache/arrow-rs/issues/6736
+
+
+## License
+
+Licensed under the Apache License, Version 2.0: <http://www.apache.org/licenses/LICENSE-2.0>.
diff --git a/parquet-variant/src/lib.rs b/parquet-variant/src/lib.rs
new file mode 100644
index 000000000000..6289f86a263f
--- /dev/null
+++ b/parquet-variant/src/lib.rs
@@ -0,0 +1,28 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Implementation of [Variant Binary Encoding] from [Apache Parquet].
+//!
+//! [Variant Binary Encoding]: https://github.com/apache/parquet-format/blob/master/VariantEncoding.md
+//! [Apache Parquet]: https://parquet.apache.org/
+//!
+//! ## 🚧 Work In Progress
+//!
+//! This crate is under active development and is not yet ready for production use.
+//! If you are interested in helping, you can find more information on the GitHub [Variant issue]
+//!
+//! [Variant issue]: https://github.com/apache/arrow-rs/issues/6736


Reviews:
user: login: wjones127
body: Looks good, though I think the readme description could be confusing to some as is.
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-08T20:06:48Z
user: login: adriangb
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T20:28:45Z
user: login: mbrobbel
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-08T20:44:48Z
user: login: alamb
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T22:45:04Z
user: login: alamb
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T22:45:45Z
user: login: alamb
body: Thank you @wjones127, @mbrobbel and @adriangb for the reviews
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T22:46:14Z
user: login: alamb
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-11T10:45:21Z

Review comments:
diff_hunk: @@ -0,0 +1,44 @@
+<!---
+  Licensed to the Apache Software Foundation (ASF) under one
+  or more contributor license agreements.  See the NOTICE file
+  distributed with this work for additional information
+  regarding copyright ownership.  The ASF licenses this file
+  to you under the Apache License, Version 2.0 (the
+  "License"); you may not use this file except in compliance
+  with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing,
+  software distributed under the License is distributed on an
+  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  KIND, either express or implied.  See the License for the
+  specific language governing permissions and limitations
+  under the License.
+-->
+
+# Apache Parquet Variant Rust Implementation
+
+[![crates.io](https://img.shields.io/crates/v/parquet-variant.svg)](https://crates.io/crates/parquet-variant)
+[![docs.rs](https://img.shields.io/docsrs/parquet-variant.svg)](https://docs.rs/parquet/latest/parquet-variant/)
+
+This crate contains an implementation of [Variant Binary Encoding], part of
+[Apache Parquet], which is part of the [Apache Arrow] project.
path: parquet-variant/README.md
user: login: wjones127
body: This sentence is a bit confusing. I suppose we mean to say the encoding is part of the Apache Parquet project, but this implementation is part of the Apache Arrow project, right?
created_at: 2025-05-08T20:06:03Z
author_association: MEMBER
diff_hunk: @@ -100,4 +101,4 @@ chrono = { version = "0.4.40", default-features = false, features = ["clock"] }
 [profile.profiling]
 inherits = "release"
 debug = true
-strip = false
\ No newline at end of file
+strip = false
path: Cargo.toml
user: login: adriangb
body: Seems like an unnecessary change
created_at: 2025-05-08T20:26:38Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,29 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Implementation of [Variant Binary Encoding] from [Apache Parquet].
+//!
+//! [Variant Binary Encoding]: https://github.com/apache/parquet-format/blob/master/VariantEncoding.md
+//! [Apache Parquet]: https://parquet.apache.org/
+//! [Apache Arrow]: https://arrow.apache.org/
path: parquet-variant/src/lib.rs
user: login: adriangb
body: Not referenced anywhere
created_at: 2025-05-08T20:28:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,44 @@
+<!---
+  Licensed to the Apache Software Foundation (ASF) under one
+  or more contributor license agreements.  See the NOTICE file
+  distributed with this work for additional information
+  regarding copyright ownership.  The ASF licenses this file
+  to you under the Apache License, Version 2.0 (the
+  "License"); you may not use this file except in compliance
+  with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing,
+  software distributed under the License is distributed on an
+  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  KIND, either express or implied.  See the License for the
+  specific language governing permissions and limitations
+  under the License.
+-->
+
+# Apache Parquet Variant Rust Implementation
+
+[![crates.io](https://img.shields.io/crates/v/parquet-variant.svg)](https://crates.io/crates/parquet-variant)
+[![docs.rs](https://img.shields.io/docsrs/parquet-variant.svg)](https://docs.rs/parquet/latest/parquet-variant/)
+
+This crate contains an implementation of [Variant Binary Encoding], part of
+[Apache Parquet], which is part of the [Apache Arrow] project.
path: parquet-variant/README.md
user: login: alamb
body: That is a good call. Here is a proposal to clarify 

```suggestion
This crate contains an implementation of [Variant Binary Encoding] from
[Apache Parquet]. This software is developed as part of the [Apache Arrow] project.
```
created_at: 2025-05-08T22:45:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,29 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Implementation of [Variant Binary Encoding] from [Apache Parquet].
+//!
+//! [Variant Binary Encoding]: https://github.com/apache/parquet-format/blob/master/VariantEncoding.md
+//! [Apache Parquet]: https://parquet.apache.org/
+//! [Apache Arrow]: https://arrow.apache.org/
path: parquet-variant/src/lib.rs
user: login: alamb
body: ```suggestion
```
created_at: 2025-05-08T22:45:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -100,4 +101,4 @@ chrono = { version = "0.4.40", default-features = false, features = ["clock"] }
 [profile.profiling]
 inherits = "release"
 debug = true
-strip = false
\ No newline at end of file
+strip = false
path: Cargo.toml
user: login: alamb
body: I think it is something my editor does by default. 
created_at: 2025-05-11T10:45:21Z
author_association: CONTRIBUTOR

PR comments:
user: login: alamb
created_at: 2025-05-08T19:45:07Z
body: @crepererum or @wjones127 or  @mbrobbel  I wonder if you have a few minutes to review this PR. I hope it is non controversial. 

cc @Weijun-H  @tustvold @etseidl @PinkCrow007
author_association: CONTRIBUTOR
user: login: alamb
created_at: 2025-05-09T21:54:03Z
body: I just found that the parquet-java implementation  also has something called `parquet-variant`: 

https://github.com/apache/parquet-java/tree/master/parquet-variant/src

Which was added a few days ago with @gene-db: 
- https://github.com/apache/parquet-java/pull/3202
author_association: CONTRIBUTOR
user: login: alamb
created_at: 2025-05-11T10:47:29Z
body: Ok, let's merge this to keep the code flowing. I tried to address @wjones127 's [comment](https://github.com/apache/arrow-rs/pull/7485#pullrequestreview-2826246185), but if it is still confusing I would be more than happy to make a follow on PR to further clarify
author_association: CONTRIBUTOR