Repository: apache/arrow-rs
Pull request #6473 (2024-09-27 13:47:24)

Update tonic-build requirement from =0.12.2 to =0.12.3
Updates the requirements on [tonic-build](https://github.com/hyperium/tonic) to permit the latest version.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a href="https://github.com/hyperium/tonic/releases">tonic-build's releases</a>.</em></p>
<blockquote>
<h2>v0.12.3</h2>
<h1><a href="https://github.com/hyperium/tonic/compare/v0.12.2...v0.12.3">0.12.3</a> (2024-08-29)</h1>
<h3>Features</h3>
<ul>
<li><strong>server:</strong> Added support for grpc max_connection_age (<a href="https://redirect.github.com/hyperium/tonic/issues/1865">#1865</a>)</li>
<li><strong>build:</strong> Add <code>#[deprecated]</code> to deprecated client methods (<a href="https://redirect.github.com/hyperium/tonic/issues/1879">#1879</a>)</li>
<li><strong>build:</strong> plumb skip_debug through prost Builder and add test (<a href="https://redirect.github.com/hyperium/tonic/issues/1900">#1900</a>)</li>
</ul>
<h3>Bug Fixes</h3>
<ul>
<li><strong>build:</strong> Revert &quot;fix tonic-build cargo build script outputs (<a href="https://redirect.github.com/hyperium/tonic/issues/1821">#1821</a>)&quot; which accidentally increases MSRV (<a href="https://redirect.github.com/hyperium/tonic/issues/1898">#1898</a>)</li>
<li><strong>server:</strong> ignore more error kinds in incoming socket stream (<a href="https://redirect.github.com/hyperium/tonic/issues/1885">#1885</a>)</li>
<li><strong>transport</strong>: do not shutdown server on broken connections (<a href="https://redirect.github.com/hyperium/tonic/issues/1948">#1948</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href="https://github.com/hyperium/tonic/blob/master/CHANGELOG.md">tonic-build's changelog</a>.</em></p>
<blockquote>
<h1><a href="https://github.com/hyperium/tonic/compare/v0.12.2...v0.12.3">0.12.3</a> (2024-08-29)</h1>
<h3>Features</h3>
<ul>
<li><strong>server:</strong> Added support for grpc max_connection_age (<a href="https://redirect.github.com/hyperium/tonic/issues/1865">#1865</a>)</li>
<li><strong>build:</strong> Add <code>#[deprecated]</code> to deprecated client methods (<a href="https://redirect.github.com/hyperium/tonic/issues/1879">#1879</a>)</li>
<li><strong>build:</strong> plumb skip_debug through prost Builder and add test (<a href="https://redirect.github.com/hyperium/tonic/issues/1900">#1900</a>)</li>
</ul>
<h3>Bug Fixes</h3>
<ul>
<li><strong>build:</strong> Revert &quot;fix tonic-build cargo build script outputs (<a href="https://redirect.github.com/hyperium/tonic/issues/1821">#1821</a>)&quot; which accidentally increases MSRV (<a href="https://redirect.github.com/hyperium/tonic/issues/1898">#1898</a>)</li>
<li><strong>server:</strong> ignore more error kinds in incoming socket stream (<a href="https://redirect.github.com/hyperium/tonic/issues/1885">#1885</a>)</li>
<li><strong>transport</strong>: do not shutdown server on broken connections (<a href="https://redirect.github.com/hyperium/tonic/issues/1948">#1948</a>)</li>
</ul>
<h1><a href="https://github.com/hyperium/tonic/compare/v0.12.1...v0.12.2">0.12.2</a> (2024-08-23)</h1>
<h3>Features</h3>
<ul>
<li>Move TimeoutExpired out of transport (<a href="https://redirect.github.com/hyperium/tonic/issues/1826">#1826</a>)</li>
<li>Move ConnectError type from transport (<a href="https://redirect.github.com/hyperium/tonic/issues/1828">#1828</a>)</li>
<li><strong>channel:</strong> allow setting max_header_list_size (<a href="https://redirect.github.com/hyperium/tonic/issues/1835">#1835</a>)</li>
<li><strong>router:</strong> Add RoutesBuilder constructor (<a href="https://redirect.github.com/hyperium/tonic/issues/1855">#1855</a>)</li>
<li><strong>tls:</strong> Rename tls-roots feature with tls-native-roots (<a href="https://redirect.github.com/hyperium/tonic/issues/1860">#1860</a>)</li>
<li><strong>router:</strong> Rename Routes::into_router with into_axum_router (<a href="https://redirect.github.com/hyperium/tonic/issues/1862">#1862</a>)</li>
<li><strong>router:</strong> Implement from axum::Router for Routes (<a href="https://redirect.github.com/hyperium/tonic/issues/1863">#1863</a>)</li>
<li><strong>channel:</strong> Re-enable TLS based on Cargo features in generated clients (<a href="https://redirect.github.com/hyperium/tonic/issues/1866">#1866</a>)</li>
<li><strong>server:</strong> allow setting max_header_list_size (<a href="https://redirect.github.com/hyperium/tonic/issues/1870">#1870</a>)</li>
<li><strong>build:</strong> Expose formatted service name (<a href="https://redirect.github.com/hyperium/tonic/issues/1684">#1684</a>)</li>
<li><strong>reflection:</strong> add back support for v1alpha reflection protocol (<a href="https://redirect.github.com/hyperium/tonic/issues/1888">#1888</a>)</li>
</ul>
<h3>Bug Fixes</h3>
<ul>
<li><strong>router:</strong> Add missing unimplemented fallback to RoutesBuilder (<a href="https://redirect.github.com/hyperium/tonic/issues/1864">#1864</a>)</li>
<li><strong>server:</strong> Prevent server from exiting on ECONNABORTED (<a href="https://redirect.github.com/hyperium/tonic/issues/1874">#1874</a>)</li>
<li><strong>web:</strong> fix panic in trailer parsing on multiple trailers (<a href="https://redirect.github.com/hyperium/tonic/issues/1880">#1880</a>)</li>
<li><strong>web:</strong> fix empty trailer parsing causing infinite parser loop (<a href="https://redirect.github.com/hyperium/tonic/issues/1883">#1883</a>)</li>
</ul>
<h1><a href="https://github.com/hyperium/tonic/compare/v0.12.0...v0.12.1">0.12.1</a> (2024-07-17)</h1>
<h3>Bug Fixes</h3>
<ul>
<li>Reduce tokio-stream feature (<a href="https://redirect.github.com/hyperium/tonic/issues/1795">#1795</a>)</li>
</ul>
<h1><a href="https://github.com/hyperium/tonic/compare/v0.11.0...v0.12.0">0.12.0</a> (2024-07-08)</h1>
<p>This breaking release updates tonic to the hyper <code>1.0</code> ecosystem and also updates
to prost <code>v0.13.0</code>.</p>
<h3>Features</h3>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="https://github.com/hyperium/tonic/commit/4b8d2c46aa57e40b1e80077f4f7b7d4679027bb5"><code>4b8d2c4</code></a> chore: prepare 0.12.3 release (<a href="https://redirect.github.com/hyperium/tonic/issues/1899">#1899</a>)</li>
<li><a href="https://github.com/hyperium/tonic/commit/517b7fc9370b58df5eed4ceffeff405d3df40389"><code>517b7fc</code></a> Use constants for header names (<a href="https://redirect.github.com/hyperium/tonic/issues/1933">#1933</a>)</li>
<li><a href="https://github.com/hyperium/tonic/commit/3c900ebd0bca00e2bae211e4f04e24512d29f5e7"><code>3c900eb</code></a> Clean up EncodeBody API (<a href="https://redirect.github.com/hyperium/tonic/issues/1924">#1924</a>)</li>
<li><a href="https://github.com/hyperium/tonic/commit/e6782fe5f987960e96e48671bbaacc802dfddab9"><code>e6782fe</code></a> fix: add <code>tower?/util</code> dep for <code>channel</code> feature only builds (<a href="https://redirect.github.com/hyperium/tonic/issues/1954">#1954</a>)</li>
<li><a href="https://github.com/hyperium/tonic/commit/6d93c1d0c1a593a5e5476d9c47a1016748acbb5f"><code>6d93c1d</code></a> chore: Make codegen independent from protoc (<a href="https://redirect.github.com/hyperium/tonic/issues/1953">#1953</a>)</li>
<li><a href="https://github.com/hyperium/tonic/commit/f074f134687391cc1df73e98db2c07c812e0a495"><code>f074f13</code></a> chore(health): Update generated code (<a href="https://redirect.github.com/hyperium/tonic/issues/1951">#1951</a>)</li>
<li><a href="https://github.com/hyperium/tonic/commit/474390bdd06ca3f0be007250b1afbfec6a655873"><code>474390b</code></a> fix(tls): do not shutdown server on broken connections (<a href="https://redirect.github.com/hyperium/tonic/issues/1948">#1948</a>)</li>
<li><a href="https://github.com/hyperium/tonic/commit/ec410141173077baba02527618097b406e47b7fa"><code>ec41014</code></a> chore: update releasing documentation (<a href="https://redirect.github.com/hyperium/tonic/issues/1807">#1807</a>)</li>
<li><a href="https://github.com/hyperium/tonic/commit/a09d45347e6815d0c35db4c30154db92a40d1d9c"><code>a09d453</code></a> optimize header name handling in <code>Grpc::map_response</code> (<a href="https://redirect.github.com/hyperium/tonic/issues/1359">#1359</a>)</li>
<li><a href="https://github.com/hyperium/tonic/commit/99b663e3a7becf007f9d06471496b8814fd67d77"><code>99b663e</code></a> chore(test): Move integration test for tonic-web to tests directory (<a href="https://redirect.github.com/hyperium/tonic/issues/1927">#1927</a>)</li>
<li>Additional commits viewable in <a href="https://github.com/hyperium/tonic/compare/v0.12.2...v0.12.3">compare view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>


Patch:
diff --git a/arrow-flight/gen/Cargo.toml b/arrow-flight/gen/Cargo.toml
index 08b53b729..aa33f1f49 100644
--- a/arrow-flight/gen/Cargo.toml
+++ b/arrow-flight/gen/Cargo.toml
@@ -34,4 +34,4 @@ publish = false
 # (and checked in) arrow.flight.protocol.rs from changing
 proc-macro2 = { version = "=1.0.86", default-features = false }
 prost-build = { version = "=0.13.3", default-features = false }
-tonic-build = { version = "=0.12.2", default-features = false, features = ["transport", "prost"] }
+tonic-build = { version = "=0.12.3", default-features = false, features = ["transport", "prost"] }
diff --git a/arrow-flight/src/arrow.flight.protocol.rs b/arrow-flight/src/arrow.flight.protocol.rs
index f1eb549d5..0cd4f6948 100644
--- a/arrow-flight/src/arrow.flight.protocol.rs
+++ b/arrow-flight/src/arrow.flight.protocol.rs
@@ -417,7 +417,13 @@ impl CancelStatus {
 }
 /// Generated client implementations.
 pub mod flight_service_client {
-    #![allow(unused_variables, dead_code, missing_docs, clippy::let_unit_value)]
+    #![allow(
+        unused_variables,
+        dead_code,
+        missing_docs,
+        clippy::wildcard_imports,
+        clippy::let_unit_value,
+    )]
     use tonic::codegen::*;
     use tonic::codegen::http::Uri;
     ///
@@ -521,8 +527,7 @@ pub mod flight_service_client {
                 .ready()
                 .await
                 .map_err(|e| {
-                    tonic::Status::new(
-                        tonic::Code::Unknown,
+                    tonic::Status::unknown(
                         format!("Service was not ready: {}", e.into()),
                     )
                 })?;
@@ -555,8 +560,7 @@ pub mod flight_service_client {
                 .ready()
                 .await
                 .map_err(|e| {
-                    tonic::Status::new(
-                        tonic::Code::Unknown,
+                    tonic::Status::unknown(
                         format!("Service was not ready: {}", e.into()),
                     )
                 })?;
@@ -590,8 +594,7 @@ pub mod flight_service_client {
                 .ready()
                 .await
                 .map_err(|e| {
-                    tonic::Status::new(
-                        tonic::Code::Unknown,
+                    tonic::Status::unknown(
                         format!("Service was not ready: {}", e.into()),
                     )
                 })?;
@@ -640,8 +643,7 @@ pub mod flight_service_client {
                 .ready()
                 .await
                 .map_err(|e| {
-                    tonic::Status::new(
-                        tonic::Code::Unknown,
+                    tonic::Status::unknown(
                         format!("Service was not ready: {}", e.into()),
                     )
                 })?;
@@ -672,8 +674,7 @@ pub mod flight_service_client {
                 .ready()
                 .await
                 .map_err(|e| {
-                    tonic::Status::new(
-                        tonic::Code::Unknown,
+                    tonic::Status::unknown(
                         format!("Service was not ready: {}", e.into()),
                     )
                 })?;
@@ -704,8 +705,7 @@ pub mod flight_service_client {
                 .ready()
                 .await
                 .map_err(|e| {
-                    tonic::Status::new(
-                        tonic::Code::Unknown,
+                    tonic::Status::unknown(
                         format!("Service was not ready: {}", e.into()),
                     )
                 })?;
@@ -736,8 +736,7 @@ pub mod flight_service_client {
                 .ready()
                 .await
                 .map_err(|e| {
-                    tonic::Status::new(
-                        tonic::Code::Unknown,
+                    tonic::Status::unknown(
                         format!("Service was not ready: {}", e.into()),
                     )
                 })?;
@@ -767,8 +766,7 @@ pub mod flight_service_client {
                 .ready()
                 .await
                 .map_err(|e| {
-                    tonic::Status::new(
-                        tonic::Code::Unknown,
+                    tonic::Status::unknown(
                         format!("Service was not ready: {}", e.into()),
                     )
                 })?;
@@ -801,8 +799,7 @@ pub mod flight_service_client {
                 .ready()
                 .await
                 .map_err(|e| {
-                    tonic::Status::new(
-                        tonic::Code::Unknown,
+                    tonic::Status::unknown(
                         format!("Service was not ready: {}", e.into()),
                     )
                 })?;
@@ -832,8 +829,7 @@ pub mod flight_service_client {
                 .ready()
                 .await
                 .map_err(|e| {
-                    tonic::Status::new(
-                        tonic::Code::Unknown,
+                    tonic::Status::unknown(
                         format!("Service was not ready: {}", e.into()),
                     )
                 })?;
@@ -852,7 +848,13 @@ pub mod flight_service_client {
 }
 /// Generated server implementations.
 pub mod flight_service_server {
-    #![allow(unused_variables, dead_code, missing_docs, clippy::let_unit_value)]
+    #![allow(
+        unused_variables,
+        dead_code,
+        missing_docs,
+        clippy::wildcard_imports,
+        clippy::let_unit_value,
+    )]
     use tonic::codegen::*;
     /// Generated trait containing gRPC methods that should be implemented for use with FlightServiceServer.
     #[async_trait]
@@ -1569,17 +1571,19 @@ pub mod flight_service_server {
                 }
                 _ => {
                     Box::pin(async move {
-                        Ok(
-                            http::Response::builder()
-                                .status(200)
-                                .header("grpc-status", tonic::Code::Unimplemented as i32)
-                                .header(
-                                    http::header::CONTENT_TYPE,
-                                    tonic::metadata::GRPC_CONTENT_TYPE,
-                                )
-                                .body(empty_body())
-                                .unwrap(),
-                        )
+                        let mut response = http::Response::new(empty_body());
+                        let headers = response.headers_mut();
+                        headers
+                            .insert(
+                                tonic::Status::GRPC_STATUS,
+                                (tonic::Code::Unimplemented as i32).into(),
+                            );
+                        headers
+                            .insert(
+                                http::header::CONTENT_TYPE,
+                                tonic::metadata::GRPC_CONTENT_TYPE,
+                            );
+                        Ok(response)
                     })
                 }
             }
diff --git a/parquet/Cargo.toml b/parquet/Cargo.toml
index b97b2a571..1d38e67a0 100644
--- a/parquet/Cargo.toml
+++ b/parquet/Cargo.toml
@@ -68,6 +68,7 @@ twox-hash = { version = "1.6", default-features = false }
 paste = { version = "1.0" }
 half = { version = "2.1", default-features = false, features = ["num-traits"] }
 sysinfo = { version = "0.31.2", optional = true, default-features = false, features = ["system"] }
+crc32fast = { version = "1.4.2", optional = true, default-features = false }
 
 [dev-dependencies]
 base64 = { version = "0.22", default-features = false, features = ["std"] }
@@ -117,6 +118,8 @@ object_store = ["dep:object_store", "async"]
 zstd = ["dep:zstd", "zstd-sys"]
 # Display memory in example/write_parquet.rs
 sysinfo = ["dep:sysinfo"]
+# Verify 32-bit CRC checksum when decoding parquet pages
+crc = ["dep:crc32fast"]
 
 [[example]]
 name = "read_parquet"
diff --git a/parquet/README.md b/parquet/README.md
index 0360d15db..a0441ee60 100644
--- a/parquet/README.md
+++ b/parquet/README.md
@@ -60,6 +60,7 @@ The `parquet` crate provides the following features which may be enabled in your
 - `zstd` (default) - support for parquet using `zstd` compression
 - `snap` (default) - support for parquet using `snappy` compression
 - `cli` - parquet [CLI tools](https://github.com/apache/arrow-rs/tree/master/parquet/src/bin)
+- `crc` - enables functionality to automatically verify checksums of each page (if present) when decoding
 - `experimental` - Experimental APIs which may change, even between minor releases
 
 ## Parquet Feature Status
@@ -82,4 +83,4 @@ The `parquet` crate provides the following features which may be enabled in your
 
 ## License
 
-Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0.
+Licensed under the Apache License, Version 2.0: <http://www.apache.org/licenses/LICENSE-2.0>.
diff --git a/parquet/src/file/serialized_reader.rs b/parquet/src/file/serialized_reader.rs
index 6fb0f78c1..b253b73a4 100644
--- a/parquet/src/file/serialized_reader.rs
+++ b/parquet/src/file/serialized_reader.rs
@@ -390,6 +390,15 @@ pub(crate) fn decode_page(
     physical_type: Type,
     decompressor: Option<&mut Box<dyn Codec>>,
 ) -> Result<Page> {
+    // Verify the 32-bit CRC checksum of the page
+    #[cfg(feature = "crc")]
+    if let Some(expected_crc) = page_header.crc {
+        let crc = crc32fast::hash(&buffer);
+        if crc != expected_crc as u32 {
+            return Err(general_err!("Page CRC checksum mismatch"));
+        }
+    }
+
     // When processing data page v2, depending on enabled compression for the
     // page, we should account for uncompressed data ('offset') of
     // repetition and definition levels.


Test patch:
diff --git a/parquet/tests/arrow_reader/checksum.rs b/parquet/tests/arrow_reader/checksum.rs
new file mode 100644
index 000000000..c60908d8b
--- /dev/null
+++ b/parquet/tests/arrow_reader/checksum.rs
@@ -0,0 +1,73 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! This file contains an end to end test for verifying checksums when reading parquet files.
+
+use std::path::PathBuf;
+
+use arrow::util::test_util::parquet_test_data;
+use parquet::arrow::arrow_reader::ArrowReaderBuilder;
+
+#[test]
+fn test_datapage_v1_corrupt_checksum() {
+    let errors = read_file_batch_errors("datapage_v1-corrupt-checksum.parquet");
+    assert_eq!(errors, [
+        Err("Parquet argument error: Parquet error: Page CRC checksum mismatch".to_string()), 
+        Ok(()),
+        Ok(()),
+        Err("Parquet argument error: Parquet error: Page CRC checksum mismatch".to_string()), 
+        Err("Parquet argument error: Parquet error: Not all children array length are the same!".to_string())
+    ]);
+}
+
+#[test]
+fn test_datapage_v1_uncompressed_checksum() {
+    let errors = read_file_batch_errors("datapage_v1-uncompressed-checksum.parquet");
+    assert_eq!(errors, [Ok(()), Ok(()), Ok(()), Ok(()), Ok(())]);
+}
+
+#[test]
+fn test_datapage_v1_snappy_compressed_checksum() {
+    let errors = read_file_batch_errors("datapage_v1-snappy-compressed-checksum.parquet");
+    assert_eq!(errors, [Ok(()), Ok(()), Ok(()), Ok(()), Ok(())]);
+}
+
+#[test]
+fn test_plain_dict_uncompressed_checksum() {
+    let errors = read_file_batch_errors("plain-dict-uncompressed-checksum.parquet");
+    assert_eq!(errors, [Ok(())]);
+}
+#[test]
+fn test_rle_dict_snappy_checksum() {
+    let errors = read_file_batch_errors("rle-dict-snappy-checksum.parquet");
+    assert_eq!(errors, [Ok(())]);
+}
+
+/// Reads a file and returns a vector with one element per record batch.
+/// The record batch data is replaced with () and errors are stringified.
+fn read_file_batch_errors(name: &str) -> Vec<Result<(), String>> {
+    let path = PathBuf::from(parquet_test_data()).join(name);
+    println!("Reading file: {:?}", path);
+    let file = std::fs::File::open(&path).unwrap();
+    let reader = ArrowReaderBuilder::try_new(file).unwrap().build().unwrap();
+    reader
+        .map(|x| match x {
+            Ok(_) => Ok(()),
+            Err(e) => Err(e.to_string()),
+        })
+        .collect()
+}
diff --git a/parquet/tests/arrow_reader/mod.rs b/parquet/tests/arrow_reader/mod.rs
index cc4c8f3c9..0e6783583 100644
--- a/parquet/tests/arrow_reader/mod.rs
+++ b/parquet/tests/arrow_reader/mod.rs
@@ -36,6 +36,8 @@ use std::sync::Arc;
 use tempfile::NamedTempFile;
 
 mod bad_data;
+#[cfg(feature = "crc")]
+mod checksum;
 mod statistics;
 
 // returns a struct array with columns "int32_col", "float32_col" and "float64_col" with the specified values