Repository: apache/arrow
Pull request #45828: GH-45827: [C++] Add io directory to Meson configuration
Author: login: WillAyd
Labels: name: Component: C++
name: awaiting merge
description: Awaiting merge
Created: 2025-03-17T18:49:32Z | Merged: 2025-03-19T03:06:02Z

### Rationale for this change

This continues building out the Meson C++ configuration

### What changes are included in this PR?

This adds the io directory to the Meson configuration

### Are these changes tested?

Yes

### Are there any user-facing changes?

No

* GitHub Issue: #45827


Diff:
diff --git a/cpp/meson.build b/cpp/meson.build
index 0fd75cc35f59..688239fdd3e4 100644
--- a/cpp/meson.build
+++ b/cpp/meson.build
@@ -58,7 +58,8 @@ endif
 
 needs_benchmarks = get_option('benchmarks')
 needs_csv = get_option('csv')
-needs_filesystem = false
+needs_hdfs = get_option('hdfs')
+needs_filesystem = false or needs_hdfs
 needs_integration = false
 needs_tests = get_option('tests')
 needs_ipc = get_option('ipc') or needs_tests or needs_benchmarks
diff --git a/cpp/meson.options b/cpp/meson.options
index 01bce102a75f..d0f3ada76aa4 100644
--- a/cpp/meson.options
+++ b/cpp/meson.options
@@ -29,6 +29,13 @@ option(
     value: false,
 )
 
+option(
+    'hdfs',
+    type: 'boolean',
+    description: 'Build the Arrow HDFS bridge',
+    value: false,
+)
+
 option(
     'ipc',
     type: 'boolean',
diff --git a/cpp/src/arrow/io/meson.build b/cpp/src/arrow/io/meson.build
new file mode 100644
index 000000000000..9a59fbaac7f4
--- /dev/null
+++ b/cpp/src/arrow/io/meson.build
@@ -0,0 +1,76 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+io_tests = ['buffered_test', 'compressed_test', 'file_test', 'memory_test']
+
+foreach io_test : io_tests
+    test_name = 'arrow-io-@0@'.format(io_test.replace('_', '-'))
+    exc = executable(
+        test_name,
+        sources: '@0@.cc'.format(io_test),
+        dependencies: [arrow_test_dep],
+        implicit_include_directories: false,
+    )
+    test(test_name, exc)
+endforeach
+
+if needs_hdfs
+    hdfs_incdir = '../../../thirdparty/hadoop/include'
+    exc = executable(
+        'arrow-io-hdfs-test',
+        sources: 'hdfs_test.cc',
+        dependencies: [arrow_test_dep, filesystem_dep],
+        implicit_include_directories: false,
+        include_directories: [hdfs_incdir],
+    )
+    test('arrow-io-hdfs-test', exc)
+endif
+
+io_benchmarks = ['file_benchmark', 'compressed_benchmark']
+
+foreach io_benchmark : io_benchmarks
+    benchmark_name = 'arrow-io-@0@'.format(io_benchmark.replace('_', '-'))
+    exc = executable(
+        benchmark_name,
+        sources: '@0@.cc'.format(io_benchmark),
+        dependencies: [arrow_test_dep, benchmark_dep],
+        implicit_include_directories: false,
+    )
+    benchmark(benchmark_name, exc)
+endforeach
+
+# TODO: Add memory benchmark with SIMD. See GH-45823
+
+install_headers(
+    [
+        'api.h',
+        'buffered.h',
+        'caching.h',
+        'compressed.h',
+        'concurrency.h',
+        'file.h',
+        'hdfs.h',
+        'interfaces.h',
+        'memory.h',
+        'mman.h',
+        'slow.h',
+        'stdio.h',
+        'transform.h',
+        'type_fwd.h',
+    ],
+    subdir: 'arrow/io',
+)
diff --git a/cpp/src/arrow/meson.build b/cpp/src/arrow/meson.build
index b2c59f1dd7e8..f8c610b3e636 100644
--- a/cpp/src/arrow/meson.build
+++ b/cpp/src/arrow/meson.build
@@ -371,7 +371,7 @@ install_headers(
         'visit_scalar_inline.h',
         'visit_type_inline.h',
     ],
-    install_dir: 'arrow',
+    subdir: 'arrow',
 )
 
 if needs_testing
@@ -531,6 +531,7 @@ pkg.generate(
 subdir('testing')
 
 subdir('c')
+subdir('io')
 subdir('util')
 
 if needs_csv
diff --git a/cpp/src/arrow/testing/meson.build b/cpp/src/arrow/testing/meson.build
index d1ebfe2eddbb..1b6e6a9540d9 100644
--- a/cpp/src/arrow/testing/meson.build
+++ b/cpp/src/arrow/testing/meson.build
@@ -35,6 +35,7 @@ install_headers(
         'util.h',
         'visibility.h',
     ],
+    subdir: 'arrow/testing',
 )
 
 if needs_tests


Reviews:
user: login: WillAyd
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-17T18:50:31Z
user: login: kou
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-18T00:35:37Z
user: login: WillAyd
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T03:37:38Z
user: login: WillAyd
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T03:49:11Z
user: login: kou
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-18T04:38:05Z
user: login: kou
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-18T04:45:21Z
user: login: WillAyd
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-18T05:00:03Z
user: login: kou
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-18T06:33:46Z
user: login: kou
body: +1
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-19T03:05:41Z

Review comments:
diff_hunk: @@ -0,0 +1,75 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+io_tests = ['buffered_test', 'compressed_test', 'file_test', 'memory_test']
+
+foreach io_test : io_tests
+    test_name = 'arrow-io-@0@'.format(io_test.replace('_', '-'))
+    exc = executable(
+        test_name,
+        sources: '@0@.cc'.format(io_test),
+        dependencies: [arrow_test_dep],
+        implicit_include_directories: false,
+    )
+    test(test_name, exc)
+endforeach
+
+if needs_hdfs
+    hdfs_incdir = '../../../thirdparty/hadoop/include'
path: cpp/src/arrow/io/meson.build
user: login: WillAyd
body: From a purity perspective I think it would be better if we used hdfs as a subproject, and could create a dependency for it. However, I wouldn't find the original source for the file `thirdparty/hadoop/include/hdfs.h`
created_at: 2025-03-17T18:50:31Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,75 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+io_tests = ['buffered_test', 'compressed_test', 'file_test', 'memory_test']
+
+foreach io_test : io_tests
+    test_name = 'arrow-io-@0@'.format(io_test.replace('_', '-'))
+    exc = executable(
+        test_name,
+        sources: '@0@.cc'.format(io_test),
+        dependencies: [arrow_test_dep],
+        implicit_include_directories: false,
+    )
+    test(test_name, exc)
+endforeach
+
+if needs_hdfs
+    hdfs_incdir = '../../../thirdparty/hadoop/include'
path: cpp/src/arrow/io/meson.build
user: login: kou
body: https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/include/hdfs/hdfs.h is the original.
created_at: 2025-03-18T00:34:45Z
author_association: MEMBER
diff_hunk: @@ -244,6 +244,30 @@ else
     rapidjson_dep = disabler()
 endif
 
+if needs_filesystem
+    arrow_fs_srcs = [
+        'filesystem/filesystem.cc',
+        'filesystem/localfs.cc',
+        'filesystem/mockfs.cc',
+        'filesystem/path_util.cc',
+        'filesystem/util_internal.cc',
+    ]
+    arrow_fs_incdir = []
+
+    if needs_hdfs
+        arrow_fs_srcs += 'filesystem/hdfs.cc'
+        hdfs_incdir = '../../thirdparty/hadoop/include'
+        arrow_fs_incdir += hdfs_incdir
+    endif
+
+    arrow_components += {
+        'arrow_filesystem': {
+            'sources': arrow_fs_srcs,
+            'include_dirs': arrow_fs_incdir,
+        },
+    }
+endif
+
path: cpp/src/arrow/meson.build
user: login: kou
body: Can we do this when we add `filesystem/` directory?
created_at: 2025-03-18T00:35:26Z
author_association: MEMBER
diff_hunk: @@ -244,6 +244,30 @@ else
     rapidjson_dep = disabler()
 endif
 
+if needs_filesystem
+    arrow_fs_srcs = [
+        'filesystem/filesystem.cc',
+        'filesystem/localfs.cc',
+        'filesystem/mockfs.cc',
+        'filesystem/path_util.cc',
+        'filesystem/util_internal.cc',
+    ]
+    arrow_fs_incdir = []
+
+    if needs_hdfs
+        arrow_fs_srcs += 'filesystem/hdfs.cc'
+        hdfs_incdir = '../../thirdparty/hadoop/include'
+        arrow_fs_incdir += hdfs_incdir
+    endif
+
+    arrow_components += {
+        'arrow_filesystem': {
+            'sources': arrow_fs_srcs,
+            'include_dirs': arrow_fs_incdir,
+        },
+    }
+endif
+
path: cpp/src/arrow/meson.build
user: login: WillAyd
body: Are you asking to remove the entire block for now?
created_at: 2025-03-18T03:37:38Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,75 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+io_tests = ['buffered_test', 'compressed_test', 'file_test', 'memory_test']
+
+foreach io_test : io_tests
+    test_name = 'arrow-io-@0@'.format(io_test.replace('_', '-'))
+    exc = executable(
+        test_name,
+        sources: '@0@.cc'.format(io_test),
+        dependencies: [arrow_test_dep],
+        implicit_include_directories: false,
+    )
+    test(test_name, exc)
+endforeach
+
+if needs_hdfs
+    hdfs_incdir = '../../../thirdparty/hadoop/include'
path: cpp/src/arrow/io/meson.build
user: login: WillAyd
body: Thanks! Does there happen to be a canonical link to download just the hadoop-hdfs-native-client? 
created_at: 2025-03-18T03:49:11Z
author_association: CONTRIBUTOR
diff_hunk: @@ -244,6 +244,30 @@ else
     rapidjson_dep = disabler()
 endif
 
+if needs_filesystem
+    arrow_fs_srcs = [
+        'filesystem/filesystem.cc',
+        'filesystem/localfs.cc',
+        'filesystem/mockfs.cc',
+        'filesystem/path_util.cc',
+        'filesystem/util_internal.cc',
+    ]
+    arrow_fs_incdir = []
+
+    if needs_hdfs
+        arrow_fs_srcs += 'filesystem/hdfs.cc'
+        hdfs_incdir = '../../thirdparty/hadoop/include'
+        arrow_fs_incdir += hdfs_incdir
+    endif
+
+    arrow_components += {
+        'arrow_filesystem': {
+            'sources': arrow_fs_srcs,
+            'include_dirs': arrow_fs_incdir,
+        },
+    }
+endif
+
path: cpp/src/arrow/meson.build
user: login: kou
body: Yes!
created_at: 2025-03-18T04:38:04Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,75 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+io_tests = ['buffered_test', 'compressed_test', 'file_test', 'memory_test']
+
+foreach io_test : io_tests
+    test_name = 'arrow-io-@0@'.format(io_test.replace('_', '-'))
+    exc = executable(
+        test_name,
+        sources: '@0@.cc'.format(io_test),
+        dependencies: [arrow_test_dep],
+        implicit_include_directories: false,
+    )
+    test(test_name, exc)
+endforeach
+
+if needs_hdfs
+    hdfs_incdir = '../../../thirdparty/hadoop/include'
path: cpp/src/arrow/io/meson.build
user: login: kou
body: Apache Hadoop source archive includes it:
https://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-3.4.1/hadoop-3.4.1-src.tar.gz?action=download

created_at: 2025-03-18T04:45:21Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,75 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+io_tests = ['buffered_test', 'compressed_test', 'file_test', 'memory_test']
+
+foreach io_test : io_tests
+    test_name = 'arrow-io-@0@'.format(io_test.replace('_', '-'))
+    exc = executable(
+        test_name,
+        sources: '@0@.cc'.format(io_test),
+        dependencies: [arrow_test_dep],
+        implicit_include_directories: false,
+    )
+    test(test_name, exc)
+endforeach
+
+if needs_hdfs
+    hdfs_incdir = '../../../thirdparty/hadoop/include'
path: cpp/src/arrow/io/meson.build
user: login: WillAyd
body: I'm not sure that there is a way with the wrapdb to natively extract just the file we need from a collection of projects like that without a top level CMakeLists.txt or meson.build configuration. I'll have to research that further if its a blocker here
created_at: 2025-03-18T05:00:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,75 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+io_tests = ['buffered_test', 'compressed_test', 'file_test', 'memory_test']
+
+foreach io_test : io_tests
+    test_name = 'arrow-io-@0@'.format(io_test.replace('_', '-'))
+    exc = executable(
+        test_name,
+        sources: '@0@.cc'.format(io_test),
+        dependencies: [arrow_test_dep],
+        implicit_include_directories: false,
+    )
+    test(test_name, exc)
+endforeach
+
+if needs_hdfs
+    hdfs_incdir = '../../../thirdparty/hadoop/include'
path: cpp/src/arrow/io/meson.build
user: login: kou
body: I don't think this is a blocker.
We can keep using the vendored `hdfs.h`.

We loads `libhdfs.so` manually by `dlload()` not compile time `-lhdfs`. So we just need `hdfs.h`. If `hdfs.h` has many backward incompatible changes, we may want to use upstream `hdfs.h`. But it's very stable so far. So we can keep using the vendored `hdfs.h`. 
created_at: 2025-03-18T06:33:46Z
author_association: MEMBER

PR comments:
user: login: github-actions[bot]
created_at: 2025-03-17T18:50:03Z
body: :warning: GitHub issue #45827 **has been automatically assigned in GitHub** to PR creator.
author_association: NONE
user: login: github-actions[bot]
created_at: 2025-03-17T18:50:04Z
body: :warning: GitHub issue #45827 **has no components**, please add labels for components.
author_association: NONE
user: login: WillAyd
created_at: 2025-03-17T18:51:17Z
body: @github-actions crossbow submit *meson
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-03-17T18:53:30Z
body: Revision: 14e5b8486efc47d01e42269c197ed5ce83e55c2f

Submitted crossbow builds: [ursacomputing/crossbow @ actions-3197a7e1ab](https://github.com/ursacomputing/crossbow/branches/all?query=actions-3197a7e1ab)

|Task|Status|
|----|------|
|test-conda-cpp-meson|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-3197a7e1ab-github-test-conda-cpp-meson)](https://github.com/ursacomputing/crossbow/actions/runs/13907547598/job/38914015641)|
author_association: NONE
user: login: WillAyd
created_at: 2025-03-17T20:18:05Z
body: @github-actions crossbow submit *meson
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-03-17T20:20:16Z
body: Revision: 9b2b04c733d9e11ef1a0fe7a699640a53affcf12

Submitted crossbow builds: [ursacomputing/crossbow @ actions-61da790a53](https://github.com/ursacomputing/crossbow/branches/all?query=actions-61da790a53)

|Task|Status|
|----|------|
|test-conda-cpp-meson|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-61da790a53-github-test-conda-cpp-meson)](https://github.com/ursacomputing/crossbow/actions/runs/13909045033/job/38918828574)|
author_association: NONE
user: login: WillAyd
created_at: 2025-03-18T15:10:30Z
body: @github-actions crossbow submit *meson
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-03-18T15:12:55Z
body: Revision: 08501813a90f9431dc7409629e6e4e24129eafd2

Submitted crossbow builds: [ursacomputing/crossbow @ actions-dc99d8a809](https://github.com/ursacomputing/crossbow/branches/all?query=actions-dc99d8a809)

|Task|Status|
|----|------|
|test-conda-cpp-meson|[![GitHub Actions](https://github.com/ursacomputing/crossbow/actions/workflows/crossbow.yml/badge.svg?branch=actions-dc99d8a809-github-test-conda-cpp-meson)](https://github.com/ursacomputing/crossbow/actions/runs/13926739798/job/38973164582)|
author_association: NONE
user: login: conbench-apache-arrow[bot]
created_at: 2025-03-19T10:07:57Z
body: After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit fc0862a25fa2d6aa17df14daa9ad37430d432425.

There were no benchmark performance regressions. 🎉

The [full Conbench report](https://github.com/apache/arrow/runs/39026617279) has more details. It also includes information about 10 possible false positives for unstable benchmarks that are known to sometimes produce them.
author_association: NONE