Repository: duckdb/duckdb-java
Pull request #162: Do not build jemalloc on Windows
Author: login: staticlibs
Created: 2025-03-14T16:16:18Z | Merged: 2025-03-15T08:52:46Z

The jemalloc extention was recently added to JDBC builds in #133. The logic for its selection in `vendor.py` does not look fully correct to me, but both Linux and Mac are passing on CI, only Windows is breaking, so this change only tries to unbreak Windows builds, and the jemalloc selection logic can be improved in subsequent PRs.


Diff:
diff --git a/.github/workflows/Java.yml b/.github/workflows/Java.yml
index d92bf2c1e..0ef02f902 100644
--- a/.github/workflows/Java.yml
+++ b/.github/workflows/Java.yml
@@ -218,22 +218,22 @@ jobs:
       - shell: bash
         run: mkdir jdbc-artifacts
 
-      - uses: actions/download-artifact@v3
+      - uses: actions/download-artifact@v4
         with:
           name: java-linux-aarch64
           path: jdbc-artifacts/java-linux-aarch64
 
-      - uses: actions/download-artifact@v3
+      - uses: actions/download-artifact@v4
         with:
           name: java-linux-amd64
           path: jdbc-artifacts/java-linux-amd64
 
-      - uses: actions/download-artifact@v3
+      - uses: actions/download-artifact@v4
         with:
           name: java-windows-amd64
           path: jdbc-artifacts/java-windows-amd64
 
-      - uses: actions/download-artifact@v3
+      - uses: actions/download-artifact@v4
         with:
           name: java-osx-universal
           path: jdbc-artifacts/java-osx-universal
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 617ed9649..153402d5b 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -32,6 +32,74 @@ set(CMAKE_JAVA_COMPILE_FLAGS -source 1.8 -target 1.8 -encoding utf-8)
 add_definitions(-DDUCKDB_BUILD_LIBRARY)
 
 if(MSVC)
+  remove_definitions(-DDUCKDB_EXTENSION_JEMALLOC_LINKED)
+  list(REMOVE_ITEM DUCKDB_SRC_FILES
+    src/duckdb/extension/jemalloc/jemalloc_extension.cpp
+    src/duckdb/extension/jemalloc/jemalloc/src/jemalloc.c
+    src/duckdb/extension/jemalloc/jemalloc/src/arena.c
+    src/duckdb/extension/jemalloc/jemalloc/src/background_thread.c
+    src/duckdb/extension/jemalloc/jemalloc/src/base.c
+    src/duckdb/extension/jemalloc/jemalloc/src/batcher.c
+    src/duckdb/extension/jemalloc/jemalloc/src/bin.c
+    src/duckdb/extension/jemalloc/jemalloc/src/bin_info.c
+    src/duckdb/extension/jemalloc/jemalloc/src/bitmap.c
+    src/duckdb/extension/jemalloc/jemalloc/src/buf_writer.c
+    src/duckdb/extension/jemalloc/jemalloc/src/cache_bin.c
+    src/duckdb/extension/jemalloc/jemalloc/src/ckh.c
+    src/duckdb/extension/jemalloc/jemalloc/src/counter.c
+    src/duckdb/extension/jemalloc/jemalloc/src/ctl.c
+    src/duckdb/extension/jemalloc/jemalloc/src/decay.c
+    src/duckdb/extension/jemalloc/jemalloc/src/div.c
+    src/duckdb/extension/jemalloc/jemalloc/src/ecache.c
+    src/duckdb/extension/jemalloc/jemalloc/src/edata.c
+    src/duckdb/extension/jemalloc/jemalloc/src/edata_cache.c
+    src/duckdb/extension/jemalloc/jemalloc/src/ehooks.c
+    src/duckdb/extension/jemalloc/jemalloc/src/emap.c
+    src/duckdb/extension/jemalloc/jemalloc/src/eset.c
+    src/duckdb/extension/jemalloc/jemalloc/src/exp_grow.c
+    src/duckdb/extension/jemalloc/jemalloc/src/extent.c
+    src/duckdb/extension/jemalloc/jemalloc/src/extent_dss.c
+    src/duckdb/extension/jemalloc/jemalloc/src/extent_mmap.c
+    src/duckdb/extension/jemalloc/jemalloc/src/fxp.c
+    src/duckdb/extension/jemalloc/jemalloc/src/san.c
+    src/duckdb/extension/jemalloc/jemalloc/src/san_bump.c
+    src/duckdb/extension/jemalloc/jemalloc/src/hook.c
+    src/duckdb/extension/jemalloc/jemalloc/src/hpa.c
+    src/duckdb/extension/jemalloc/jemalloc/src/hpa_hooks.c
+    src/duckdb/extension/jemalloc/jemalloc/src/hpdata.c
+    src/duckdb/extension/jemalloc/jemalloc/src/inspect.c
+    src/duckdb/extension/jemalloc/jemalloc/src/large.c
+    src/duckdb/extension/jemalloc/jemalloc/src/log.c
+    src/duckdb/extension/jemalloc/jemalloc/src/malloc_io.c
+    src/duckdb/extension/jemalloc/jemalloc/src/mutex.c
+    src/duckdb/extension/jemalloc/jemalloc/src/nstime.c
+    src/duckdb/extension/jemalloc/jemalloc/src/pa.c
+    src/duckdb/extension/jemalloc/jemalloc/src/pa_extra.c
+    src/duckdb/extension/jemalloc/jemalloc/src/pai.c
+    src/duckdb/extension/jemalloc/jemalloc/src/pac.c
+    src/duckdb/extension/jemalloc/jemalloc/src/pages.c
+    src/duckdb/extension/jemalloc/jemalloc/src/peak_event.c
+    src/duckdb/extension/jemalloc/jemalloc/src/prof.c
+    src/duckdb/extension/jemalloc/jemalloc/src/prof_data.c
+    src/duckdb/extension/jemalloc/jemalloc/src/prof_log.c
+    src/duckdb/extension/jemalloc/jemalloc/src/prof_recent.c
+    src/duckdb/extension/jemalloc/jemalloc/src/prof_stats.c
+    src/duckdb/extension/jemalloc/jemalloc/src/prof_sys.c
+    src/duckdb/extension/jemalloc/jemalloc/src/psset.c
+    src/duckdb/extension/jemalloc/jemalloc/src/rtree.c
+    src/duckdb/extension/jemalloc/jemalloc/src/safety_check.c
+    src/duckdb/extension/jemalloc/jemalloc/src/sc.c
+    src/duckdb/extension/jemalloc/jemalloc/src/sec.c
+    src/duckdb/extension/jemalloc/jemalloc/src/stats.c
+    src/duckdb/extension/jemalloc/jemalloc/src/sz.c
+    src/duckdb/extension/jemalloc/jemalloc/src/tcache.c
+    src/duckdb/extension/jemalloc/jemalloc/src/test_hooks.c
+    src/duckdb/extension/jemalloc/jemalloc/src/thread_event.c
+    src/duckdb/extension/jemalloc/jemalloc/src/ticker.c
+    src/duckdb/extension/jemalloc/jemalloc/src/tsd.c
+    src/duckdb/extension/jemalloc/jemalloc/src/util.c
+    src/duckdb/extension/jemalloc/jemalloc/src/witness.c
+    src/duckdb/extension/jemalloc/jemalloc/src/zone.c )
   add_definitions(/bigobj /D_DISABLE_CONSTEXPR_MUTEX_CONSTRUCTOR)
 else()
   set(CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS_RELEASE} -O3 -DNDEBUG ")
diff --git a/CMakeLists.txt.in b/CMakeLists.txt.in
index 7f6669bfb..d16640de7 100644
--- a/CMakeLists.txt.in
+++ b/CMakeLists.txt.in
@@ -32,6 +32,74 @@ set(CMAKE_JAVA_COMPILE_FLAGS -source 1.8 -target 1.8 -encoding utf-8)
 add_definitions(-DDUCKDB_BUILD_LIBRARY)
 
 if(MSVC)
+  remove_definitions(-DDUCKDB_EXTENSION_JEMALLOC_LINKED)
+  list(REMOVE_ITEM DUCKDB_SRC_FILES
+    src/duckdb/extension/jemalloc/jemalloc_extension.cpp
+    src/duckdb/extension/jemalloc/jemalloc/src/jemalloc.c
+    src/duckdb/extension/jemalloc/jemalloc/src/arena.c
+    src/duckdb/extension/jemalloc/jemalloc/src/background_thread.c
+    src/duckdb/extension/jemalloc/jemalloc/src/base.c
+    src/duckdb/extension/jemalloc/jemalloc/src/batcher.c
+    src/duckdb/extension/jemalloc/jemalloc/src/bin.c
+    src/duckdb/extension/jemalloc/jemalloc/src/bin_info.c
+    src/duckdb/extension/jemalloc/jemalloc/src/bitmap.c
+    src/duckdb/extension/jemalloc/jemalloc/src/buf_writer.c
+    src/duckdb/extension/jemalloc/jemalloc/src/cache_bin.c
+    src/duckdb/extension/jemalloc/jemalloc/src/ckh.c
+    src/duckdb/extension/jemalloc/jemalloc/src/counter.c
+    src/duckdb/extension/jemalloc/jemalloc/src/ctl.c
+    src/duckdb/extension/jemalloc/jemalloc/src/decay.c
+    src/duckdb/extension/jemalloc/jemalloc/src/div.c
+    src/duckdb/extension/jemalloc/jemalloc/src/ecache.c
+    src/duckdb/extension/jemalloc/jemalloc/src/edata.c
+    src/duckdb/extension/jemalloc/jemalloc/src/edata_cache.c
+    src/duckdb/extension/jemalloc/jemalloc/src/ehooks.c
+    src/duckdb/extension/jemalloc/jemalloc/src/emap.c
+    src/duckdb/extension/jemalloc/jemalloc/src/eset.c
+    src/duckdb/extension/jemalloc/jemalloc/src/exp_grow.c
+    src/duckdb/extension/jemalloc/jemalloc/src/extent.c
+    src/duckdb/extension/jemalloc/jemalloc/src/extent_dss.c
+    src/duckdb/extension/jemalloc/jemalloc/src/extent_mmap.c
+    src/duckdb/extension/jemalloc/jemalloc/src/fxp.c
+    src/duckdb/extension/jemalloc/jemalloc/src/san.c
+    src/duckdb/extension/jemalloc/jemalloc/src/san_bump.c
+    src/duckdb/extension/jemalloc/jemalloc/src/hook.c
+    src/duckdb/extension/jemalloc/jemalloc/src/hpa.c
+    src/duckdb/extension/jemalloc/jemalloc/src/hpa_hooks.c
+    src/duckdb/extension/jemalloc/jemalloc/src/hpdata.c
+    src/duckdb/extension/jemalloc/jemalloc/src/inspect.c
+    src/duckdb/extension/jemalloc/jemalloc/src/large.c
+    src/duckdb/extension/jemalloc/jemalloc/src/log.c
+    src/duckdb/extension/jemalloc/jemalloc/src/malloc_io.c
+    src/duckdb/extension/jemalloc/jemalloc/src/mutex.c
+    src/duckdb/extension/jemalloc/jemalloc/src/nstime.c
+    src/duckdb/extension/jemalloc/jemalloc/src/pa.c
+    src/duckdb/extension/jemalloc/jemalloc/src/pa_extra.c
+    src/duckdb/extension/jemalloc/jemalloc/src/pai.c
+    src/duckdb/extension/jemalloc/jemalloc/src/pac.c
+    src/duckdb/extension/jemalloc/jemalloc/src/pages.c
+    src/duckdb/extension/jemalloc/jemalloc/src/peak_event.c
+    src/duckdb/extension/jemalloc/jemalloc/src/prof.c
+    src/duckdb/extension/jemalloc/jemalloc/src/prof_data.c
+    src/duckdb/extension/jemalloc/jemalloc/src/prof_log.c
+    src/duckdb/extension/jemalloc/jemalloc/src/prof_recent.c
+    src/duckdb/extension/jemalloc/jemalloc/src/prof_stats.c
+    src/duckdb/extension/jemalloc/jemalloc/src/prof_sys.c
+    src/duckdb/extension/jemalloc/jemalloc/src/psset.c
+    src/duckdb/extension/jemalloc/jemalloc/src/rtree.c
+    src/duckdb/extension/jemalloc/jemalloc/src/safety_check.c
+    src/duckdb/extension/jemalloc/jemalloc/src/sc.c
+    src/duckdb/extension/jemalloc/jemalloc/src/sec.c
+    src/duckdb/extension/jemalloc/jemalloc/src/stats.c
+    src/duckdb/extension/jemalloc/jemalloc/src/sz.c
+    src/duckdb/extension/jemalloc/jemalloc/src/tcache.c
+    src/duckdb/extension/jemalloc/jemalloc/src/test_hooks.c
+    src/duckdb/extension/jemalloc/jemalloc/src/thread_event.c
+    src/duckdb/extension/jemalloc/jemalloc/src/ticker.c
+    src/duckdb/extension/jemalloc/jemalloc/src/tsd.c
+    src/duckdb/extension/jemalloc/jemalloc/src/util.c
+    src/duckdb/extension/jemalloc/jemalloc/src/witness.c
+    src/duckdb/extension/jemalloc/jemalloc/src/zone.c )
   add_definitions(/bigobj /D_DISABLE_CONSTEXPR_MUTEX_CONSTRUCTOR)
 else()
   set(CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS_RELEASE} -O3 -DNDEBUG ")


Reviews:
user: login: Mytherin
body: Thanks for the PR! Looks good - one comment
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-15T08:43:15Z
user: login: staticlibs
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-15T09:48:48Z
user: login: Mytherin
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-15T10:28:22Z

Review comments:
diff_hunk: @@ -32,6 +32,74 @@ set(CMAKE_JAVA_COMPILE_FLAGS -source 1.8 -target 1.8 -encoding utf-8)
 add_definitions(-DDUCKDB_BUILD_LIBRARY)
 
 if(MSVC)
+  remove_definitions(-DDUCKDB_EXTENSION_JEMALLOC_LINKED)
path: CMakeLists.txt
user: login: Mytherin
body: Instead of removing them can we perhaps not add them in the first place? That seems like it would be cleaner. Or is the above list of files auto-generated?
created_at: 2025-03-15T08:43:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -32,6 +32,74 @@ set(CMAKE_JAVA_COMPILE_FLAGS -source 1.8 -target 1.8 -encoding utf-8)
 add_definitions(-DDUCKDB_BUILD_LIBRARY)
 
 if(MSVC)
+  remove_definitions(-DDUCKDB_EXTENSION_JEMALLOC_LINKED)
path: CMakeLists.txt
user: login: staticlibs
body: Yes, these definitions and the sources list is auto-generated by `vendor.py`, regarding jemalloc it contains the [following logic](https://github.com/duckdb/duckdb-java/blob/a179e869e51f43fda865cc37459cae8dad533fe2/vendor.py#L25). I cannot make sense of it, because these platform checks are done when the `vendor.py` itself is being run (how is it supposed to check for Android and WASM at that time, why checking for WASM for JDBC sources etc - it was added in #133, I asked Laurens in Discord about it). I assume these checks for jemalloc need to be run at CMake configure time instead. And the list of sources needs to be added by `vendor.py` into a separate CMake variable. But for now, without touching `vendor.py` this "filter out on Windows" logic looks the easiest.
created_at: 2025-03-15T09:48:48Z
author_association: COLLABORATOR
diff_hunk: @@ -32,6 +32,74 @@ set(CMAKE_JAVA_COMPILE_FLAGS -source 1.8 -target 1.8 -encoding utf-8)
 add_definitions(-DDUCKDB_BUILD_LIBRARY)
 
 if(MSVC)
+  remove_definitions(-DDUCKDB_EXTENSION_JEMALLOC_LINKED)
path: CMakeLists.txt
user: login: Mytherin
body: That makes sense. It looks like these checks were copied from the Python build, which indeed doesn't work because the vendor script does not get run per configuration. The correct checks should be in the CMake. I would copy over the checks [from here](https://github.com/duckdb/duckdb/blob/main/extension/extension_config.cmake#L16).
created_at: 2025-03-15T10:28:22Z
author_association: CONTRIBUTOR

PR comments:
user: login: Mytherin
created_at: 2025-03-15T08:52:57Z
body: We can clean up the logic in a subsequent PR - thanks for the fix!
author_association: CONTRIBUTOR
user: login: staticlibs
created_at: 2025-03-15T09:54:00Z
body: Thanks for merging this! CI is passing now in main, I've rebased other PRs onto the latest main.
author_association: COLLABORATOR