Repository: bytecodealliance/wasmtime
Pull request #8531 (2024-05-02 22:39:29)

wasmtime: Make table lazy-init configurable
Lazy initialization of tables has trade-offs that we haven't explored in a while. Making it configurable makes it easier to test the effects of these trade-offs on a variety of WebAssembly programs, and allows embedders to decide whether the trade-offs are worth-while for their use cases.

TODO:
- [x] Finish documentation comments
- [x] Add CLI flag in the `-O` group
- [x] Either implement support in Winch, or check in `Config::validate` that when Winch is enabled so is lazy-init
- [x] Add `disas` tests with lazy init disabled
- [x] Hook up this flag in fuzzing

I've tested this by temporarily setting the flag to disable lazy-init by default (in `crates/environ/src/tunables.rs`) and verifying that the test suite still passes, except for a bunch of Winch tests (as expected), as well as the amusing case of `code_too_large::code_too_large_without_panic`, which fails because disabling lazy-init deletes enough code that the code is no longer too large.

With lazy-init disabled, the `disas` tests show that this shaves off six or seven CLIF instructions and a cold block per `call_indirect` or `table_get`. The `typed-funcrefs.wat` test becomes branch-free, with two loads and an indirect call per `call_indirect`, making it equivalent to the version that uses globals instead of tables.

cc: @cfallin

github-actions[bot]: #### Subscribe to Label Action

cc @saulecabrera

<details>
This issue or pull request has been labeled: "wasmtime:api", "wasmtime:config", "winch"

Thus the following users have been cc'd because of the following labels:

* saulecabrera: winch

To subscribe or unsubscribe from this label, edit the <code>.github/subscribe-to-label.json</code> configuration file.

[Learn more.](https://github.com/bytecodealliance/subscribe-to-label-action)
</details>
github-actions[bot]: #### Label Messager: wasmtime:config

It looks like you are changing Wasmtime's configuration options. Make sure to
complete this check list:

* [ ] If you added a new `Config` method, you wrote extensive documentation for
      it.

  <details>

  Our documentation should be of the following form:

  ```text
  Short, simple summary sentence.

  More details. These details can be multiple paragraphs. There should be
  information about not just the method, but its parameters and results as
  well.

  Is this method fallible? If so, when can it return an error?

  Can this method panic? If so, when does it panic?

  # Example

  Optional example here.
  ```

  </details>

* [ ] If you added a new `Config` method, or modified an existing one, you
  ensured that this configuration is exercised by the fuzz targets.

  <details>

  For example, if you expose a new strategy for allocating the next instance
  slot inside the pooling allocator, you should ensure that at least one of our
  fuzz targets exercises that new strategy.

  Often, all that is required of you is to ensure that there is a knob for this
  configuration option in [`wasmtime_fuzzing::Config`][fuzzing-config] (or one
  of its nested `struct`s).

  Rarely, this may require authoring a new fuzz target to specifically test this
  configuration. See [our docs on fuzzing][fuzzing-docs] for more details.

  </details>

* [ ] If you are enabling a configuration option by default, make sure that it
  has been fuzzed for at least two weeks before turning it on by default.

[fuzzing-config]: https://github.com/bytecodealliance/wasmtime/blob/ca0e8d0a1d8cefc0496dba2f77a670571d8fdcab/crates/fuzzing/src/generators.rs#L182-L194
[fuzzing-docs]: https://docs.wasmtime.dev/contributing-fuzzing.html


--------------------------------------------------------------------------------

<details>

To modify this label's message, edit the <code>.github/label-messager/wasmtime-config.md</code> file.

To add new label messages or remove existing label messages, edit the
<code>.github/label-messager.json</code> configuration file.

[Learn more.](https://github.com/bytecodealliance/label-messager-action)

</details>
jameysharp: I think I've addressed all of your suggestions, Alex; thank you for them!

I'm trying to coax the fuzzer to discover an artificial bug I added in the non-lazy-init configuration just to make sure it can, and it's been running for a while without finding it. So maybe you (or whoever reviews this) can double-check if I did something wrong in the fuzzer configuration?
jameysharp: Okay, I've learned that the `table_ops` fuzzer can't test this because it only uses `externref` tables, but lazy-init only applies to `funcref` tables. And the `single-inst` module-builder for `differential` doesn't generate tables or table instructions at all.

But I'm still not sure why `wasm-smith` isn't finding my asserts with either `differential` or `instantiate`. I tried both asserting that lazy-init is enabled, and that it's disabled, during codegen in wasmtime-cranelift, and neither one is getting hit.
jameysharp: Nevermind, `instantiate` found my assert a few minutes after I wrote that, and then found the opposite assert after another 20 minutes. And I've also once again run `cargo test --test all` with `table_lazy_init` set false, and the only tests that failed were Winch-related (now with the error message from `Config::validate`) and the `code_too_large_without_panic` test again.

So I think this is ready to land.
github-actions[bot]: #### Subscribe to Label Action

cc @fitzgen

<details>
This issue or pull request has been labeled: "fuzzing"

Thus the following users have been cc'd because of the following labels:

* fitzgen: fuzzing

To subscribe or unsubscribe from this label, edit the <code>.github/subscribe-to-label.json</code> configuration file.

[Learn more.](https://github.com/bytecodealliance/subscribe-to-label-action)
</details>

Patch:
diff --git a/Cargo.toml b/Cargo.toml
index 80e362307..9efec7e6b 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -263,7 +263,7 @@ async-trait = "0.1.71"
 heck = "0.4"
 similar = "2.1.0"
 toml = "0.8.10"
-mach2 = "0.4.1"
+mach2 = "0.4.2"
 memfd = "0.6.2"
 memoffset = "0.9.0"
 psm = "0.1.11"
diff --git a/crates/cli-flags/src/lib.rs b/crates/cli-flags/src/lib.rs
index 9f196f459..fce7d87e1 100644
--- a/crates/cli-flags/src/lib.rs
+++ b/crates/cli-flags/src/lib.rs
@@ -63,6 +63,12 @@ wasmtime_option_group! {
         /// linear memories.
         pub guard_before_linear_memory: Option<bool>,
 
+        /// Whether to initialize tables lazily, so that instantiation is
+        /// fast but indirect calls are a little slower. If no, tables are
+        /// initialized eagerly from any active element segments that apply to
+        /// them during instantiation. (default: yes)
+        pub table_lazy_init: Option<bool>,
+
         /// Enable the pooling allocator, in place of the on-demand allocator.
         pub pooling_allocator: Option<bool>,
 
@@ -548,6 +554,9 @@ impl CommonOptions {
         if let Some(enable) = self.opts.guard_before_linear_memory {
             config.guard_before_linear_memory(enable);
         }
+        if let Some(enable) = self.opts.table_lazy_init {
+            config.table_lazy_init(enable);
+        }
 
         // If fuel has been configured, set the `consume fuel` flag on the config.
         if self.wasm.fuel.is_some() {
diff --git a/crates/cranelift/src/func_environ.rs b/crates/cranelift/src/func_environ.rs
index b8319cbfe..869a9874c 100644
--- a/crates/cranelift/src/func_environ.rs
+++ b/crates/cranelift/src/func_environ.rs
@@ -817,6 +817,7 @@ impl<'module_environment> FuncEnvironment<'module_environment> {
         table_index: TableIndex,
         index: ir::Value,
         cold_blocks: bool,
+        lazy_init: bool,
     ) -> ir::Value {
         let pointer_type = self.pointer_type();
         self.ensure_table_exists(builder.func, table_index);
@@ -833,6 +834,11 @@ impl<'module_environment> FuncEnvironment<'module_environment> {
             self.isa.flags().enable_table_access_spectre_mitigation(),
         );
         let value = builder.ins().load(pointer_type, flags, table_entry_addr, 0);
+
+        if !lazy_init {
+            return value;
+        }
+
         // Mask off the "initialized bit". See documentation on
         // FUNCREF_INIT_BIT in crates/environ/src/ref_bits.rs for more
         // details. Note that `FUNCREF_MASK` has type `usize` which may not be
@@ -1354,11 +1360,14 @@ impl<'a, 'func, 'module_env> Call<'a, 'func, 'module_env> {
         cold_blocks: bool,
     ) -> WasmResult<Option<(ir::Value, ir::Value)>> {
         // Get the funcref pointer from the table.
+        let table = &self.env.module.table_plans[table_index];
+        let TableStyle::CallerChecksSignature { lazy_init } = table.style;
         let funcref_ptr = self.env.get_or_init_func_ref_table_elem(
             self.builder,
             table_index,
             callee,
             cold_blocks,
+            lazy_init,
         );
 
         // If necessary, check the signature.
@@ -1407,7 +1416,7 @@ impl<'a, 'func, 'module_env> Call<'a, 'func, 'module_env> {
 
         // Generate a rustc compile error here if more styles are added in
         // the future as the following code is tailored to just this style.
-        let TableStyle::CallerChecksSignature = table.style;
+        let TableStyle::CallerChecksSignature { .. } = table.style;
 
         // Test if a type check is necessary for this table. If this table is a
         // table of typed functions and that type matches `ty_index`, then
@@ -1752,9 +1761,14 @@ impl<'module_environment> cranelift_wasm::FuncEnvironment for FuncEnvironment<'m
 
             // Function types.
             WasmHeapTopType::Func => match plan.style {
-                TableStyle::CallerChecksSignature => {
-                    Ok(self.get_or_init_func_ref_table_elem(builder, table_index, index, false))
-                }
+                TableStyle::CallerChecksSignature { lazy_init } => Ok(self
+                    .get_or_init_func_ref_table_elem(
+                        builder,
+                        table_index,
+                        index,
+                        false,
+                        lazy_init,
+                    )),
             },
         }
     }
@@ -1805,7 +1819,7 @@ impl<'module_environment> cranelift_wasm::FuncEnvironment for FuncEnvironment<'m
             // Function types.
             WasmHeapTopType::Func => {
                 match plan.style {
-                    TableStyle::CallerChecksSignature => {
+                    TableStyle::CallerChecksSignature { lazy_init } => {
                         let (elem_addr, flags) = table_data.prepare_table_addr(
                             builder,
                             index,
@@ -1815,9 +1829,13 @@ impl<'module_environment> cranelift_wasm::FuncEnvironment for FuncEnvironment<'m
                         // Set the "initialized bit". See doc-comment on
                         // `FUNCREF_INIT_BIT` in
                         // crates/environ/src/ref_bits.rs for details.
-                        let value_with_init_bit = builder
-                            .ins()
-                            .bor_imm(value, Imm64::from(FUNCREF_INIT_BIT as i64));
+                        let value_with_init_bit = if lazy_init {
+                            builder
+                                .ins()
+                                .bor_imm(value, Imm64::from(FUNCREF_INIT_BIT as i64))
+                        } else {
+                            value
+                        };
                         builder
                             .ins()
                             .store(flags, value_with_init_bit, elem_addr, 0);
diff --git a/crates/environ/src/module.rs b/crates/environ/src/module.rs
index 9e25280ef..6e505b4f3 100644
--- a/crates/environ/src/module.rs
+++ b/crates/environ/src/module.rs
@@ -315,13 +315,19 @@ pub trait InitMemory {
 #[derive(Debug, Clone, Hash, Serialize, Deserialize)]
 pub enum TableStyle {
     /// Signatures are stored in the table and checked in the caller.
-    CallerChecksSignature,
+    CallerChecksSignature {
+        /// Whether this table is initialized lazily and requires an
+        /// initialization check on every access.
+        lazy_init: bool,
+    },
 }
 
 impl TableStyle {
     /// Decide on an implementation style for the given `Table`.
-    pub fn for_table(_table: Table, _tunables: &Tunables) -> Self {
-        Self::CallerChecksSignature
+    pub fn for_table(_table: Table, tunables: &Tunables) -> Self {
+        Self::CallerChecksSignature {
+            lazy_init: tunables.table_lazy_init,
+        }
     }
 }
 
diff --git a/crates/environ/src/tunables.rs b/crates/environ/src/tunables.rs
index 440c593e9..f945b858b 100644
--- a/crates/environ/src/tunables.rs
+++ b/crates/environ/src/tunables.rs
@@ -42,6 +42,12 @@ pub struct Tunables {
     /// beginning of the allocation in addition to the end.
     pub guard_before_linear_memory: bool,
 
+    /// Whether to initialize tables lazily, so that instantiation is fast but
+    /// indirect calls are a little slower. If false, tables are initialized
+    /// eagerly from any active element segments that apply to them during
+    /// instantiation.
+    pub table_lazy_init: bool,
+
     /// Indicates whether an address map from compiled native code back to wasm
     /// offsets in the original file is generated.
     pub generate_address_map: bool,
@@ -113,6 +119,7 @@ impl Tunables {
             epoch_interruption: false,
             static_memory_bound_is_maximum: false,
             guard_before_linear_memory: true,
+            table_lazy_init: true,
             generate_address_map: true,
             debug_adapter_modules: false,
             relaxed_simd_deterministic: false,
diff --git a/crates/fuzzing/src/generators/config.rs b/crates/fuzzing/src/generators/config.rs
index 13ba248a0..b77dfa026 100644
--- a/crates/fuzzing/src/generators/config.rs
+++ b/crates/fuzzing/src/generators/config.rs
@@ -227,6 +227,9 @@ impl Config {
             }
 
             cfg.cranelift_pcc(pcc);
+
+            // Eager init is currently only supported on Cranelift, not Winch.
+            cfg.table_lazy_init(self.wasmtime.table_lazy_init);
         }
 
         self.wasmtime.async_config.configure(&mut cfg);
@@ -483,6 +486,7 @@ pub struct WasmtimeConfig {
     cache_call_indirects: bool,
     /// The maximum number of call-indirect cache slots.
     max_call_indirect_cache_slots: usize,
+    table_lazy_init: bool,
 
     /// Whether or not fuzzing should enable PCC.
     pcc: bool,
diff --git a/crates/wasmtime/src/compile.rs b/crates/wasmtime/src/compile.rs
index 65449702d..62f910d86 100644
--- a/crates/wasmtime/src/compile.rs
+++ b/crates/wasmtime/src/compile.rs
@@ -807,7 +807,9 @@ impl FunctionIndices {
                 // Attempt to convert table initializer segments to
                 // FuncTable representation where possible, to enable
                 // table lazy init.
-                translation.try_func_table_init();
+                if engine.tunables().table_lazy_init {
+                    translation.try_func_table_init();
+                }
 
                 let funcs: PrimaryMap<DefinedFuncIndex, CompiledFunctionInfo> =
                     wasm_functions_for_module(&mut wasm_functions, module)
diff --git a/crates/wasmtime/src/config.rs b/crates/wasmtime/src/config.rs
index 8b652565d..b9d1448e8 100644
--- a/crates/wasmtime/src/config.rs
+++ b/crates/wasmtime/src/config.rs
@@ -149,6 +149,7 @@ struct ConfigTunables {
     epoch_interruption: Option<bool>,
     static_memory_bound_is_maximum: Option<bool>,
     guard_before_linear_memory: Option<bool>,
+    table_lazy_init: Option<bool>,
     generate_address_map: Option<bool>,
     debug_adapter_modules: Option<bool>,
     relaxed_simd_deterministic: Option<bool>,
@@ -1541,6 +1542,19 @@ impl Config {
         self
     }
 
+    /// Indicates whether to initialize tables lazily, so that instantiation
+    /// is fast but indirect calls are a little slower. If false, tables
+    /// are initialized eagerly during instantiation from any active element
+    /// segments that apply to them.
+    ///
+    /// ## Default
+    ///
+    /// This value defaults to `true`.
+    pub fn table_lazy_init(&mut self, table_lazy_init: bool) -> &mut Self {
+        self.tunables.table_lazy_init = Some(table_lazy_init);
+        self
+    }
+
     /// Configure the version information used in serialized and deserialzied [`crate::Module`]s.
     /// This effects the behavior of [`crate::Module::serialize()`], as well as
     /// [`crate::Module::deserialize()`] and related functions.
@@ -1810,6 +1824,7 @@ impl Config {
             epoch_interruption
             static_memory_bound_is_maximum
             guard_before_linear_memory
+            table_lazy_init
             generate_address_map
             debug_adapter_modules
             relaxed_simd_deterministic
@@ -1830,6 +1845,10 @@ impl Config {
             if tunables.winch_callable && tunables.tail_callable {
                 bail!("Winch does not support the WebAssembly tail call proposal");
             }
+
+            if tunables.winch_callable && !tunables.table_lazy_init {
+                bail!("Winch requires the table-lazy-init configuration option");
+            }
         }
 
         if tunables.static_memory_offset_guard_size < tunables.dynamic_memory_offset_guard_size {
diff --git a/crates/wasmtime/src/engine/serialization.rs b/crates/wasmtime/src/engine/serialization.rs
index 1415aa19a..e62ceb606 100644
--- a/crates/wasmtime/src/engine/serialization.rs
+++ b/crates/wasmtime/src/engine/serialization.rs
@@ -355,6 +355,7 @@ impl Metadata<'_> {
             epoch_interruption,
             static_memory_bound_is_maximum,
             guard_before_linear_memory,
+            table_lazy_init,
             relaxed_simd_deterministic,
             tail_callable,
             winch_callable,
@@ -415,6 +416,7 @@ impl Metadata<'_> {
             other.guard_before_linear_memory,
             "guard before linear memory",
         )?;
+        Self::check_bool(table_lazy_init, other.table_lazy_init, "table lazy init")?;
         Self::check_bool(
             relaxed_simd_deterministic,
             other.relaxed_simd_deterministic,
diff --git a/crates/wasmtime/src/runtime/vm/table.rs b/crates/wasmtime/src/runtime/vm/table.rs
index 8f826f310..f8e9b4124 100644
--- a/crates/wasmtime/src/runtime/vm/table.rs
+++ b/crates/wasmtime/src/runtime/vm/table.rs
@@ -14,7 +14,7 @@ use core::ptr::{self, NonNull};
 use core::slice;
 use sptr::Strict;
 use wasmtime_environ::{
-    TablePlan, Trap, WasmHeapTopType, WasmRefType, FUNCREF_INIT_BIT, FUNCREF_MASK,
+    TablePlan, TableStyle, Trap, WasmHeapTopType, WasmRefType, FUNCREF_INIT_BIT, FUNCREF_MASK,
 };
 
 /// An element going into or coming out of a table.
@@ -113,18 +113,24 @@ impl TaggedFuncRef {
 
     /// Converts the given `ptr`, a valid funcref pointer, into a tagged pointer
     /// by adding in the `FUNCREF_INIT_BIT`.
-    fn from(ptr: *mut VMFuncRef) -> Self {
-        let masked = Strict::map_addr(ptr, |a| a | FUNCREF_INIT_BIT);
-        TaggedFuncRef(masked)
+    fn from(ptr: *mut VMFuncRef, lazy_init: bool) -> Self {
+        if lazy_init {
+            let masked = Strict::map_addr(ptr, |a| a | FUNCREF_INIT_BIT);
+            TaggedFuncRef(masked)
+        } else {
+            TaggedFuncRef(ptr)
+        }
     }
 
     /// Converts a tagged pointer into a `TableElement`, returning `UninitFunc`
     /// for null (not a tagged value) or `FuncRef` for otherwise tagged values.
-    fn into_table_element(self) -> TableElement {
+    fn into_table_element(self, lazy_init: bool) -> TableElement {
         let ptr = self.0;
-        if ptr.is_null() {
+        if lazy_init && ptr.is_null() {
             TableElement::UninitFunc
         } else {
+            // Masking off the tag bit is harmless whether the table uses lazy
+            // init or not.
             let unmasked = Strict::map_addr(ptr, |a| a & FUNCREF_MASK);
             TableElement::FuncRef(unmasked)
         }
@@ -156,6 +162,8 @@ pub struct StaticFuncTable {
     data: SendSyncPtr<[FuncTableElem]>,
     /// The current size of the table.
     size: u32,
+    /// Whether elements of this table are initialized lazily.
+    lazy_init: bool,
 }
 
 pub struct StaticGcRefTable {
@@ -189,6 +197,8 @@ pub struct DynamicFuncTable {
     elements: Vec<FuncTableElem>,
     /// Maximum size that `elements` can grow to.
     maximum: Option<u32>,
+    /// Whether elements of this table are initialized lazily.
+    lazy_init: bool,
 }
 
 pub struct DynamicGcRefTable {
@@ -260,10 +270,12 @@ impl Table {
     /// Create a new dynamic (movable) table instance for the specified table plan.
     pub fn new_dynamic(plan: &TablePlan, store: &mut dyn Store) -> Result<Self> {
         Self::limit_new(plan, store)?;
+        let TableStyle::CallerChecksSignature { lazy_init } = plan.style;
         match wasm_to_table_type(plan.table.wasm_ty) {
             TableElementType::Func => Ok(Self::from(DynamicFuncTable {
                 elements: vec![None; usize::try_from(plan.table.minimum).unwrap()],
                 maximum: plan.table.maximum,
+                lazy_init,
             })),
             TableElementType::GcRef => Ok(Self::from(DynamicGcRefTable {
                 elements: (0..usize::try_from(plan.table.minimum).unwrap())
@@ -307,7 +319,12 @@ impl Table {
                     data.as_non_null().cast::<FuncTableElem>(),
                     cmp::min(len, max),
                 ));
-                Ok(Self::from(StaticFuncTable { data, size }))
+                let TableStyle::CallerChecksSignature { lazy_init } = plan.style;
+                Ok(Self::from(StaticFuncTable {
+                    data,
+                    size,
+                    lazy_init,
+                }))
             }
             TableElementType::GcRef => {
                 let len = {
@@ -405,14 +422,14 @@ impl Table {
     ) -> Result<(), Trap> {
         let dst = usize::try_from(dst).map_err(|_| Trap::TableOutOfBounds)?;
 
-        let elements = self
-            .funcrefs_mut()
+        let (funcrefs, lazy_init) = self.funcrefs_mut();
+        let elements = funcrefs
             .get_mut(dst..)
             .and_then(|s| s.get_mut(..items.len()))
             .ok_or(Trap::TableOutOfBounds)?;
 
         for (item, slot) in items.zip(elements) {
-            *slot = TaggedFuncRef::from(item);
+            *slot = TaggedFuncRef::from(item, lazy_init);
         }
         Ok(())
     }
@@ -464,7 +481,8 @@ impl Table {
 
         match val {
             TableElement::FuncRef(f) => {
-                self.funcrefs_mut()[start..end].fill(TaggedFuncRef::from(f));
+                let (funcrefs, lazy_init) = self.funcrefs_mut();
+                funcrefs[start..end].fill(TaggedFuncRef::from(f, lazy_init));
             }
             TableElement::GcRef(r) => {
                 // Clone the init GC reference into each table slot.
@@ -479,7 +497,8 @@ impl Table {
                 }
             }
             TableElement::UninitFunc => {
-                self.funcrefs_mut()[start..end].fill(TaggedFuncRef::UNINIT);
+                let (funcrefs, _lazy_init) = self.funcrefs_mut();
+                funcrefs[start..end].fill(TaggedFuncRef::UNINIT);
             }
         }
 
@@ -546,7 +565,7 @@ impl Table {
 
         // First resize the storage and then fill with the init value
         match self {
-            Table::Static(StaticTable::Func(StaticFuncTable { data, size })) => {
+            Table::Static(StaticTable::Func(StaticFuncTable { data, size, .. })) => {
                 unsafe {
                     debug_assert!(data.as_ref()[*size as usize..new_size as usize]
                         .iter()
@@ -590,11 +609,13 @@ impl Table {
     pub fn get(&self, gc_store: &mut GcStore, index: u32) -> Option<TableElement> {
         let index = usize::try_from(index).ok()?;
         match self.element_type() {
-            TableElementType::Func => self
-                .funcrefs()
-                .get(index)
-                .copied()
-                .map(|e| e.into_table_element()),
+            TableElementType::Func => {
+                let (funcrefs, lazy_init) = self.funcrefs();
+                funcrefs
+                    .get(index)
+                    .copied()
+                    .map(|e| e.into_table_element(lazy_init))
+            }
             TableElementType::GcRef => self.gc_refs().get(index).map(|r| {
                 let r = r.as_ref().map(|r| gc_store.clone_gc_ref(r));
                 TableElement::GcRef(r)
@@ -616,10 +637,12 @@ impl Table {
         let index = usize::try_from(index).map_err(|_| ())?;
         match elem {
             TableElement::FuncRef(f) => {
-                *self.funcrefs_mut().get_mut(index).ok_or(())? = TaggedFuncRef::from(f);
+                let (funcrefs, lazy_init) = self.funcrefs_mut();
+                *funcrefs.get_mut(index).ok_or(())? = TaggedFuncRef::from(f, lazy_init);
             }
             TableElement::UninitFunc => {
-                *self.funcrefs_mut().get_mut(index).ok_or(())? = TaggedFuncRef::UNINIT;
+                let (funcrefs, _lazy_init) = self.funcrefs_mut();
+                *funcrefs.get_mut(index).ok_or(())? = TaggedFuncRef::UNINIT;
             }
             TableElement::GcRef(e) => {
                 *self.gc_refs_mut().get_mut(index).ok_or(())? = e;
@@ -675,10 +698,12 @@ impl Table {
     /// Return a `VMTableDefinition` for exposing the table to compiled wasm code.
     pub fn vmtable(&mut self) -> VMTableDefinition {
         match self {
-            Table::Static(StaticTable::Func(StaticFuncTable { data, size })) => VMTableDefinition {
-                base: data.as_ptr().cast(),
-                current_elements: *size,
-            },
+            Table::Static(StaticTable::Func(StaticFuncTable { data, size, .. })) => {
+                VMTableDefinition {
+                    base: data.as_ptr().cast(),
+                    current_elements: *size,
+                }
+            }
             Table::Static(StaticTable::GcRef(StaticGcRefTable { data, size })) => {
                 VMTableDefinition {
                     base: data.as_ptr().cast(),
@@ -704,28 +729,52 @@ impl Table {
         self.element_type().matches(val)
     }
 
-    fn funcrefs(&self) -> &[TaggedFuncRef] {
+    fn funcrefs(&self) -> (&[TaggedFuncRef], bool) {
         assert_eq!(self.element_type(), TableElementType::Func);
         match self {
-            Self::Dynamic(DynamicTable::Func(DynamicFuncTable { elements, .. })) => unsafe {
-                slice::from_raw_parts(elements.as_ptr().cast(), elements.len())
-            },
-            Self::Static(StaticTable::Func(StaticFuncTable { data, size })) => unsafe {
-                slice::from_raw_parts(data.as_ptr().cast(), usize::try_from(*size).unwrap())
-            },
+            Self::Dynamic(DynamicTable::Func(DynamicFuncTable {
+                elements,
+                lazy_init,
+                ..
+            })) => (
+                unsafe { slice::from_raw_parts(elements.as_ptr().cast(), elements.len()) },
+                *lazy_init,
+            ),
+            Self::Static(StaticTable::Func(StaticFuncTable {
+                data,
+                size,
+                lazy_init,
+            })) => (
+                unsafe {
+                    slice::from_raw_parts(data.as_ptr().cast(), usize::try_from(*size).unwrap())
+                },
+                *lazy_init,
+            ),
             _ => unreachable!(),
         }
     }
 
-    fn funcrefs_mut(&mut self) -> &mut [TaggedFuncRef] {
+    fn funcrefs_mut(&mut self) -> (&mut [TaggedFuncRef], bool) {
         assert_eq!(self.element_type(), TableElementType::Func);
         match self {
-            Self::Dynamic(DynamicTable::Func(DynamicFuncTable { elements, .. })) => unsafe {
-                slice::from_raw_parts_mut(elements.as_mut_ptr().cast(), elements.len())
-            },
-            Self::Static(StaticTable::Func(StaticFuncTable { data, size })) => unsafe {
-                slice::from_raw_parts_mut(data.as_ptr().cast(), usize::try_from(*size).unwrap())
-            },
+            Self::Dynamic(DynamicTable::Func(DynamicFuncTable {
+                elements,
+                lazy_init,
+                ..
+            })) => (
+                unsafe { slice::from_raw_parts_mut(elements.as_mut_ptr().cast(), elements.len()) },
+                *lazy_init,
+            ),
+            Self::Static(StaticTable::Func(StaticFuncTable {
+                data,
+                size,
+                lazy_init,
+            })) => (
+                unsafe {
+                    slice::from_raw_parts_mut(data.as_ptr().cast(), usize::try_from(*size).unwrap())
+                },
+                *lazy_init,
+            ),
             _ => unreachable!(),
         }
     }
@@ -770,8 +819,9 @@ impl Table {
         match ty {
             TableElementType::Func => {
                 // `funcref` are `Copy`, so just do a mempcy
-                dst_table.funcrefs_mut()[dst_range]
-                    .copy_from_slice(&src_table.funcrefs()[src_range]);
+                let (dst_funcrefs, _lazy_init) = dst_table.funcrefs_mut();
+                let (src_funcrefs, _lazy_init) = src_table.funcrefs();
+                dst_funcrefs[dst_range].copy_from_slice(&src_funcrefs[src_range]);
             }
             TableElementType::GcRef => {
                 assert_eq!(
@@ -810,7 +860,8 @@ impl Table {
         match ty {
             TableElementType::Func => {
                 // `funcref` are `Copy`, so just do a memmove
-                self.funcrefs_mut().copy_within(src_range, dst_range.start);
+                let (funcrefs, _lazy_init) = self.funcrefs_mut();
+                funcrefs.copy_within(src_range, dst_range.start);
             }
             TableElementType::GcRef => {
                 // We need to clone each `externref` while handling overlapping
@@ -842,6 +893,7 @@ impl Default for Table {
         Self::from(StaticFuncTable {
             data: SendSyncPtr::new(NonNull::from(&mut [])),
             size: 0,
+            lazy_init: false,
         })
     }
 }
diff --git a/src/commands/serve.rs b/src/commands/serve.rs
index e50e5228b..37ab377d7 100644
--- a/src/commands/serve.rs
+++ b/src/commands/serve.rs
@@ -10,7 +10,7 @@ use std::{
     },
 };
 use wasmtime::component::{InstancePre, Linker};
-use wasmtime::{Engine, Store, StoreLimits};
+use wasmtime::{Config, Engine, Memory, MemoryType, Store, StoreLimits};
 use wasmtime_wasi::{StreamError, StreamResult, WasiCtx, WasiCtxBuilder, WasiView};
 use wasmtime_wasi_http::io::TokioIo;
 use wasmtime_wasi_http::{
@@ -247,12 +247,10 @@ impl ServeCommand {
     async fn serve(mut self) -> Result<()> {
         use hyper::server::conn::http1;
 
-        // If not explicitly specified then always enable the pooling allocator.
-        // The `wasmtime serve` use case is tailor-made for pooling allocation
-        // and there's not much downside to enabling it.
-        let pooling_allocator_default = Some(true);
-
-        let mut config = self.run.common.config(None, pooling_allocator_default)?;
+        let mut config = self
+            .run
+            .common
+            .config(None, use_pooling_allocator_by_default().unwrap_or(None))?;
         config.wasm_component_model(true);
         config.async_support(true);
 
@@ -555,3 +553,41 @@ impl wasmtime_wasi::HostOutputStream for LogStream {
 impl wasmtime_wasi::Subscribe for LogStream {
     async fn ready(&mut self) {}
 }
+
+/// The pooling allocator is tailor made for the `wasmtime serve` use case, so
+/// try to use it when we can. The main cost of the pooling allocator, however,
+/// is the virtual memory required to run it. Not all systems support the same
+/// amount of virtual memory, for example some aarch64 and riscv64 configuration
+/// only support 39 bits of virtual address space.
+///
+/// The pooling allocator, by default, will request 1000 linear memories each
+/// sized at 6G per linear memory. This is 6T of virtual memory which ends up
+/// being about 42 bits of the address space. This exceeds the 39 bit limit of
+/// some systems, so there the pooling allocator will fail by default.
+///
+/// This function attempts to dynamically determine the hint for the pooling
+/// allocator. This returns `Some(true)` if the pooling allocator should be used
+/// by default, or `None` or an error otherwise.
+///
+/// The method for testing this is to allocate a 0-sized 64-bit linear memory
+/// with a maximum size that's N bits large where we force all memories to be
+/// static. This should attempt to acquire N bits of the virtual address space.
+/// If successful that should mean that the pooling allocator is OK to use, but
+/// if it fails then the pooling allocator is not used and the normal mmap-based
+/// implementation is used instead.
+fn use_pooling_allocator_by_default() -> Result<Option<bool>> {
+    const BITS_TO_TEST: u32 = 42;
+    let mut config = Config::new();
+    config.wasm_memory64(true);
+    config.static_memory_maximum_size(1 << BITS_TO_TEST);
+    let engine = Engine::new(&config)?;
+    let mut store = Store::new(&engine, ());
+    // NB: the maximum size is in wasm pages to take out the 16-bits of wasm
+    // page size here from the maximum size.
+    let ty = MemoryType::new64(0, Some(1 << (BITS_TO_TEST - 16)));
+    if Memory::new(&mut store, ty).is_ok() {
+        Ok(Some(true))
+    } else {
+        Ok(None)
+    }
+}
diff --git a/winch/codegen/src/visitor.rs b/winch/codegen/src/visitor.rs
index 88fa594c3..75c514748 100644
--- a/winch/codegen/src/visitor.rs
+++ b/winch/codegen/src/visitor.rs
@@ -1375,12 +1375,13 @@ where
         // This code assumes that [`Self::emit_lazy_init_funcref`] will
         // push the funcref to the value stack.
         match self.env.translation.module.table_plans[table_index].style {
-            TableStyle::CallerChecksSignature => {
+            TableStyle::CallerChecksSignature { lazy_init: true } => {
                 let funcref_ptr = self.context.stack.peek().map(|v| v.unwrap_reg()).unwrap();
                 self.masm
                     .trapz(funcref_ptr.into(), TrapCode::IndirectCallToNull);
                 self.emit_typecheck_funcref(funcref_ptr.into(), type_index);
             }
+            _ => unimplemented!("Support for eager table init"),
         }
 
         let callee = self.env.funcref(type_index);
@@ -1428,7 +1429,10 @@ where
 
         match heap_type {
             WasmHeapType::Func => match style {
-                TableStyle::CallerChecksSignature => self.emit_lazy_init_funcref(table_index),
+                TableStyle::CallerChecksSignature { lazy_init: true } => {
+                    self.emit_lazy_init_funcref(table_index)
+                }
+                _ => unimplemented!("Support for eager table init"),
             },
             t => unimplemented!("Support for WasmHeapType: {t}"),
         }
@@ -1501,7 +1505,7 @@ where
         let plan = self.env.table_plan(table_index);
         match plan.table.wasm_ty.heap_type {
             WasmHeapType::Func => match plan.style {
-                TableStyle::CallerChecksSignature => {
+                TableStyle::CallerChecksSignature { lazy_init: true } => {
                     let value = self.context.pop_to_reg(self.masm, None);
                     let index = self.context.pop_to_reg(self.masm, None);
                     let base = self.context.any_gpr(self.masm);
@@ -1521,6 +1525,7 @@ where
                     self.context.free_reg(index);
                     self.context.free_reg(base);
                 }
+                _ => unimplemented!("Support for eager table init"),
             },
             ty => unimplemented!("Support for WasmHeapType: {ty}"),
         };


Test patch:
diff --git a/tests/disas/typed-funcrefs-eager-init.wat b/tests/disas/typed-funcrefs-eager-init.wat
new file mode 100644
index 000000000..dc044749c
--- /dev/null
+++ b/tests/disas/typed-funcrefs-eager-init.wat
@@ -0,0 +1,211 @@
+;;! target = "x86_64"
+;;! test = "optimize"
+;;! flags = [ "-Wfunction-references=y", "-Otable-lazy-init=n" ]
+
+;; This test is meant to simulate how typed funcrefs in a table may be
+;; used for ICs (inline caches) in a Wasm module compiled from a dynamic
+;; language. In native JIT engines, IC chains have head pointers that
+;; are raw code pointers and IC-using code can call each with a few ops
+;; (load pointer, call indirect). We'd like similar efficiency by
+;; storing funcrefs for the first IC in each chain in a typed-funcref
+;; table.
+
+(module
+  (type $ic-stub (func (param i32 i32 i32 i32) (result i32)))
+
+  ;; This syntax declares a table that is exactly 100 elements, whose
+  ;; elements are nullable function references, and whose default
+  ;; value is `null`.
+  (table $ic-sites 100 100 (ref null $ic-stub))
+
+  (func $ic1 (param i32 i32 i32 i32) (result i32)
+        local.get 0)
+
+  ;; A function which uses ICs through `table.get` plus `call_ref`
+  (func $call-ics-with-call-ref (param i32 i32 i32 i32) (result i32)
+        (local $sum i32)
+
+        ;; IC callsite index 1 (arbitrary).
+        local.get 0
+        local.get 1
+        local.get 2
+        local.get 3
+        i32.const 1
+        table.get $ic-sites
+        call_ref $ic-stub
+        local.get $sum
+        i32.add
+        local.set $sum
+
+        ;; IC callsite index 2 (arbitrary).
+        local.get 0
+        local.get 1
+        local.get 2
+        local.get 3
+        i32.const 2
+        table.get $ic-sites
+        call_ref $ic-stub
+        local.get $sum
+        i32.add
+        local.set $sum
+
+        local.get $sum)
+
+  ;; Same as the above function, but uses `call_indirect` rather than
+  ;; `call_ref`.
+  (func $call-ics-with-call-indirect (param i32 i32 i32 i32) (result i32)
+        (local $sum i32)
+
+        ;; IC callsite index 1 (arbitrary).
+        local.get 0
+        local.get 1
+        local.get 2
+        local.get 3
+        i32.const 1
+        call_indirect $ic-sites (type $ic-stub)
+        local.get $sum
+        i32.add
+        local.set $sum
+
+        ;; IC callsite index 2 (arbitrary).
+        local.get 0
+        local.get 1
+        local.get 2
+        local.get 3
+        i32.const 2
+        call_indirect $ic-sites (type $ic-stub)
+        local.get $sum
+        i32.add
+        local.set $sum
+
+        local.get $sum)
+
+  (global $ic-site0 (mut (ref $ic-stub)) (ref.func $ic1))
+  (global $ic-site1 (mut (ref $ic-stub)) (ref.func $ic1))
+
+  ;; Sort of similar to the previous two functions, but uses globals instead of
+  ;; tables to store ICs. Mostly just here for comparison in terms of codegen.
+  (func $call-ics-with-global-get (param i32 i32 i32 i32) (result i32)
+        (local $sum i32)
+
+        ;; IC callsite index 1 (arbitrary).
+        local.get 0
+        local.get 1
+        local.get 2
+        local.get 3
+        global.get $ic-site0
+        call_ref $ic-stub
+        local.get $sum
+        i32.add
+        local.set $sum
+
+        ;; IC callsite index 2 (arbitrary).
+        local.get 0
+        local.get 1
+        local.get 2
+        local.get 3
+        global.get $ic-site1
+        call_ref $ic-stub
+        local.get $sum
+        i32.add
+        local.set $sum
+
+        local.get $sum)
+)
+
+;; function u0:0(i64 vmctx, i64, i32, i32, i32, i32) -> i32 fast {
+;;     gv0 = vmctx
+;;     gv1 = load.i64 notrap aligned readonly gv0+8
+;;     gv2 = load.i64 notrap aligned gv1
+;;     stack_limit = gv2
+;;
+;;                                 block0(v0: i64, v1: i64, v2: i32, v3: i32, v4: i32, v5: i32):
+;; @0039                               jump block1
+;;
+;;                                 block1:
+;; @0039                               return v2
+;; }
+;;
+;; function u0:1(i64 vmctx, i64, i32, i32, i32, i32) -> i32 fast {
+;;     gv0 = vmctx
+;;     gv1 = load.i64 notrap aligned readonly gv0+8
+;;     gv2 = load.i64 notrap aligned gv1
+;;     gv3 = vmctx
+;;     gv4 = load.i64 notrap aligned readonly gv3+88
+;;     sig0 = (i64 vmctx, i64, i32, i32, i32, i32) -> i32 fast
+;;     stack_limit = gv2
+;;
+;;                                 block0(v0: i64, v1: i64, v2: i32, v3: i32, v4: i32, v5: i32):
+;; @0048                               v12 = load.i64 notrap aligned readonly v0+88
+;;                                     v48 = iconst.i64 8
+;; @0048                               v14 = iadd v12, v48  ; v48 = 8
+;; @0048                               v17 = load.i64 table_oob aligned table v14
+;; @004a                               v18 = load.i64 null_reference aligned readonly v17+16
+;; @004a                               v19 = load.i64 notrap aligned readonly v17+32
+;; @004a                               v20 = call_indirect sig0, v18(v19, v0, v2, v3, v4, v5)
+;;                                     v56 = iconst.i64 16
+;; @005b                               v28 = iadd v12, v56  ; v56 = 16
+;; @005b                               v31 = load.i64 table_oob aligned table v28
+;; @005d                               v32 = load.i64 null_reference aligned readonly v31+16
+;; @005d                               v33 = load.i64 notrap aligned readonly v31+32
+;; @005d                               v34 = call_indirect sig0, v32(v33, v0, v2, v3, v4, v5)
+;; @0066                               jump block1
+;;
+;;                                 block1:
+;; @0061                               v35 = iadd.i32 v34, v20
+;; @0066                               return v35
+;; }
+;;
+;; function u0:2(i64 vmctx, i64, i32, i32, i32, i32) -> i32 fast {
+;;     gv0 = vmctx
+;;     gv1 = load.i64 notrap aligned readonly gv0+8
+;;     gv2 = load.i64 notrap aligned gv1
+;;     gv3 = vmctx
+;;     gv4 = load.i64 notrap aligned readonly gv3+88
+;;     sig0 = (i64 vmctx, i64, i32, i32, i32, i32) -> i32 fast
+;;     stack_limit = gv2
+;;
+;;                                 block0(v0: i64, v1: i64, v2: i32, v3: i32, v4: i32, v5: i32):
+;; @0075                               v12 = load.i64 notrap aligned readonly v0+88
+;;                                     v48 = iconst.i64 8
+;; @0075                               v14 = iadd v12, v48  ; v48 = 8
+;; @0075                               v17 = load.i64 table_oob aligned table v14
+;; @0075                               v18 = load.i64 icall_null aligned readonly v17+16
+;; @0075                               v19 = load.i64 notrap aligned readonly v17+32
+;; @0075                               v20 = call_indirect sig0, v18(v19, v0, v2, v3, v4, v5)
+;;                                     v56 = iconst.i64 16
+;; @0087                               v28 = iadd v12, v56  ; v56 = 16
+;; @0087                               v31 = load.i64 table_oob aligned table v28
+;; @0087                               v32 = load.i64 icall_null aligned readonly v31+16
+;; @0087                               v33 = load.i64 notrap aligned readonly v31+32
+;; @0087                               v34 = call_indirect sig0, v32(v33, v0, v2, v3, v4, v5)
+;; @0091                               jump block1
+;;
+;;                                 block1:
+;; @008c                               v35 = iadd.i32 v34, v20
+;; @0091                               return v35
+;; }
+;;
+;; function u0:3(i64 vmctx, i64, i32, i32, i32, i32) -> i32 fast {
+;;     gv0 = vmctx
+;;     gv1 = load.i64 notrap aligned readonly gv0+8
+;;     gv2 = load.i64 notrap aligned gv1
+;;     gv3 = vmctx
+;;     sig0 = (i64 vmctx, i64, i32, i32, i32, i32) -> i32 fast
+;;     stack_limit = gv2
+;;
+;;                                 block0(v0: i64, v1: i64, v2: i32, v3: i32, v4: i32, v5: i32):
+;; @009e                               v9 = load.i64 notrap aligned table v0+112
+;; @00a0                               v10 = load.i64 null_reference aligned readonly v9+16
+;; @00a0                               v11 = load.i64 notrap aligned readonly v9+32
+;; @00a0                               v12 = call_indirect sig0, v10(v11, v0, v2, v3, v4, v5)
+;; @00af                               v15 = load.i64 notrap aligned table v0+128
+;; @00b1                               v16 = load.i64 null_reference aligned readonly v15+16
+;; @00b1                               v17 = load.i64 notrap aligned readonly v15+32
+;; @00b1                               v18 = call_indirect sig0, v16(v17, v0, v2, v3, v4, v5)
+;; @00ba                               jump block1
+;;
+;;                                 block1:
+;; @00b5                               v19 = iadd.i32 v18, v12
+;; @00ba                               return v19
+;; }
diff --git a/tests/disas/typed-funcrefs.wat b/tests/disas/typed-funcrefs.wat
index a9567d539..7877c32dc 100644
--- a/tests/disas/typed-funcrefs.wat
+++ b/tests/disas/typed-funcrefs.wat
@@ -1,6 +1,6 @@
 ;;! target = "x86_64"
 ;;! test = "optimize"
-;;! flags = [ "-Wfunction-references=y" ]
+;;! flags = [ "-Wfunction-references=y", "-Otable-lazy-init=y" ]
 
 ;; This test is meant to simulate how typed funcrefs in a table may be
 ;; used for ICs (inline caches) in a Wasm module compiled from a dynamic