Repository: JuliaLang/PrecompileTools.jl
Pull request #53: Ensure compilation happens after logging starts
Author: login: timholy
Created: 2025-04-02T09:22:21Z | Merged: 2025-04-03T07:08:21Z

If compilation of the workload happens before the logging system is
initialized, many CodeInstances may be missed. This exploits
`Core.@latestworld` to prevent inference from proceeding into the
workload. It removes the `@force_compile`, which introduces
some risk that even the workload will be interpreted.

Also wraps the `@setup_workload` block with a `let`.

Fixes #17
Fixes #26
Fixes #37
Closes #40
Closes https://github.com/JuliaLang/julia/issues/57957

CC @vtjnash

Linked issue #17: Understand why `let` wrapping the compile workload can lead to worse latency
This was observed in https://github.com/JuliaLang/PrecompileTools.jl/pull/16. It would be good to understand the interaction with `let` here since ideally, it would be good to keep it.


Linked issue comments: user: login: timholy
created_at: 2023-05-08T17:33:04Z
body: I suspect it changes the interpreter/compiler heuristics in `src/toplevel.c` but I haven't looked in detail. And yes, ideally it would be best to be able to use `let`.
author_association: MEMBER
user: login: adknudson
created_at: 2024-03-18T06:07:26Z
body: Is this issue related to why symbols in the `@setup_workload` block are visible in the package namespace? I.e. if I have the following:

```julia
module MyPackage

using PrecompileTools: @setup_workload, @compile_workload

@setup_workload begin
    # these become visible in the namespace
    some_variable = rand(4, 4)
    b = rand(4)
    temp = :U

    @compile_workload begin
        some_variable * b
        some_func(temp)
    end
end

end
```

The variables `some_variable`, `b`, and `temp` are visible if a user types in `MyPackage.[TAB]`. I could wrap the contents of `@setup_workload` in a let-block and then the setup variables would no longer be in the namespace, but that feels wrong.
author_association: NONE
user: login: timholy
created_at: 2025-03-31T11:05:15Z
body: Yes, exactly. We tried to have `@setup_workload` do that `let`-wrapping for you, but it had side effects on the ordering of execution and compilation. If compilation of the workload happens before `@compile_workload` "tagging" gets turned on, it defeats the purpose of `@compile_workload`.
author_association: MEMBER

Diff:
diff --git a/.github/workflows/CI.yml b/.github/workflows/CI.yml
index 9120914..6e90189 100644
--- a/.github/workflows/CI.yml
+++ b/.github/workflows/CI.yml
@@ -33,6 +33,7 @@ jobs:
       - uses: julia-actions/cache@v2
       - uses: julia-actions/julia-buildpkg@v1
       - uses: julia-actions/julia-runtest@v1
+      - run: julia --code-coverage=none --startup=no --project test/without_coverage.jl
       - uses: julia-actions/julia-processcoverage@v1
       - uses: codecov/codecov-action@v5
         with:
diff --git a/.gitignore b/.gitignore
index 20fe29d..b5ac9c8 100644
--- a/.gitignore
+++ b/.gitignore
@@ -3,3 +3,4 @@
 *.jl.mem
 /Manifest.toml
 /docs/build/
+.vscode/
\ No newline at end of file
diff --git a/Project.toml b/Project.toml
index 34a5224..89882f7 100644
--- a/Project.toml
+++ b/Project.toml
@@ -1,19 +1,21 @@
 name = "PrecompileTools"
 uuid = "aea7be01-6a6a-4083-8856-8a6e6704d82a"
-authors = ["Tim Holy <tim.holy@gmail.com>", "t-bltg <tf.bltg@gmail.com>", "and contributors"]
 version = "1.3.0"
+authors = ["Tim Holy <tim.holy@gmail.com>", "t-bltg <tf.bltg@gmail.com>", "and contributors"]
 
 [deps]
 Preferences = "21216c6a-2e73-6563-6e65-726566657250"
 
 [compat]
+InteractiveUtils = "1.11.0"
 Preferences = "1"
 julia = "1.12"
 
 [extras]
+InteractiveUtils = "b77e0a4c-d291-57a0-90e8-8db25a27a240"
 Pkg = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"
 Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
 UUIDs = "cf7118a7-6976-5b1a-9a39-7adc72f591a4"
 
 [targets]
-test = ["Pkg", "Test", "UUIDs"]
+test = ["InteractiveUtils", "Pkg", "Test", "UUIDs"]
diff --git a/src/workloads.jl b/src/workloads.jl
index 806f1f9..0cc669e 100644
--- a/src/workloads.jl
+++ b/src/workloads.jl
@@ -12,6 +12,8 @@ function workload_enabled(mod::Module)
     end
 end
 
+@noinline is_generating_output() = ccall(:jl_generating_output, Cint, ()) == 1
+
 function tag_newly_inferred_enable()
     ccall(:jl_tag_newly_inferred_enable, Cvoid, ())
     if !Base.generating_output()   # for verbose[]
@@ -60,10 +62,10 @@ end
         - indirect runtime-dispatched calls to such methods.
 """
 macro compile_workload(ex::Expr)
-    local iscompiling = :((ccall(:jl_generating_output, Cint, ()) == 1 && $PrecompileTools.workload_enabled(@__MODULE__)))
+    local iscompiling = :($PrecompileTools.is_generating_output() && $PrecompileTools.workload_enabled(@__MODULE__))
     ex = quote
         begin
-            Base.Experimental.@force_compile
+            Core.@latestworld  # block inference from proceeding beyond this point (xref https://github.com/JuliaLang/julia/issues/57957)
             $(esc(ex))
         end
     end
@@ -110,7 +112,9 @@ macro setup_workload(ex::Expr)
     # trigger inference & codegen in undesirable ways (see #16).
     return quote
         if $iscompiling || $PrecompileTools.verbose[]
-            $(esc(ex))
+            let
+                $(esc(ex))
+            end
         end
     end
 end
diff --git a/test/AliasTables/Manifest.toml b/test/AliasTables/Manifest.toml
new file mode 100644
index 0000000..6065c23
--- /dev/null
+++ b/test/AliasTables/Manifest.toml
@@ -0,0 +1,51 @@
+# This file is machine-generated - editing it directly is not advised
+
+julia_version = "1.12.0-DEV"
+manifest_format = "2.0"
+project_hash = "08b5cd69f1c3c631130172e148f3d7bf0afe42df"
+
+[[deps.AliasTables]]
+deps = ["PrecompileTools"]
+path = "."
+uuid = "beccad7b-8a6a-4466-a02c-5791472d17ea"
+version = "0.1.0"
+
+[[deps.Dates]]
+deps = ["Printf"]
+uuid = "ade2ca70-3891-5945-98fb-dc099432e06a"
+version = "1.11.0"
+
+[[deps.PrecompileTools]]
+deps = ["Preferences"]
+path = "../.."
+uuid = "aea7be01-6a6a-4083-8856-8a6e6704d82a"
+version = "1.3.0"
+
+[[deps.Preferences]]
+deps = ["TOML"]
+git-tree-sha1 = "9306f6085165d270f7e3db02af26a400d580f5c6"
+uuid = "21216c6a-2e73-6563-6e65-726566657250"
+version = "1.4.3"
+
+[[deps.Printf]]
+deps = ["Unicode"]
+uuid = "de0858da-6303-5e67-8744-51eddeeeb8d7"
+version = "1.11.0"
+
+[[deps.Random]]
+deps = ["SHA"]
+uuid = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
+version = "1.11.0"
+
+[[deps.SHA]]
+uuid = "ea8e919c-243c-51af-8825-aaa63cd721ce"
+version = "0.7.0"
+
+[[deps.TOML]]
+deps = ["Dates"]
+uuid = "fa267f1f-6049-4f14-aa54-33bafae1ed76"
+version = "1.0.3"
+
+[[deps.Unicode]]
+uuid = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5"
+version = "1.11.0"
diff --git a/test/AliasTables/Project.toml b/test/AliasTables/Project.toml
new file mode 100644
index 0000000..6eeaad7
--- /dev/null
+++ b/test/AliasTables/Project.toml
@@ -0,0 +1,11 @@
+name = "AliasTables"
+uuid = "beccad7b-8a6a-4466-a02c-5791472d17ea"
+version = "0.1.0"
+authors = ["Tim Holy <tim.holy@gmail.com>"]
+
+[deps]
+PrecompileTools = "aea7be01-6a6a-4083-8856-8a6e6704d82a"
+Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
+
+[compat]
+Random = "1.11.0"
diff --git a/test/AliasTables/src/AliasTables.jl b/test/AliasTables/src/AliasTables.jl
new file mode 100644
index 0000000..9060ecb
--- /dev/null
+++ b/test/AliasTables/src/AliasTables.jl
@@ -0,0 +1,18 @@
+module AliasTables
+
+using Random
+using PrecompileTools
+
+export AliasTable
+
+struct AliasTable{T}
+    x::T
+end
+Base.rand(x::AliasTable) = rand(x.x)
+
+PrecompileTools.@compile_workload begin
+    at = AliasTable([1.0, 2.0])
+    rand(at)
+end
+
+end
diff --git a/test/MSort/Manifest.toml b/test/MSort/Manifest.toml
new file mode 100644
index 0000000..f613e09
--- /dev/null
+++ b/test/MSort/Manifest.toml
@@ -0,0 +1,42 @@
+# This file is machine-generated - editing it directly is not advised
+
+julia_version = "1.12.0-DEV"
+manifest_format = "2.0"
+project_hash = "54196cb61d86afd465157edc4da76dcf0e248920"
+
+[[deps.Dates]]
+deps = ["Printf"]
+uuid = "ade2ca70-3891-5945-98fb-dc099432e06a"
+version = "1.11.0"
+
+[[deps.MSort]]
+deps = ["PrecompileTools"]
+path = "."
+uuid = "75c9a4e7-9bfb-42c4-89fa-37abaaaaefd3"
+version = "0.1.0"
+
+[[deps.PrecompileTools]]
+deps = ["Preferences"]
+path = "../.."
+uuid = "aea7be01-6a6a-4083-8856-8a6e6704d82a"
+version = "1.3.0"
+
+[[deps.Preferences]]
+deps = ["TOML"]
+git-tree-sha1 = "9306f6085165d270f7e3db02af26a400d580f5c6"
+uuid = "21216c6a-2e73-6563-6e65-726566657250"
+version = "1.4.3"
+
+[[deps.Printf]]
+deps = ["Unicode"]
+uuid = "de0858da-6303-5e67-8744-51eddeeeb8d7"
+version = "1.11.0"
+
+[[deps.TOML]]
+deps = ["Dates"]
+uuid = "fa267f1f-6049-4f14-aa54-33bafae1ed76"
+version = "1.0.3"
+
+[[deps.Unicode]]
+uuid = "4ec0a83e-493e-50e2-b9ac-8f72acf5a8f5"
+version = "1.11.0"
diff --git a/test/MSort/Project.toml b/test/MSort/Project.toml
new file mode 100644
index 0000000..1b085bd
--- /dev/null
+++ b/test/MSort/Project.toml
@@ -0,0 +1,7 @@
+name = "MSort"
+uuid = "75c9a4e7-9bfb-42c4-89fa-37abaaaaefd3"
+version = "0.1.0"
+authors = ["Tim Holy <tim.holy@gmail.com>"]
+
+[deps]
+PrecompileTools = "aea7be01-6a6a-4083-8856-8a6e6704d82a"
diff --git a/test/MSort/src/MSort.jl b/test/MSort/src/MSort.jl
new file mode 100644
index 0000000..fccb960
--- /dev/null
+++ b/test/MSort/src/MSort.jl
@@ -0,0 +1,57 @@
+module MSort
+
+# issue #26
+
+using PrecompileTools
+
+function quicksort(
+    v::Vector{T};
+    lo::Int = 1,
+    hi::Int = length(v),
+) where {T <: Union{Int64, Float64}}
+    x = copy(v)
+    quick!(x, lo, hi)
+    x
+end
+
+function partition(
+    xp::Vector{T},
+    pivot::T,
+    left::Int,
+    right::Int,
+) where {T <: Union{Int64, Float64}}
+    while left <= right
+        while xp[left] < pivot
+            left += 1
+        end
+        while pivot < xp[right]
+            right -= 1
+        end
+        if left <= right
+            xp[left], xp[right] = xp[right], xp[left]
+            left += 1
+            right -= 1
+        end
+    end
+    left, right
+end
+
+function quick!(
+    xp::Vector{T},
+    i::Int,
+    j::Int,
+) where {T <: Union{Int64, Float64}}
+    if j > i
+        left, right = partition(xp, xp[(j+i)>>>1], i, j)
+        quick!(xp, i, right)
+        quick!(xp, left, j)
+    end
+end
+
+@setup_workload begin
+    @compile_workload begin
+        quicksort(rand(64))
+    end
+end
+
+end
diff --git a/test/runtests.jl b/test/runtests.jl
index 25451a3..f9db701 100644
--- a/test/runtests.jl
+++ b/test/runtests.jl
@@ -82,6 +82,8 @@ using Base: specializations
     end
     PrecompileTools.verbose[] = oldval
 
+    ## @recompile_invalidations
+
     # Mimic the format written to `_jl_debug_method_invalidation`
     # As a source of MethodInstances, `getproperty` has lots
     m = which(getproperty, (Any, Symbol))
diff --git a/test/without_coverage.jl b/test/without_coverage.jl
new file mode 100644
index 0000000..2fdeba0
--- /dev/null
+++ b/test/without_coverage.jl
@@ -0,0 +1,27 @@
+using PrecompileTools
+using Test
+using InteractiveUtils
+
+@testset "No-coverage tests" begin
+    @assert Base.JLOptions().code_coverage == 0 "These tests should be run with coverage disabled"
+
+    push!(LOAD_PATH, @__DIR__)
+
+    using MSort
+    using AliasTables
+    x = rand(64)
+    pipe = Pipe()
+    oldstderr = stderr
+    redirect_stderr(pipe)
+    @trace_compile begin
+        MSort.quicksort(x)
+        at = AliasTable([1.0, 2.0])
+        rand(at)
+    end
+    close(pipe.in)
+    redirect_stderr(oldstderr)
+    str = read(pipe.out, String)
+    @test isempty(str)
+
+    pop!(LOAD_PATH)
+end


Reviews:
user: login: KristofferC
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-03T09:14:43Z
user: login: timholy
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-03T10:08:16Z
user: login: KristofferC
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-03T12:01:29Z

Review comments:
diff_hunk: @@ -0,0 +1,27 @@
+using PrecompileTools
+using Test
+using InteractiveUtils
+
+@testset "No-coverage tests" begin
+    @assert Base.JLOptions().code_coverage == 0 "These tests should be run with coverage disabled"
+
+    push!(LOAD_PATH, @__DIR__)
+
+    using MSort
+    using AliasTables
+    x = rand(64)
+    pipe = Pipe()
+    oldstderr = stderr
+    redirect_stderr(pipe)
+    @trace_compile begin
path: test/without_coverage.jl
user: login: KristofferC
body: You have to be a bit careful here with the exact same thing this PR attempts to fix, that Julia doesn't compile the code in the `begin` block before the tracing has been enabled (as discussed in https://github.com/JuliaLang/julia/issues/57899). So maybe there should be an `@eval` here or something like that?
created_at: 2025-04-03T09:14:43Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,27 @@
+using PrecompileTools
+using Test
+using InteractiveUtils
+
+@testset "No-coverage tests" begin
+    @assert Base.JLOptions().code_coverage == 0 "These tests should be run with coverage disabled"
+
+    push!(LOAD_PATH, @__DIR__)
+
+    using MSort
+    using AliasTables
+    x = rand(64)
+    pipe = Pipe()
+    oldstderr = stderr
+    redirect_stderr(pipe)
+    @trace_compile begin
path: test/without_coverage.jl
user: login: timholy
body: Ah interesting. It does work, in the sense that I verified that the test failed when there was something wrong (either with the precompilation itself or if coverage was on). But sure, we can do this: #57
created_at: 2025-04-03T10:08:15Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,27 @@
+using PrecompileTools
+using Test
+using InteractiveUtils
+
+@testset "No-coverage tests" begin
+    @assert Base.JLOptions().code_coverage == 0 "These tests should be run with coverage disabled"
+
+    push!(LOAD_PATH, @__DIR__)
+
+    using MSort
+    using AliasTables
+    x = rand(64)
+    pipe = Pipe()
+    oldstderr = stderr
+    redirect_stderr(pipe)
+    @trace_compile begin
path: test/without_coverage.jl
user: login: KristofferC
body: It might work because (AFAIU), `@testset` has special code to insert world age bumps between statements, or something to that effect. Or because of some other completely different reason... 🤷 
created_at: 2025-04-03T12:01:29Z
author_association: MEMBER

PR comments:
user: login: timholy
created_at: 2025-04-02T09:27:27Z
body: Weird. Those tests pass locally for me. I'm guessing it's a coverage thing.
author_association: MEMBER
user: login: codecov[bot]
created_at: 2025-04-02T09:42:45Z
body: ## [Codecov](https://app.codecov.io/gh/JuliaLang/PrecompileTools.jl/pull/53?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JuliaLang) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 81.81%. Comparing base [(`1e738ac`)](https://app.codecov.io/gh/JuliaLang/PrecompileTools.jl/commit/1e738ac342c68af25597354fb9d2a4e94981c059?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JuliaLang) to head [(`e5ec2f6`)](https://app.codecov.io/gh/JuliaLang/PrecompileTools.jl/commit/e5ec2f63759ecbcfedfc32b4344aa0583c2dfc03?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JuliaLang).
> Report is 4 commits behind head on main.

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main      #53   +/-   ##
=======================================
  Coverage   81.81%   81.81%           
=======================================
  Files           3        3           
  Lines          77       77           
=======================================
  Hits           63       63           
  Misses         14       14           
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/JuliaLang/PrecompileTools.jl/pull/53?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JuliaLang).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=JuliaLang).

<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
author_association: NONE
user: login: KristofferC
created_at: 2025-04-02T11:16:47Z
body: Is there something that can be backported to a pre-1.12+ julia PrecompileTools version so that the compile fixes are also available there?
author_association: MEMBER
user: login: timholy
created_at: 2025-04-02T11:51:41Z
body: No, `Core.@invokelatest` was only added in Julia 1.12. EDIT: although it's possible that hiding the `ccall` behind a `@noinline` could help a bit.
author_association: MEMBER
user: login: KristofferC
created_at: 2025-04-02T11:58:05Z
body: `Core.@latestworld` I guess we mean. 

> although it's possible that hiding the ccall behind a @noinline could help a bit.

Or maybe `@invokelatest tag_newly_inferred_enable()`... I'm just throwing out stuff
author_association: MEMBER
user: login: timholy
created_at: 2025-04-02T12:26:12Z
body: Yes I meant `Core.@latestworld`, sorry. I (strongly) doubt the `@invokelatest` would help.
author_association: MEMBER
user: login: timholy
created_at: 2025-04-02T12:27:57Z
body: I could add a branch for pre-Julia 1.12 to make it easier to continue to support older releases. Since this repo is JuliaLang, any preferred naming convention? If not I may use `before-julia-1.12`.
author_association: MEMBER
user: login: vtjnash
created_at: 2025-04-02T13:45:53Z
body: Before v1.12, you could conditionally define something like `macro latestworld() esc(Expr(:toplevel, nothing)) end` to attempt to influence the heuristics (or perhaps `Expr(:copyast, nothing)` if forced toplevel is too strong)
author_association: MEMBER