Repository: CliMA/ClimaComms.jl
Pull request #111: Extend threaded macro to CUDADevice
Author: login: dennisYatunin
Created: 2025-05-03T02:52:03Z | Merged: 2025-05-08T01:59:02Z

<!--- THESE LINES ARE COMMENTED -->
## Purpose 
<!--- One sentence to describe the purpose of this PR, refer to any linked issues:
#14 -- this will link to issue 14
Closes #2 -- this will automatically close issue 2 on PR merge
-->
This PR extends `ClimaComms.@threaded` to work with `CUDADevice`, and it adds an option to control thread coarsening. This will allow us to implement simple kernels in a device-agnostic way (i.e., without needing to explicitly import CUDA). In particular, this functionality is needed to implement LU factorization and LU solver kernels in CliMA/ClimaAtmos.jl#2730. It can also be used to generalize most of the kernels defined in `ClimaCoreCUDAExt.jl` to multi-threaded CPUs.

The current implementation of this macro only performs simple one-dimensional parallelization, but it can be extended to support shared memory and multi-dimensional thread/block configurations in the future. Eventually, we might be able to eliminate CUDA extensions from other packages altogether, replacing them with device-agnostic macros from ClimaComms. The main benefit of this will be that all of our kernel code is defined in one location, with clear documentation that informs developers about important design patterns.

This PR also bumps the version of the cache action in the test-os workflow from `v1` to `v4`, since an error was being thrown for `v1`.

<!---
Review checklist

I have:
- followed the codebase contribution guide: https://clima.github.io/ClimateMachine.jl/latest/Contributing/
- followed the style guide: https://clima.github.io/ClimateMachine.jl/latest/DevDocs/CodeStyle/
- followed the documentation policy: https://github.com/CliMA/policies/wiki/Documentation-Policy
- checked that this PR does not duplicate an open PR.

In the Content, I have included 
- relevant unit tests, and integration tests, 
- appropriate docstrings on all functions, structs, and modules, and included relevant documentation.

-->

----
- [x] I have read and checked the items on the review checklist.



Diff:
diff --git a/.github/workflows/OS-Tests.yml b/.github/workflows/OS-Tests.yml
index f2804027..d83cff8e 100644
--- a/.github/workflows/OS-Tests.yml
+++ b/.github/workflows/OS-Tests.yml
@@ -43,7 +43,7 @@ jobs:
         version: '1.11'
 
     - name: Cache artifacts
-      uses: actions/cache@v1
+      uses: actions/cache@v4
       if: steps.filter.outputs.run_test == 'true'
       env:
         cache-name: cache-artifacts
diff --git a/docs/src/index.md b/docs/src/index.md
index 58383e7b..84f30d36 100644
--- a/docs/src/index.md
+++ b/docs/src/index.md
@@ -17,12 +17,9 @@ ClimaComms.AbstractDevice) and the [`Context`](@ref ClimaComms.AbstractCommsCont
 
 A `Device` identifies a computing device, a piece of hardware that will be
 executing some code. The `Device`s currently implemented are
-- [`CPUSingleThreaded`](@ref ClimaComms.CPUSingleThreaded), for a CPU core with a single thread;
-- [`CUDADevice`](@ref ClimaComms.CUDADevice), for a single CUDA-enabled GPU.
-
-!!! warn
-    [`CPUMultiThreaded`](@ref ClimaComms.CPUMultiThreaded) is also available, but this device is not
-    actively used or developed.
+- [`CPUSingleThreaded`](@ref ClimaComms.CPUSingleThreaded) for a CPU core with a single thread,
+- [`CPUMultiThreaded`](@ref ClimaComms.CPUMultiThreaded) for a CPU core with multiple threads,
+- [`CUDADevice`](@ref ClimaComms.CUDADevice) for a single CUDA-enabled GPU.
 
 `Device`s are part of [`Context`](@ref ClimaComms.AbstractCommsContext)s,
 objects that contain information require for multiple `Device`s to communicate.
@@ -63,3 +60,34 @@ julia -E "using Pkg; Pkg.add(\"CUDA\"); Pkg.add(\"MPI\")"
 Some packages come with environments that includes all possible backends
 (typically `.buildkite`). You can also consider directly using those
 environments.
+
+# Writing generic kernels
+
+To implement kernels that work across all `Device`s, `ClimaComms.jl` provides
+the [`@threaded`](@ref ClimaComms.@threaded) macro. Like the standard library's
+[`Threads.@threads`](https://docs.julialang.org/en/v1/base/multi-threading/#Base.Threads.@threads)
+macro, this can be placed in front of any `for`-loop to parallelize its
+iterations across threads. For example, given two vectors `a` and `b` of equal
+size, a threaded version of the `copyto!` function can be implemented as
+```julia-repl
+julia> threaded_copyto!(a, b) = ClimaComms.@threaded for i in axes(a, 1)
+           a[i] = b[i]
+       end
+threaded_copyto! (generic function with 1 method)
+
+julia> threaded_copyto!(a, b)
+```
+This macro offers several options for fine-tuning performance:
+- the `Device` can be specified to reduce compilation time
+- the level of thread coarsening (number of loop iterations evaluated in each
+  thread) can be specified to reduce the overhead of launching many threads
+- on GPUs, the size of each block (a collection of threads launched on a single
+  multiprocessor) can be specified to improve GPU
+  [utilization](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#multiprocessor-level)
+
+!!! note
+    Executing code on GPUs requires *static compilation*, which means that the
+    compiler must be able to infer the types of all variables used in a threaded
+    loop. For example, global variables and type variables defined outside a
+    loop should be avoided when the loop is parallelized on a GPU. See the
+    docstring of [`@threaded`](@ref ClimaComms.@threaded) for more information.
diff --git a/ext/ClimaCommsCUDAExt.jl b/ext/ClimaCommsCUDAExt.jl
index 59d0a1c7..c9ca2802 100644
--- a/ext/ClimaCommsCUDAExt.jl
+++ b/ext/ClimaCommsCUDAExt.jl
@@ -4,7 +4,7 @@ import CUDA
 
 import Adapt
 import ClimaComms
-import ClimaComms: CUDADevice
+import ClimaComms: CUDADevice, threaded
 
 function ClimaComms._assign_device(::CUDADevice, rank_number)
     CUDA.device!(rank_number % CUDA.ndevices())
@@ -50,4 +50,83 @@ ClimaComms.elapsed(f::F, ::CUDADevice, args...; kwargs...) where {F} =
 ClimaComms.assert(::CUDADevice, cond::C, text::T) where {C, T} =
     isnothing(text) ? (CUDA.@cuassert cond()) : (CUDA.@cuassert cond() text())
 
+# TODO: Generalize all of the following code to multi-dimensional thread blocks
+# and multiple iterators.
+
+# The number of threads in the kernel being executed by the calling thread.
+threads_in_kernel() = CUDA.blockDim().x * CUDA.gridDim().x
+
+# The index of the calling thread, which is between 1 and threads_in_kernel().
+thread_index() =
+    (CUDA.blockIdx().x - 1) * CUDA.blockDim().x + CUDA.threadIdx().x
+
+# The maximum number of blocks that can fit on the GPU used for this kernel.
+grid_size_limit(kernel) = CUDA.attribute(
+    CUDA.device(kernel.fun.mod.ctx),
+    CUDA.DEVICE_ATTRIBUTE_MAX_GRID_DIM_X,
+)
+
+# Either the first value if it is available, or the maximum number of threads
+# that can fit in one block of this kernel (cuOccupancyMaxPotentialBlockSize).
+# With enough blocks, the latter value will maximize the occupancy of the GPU.
+block_size_limit(max_threads_in_block::Int, _) = max_threads_in_block
+block_size_limit(::Val{:auto}, kernel) =
+    CUDA.launch_configuration(kernel.fun).threads
+
+function threaded(f::F, ::CUDADevice, ::Val, itr; block_size) where {F}
+    length(itr) > 0 || return nothing
+    Base.require_one_based_indexing(itr)
+
+    function call_f_once_from_thread()
+        item_index = thread_index()
+        item_index <= length(itr) && @inbounds f(itr[item_index])
+        return nothing
+    end
+    kernel = CUDA.@cuda launch=false call_f_once_from_thread()
+    max_blocks = grid_size_limit(kernel)
+    max_threads_in_block = block_size_limit(block_size, kernel)
+
+    # If there are too many items, coarsen by the smallest possible amount.
+    length(itr) <= max_blocks * max_threads_in_block ||
+        return threaded(f, CUDADevice(), 1, itr)
+
+    threads_in_block = min(max_threads_in_block, length(itr))
+    blocks = cld(length(itr), threads_in_block)
+    CUDA.@sync kernel(; blocks, threads = threads_in_block)
+end
+
+function threaded(
+    f::F,
+    ::CUDADevice,
+    min_items_in_thread::Int,
+    itr;
+    block_size,
+) where {F}
+    min_items_in_thread > 0 || throw(ArgumentError("`coarsen` is not positive"))
+    length(itr) > 0 || return nothing
+    Base.require_one_based_indexing(itr)
+
+    # Maximize memory coalescing with a "grid-stride loop"; for reference, see
+    # https://developer.nvidia.com/blog/cuda-pro-tip-write-flexible-kernels-grid-stride-loops
+    call_f_multiple_times_from_thread() =
+        for item_index in thread_index():threads_in_kernel():length(itr)
+            @inbounds f(itr[item_index])
+        end
+    kernel = CUDA.@cuda launch=false call_f_multiple_times_from_thread()
+    max_blocks = grid_size_limit(kernel)
+    max_threads_in_block = block_size_limit(block_size, kernel)
+
+    # If there are too many items to use the specified coarsening, increase it
+    # by the smallest possible amount.
+    max_required_threads = cld(length(itr), min_items_in_thread)
+    items_in_thread =
+        max_required_threads <= max_blocks * max_threads_in_block ?
+        min_items_in_thread :
+        cld(length(itr), max_blocks * max_threads_in_block)
+
+    threads_in_block = min(max_threads_in_block, max_required_threads)
+    blocks = cld(length(itr), items_in_thread * threads_in_block)
+    CUDA.@sync kernel(; blocks, threads = threads_in_block)
+end
+
 end
diff --git a/src/devices.jl b/src/devices.jl
index 143b644b..76ce0669 100644
--- a/src/devices.jl
+++ b/src/devices.jl
@@ -101,36 +101,6 @@ Currently used to assign CUDADevices to MPI ranks.
 """
 _assign_device(device, id) = nothing
 
-"""
-    @threaded device for ... end
-
-A threading macro that uses Julia native threading if the
-device is a `CPUMultiThreaded` type, otherwise return
-the original expression without `Threads.@threads`. This is
-done to avoid overhead from `Threads.@threads`, and the device
-is used (instead of checking `Threads.nthreads() == 1`) so
-that this is statically inferred.
-
-## References
-
- - https://discourse.julialang.org/t/threads-threads-with-one-thread-how-to-remove-the-overhead/58435
- - https://discourse.julialang.org/t/overhead-of-threads-threads/53964
-"""
-macro threaded(device, loop)
-    quote
-        if $(esc(device)) isa CPUMultiThreaded
-            Threads.@threads $(Expr(
-                loop.head,
-                Expr(loop.args[1].head, esc.(loop.args[1].args)...),
-                esc(loop.args[2]),
-            ))
-        else
-            Base.@assert $(esc(device)) isa AbstractDevice
-            $(esc(loop))
-        end
-    end
-end
-
 """
     @time f(args...; kwargs...)
 
@@ -376,3 +346,189 @@ macro assert(device, cond, text = nothing)
 end
 assert(::AbstractCPUDevice, cond::C, text::T) where {C, T} =
     isnothing(text) ? (Base.@assert cond()) : (Base.@assert cond() text())
+
+
+"""
+    @threaded [device] [coarsen=...] [block_size=...] for ... end
+
+Device-flexible generalization of `Threads.@threads`, which distributes the
+iterations of a for-loop across multiple threads, with the option to control
+thread coarsening and GPU kernel configuration. Coarsening makes each thread
+evaluate more than one iteration of the loop, which can improve performance by
+reducing the runtime overhead of launching additional threads (though too much
+coarsening worsens performance because it reduces parallelization). The `device`
+is either inferred by calling `ClimaComms.device()`, or it can be specified
+manually, with the following device-dependent behavior:
+
+ - When `device` is a `CPUSingleThreaded()`, the loop is evaluated as-is. This
+   avoids the runtime overhead of calling `Threads.@threads` with a single
+   thread, and, when the device type is statically inferrable, it also avoids
+   compilation overhead.
+
+ - When `device` is a `CPUMultiThreaded()`, the loop is passed to
+   `Threads.@threads`. This supports three different kinds of "schedulers" for
+   determining how many iterations of the loop to evaluate in each thread:
+     1) (default) a "dynamic" scheduler that changes the number of iterations as
+        new threads are launched,
+     2) a "static" scheduler that evaluates a fixed number of iterations per
+        thread, and
+     3) a "greedy" scheduler that uses a small number of threads, continuously
+        evaluating iterations in each thread until the loop is completed (only
+        available as of Julia 1.11).
+   Setting `coarsen` to `:dynamic` or `:greedy` launches threads with those
+   schedulers. Setting it to `:static` or an integer value launches threads with
+   static scheduling (using `:static` is similar to using `1`, but slightly more
+   performant). To read more about multi-threading, see the documentation for
+   [`Threads.@threads`](https://docs.julialang.org/en/v1/base/multi-threading/#Base.Threads.@threads).
+
+ - When `device` is a `CUDADevice()`, the loop is compiled with `CUDA.@cuda` and
+   run with `CUDA.@sync`. Since CUDA launches all threads at the same time, only
+   static scheduling can be used. Setting `coarsen` to any symbol causes each
+   thread to evaluate a single iteration (default), and setting it to an integer
+   value causes each thread to evaluate that number of iterations (the default
+   is similar to using `1`, but slightly more performant). If the total number
+   of iterations in the loop is extremely large, the specified coarsening may
+   require more threads than can be simultaneously launched on the GPU, in which
+   case the amount of coarsening is automatically increased.
+
+   The optional argument `block_size` is also available for manually specifying
+   the size of each block on a GPU. The default value of `:auto` sets the number
+   of threads in each block to the largest possible value that permits a high
+   GPU "occupancy" (the number of active thread warps in each multiprocessor
+   executing the kernel). An integer can be used instead of `:auto` to override
+   this default value. If the specified value exceeds the total number of
+   threads, it is automatically decreased to avoid idle threads.
+
+NOTE: When a value in the body of the loop has a type that cannot be inferred by
+the compiler, an `InvalidIRError` will be thrown during compilation for a
+`CUDADevice()`. In particular, global variables are not inferrable, so
+`@threaded` must be wrapped in a function whenever it is used in the REPL:
+
+```julia-repl
+julia> a = CUDA.CuArray{Int}(undef, 100); b = similar(a);
+
+julia> threaded_copyto!(a, b) = ClimaComms.@threaded for i in axes(a, 1)
+           a[i] = b[i]
+       end
+threaded_copyto! (generic function with 1 method)
+
+julia> threaded_copyto!(a, b)
+
+julia> ClimaComms.@threaded for i in axes(a, 1)
+           a[i] = b[i]
+       end
+ERROR: InvalidIRError: ...
+```
+
+Moreover, type variables are not inferrable across function boundaries, so types
+used in a threaded loop cannot be precomputed before the loop:
+
+```julia-repl
+julia> threaded_add_epsilon!(a) = ClimaComms.@threaded for i in axes(a, 1)
+           FT = eltype(a)
+           a[i] += eps(FT)
+       end
+threaded_add_epsilon! (generic function with 1 method)
+
+julia> threaded_add_epsilon!(a)
+
+julia> function threaded_add_epsilon!(a)
+           FT = eltype(a)
+           ClimaComms.@threaded for i in axes(a, 1)
+               a[i] += eps(FT)
+           end
+       end
+threaded_add_epsilon! (generic function with 1 method)
+
+julia> threaded_add_epsilon!(a)
+ERROR: InvalidIRError: ...
+```
+
+To fix other kinds of inference issues on GPUs, especially ones brought about by
+indexing into iterators with nonuniform element types, see
+[`UnrolledUtilities.jl`](https://github.com/CliMA/UnrolledUtilities.jl).
+"""
+macro threaded(args...)
+    usage_string = "Usage: @threaded [device] [coarsen=...] [block_size=...] for ... end"
+    n_args = length(args)
+    1 <= n_args <= 3 || throw(ArgumentError(usage_string))
+
+    device_expr = :($ClimaComms.device())
+    coarsen_expr = :(:dynamic)
+    block_size_expr = :(:auto)
+    for arg_number in 1:(n_args - 1)
+        device_or_coarsen_expr = args[arg_number]
+        if Meta.isexpr(device_or_coarsen_expr, :(=))
+            (kwarg_name, kwarg_value) = device_or_coarsen_expr.args
+            if kwarg_name == :coarsen
+                coarsen_expr = kwarg_value
+            elseif kwarg_name == :block_size
+                block_size_expr = kwarg_value
+            else
+                throw(ArgumentError(usage_string))
+            end
+        else
+            arg_number == 1 || throw(ArgumentError(usage_string))
+            device_expr = device_or_coarsen_expr
+        end
+    end
+    if coarsen_expr isa QuoteNode
+        coarsen_expr = :(Val($(coarsen_expr)))
+    end
+    if block_size_expr isa QuoteNode
+        block_size_expr = :(Val($(block_size_expr)))
+    end
+
+    loop_expr = args[n_args]
+    Meta.isexpr(loop_expr, :for) || throw(ArgumentError(usage_string))
+    (var_and_itr_expr, loop_body) = loop_expr.args
+    Meta.isexpr(var_and_itr_expr, :(=)) ||
+        throw(ArgumentError("@threaded does not support nested loops"))
+    (var_expr, itr_expr) = var_and_itr_expr.args
+
+    return quote
+        device = $(esc(device_expr))
+        device isa CPUSingleThreaded ? $(esc(loop_expr)) :
+        threaded(
+            $(esc(var_expr)) -> $(esc(loop_body)),
+            device,
+            $(esc(coarsen_expr)),
+            $(esc(itr_expr));
+            block_size = $(esc(block_size_expr)),
+        )
+    end
+end
+
+threaded(f::F, device, coarsen, itr; gpu_kwargs...) where {F} =
+    threaded(f, device, coarsen, itr) # Drop kwargs that are only used for GPUs.
+
+threaded(f::F, ::CPUMultiThreaded, ::Val{:dynamic}, itr) where {F} =
+    Threads.@threads :dynamic for item in itr
+        f(item)
+    end
+
+threaded(f::F, ::CPUMultiThreaded, ::Val{:static}, itr) where {F} =
+    Threads.@threads :static for item in itr
+        f(item)
+    end
+
+@static if VERSION >= v"1.11"
+    threaded(f::F, ::CPUMultiThreaded, ::Val{:greedy}, itr) where {F} =
+        Threads.@threads :greedy for item in itr
+            f(item)
+        end
+end
+
+function threaded(f::F, ::CPUMultiThreaded, items_in_thread::Int, itr) where {F}
+    items_in_thread > 0 || throw(ArgumentError("`coarsen` is not positive"))
+    Base.require_one_based_indexing(itr)
+
+    threads = cld(length(itr), items_in_thread)
+    Threads.@threads :static for thread_index in 1:threads
+        first_item_index = items_in_thread * (thread_index - 1) + 1
+        last_item_index = items_in_thread * thread_index
+        for item_index in first_item_index:min(last_item_index, length(itr))
+            @inbounds f(itr[item_index])
+        end
+    end
+end
diff --git a/test/hygiene.jl b/test/hygiene.jl
index 24ec0b2d..42c0d2c6 100644
--- a/test/hygiene.jl
+++ b/test/hygiene.jl
@@ -3,10 +3,10 @@ CC.@import_required_backends
 function test_macro_hyhiene(dev)
     AT = CC.array_type(dev)
     n = 3 # tests that we can reach variables defined in scope
+
+    array = AT(rand(10))
     CC.@threaded dev for i in 1:n
-        if Threads.nthreads() > 1
-            @show Threads.threadid()
-        end
+        array[i] = sin(array[i])
     end
 
     CC.time(dev) do
diff --git a/test/runtests.jl b/test/runtests.jl
index 5553c581..26fb92bd 100644
--- a/test/runtests.jl
+++ b/test/runtests.jl
@@ -228,6 +228,41 @@ end
     @test x == Array(a)[1]
 end
 
+@testset "threaded" begin
+    a = AT(rand(100))
+    b = AT(rand(100))
+
+    kernel1!(a, b) = ClimaComms.@threaded for i in axes(a, 1)
+        a[i] = b[i]
+    end
+    kernel1!(a, b)
+    @test a == b
+
+    kernel2!(a, b) = ClimaComms.@threaded coarsen=:static for i in axes(a, 1)
+        a[i] = 2 * b[i]
+    end
+    kernel2!(a, b)
+    @test a == 2 * b
+
+    kernel3!(a, b) = ClimaComms.@threaded device coarsen=3 for i in axes(a, 1)
+        a[i] = 3 * b[i]
+    end
+    kernel3!(a, b)
+    @test a == 3 * b
+
+    kernel4!(a, b) = ClimaComms.@threaded device coarsen=400 for i in axes(a, 1)
+        a[i] = 4 * b[i]
+    end
+    kernel4!(a, b)
+    @test a == 4 * b
+
+    kernel5!(a, b) = ClimaComms.@threaded block_size=50 for i in axes(a, 1)
+        a[i] = 5 * b[i]
+    end
+    kernel5!(a, b)
+    @test a == 5 * b
+end
+
 import Adapt
 @testset "Adapt" begin
     @test Adapt.adapt(Array, ClimaComms.CUDADevice()) ==


Reviews:
user: login: coderabbitai[bot]
body: **Actionable comments posted: 3**

<details>
<summary>🧹 Nitpick comments (1)</summary><blockquote>

<details>
<summary>test/runtests.jl (1)</summary><blockquote>

`231-258`: **Add a zero-length case to the `@threaded` tests**

The new CUDA backend short-circuits on `length(itr)==0`; adding

```julia
empty_a = AT(undef, 0); empty_b = AT(undef, 0)
kernel1!(empty_a, empty_b)   # should be a no-op, not an error
@test length(empty_a) == 0
```

would catch regressions quickly.

</blockquote></details>

</blockquote></details>

<details>
<summary>📜 Review details</summary>

**Configuration used: CodeRabbit UI**
**Review profile: CHILL**
**Plan: Pro**

<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between c968350313d5dc81d007721bc27a4bc9c97a79e3 and 5a6b095ad5a241923950f111a5b28cf8074242b5.

</details>

<details>
<summary>📒 Files selected for processing (3)</summary>

* `ext/ClimaCommsCUDAExt.jl` (2 hunks)
* `src/devices.jl` (1 hunks)
* `test/runtests.jl` (1 hunks)

</details>

<details>
<summary>⏰ Context from checks skipped due to timeout of 90000ms (1)</summary>

* GitHub Check: docbuild

</details>

<details>
<summary>🔇 Additional comments (1)</summary><blockquote>

<details>
<summary>src/devices.jl (1)</summary>

`456-462`: **Handle `:static` scheduler explicitly**  

Generated function only admits `:dynamic` / `:greedy` because those are the only symbols passed by the macro. If a user writes `@threaded coarsen=:static ...`, the macro will substitute `Val(:static)` and dispatch here, but `Threads.@threads :static` is valid. Consider adding it to the docs or throwing if `S == :static` to prevent confusion.

</details>

</blockquote></details>

</details>

<!-- This is an auto-generated comment by CodeRabbit for review status -->
state: COMMENTED
author_association: NONE
submitted_at: 2025-05-03T02:56:57Z
user: login: dennisYatunin
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-03T03:04:13Z
user: login: coderabbitai[bot]
body: **Actionable comments posted: 0**

<details>
<summary>♻️ Duplicate comments (1)</summary><blockquote>

<details>
<summary>src/devices.jl (1)</summary><blockquote>

`445-453`: **Fix inconsistent return type in single-threaded branch.**

The single-threaded branch returns the value of the loop while other branches return `nothing`. Also, `device_expr` is evaluated multiple times.


```diff
- return quote
-     $(esc(device_expr)) isa CPUSingleThreaded ? $(esc(loop_expr)) :
-     threaded(
-         $(esc(var_expr)) -> $(esc(loop_body)),
-         $(esc(device_expr)),
-         $(esc(coarsen_expr)),
-         $(esc(itr_expr)),
-     )
- end
+ return quote
+     let _dev = $(esc(device_expr))               # evaluate once
+         if _dev isa CPUSingleThreaded
+             $(esc(loop_expr))
+             nothing                               # keep return type consistent
+         else
+             threaded(
+                 $(esc(var_expr)) -> $(esc(loop_body)),
+                 _dev,
+                 $(esc(coarsen_expr)),
+                 $(esc(itr_expr)),
+             )
+         end
+     end
+ end
```

</blockquote></details>

</blockquote></details>

<details>
<summary>🧹 Nitpick comments (2)</summary><blockquote>

<details>
<summary>src/devices.jl (2)</summary><blockquote>

`463-472`: **Add validation for integer-based coarsening.**

The function should check that `items_in_thread` is positive.


```diff
 function threaded(f::F, ::CPUMultiThreaded, items_in_thread::Int, itr) where {F}
+    items_in_thread > 0 || throw(ArgumentError("Number of items per thread must be positive"))
     n_threads = cld(length(itr), items_in_thread)
     Threads.@threads :static for thread_index in 1:n_threads
         first_item_index = items_in_thread * (thread_index - 1) + 1
         last_item_index = items_in_thread * thread_index
         for item_index in first_item_index:min(last_item_index, length(itr))
             f(itr[item_index])
         end
     end
 end
```

---

`416-436`: **Consider validating coarsening value when it's a literal integer.**

For direct integer inputs to the coarsening parameter, add validation to ensure positive values.


```diff
     if coarsen_expr isa QuoteNode
         coarsen_expr = :(Val($(coarsen_expr)))
+    elseif coarsen_expr isa Integer
+        coarsen_expr > 0 || throw(ArgumentError("Integer coarsening value must be positive"))
     end
```

</blockquote></details>

</blockquote></details>

<details>
<summary>📜 Review details</summary>

**Configuration used: CodeRabbit UI**
**Review profile: CHILL**
**Plan: Pro**

<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between 5a6b095ad5a241923950f111a5b28cf8074242b5 and 05ffda23f833ac9a574bf2079636d1b981733266.

</details>

<details>
<summary>📒 Files selected for processing (4)</summary>

* `.github/workflows/OS-Tests.yml` (1 hunks)
* `ext/ClimaCommsCUDAExt.jl` (2 hunks)
* `src/devices.jl` (1 hunks)
* `test/runtests.jl` (1 hunks)

</details>

<details>
<summary>✅ Files skipped from review due to trivial changes (1)</summary>

* .github/workflows/OS-Tests.yml

</details>

<details>
<summary>🚧 Files skipped from review as they are similar to previous changes (2)</summary>

* test/runtests.jl
* ext/ClimaCommsCUDAExt.jl

</details>

<details>
<summary>🔇 Additional comments (2)</summary><blockquote>

<details>
<summary>src/devices.jl (2)</summary>

`349-454`: **Well-documented and comprehensive `@threaded` macro.**

The macro effectively extends threading capabilities to CUDA devices while maintaining compatibility with CPU execution. Documentation clearly explains behavior across device types and coarsening strategies.

---

`456-461`: **Good use of @generated function for clean compile-time specialization.**

The generated function approach cleanly dispatches to `Threads.@threads` with the appropriate scheduler symbol.

</details>

</blockquote></details>

</details>

<!-- This is an auto-generated comment by CodeRabbit for review status -->
state: COMMENTED
author_association: NONE
submitted_at: 2025-05-03T03:04:35Z
user: login: coderabbitai[bot]
state: COMMENTED
author_association: NONE
submitted_at: 2025-05-03T03:04:42Z
user: login: coderabbitai[bot]
body: **Actionable comments posted: 1**

<details>
<summary>🧹 Nitpick comments (1)</summary><blockquote>

<details>
<summary>src/devices.jl (1)</summary><blockquote>

`417-452`: **Consider evaluating device_expr once**

The ternary expression on line 449-450 evaluates `device_expr` twice - once for the condition check and once in the `threaded` call. For complex expressions, this could be inefficient.


```diff
-        device = $(esc(device_expr))
-        device isa CPUSingleThreaded ? $(esc(loop_expr)) :
-        threaded($function_expr, device, $(esc(coarsen_expr)), $(esc(itr_expr)))
+        let device = $(esc(device_expr))
+            device isa CPUSingleThreaded ? $(esc(loop_expr)) :
+            threaded($function_expr, device, $(esc(coarsen_expr)), $(esc(itr_expr)))
+        end
```

</blockquote></details>

</blockquote></details>

<details>
<summary>📜 Review details</summary>

**Configuration used: CodeRabbit UI**
**Review profile: CHILL**
**Plan: Pro**

<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between 670b793424c6cef6fea897c389a9e2c140742764 and 974ca466dea8a4b746db0e277da9a02c62513973.

</details>

<details>
<summary>📒 Files selected for processing (5)</summary>

* `.github/workflows/OS-Tests.yml` (1 hunks)
* `ext/ClimaCommsCUDAExt.jl` (2 hunks)
* `src/devices.jl` (1 hunks)
* `test/hygiene.jl` (1 hunks)
* `test/runtests.jl` (1 hunks)

</details>

<details>
<summary>🚧 Files skipped from review as they are similar to previous changes (4)</summary>

* .github/workflows/OS-Tests.yml
* test/hygiene.jl
* test/runtests.jl
* ext/ClimaCommsCUDAExt.jl

</details>

<details>
<summary>🧰 Additional context used</summary>

<details>
<summary>🧠 Learnings (1)</summary>

<details>
<summary>src/devices.jl (1)</summary>

```
Learnt from: dennisYatunin
PR: CliMA/ClimaComms.jl#111
File: src/devices.jl:445-453
Timestamp: 2025-05-03T03:04:41.544Z
Learning: In Julia, a for-loop evaluates to `nothing` when it completes execution, so there's no need to explicitly add `nothing` at the end of a for-loop to maintain consistent return values in control flow branches.
```

</details>

</details>

</details>

<details>
<summary>🔇 Additional comments (3)</summary><blockquote>

<details>
<summary>src/devices.jl (3)</summary>

`351-416`: **Great documentation!**

The comprehensive documentation clearly explains how the macro behaves across different device types, including edge cases and inference limitations. Good callout about the `InvalidIRError` on `CUDADevice` with non-inferrable types.

---

`453-477`: **CPU implementations look good**

The implementations for `CPUMultiThreaded` cover all three schedulers plus numeric coarsening. The `items_in_thread > 0` check prevents invalid coarsening values.

---

`446-452`: **Consistent with previous learnings**

The implementation correctly follows the principle that Julia for-loops evaluate to `nothing`, so there's no need to explicitly return `nothing` after the for-loop.

</details>

</blockquote></details>

</details>

<!-- This is an auto-generated comment by CodeRabbit for review status -->
state: COMMENTED
author_association: NONE
submitted_at: 2025-05-03T21:03:52Z
user: login: dennisYatunin
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-03T21:12:54Z
user: login: coderabbitai[bot]
state: COMMENTED
author_association: NONE
submitted_at: 2025-05-03T21:13:23Z
user: login: coderabbitai[bot]
body: **Actionable comments posted: 0**

<details>
<summary>🧹 Nitpick comments (1)</summary><blockquote>

<details>
<summary>src/devices.jl (1)</summary><blockquote>

`419-454`: **Consider refactoring the device evaluation to avoid duplicated evaluation.**

The ternary expression on lines 451-453 evaluates `device_expr` twice if it's not a `CPUSingleThreaded`.


```diff
-        device = $(esc(device_expr))
-        device isa CPUSingleThreaded ? $(esc(loop_expr)) :
-        threaded($function_expr, device, $(esc(coarsen_expr)), $(esc(itr_expr)))
+        device = $(esc(device_expr))
+        if device isa CPUSingleThreaded
+            $(esc(loop_expr))
+        else
+            threaded($function_expr, device, $(esc(coarsen_expr)), $(esc(itr_expr)))
+        end
```

</blockquote></details>

</blockquote></details>

<details>
<summary>📜 Review details</summary>

**Configuration used: CodeRabbit UI**
**Review profile: CHILL**
**Plan: Pro**

<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between fe5f827a739605b58204c365ee7e3b6445b4fc89 and a446da158e5e430a0d851705d05cf240faf1f68c.

</details>

<details>
<summary>📒 Files selected for processing (5)</summary>

* `.github/workflows/OS-Tests.yml` (1 hunks)
* `ext/ClimaCommsCUDAExt.jl` (2 hunks)
* `src/devices.jl` (1 hunks)
* `test/hygiene.jl` (1 hunks)
* `test/runtests.jl` (1 hunks)

</details>

<details>
<summary>🚧 Files skipped from review as they are similar to previous changes (4)</summary>

* .github/workflows/OS-Tests.yml
* test/hygiene.jl
* test/runtests.jl
* ext/ClimaCommsCUDAExt.jl

</details>

<details>
<summary>🧰 Additional context used</summary>

<details>
<summary>🧠 Learnings (1)</summary>

<details>
<summary>src/devices.jl (2)</summary>

```
Learnt from: dennisYatunin
PR: CliMA/ClimaComms.jl#111
File: src/devices.jl:445-453
Timestamp: 2025-05-03T03:04:41.544Z
Learning: In Julia, a for-loop evaluates to `nothing` when it completes execution, so there's no need to explicitly add `nothing` at the end of a for-loop to maintain consistent return values in control flow branches.
```

```
Learnt from: dennisYatunin
PR: CliMA/ClimaComms.jl#111
File: src/devices.jl:472-482
Timestamp: 2025-05-03T21:13:22.142Z
Learning: ClimaComms.jl is designed to work with iterators that support both `length` and `getindex` methods, as these capabilities are required for GPU compatibility. Iterators without these methods are intentionally not supported.
```

</details>

</details>

</details>

<details>
<summary>🔇 Additional comments (3)</summary><blockquote>

<details>
<summary>src/devices.jl (3)</summary>

`351-418`: **Well-documented macro with comprehensive instructions.**

The docstring for `@threaded` provides excellent guidance on device-specific behaviors, coarsening strategies, and potential pitfalls like inference errors on GPUs.

---

`456-471`: **Clean implementation of threading schedulers.**

The different scheduler implementations for CPU multithreaded execution match the documented behavior and properly utilize Julia's `Threads.@threads` capabilities.

---

`473-483`: **Static coarsening implementation looks good.**

The manual partitioning for static coarsening correctly divides work across threads and handles boundary conditions.

</details>

</blockquote></details>

</details>

<!-- This is an auto-generated comment by CodeRabbit for review status -->
state: COMMENTED
author_association: NONE
submitted_at: 2025-05-03T22:08:34Z
user: login: coderabbitai[bot]
body: **Actionable comments posted: 0**

<details>
<summary>🧹 Nitpick comments (1)</summary><blockquote>

<details>
<summary>src/devices.jl (1)</summary><blockquote>

`451-453`: **Consider caching device evaluation.**

The device expression is evaluated twice in the ternary. While for-loops return `nothing` in Julia (as noted in your feedback), caching the device in a local variable would avoid duplicate evaluation.

```diff
-        device = $(esc(device_expr))
-        device isa CPUSingleThreaded ? $(esc(loop_expr)) :
-        threaded($function_expr, device, $(esc(coarsen_expr)), $(esc(itr_expr)))
+        device = $(esc(device_expr))
+        if device isa CPUSingleThreaded
+            $(esc(loop_expr))
+        else
+            threaded($function_expr, device, $(esc(coarsen_expr)), $(esc(itr_expr)))
+        end
```

</blockquote></details>

</blockquote></details>

<details>
<summary>📜 Review details</summary>

**Configuration used: CodeRabbit UI**
**Review profile: CHILL**
**Plan: Pro**

<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between a446da158e5e430a0d851705d05cf240faf1f68c and 1eea774d14798e3d70884365da993b80fc5f77a7.

</details>

<details>
<summary>📒 Files selected for processing (5)</summary>

* `.github/workflows/OS-Tests.yml` (1 hunks)
* `ext/ClimaCommsCUDAExt.jl` (2 hunks)
* `src/devices.jl` (1 hunks)
* `test/hygiene.jl` (1 hunks)
* `test/runtests.jl` (1 hunks)

</details>

<details>
<summary>🚧 Files skipped from review as they are similar to previous changes (4)</summary>

* .github/workflows/OS-Tests.yml
* test/hygiene.jl
* ext/ClimaCommsCUDAExt.jl
* test/runtests.jl

</details>

<details>
<summary>🧰 Additional context used</summary>

<details>
<summary>🧠 Learnings (1)</summary>

<details>
<summary>src/devices.jl (2)</summary>

```
Learnt from: dennisYatunin
PR: CliMA/ClimaComms.jl#111
File: src/devices.jl:445-453
Timestamp: 2025-05-03T03:04:41.544Z
Learning: In Julia, a for-loop evaluates to `nothing` when it completes execution, so there's no need to explicitly add `nothing` at the end of a for-loop to maintain consistent return values in control flow branches.
```

```
Learnt from: dennisYatunin
PR: CliMA/ClimaComms.jl#111
File: src/devices.jl:472-482
Timestamp: 2025-05-03T21:13:22.142Z
Learning: ClimaComms.jl is designed to work with iterators that support both `length` and `getindex` methods, as these capabilities are required for GPU compatibility. Iterators without these methods are intentionally not supported.
```

</details>

</details>

</details>

<details>
<summary>🔇 Additional comments (4)</summary><blockquote>

<details>
<summary>src/devices.jl (4)</summary>

`349-418`: **Well-documented device-aware macro.**

The documentation comprehensively explains how `@threaded` works across different devices and coarsening strategies. Good inclusion of practical examples and error handling notes.

---

`456-471`: **Appropriate threading strategy implementations.**

The implementations for dynamic, static, and greedy scheduling directly leverage `Threads.@threads` with proper versioning guard for Julia 1.11+ features.

---

`473-483`: **Robust implementation of custom coarsening.**

The function correctly distributes work across threads with proper bounds checking. As noted in past discussions, the implementation intentionally requires iterators that support both `length` and `getindex` for GPU compatibility.

---

`419-454`: <details>
<summary>✅ Verification successful</summary>

**Clean macro implementation with proper argument handling.**

The macro correctly parses optional arguments, validates inputs, and handles device-specific code generation.


---

🏁 Script executed:

```shell
#!/bin/bash
# Check if the device expression is evaluated twice in actual usage
rg -A 5 "@threaded.*CPUSingleThreaded.*threaded" --type julia
```

Length of output: 63

---



</details>

**Approve: `device_expr` evaluated only once**  
Local binding to `device` ensures your `device_expr` is evaluated exactly once. Macro argument parsing, validation, and device-specific code generation all look solid.

</details>

</blockquote></details>

</details>

<!-- This is an auto-generated comment by CodeRabbit for review status -->
state: COMMENTED
author_association: NONE
submitted_at: 2025-05-04T00:38:10Z
user: login: coderabbitai[bot]
body: **Actionable comments posted: 0**

<details>
<summary>🧹 Nitpick comments (1)</summary><blockquote>

<details>
<summary>src/devices.jl (1)</summary><blockquote>

`419-454`: **Consider avoiding double evaluation of device expression.**

The ternary operator evaluates `device_expr` twice when it's not a `CPUSingleThreaded`.


```diff
-        device = $(esc(device_expr))
-        device isa CPUSingleThreaded ? $(esc(loop_expr)) :
-        threaded($function_expr, device, $(esc(coarsen_expr)), $(esc(itr_expr)))
+        let device = $(esc(device_expr))
+            device isa CPUSingleThreaded ? $(esc(loop_expr)) :
+            threaded($function_expr, device, $(esc(coarsen_expr)), $(esc(itr_expr)))
+        end
```

</blockquote></details>

</blockquote></details>

<details>
<summary>📜 Review details</summary>

**Configuration used: CodeRabbit UI**
**Review profile: CHILL**
**Plan: Pro**

<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between 1eea774d14798e3d70884365da993b80fc5f77a7 and 340ace9b84adb08b1023696b28151c05df4dece9.

</details>

<details>
<summary>📒 Files selected for processing (5)</summary>

* `.github/workflows/OS-Tests.yml` (1 hunks)
* `ext/ClimaCommsCUDAExt.jl` (2 hunks)
* `src/devices.jl` (1 hunks)
* `test/hygiene.jl` (1 hunks)
* `test/runtests.jl` (1 hunks)

</details>

<details>
<summary>🚧 Files skipped from review as they are similar to previous changes (4)</summary>

* .github/workflows/OS-Tests.yml
* test/hygiene.jl
* test/runtests.jl
* ext/ClimaCommsCUDAExt.jl

</details>

<details>
<summary>🧰 Additional context used</summary>

<details>
<summary>🧠 Learnings (1)</summary>

<details>
<summary>src/devices.jl (2)</summary>

```
Learnt from: dennisYatunin
PR: CliMA/ClimaComms.jl#111
File: src/devices.jl:445-453
Timestamp: 2025-05-03T03:04:41.544Z
Learning: In Julia, a for-loop evaluates to `nothing` when it completes execution, so there's no need to explicitly add `nothing` at the end of a for-loop to maintain consistent return values in control flow branches.
```

```
Learnt from: dennisYatunin
PR: CliMA/ClimaComms.jl#111
File: src/devices.jl:472-482
Timestamp: 2025-05-03T21:13:22.142Z
Learning: ClimaComms.jl is designed to work with iterators that support both `length` and `getindex` methods, as these capabilities are required for GPU compatibility. Iterators without these methods are intentionally not supported.
```

</details>

</details>

</details>

<details>
<summary>🔇 Additional comments (5)</summary><blockquote>

<details>
<summary>src/devices.jl (5)</summary>

`351-418`: **Comprehensive docstring with clear explanations.**

The docstring effectively explains the purpose, behavior across different devices, and limitations. Good job highlighting type inference issues and providing REPL examples.

---

`456-471`: **Well-implemented scheduler variants for CPU multi-threading.**

The implementations properly handle different scheduling strategies with clean, consistent interfaces.

---

`473-485`: **Static scheduling implementation looks good.**

The manual partitioning of work items across threads is handled correctly, with proper bounds checking.


Note: Based on the retrieved learning, the code correctly works with iterators that support both `length` and `getindex` methods, which is required for GPU compatibility.

---

`445-446`: **Clear error message for nested loops.**

Good job providing a specific error message that explicitly states the limitation.

---

`466-471`: **Appropriate version check for greedy scheduler.**

The conditional compilation block ensures compatibility with different Julia versions.

</details>

</blockquote></details>

</details>

<!-- This is an auto-generated comment by CodeRabbit for review status -->
state: COMMENTED
author_association: NONE
submitted_at: 2025-05-04T18:44:36Z
user: login: sriharshakandala
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-05T18:02:12Z
user: login: sriharshakandala
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-05T20:44:46Z
user: login: coderabbitai[bot]
body: **Actionable comments posted: 0**

<details>
<summary>🧹 Nitpick comments (3)</summary><blockquote>

<details>
<summary>ext/ClimaCommsCUDAExt.jl (2)</summary><blockquote>

`53-54`: **Consider adding issue reference to TODOs**

These TODOs indicate planned features, but should ideally link to a tracking issue for clarity.

---

`112-114`: **Consider adding a comment about memory coalescing**

The grid-stride loop pattern is excellent for coalescing, but a brief comment explaining why this approach was chosen over alternatives would help future maintainers.

<details>
<summary>🧰 Tools</summary>

<details>
<summary>🪛 GitHub Actions: JuliaFormatter</summary>

[error] 112-119: Git diff detected changes in ext/ClimaCommsCUDAExt.jl. The pipeline failed with exit code 1, indicating an issue with uncommitted or unexpected changes.

</details>

</details>

</blockquote></details>
<details>
<summary>src/devices.jl (1)</summary><blockquote>

`467-476`: **Consider using let block for device evaluation**

The device expression is evaluated twice in the ternary. While you noted that for-loops return `nothing` in Julia, evaluating `device_expr` once in a let block would be cleaner:

```diff
-        device = $(esc(device_expr))
-        device isa CPUSingleThreaded ? $(esc(loop_expr)) :
-        threaded(
-            $(esc(var_expr)) -> $(esc(loop_body)),
-            device,
-            $(esc(coarsen_expr)),
-            $(esc(itr_expr));
-            blockdim = $(esc(blockdim_expr)),
-        )
+        let device = $(esc(device_expr))
+            if device isa CPUSingleThreaded
+                $(esc(loop_expr))
+            else
+                threaded(
+                    $(esc(var_expr)) -> $(esc(loop_body)),
+                    device,
+                    $(esc(coarsen_expr)),
+                    $(esc(itr_expr));
+                    blockdim = $(esc(blockdim_expr)),
+                )
+            end
+        end
```

</blockquote></details>

</blockquote></details>

<details>
<summary>📜 Review details</summary>

**Configuration used: CodeRabbit UI**
**Review profile: CHILL**
**Plan: Pro**

<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between 340ace9b84adb08b1023696b28151c05df4dece9 and bf2b840864d94940bfcc589a74c6b04c811f1c85.

</details>

<details>
<summary>📒 Files selected for processing (3)</summary>

* `ext/ClimaCommsCUDAExt.jl` (2 hunks)
* `src/devices.jl` (1 hunks)
* `test/runtests.jl` (1 hunks)

</details>

<details>
<summary>🚧 Files skipped from review as they are similar to previous changes (1)</summary>

* test/runtests.jl

</details>

<details>
<summary>🧰 Additional context used</summary>

<details>
<summary>🧠 Learnings (1)</summary>

<details>
<summary>src/devices.jl (2)</summary>

```
Learnt from: dennisYatunin
PR: CliMA/ClimaComms.jl#111
File: src/devices.jl:445-453
Timestamp: 2025-05-03T03:04:41.565Z
Learning: In Julia, a for-loop evaluates to `nothing` when it completes execution, so there's no need to explicitly add `nothing` at the end of a for-loop to maintain consistent return values in control flow branches.
```

```
Learnt from: dennisYatunin
PR: CliMA/ClimaComms.jl#111
File: src/devices.jl:472-482
Timestamp: 2025-05-03T21:13:22.170Z
Learning: ClimaComms.jl is designed to work with iterators that support both `length` and `getindex` methods, as these capabilities are required for GPU compatibility. Iterators without these methods are intentionally not supported.
```

</details>

</details><details>
<summary>🪛 GitHub Actions: JuliaFormatter</summary>

<details>
<summary>ext/ClimaCommsCUDAExt.jl</summary>

[error] 112-119: Git diff detected changes in ext/ClimaCommsCUDAExt.jl. The pipeline failed with exit code 1, indicating an issue with uncommitted or unexpected changes.

</details>

</details>

</details>

<details>
<summary>⏰ Context from checks skipped due to timeout of 90000ms (2)</summary>

* GitHub Check: test-os (macos-latest)
* GitHub Check: docbuild

</details>

<details>
<summary>🔇 Additional comments (8)</summary><blockquote>

<details>
<summary>ext/ClimaCommsCUDAExt.jl (4)</summary>

`56-67`: **Well-structured helper functions**

These helper functions nicely abstract common CUDA thread management patterns.

---

`76-96`: **Good handling of edge cases**

The uncoarsened implementation correctly handles:
- Empty iterators
- Device limits
- Threads/blocks configuration
- Fallback to coarsening when needed

One edge case is appropriately checked with early return on line 77.

---

`98-131`: **Efficient coarsened implementation**

The implementation uses the grid-stride loop pattern with adaptive coarsening. The approach correctly validates inputs and handles memory coalescing concerns.

<details>
<summary>🧰 Tools</summary>

<details>
<summary>🪛 GitHub Actions: JuliaFormatter</summary>

[error] 112-119: Git diff detected changes in ext/ClimaCommsCUDAExt.jl. The pipeline failed with exit code 1, indicating an issue with uncommitted or unexpected changes.

</details>

</details>

---

`1-133`: <details>
<summary>❓ Verification inconclusive</summary>

**Fix JuliaFormatter pipeline failure**

There's a formatter error in the pipeline. Run JuliaFormatter locally before committing.

---

🏁 Script executed:

```shell
#!/bin/bash
# Check if JuliaFormatter can be run locally
julia -e 'try; using JuliaFormatter; println("JuliaFormatter available"); catch; println("JuliaFormatter not found"); end'
```

Length of output: 220

---



</details>

**Run JuliaFormatter locally to fix formatting errors**

CI’s formatting check is failing on `ext/ClimaCommsCUDAExt.jl`. Please run JuliaFormatter before committing, for example:

```bash
julia --project=. -e 'using JuliaFormatter; format("ext/ClimaCommsCUDAExt.jl"; overwrite=true)'
# or to format the entire repo:
julia --project=. -e 'using JuliaFormatter; format(".", recursive=true; overwrite=true)'
```

<details>
<summary>🧰 Tools</summary>

<details>
<summary>🪛 GitHub Actions: JuliaFormatter</summary>

[error] 112-119: Git diff detected changes in ext/ClimaCommsCUDAExt.jl. The pipeline failed with exit code 1, indicating an issue with uncommitted or unexpected changes.

</details>

</details>

</details>
<details>
<summary>src/devices.jl (4)</summary>

`351-427`: **Excellent comprehensive docstring**

The docstring clearly explains:
- Device-specific behaviors
- Coarsening strategies and their implications
- Common pitfalls and debugging tips

---

`428-477`: **Clean and robust macro implementation**

The macro properly:
- Parses arguments with sensible defaults
- Validates inputs
- Prevents nested loops
- Dispatches correctly based on device

---

`482-497`: **Good support for different CPU schedulers**

The implementations correctly match Julia's threading schedulers, including conditional compilation for the newer `:greedy` scheduler in Julia 1.11+.

---

`499-511`: **Robust manual CPU coarsening implementation**

The manual coarsening implementation:
- Validates inputs
- Ensures one-based indexing
- Uses static scheduling for threads
- Correctly handles the last partial chunk

</details>

</blockquote></details>

</details>

<!-- This is an auto-generated comment by CodeRabbit for review status -->
state: COMMENTED
author_association: NONE
submitted_at: 2025-05-07T11:54:22Z
user: login: coderabbitai[bot]
body: **Actionable comments posted: 0**

<details>
<summary>🧹 Nitpick comments (1)</summary><blockquote>

<details>
<summary>src/devices.jl (1)</summary><blockquote>

`499-511`: **Missing word in error message and potential performance optimization**

Two minor issues:
1. Error message missing a word 
2. `min()` called repeatedly in the loop


```diff
- items_in_thread > 0 || throw(ArgumentError("`coarsen` must positive"))
+ items_in_thread > 0 || throw(ArgumentError("`coarsen` must be positive"))
```


Also consider caching the min calculation:
```diff
        first_item_index = items_in_thread * (thread_index - 1) + 1
        last_item_index = items_in_thread * thread_index
-       for item_index in first_item_index:min(last_item_index, length(itr))
+       end_idx = min(last_item_index, length(itr))
+       for item_index in first_item_index:end_idx
            @inbounds f(itr[item_index])
        end
```

</blockquote></details>

</blockquote></details>

<details>
<summary>📜 Review details</summary>

**Configuration used: CodeRabbit UI**
**Review profile: CHILL**
**Plan: Pro**

<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between bf2b840864d94940bfcc589a74c6b04c811f1c85 and 47da27fca0009ab11dfd900887966996a2f9e941.

</details>

<details>
<summary>📒 Files selected for processing (3)</summary>

* `ext/ClimaCommsCUDAExt.jl` (2 hunks)
* `src/devices.jl` (1 hunks)
* `test/runtests.jl` (1 hunks)

</details>

<details>
<summary>🚧 Files skipped from review as they are similar to previous changes (2)</summary>

* ext/ClimaCommsCUDAExt.jl
* test/runtests.jl

</details>

<details>
<summary>🧰 Additional context used</summary>

<details>
<summary>🧠 Learnings (1)</summary>

<details>
<summary>src/devices.jl (2)</summary>

```
Learnt from: dennisYatunin
PR: CliMA/ClimaComms.jl#111
File: src/devices.jl:445-453
Timestamp: 2025-05-03T03:04:41.565Z
Learning: In Julia, a for-loop evaluates to `nothing` when it completes execution, so there's no need to explicitly add `nothing` at the end of a for-loop to maintain consistent return values in control flow branches.
```

```
Learnt from: dennisYatunin
PR: CliMA/ClimaComms.jl#111
File: src/devices.jl:472-482
Timestamp: 2025-05-03T21:13:22.170Z
Learning: ClimaComms.jl is designed to work with iterators that support both `length` and `getindex` methods, as these capabilities are required for GPU compatibility. Iterators without these methods are intentionally not supported.
```

</details>

</details>

</details>

<details>
<summary>⏰ Context from checks skipped due to timeout of 90000ms (1)</summary>

* GitHub Check: docbuild

</details>

<details>
<summary>🔇 Additional comments (5)</summary><blockquote>

<details>
<summary>src/devices.jl (5)</summary>

`428-464`: **Well-structured macro with clear validation and defaults**

The macro implementation is clean with good argument parsing and error handling. It correctly handles various input forms and provides meaningful defaults.

---

`466-477`: **Device expression might be evaluated twice**

The ternary operator evaluates `device_expr` twice if it's not a `CPUSingleThreaded`. This could cause unexpected behavior if the expression has side effects.

Consider using a temporary variable:

```diff
- device = $(esc(device_expr))
- device isa CPUSingleThreaded ? $(esc(loop_expr)) :
- threaded(
+ let device = $(esc(device_expr))
+     device isa CPUSingleThreaded ? $(esc(loop_expr)) :
+     threaded(
```

---

`479-481`: **LGTM: Good separation for GPU-specific kwargs**

Clean approach to drop GPU-specific keyword arguments for non-GPU devices.

---

`482-497`: **Clean implementation of CPU scheduling strategies**

The threaded function implementations for dynamic, static, and greedy scheduling are concise and correct.

---

`351-427`: **Excellent detailed docstring for @threaded macro**

The documentation is comprehensive, covering:
- Device-specific behaviors
- Coarsening options and their performance implications
- Block dimension configuration
- Practical usage notes including error handling

This level of detail will help users correctly utilize the macro across different devices.

</details>

</blockquote></details>

</details>

<!-- This is an auto-generated comment by CodeRabbit for review status -->
state: COMMENTED
author_association: NONE
submitted_at: 2025-05-07T11:57:25Z
user: login: coderabbitai[bot]
body: **Actionable comments posted: 0**

<details>
<summary>♻️ Duplicate comments (1)</summary><blockquote>

<details>
<summary>src/devices.jl (1)</summary><blockquote>

`493-501`: **Consider using a let block to avoid evaluating device twice.**

The ternary expression evaluates `device_expr` twice, which could be inefficient if it has side effects.

```diff
-        device = $(esc(device_expr))
-        device isa CPUSingleThreaded ? $(esc(loop_expr)) :
-        threaded(
-            $(esc(var_expr)) -> $(esc(loop_body)),
-            device,
-            $(esc(coarsen_expr)),
-            $(esc(itr_expr));
-            block_size = $(esc(block_size_expr)),
-        )
+        let device = $(esc(device_expr))
+            if device isa CPUSingleThreaded
+                $(esc(loop_expr))
+            else
+                threaded(
+                    $(esc(var_expr)) -> $(esc(loop_body)),
+                    device,
+                    $(esc(coarsen_expr)),
+                    $(esc(itr_expr));
+                    block_size = $(esc(block_size_expr)),
+                )
+            end
+        end
```

</blockquote></details>

</blockquote></details>

<details>
<summary>🧹 Nitpick comments (3)</summary><blockquote>

<details>
<summary>docs/src/index.md (3)</summary><blockquote>

`67-67`: **Missing comma before "provides".**

```diff
-To implement kernels that work across all `Device`s, `ClimaComms.jl` provides
+To implement kernels that work across all `Device`s, `ClimaComms.jl` provides
```

<details>
<summary>🧰 Tools</summary>

<details>
<summary>🪛 LanguageTool</summary>

[uncategorized] ~67-~67: A punctuation mark might be missing here.
Context: ...Device`s, `ClimaComms.jl` provides the [`@threaded`](@ref ClimaComms.@threaded) m...

(AI_EN_LECTOR_MISSING_PUNCTUATION)

</details>

</details>

---

`91-91`: **Simplify phrasing.**

```diff
-global variables and type variables defined outside of a
+global variables and type variables defined outside a
```

<details>
<summary>🧰 Tools</summary>

<details>
<summary>🪛 LanguageTool</summary>

[style] ~91-~91: This phrase is redundant. Consider using “outside”.
Context: ...al variables and type variables defined outside of a     loop should be avoided when the l...

(OUTSIDE_OF)

</details>

</details>

---

`93-93`: **Missing comma before "for".**

```diff
-docstring of [`@threaded`](@ref ClimaComms.@threaded) for more information.
+docstring of [`@threaded`](@ref ClimaComms.@threaded), for more information.
```

<details>
<summary>🧰 Tools</summary>

<details>
<summary>🪛 LanguageTool</summary>

[uncategorized] ~93-~93: A punctuation mark might be missing here.
Context: ...zed on a GPU. See the     docstring of [`@threaded`](@ref ClimaComms.@threaded) f...

(AI_EN_LECTOR_MISSING_PUNCTUATION)

</details>

</details>

</blockquote></details>

</blockquote></details>

<details>
<summary>📜 Review details</summary>

**Configuration used: CodeRabbit UI**
**Review profile: CHILL**
**Plan: Pro**

<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between 47da27fca0009ab11dfd900887966996a2f9e941 and 9e3bf307356ff9d0837205a3769e06e5f034e492.

</details>

<details>
<summary>📒 Files selected for processing (6)</summary>

* `.github/workflows/OS-Tests.yml` (1 hunks)
* `docs/src/index.md` (2 hunks)
* `ext/ClimaCommsCUDAExt.jl` (2 hunks)
* `src/devices.jl` (1 hunks)
* `test/hygiene.jl` (1 hunks)
* `test/runtests.jl` (1 hunks)

</details>

<details>
<summary>🚧 Files skipped from review as they are similar to previous changes (4)</summary>

* .github/workflows/OS-Tests.yml
* test/hygiene.jl
* ext/ClimaCommsCUDAExt.jl
* test/runtests.jl

</details>

<details>
<summary>🧰 Additional context used</summary>

<details>
<summary>🧠 Learnings (1)</summary>

<details>
<summary>src/devices.jl (2)</summary>

```
Learnt from: dennisYatunin
PR: CliMA/ClimaComms.jl#111
File: src/devices.jl:445-453
Timestamp: 2025-05-03T03:04:41.565Z
Learning: In Julia, a for-loop evaluates to `nothing` when it completes execution, so there's no need to explicitly add `nothing` at the end of a for-loop to maintain consistent return values in control flow branches.
```

```
Learnt from: dennisYatunin
PR: CliMA/ClimaComms.jl#111
File: src/devices.jl:472-482
Timestamp: 2025-05-03T21:13:22.170Z
Learning: ClimaComms.jl is designed to work with iterators that support both `length` and `getindex` methods, as these capabilities are required for GPU compatibility. Iterators without these methods are intentionally not supported.
```

</details>

</details><details>
<summary>🪛 LanguageTool</summary>

<details>
<summary>docs/src/index.md</summary>

[uncategorized] ~67-~67: A punctuation mark might be missing here.
Context: ...Device`s, `ClimaComms.jl` provides the [`@threaded`](@ref ClimaComms.@threaded) m...

(AI_EN_LECTOR_MISSING_PUNCTUATION)

---

[style] ~87-~87: Using many exclamation marks might seem excessive (in this case: 7 exclamation marks for a text that’s 4130 characters long)
Context: ...guide/index.html#multiprocessor-level)  !!! note     Executing code on GPUs require...

(EN_EXCESSIVE_EXCLAMATION)

---

[style] ~91-~91: This phrase is redundant. Consider using “outside”.
Context: ...al variables and type variables defined outside of a     loop should be avoided when the l...

(OUTSIDE_OF)

---

[uncategorized] ~93-~93: A punctuation mark might be missing here.
Context: ...zed on a GPU. See the     docstring of [`@threaded`](@ref ClimaComms.@threaded) f...

(AI_EN_LECTOR_MISSING_PUNCTUATION)

</details>

</details>

</details>

<details>
<summary>⏰ Context from checks skipped due to timeout of 90000ms (1)</summary>

* GitHub Check: docbuild

</details>

<details>
<summary>🔇 Additional comments (4)</summary><blockquote>

<details>
<summary>docs/src/index.md (1)</summary>

`63-93`: **Clear and useful documentation for the new `@threaded` macro.**

The new section explaining the `@threaded` macro provides excellent guidance for writing device-agnostic kernels. The example is succinct and the performance tuning options are well explained.

<details>
<summary>🧰 Tools</summary>

<details>
<summary>🪛 LanguageTool</summary>

[uncategorized] ~67-~67: A punctuation mark might be missing here.
Context: ...Device`s, `ClimaComms.jl` provides the [`@threaded`](@ref ClimaComms.@threaded) m...

(AI_EN_LECTOR_MISSING_PUNCTUATION)

---

[style] ~87-~87: Using many exclamation marks might seem excessive (in this case: 7 exclamation marks for a text that’s 4130 characters long)
Context: ...guide/index.html#multiprocessor-level)  !!! note     Executing code on GPUs require...

(EN_EXCESSIVE_EXCLAMATION)

---

[style] ~91-~91: This phrase is redundant. Consider using “outside”.
Context: ...al variables and type variables defined outside of a     loop should be avoided when the l...

(OUTSIDE_OF)

---

[uncategorized] ~93-~93: A punctuation mark might be missing here.
Context: ...zed on a GPU. See the     docstring of [`@threaded`](@ref ClimaComms.@threaded) f...

(AI_EN_LECTOR_MISSING_PUNCTUATION)

</details>

</details>

</details>
<details>
<summary>src/devices.jl (3)</summary>

`351-452`: **Excellent docstring with clear examples.**

The docstring comprehensively explains the `@threaded` macro behavior across different devices and includes examples illustrating proper usage patterns and common pitfalls.

---

`453-502`: **Well-structured macro implementation.**

The macro implementation properly handles different argument combinations and dispatches based on device type. The code is clean and readable.

---

`504-536`: **Good implementation of coarsening strategies.**

The threaded function overloads properly handle different coarsening strategies and Julia version compatibility.

</details>

</blockquote></details>

</details>

<!-- This is an auto-generated comment by CodeRabbit for review status -->
state: COMMENTED
author_association: NONE
submitted_at: 2025-05-07T23:37:48Z
user: login: coderabbitai[bot]
body: **Actionable comments posted: 0**

<details>
<summary>🧹 Nitpick comments (1)</summary><blockquote>

<details>
<summary>src/devices.jl (1)</summary><blockquote>

`453-502`: **Consider evaluating device_expr only once.**

The macro implementation looks robust, but there's a potential inefficiency in line 493 where `device_expr` could be evaluated twice if it contains complex expressions with side effects.

```diff
-        device = $(esc(device_expr))
-        device isa CPUSingleThreaded ? $(esc(loop_expr)) :
+        let device = $(esc(device_expr))
+            device isa CPUSingleThreaded ? $(esc(loop_expr)) :
+            threaded(
+                $(esc(var_expr)) -> $(esc(loop_body)),
+                device,
+                $(esc(coarsen_expr)),
+                $(esc(itr_expr));
+                block_size = $(esc(block_size_expr)),
+            )
+        end
-        threaded(
-            $(esc(var_expr)) -> $(esc(loop_body)),
-            device,
-            $(esc(coarsen_expr)),
-            $(esc(itr_expr));
-            block_size = $(esc(block_size_expr)),
-        )
```

</blockquote></details>

</blockquote></details>

<details>
<summary>📜 Review details</summary>

**Configuration used: CodeRabbit UI**
**Review profile: CHILL**
**Plan: Pro**

<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between 9e3bf307356ff9d0837205a3769e06e5f034e492 and a0b33fab990d200635305dbca9e2bffe44429aaa.

</details>

<details>
<summary>📒 Files selected for processing (6)</summary>

* `.github/workflows/OS-Tests.yml` (1 hunks)
* `docs/src/index.md` (2 hunks)
* `ext/ClimaCommsCUDAExt.jl` (2 hunks)
* `src/devices.jl` (1 hunks)
* `test/hygiene.jl` (1 hunks)
* `test/runtests.jl` (1 hunks)

</details>

<details>
<summary>🚧 Files skipped from review as they are similar to previous changes (4)</summary>

* test/hygiene.jl
* .github/workflows/OS-Tests.yml
* ext/ClimaCommsCUDAExt.jl
* test/runtests.jl

</details>

<details>
<summary>🧰 Additional context used</summary>

<details>
<summary>🧠 Learnings (1)</summary>

<details>
<summary>src/devices.jl (2)</summary>

```
Learnt from: dennisYatunin
PR: CliMA/ClimaComms.jl#111
File: src/devices.jl:445-453
Timestamp: 2025-05-03T03:04:41.565Z
Learning: In Julia, a for-loop evaluates to `nothing` when it completes execution, so there's no need to explicitly add `nothing` at the end of a for-loop to maintain consistent return values in control flow branches.
```

```
Learnt from: dennisYatunin
PR: CliMA/ClimaComms.jl#111
File: src/devices.jl:472-482
Timestamp: 2025-05-03T21:13:22.170Z
Learning: ClimaComms.jl is designed to work with iterators that support both `length` and `getindex` methods, as these capabilities are required for GPU compatibility. Iterators without these methods are intentionally not supported.
```

</details>

</details><details>
<summary>🪛 LanguageTool</summary>

<details>
<summary>docs/src/index.md</summary>

[uncategorized] ~67-~67: A punctuation mark might be missing here.
Context: ...Device`s, `ClimaComms.jl` provides the [`@threaded`](@ref ClimaComms.@threaded) m...

(AI_EN_LECTOR_MISSING_PUNCTUATION)

---

[style] ~87-~87: Using many exclamation marks might seem excessive (in this case: 7 exclamation marks for a text that’s 4127 characters long)
Context: ...guide/index.html#multiprocessor-level)  !!! note     Executing code on GPUs require...

(EN_EXCESSIVE_EXCLAMATION)

---

[uncategorized] ~93-~93: A punctuation mark might be missing here.
Context: ...zed on a GPU. See the     docstring of [`@threaded`](@ref ClimaComms.@threaded) f...

(AI_EN_LECTOR_MISSING_PUNCTUATION)

</details>

</details>

</details>

<details>
<summary>⏰ Context from checks skipped due to timeout of 90000ms (1)</summary>

* GitHub Check: docbuild

</details>

<details>
<summary>🔇 Additional comments (4)</summary><blockquote>

<details>
<summary>docs/src/index.md (2)</summary>

`20-22`: **Well-structured device list update.**

The device list now properly includes `CPUMultiThreaded` as a fully supported option alongside other devices.

---

`64-93`: **Excellent documentation for the generic kernels implementation.**

The new "Writing generic kernels" section provides clear, concise guidance on using the `@threaded` macro across different devices with helpful examples and performance tuning options.

<details>
<summary>🧰 Tools</summary>

<details>
<summary>🪛 LanguageTool</summary>

[uncategorized] ~67-~67: A punctuation mark might be missing here.
Context: ...Device`s, `ClimaComms.jl` provides the [`@threaded`](@ref ClimaComms.@threaded) m...

(AI_EN_LECTOR_MISSING_PUNCTUATION)

---

[style] ~87-~87: Using many exclamation marks might seem excessive (in this case: 7 exclamation marks for a text that’s 4127 characters long)
Context: ...guide/index.html#multiprocessor-level)  !!! note     Executing code on GPUs require...

(EN_EXCESSIVE_EXCLAMATION)

---

[uncategorized] ~93-~93: A punctuation mark might be missing here.
Context: ...zed on a GPU. See the     docstring of [`@threaded`](@ref ClimaComms.@threaded) f...

(AI_EN_LECTOR_MISSING_PUNCTUATION)

</details>

</details>

</details>
<details>
<summary>src/devices.jl (2)</summary>

`351-452`: **Comprehensive docstring with great examples and warnings.**

The docstring thoroughly explains different device behaviors, coarsening options, and includes practical examples showing GPU inference pitfalls. Clear guidance for users on both usage and potential issues.

---

`504-536`: **Well-structured threaded implementations.**

The implementation correctly handles different threading strategies across devices, with appropriate version checking for Julia 1.11+ features. The integer-based coarsening implementation is particularly well thought out.

</details>

</blockquote></details>

</details>

<!-- This is an auto-generated comment by CodeRabbit for review status -->
state: COMMENTED
author_association: NONE
submitted_at: 2025-05-07T23:49:04Z
user: login: coderabbitai[bot]
body: **Actionable comments posted: 0**

<details>
<summary>🧹 Nitpick comments (2)</summary><blockquote>

<details>
<summary>src/devices.jl (1)</summary><blockquote>

`452-501`: **Well-designed macro implementation**

The macro implementation effectively parses arguments, validates the loop expression, and dispatches to appropriate execution paths based on device type. The ternary expression on line 492 evaluates `device_expr` twice, which could be inefficient if it's complex or has side effects.



```diff
-        device = $(esc(device_expr))
-        device isa CPUSingleThreaded ? $(esc(loop_expr)) :
-        threaded(
-            $(esc(var_expr)) -> $(esc(loop_body)),
-            device,
-            $(esc(coarsen_expr)),
-            $(esc(itr_expr));
-            block_size = $(esc(block_size_expr)),
-        )
+        let device = $(esc(device_expr))
+            if device isa CPUSingleThreaded
+                $(esc(loop_expr))
+            else
+                threaded(
+                    $(esc(var_expr)) -> $(esc(loop_body)),
+                    device,
+                    $(esc(coarsen_expr)),
+                    $(esc(itr_expr));
+                    block_size = $(esc(block_size_expr)),
+                )
+            end
+        end
```

</blockquote></details>
<details>
<summary>docs/src/index.md (1)</summary><blockquote>

`64-94`: **Clear and concise documentation**

The "Writing generic kernels" section effectively explains the purpose and usage of the `@threaded` macro with a practical example. The note about GPU static compilation requirements is particularly valuable.



Consider adding commas after "To implement kernels that work across all `Device`s" and after "See the docstring of [`@threaded`](@ref ClimaComms.@threaded)" to improve readability.

<details>
<summary>🧰 Tools</summary>

<details>
<summary>🪛 LanguageTool</summary>

[uncategorized] ~67-~67: A punctuation mark might be missing here.
Context: ...Device`s, `ClimaComms.jl` provides the [`@threaded`](@ref ClimaComms.@threaded) m...

(AI_EN_LECTOR_MISSING_PUNCTUATION)

---

[style] ~87-~87: Using many exclamation marks might seem excessive (in this case: 7 exclamation marks for a text that’s 4127 characters long)
Context: ...guide/index.html#multiprocessor-level)  !!! note     Executing code on GPUs require...

(EN_EXCESSIVE_EXCLAMATION)

---

[uncategorized] ~93-~93: A punctuation mark might be missing here.
Context: ...zed on a GPU. See the     docstring of [`@threaded`](@ref ClimaComms.@threaded) f...

(AI_EN_LECTOR_MISSING_PUNCTUATION)

</details>

</details>

</blockquote></details>

</blockquote></details>

<details>
<summary>📜 Review details</summary>

**Configuration used: CodeRabbit UI**
**Review profile: CHILL**
**Plan: Pro**

<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between a0b33fab990d200635305dbca9e2bffe44429aaa and 6ea2975e4d0355f9912e00860e033dc945972413.

</details>

<details>
<summary>📒 Files selected for processing (6)</summary>

* `.github/workflows/OS-Tests.yml` (1 hunks)
* `docs/src/index.md` (2 hunks)
* `ext/ClimaCommsCUDAExt.jl` (2 hunks)
* `src/devices.jl` (1 hunks)
* `test/hygiene.jl` (1 hunks)
* `test/runtests.jl` (1 hunks)

</details>

<details>
<summary>🚧 Files skipped from review as they are similar to previous changes (3)</summary>

* .github/workflows/OS-Tests.yml
* test/hygiene.jl
* test/runtests.jl

</details>

<details>
<summary>🧰 Additional context used</summary>

<details>
<summary>🧠 Learnings (1)</summary>

<details>
<summary>src/devices.jl (2)</summary>

```
Learnt from: dennisYatunin
PR: CliMA/ClimaComms.jl#111
File: src/devices.jl:445-453
Timestamp: 2025-05-03T03:04:41.565Z
Learning: In Julia, a for-loop evaluates to `nothing` when it completes execution, so there's no need to explicitly add `nothing` at the end of a for-loop to maintain consistent return values in control flow branches.
```

```
Learnt from: dennisYatunin
PR: CliMA/ClimaComms.jl#111
File: src/devices.jl:472-482
Timestamp: 2025-05-03T21:13:22.170Z
Learning: ClimaComms.jl is designed to work with iterators that support both `length` and `getindex` methods, as these capabilities are required for GPU compatibility. Iterators without these methods are intentionally not supported.
```

</details>

</details><details>
<summary>🪛 LanguageTool</summary>

<details>
<summary>docs/src/index.md</summary>

[uncategorized] ~67-~67: A punctuation mark might be missing here.
Context: ...Device`s, `ClimaComms.jl` provides the [`@threaded`](@ref ClimaComms.@threaded) m...

(AI_EN_LECTOR_MISSING_PUNCTUATION)

---

[style] ~87-~87: Using many exclamation marks might seem excessive (in this case: 7 exclamation marks for a text that’s 4127 characters long)
Context: ...guide/index.html#multiprocessor-level)  !!! note     Executing code on GPUs require...

(EN_EXCESSIVE_EXCLAMATION)

---

[uncategorized] ~93-~93: A punctuation mark might be missing here.
Context: ...zed on a GPU. See the     docstring of [`@threaded`](@ref ClimaComms.@threaded) f...

(AI_EN_LECTOR_MISSING_PUNCTUATION)

</details>

</details>

</details>

<details>
<summary>⏰ Context from checks skipped due to timeout of 90000ms (1)</summary>

* GitHub Check: docbuild

</details>

<details>
<summary>🔇 Additional comments (4)</summary><blockquote>

<details>
<summary>src/devices.jl (1)</summary>

`503-535`: **Comprehensive implementation of threaded methods**

The implementation covers various scheduling strategies and coarsening options for CPU multi-threaded devices. The code is well-structured with appropriate error handling and boundary checks.

</details>
<details>
<summary>ext/ClimaCommsCUDAExt.jl (3)</summary>

`53-75`: **Well-implemented helper functions for CUDA operations**

These helper functions abstract away the complexities of CUDA thread indexing and device limits. The TODO comment appropriately notes future work for multi-dimensional thread blocks.

---

`76-96`: **Effective single-item-per-thread implementation**

The implementation handles edge cases properly, including empty iterators and large workloads that exceed device limits. The automatic fallback to coarsened execution is a good design choice.

---

`98-130`: **Grid-stride loop implementation follows CUDA best practices**

The implementation uses a grid-stride loop pattern for better memory coalescing, which is a recommended approach for CUDA kernels. It correctly adjusts coarsening based on device constraints.

</details>

</blockquote></details>

</details>

<!-- This is an auto-generated comment by CodeRabbit for review status -->
state: COMMENTED
author_association: NONE
submitted_at: 2025-05-08T00:15:40Z
user: login: coderabbitai[bot]
body: **Actionable comments posted: 0**

<details>
<summary>🧹 Nitpick comments (1)</summary><blockquote>

<details>
<summary>docs/src/index.md (1)</summary><blockquote>

`64-94`: **Excellent documentation for the new @threaded feature.**

The section clearly explains the macro's capabilities, provides a concrete example, and highlights important considerations for GPU compatibility. 

Minor suggestion: Add commas after "Device`s," on line 67 and after "GPU." on line 93 for better readability.

<details>
<summary>🧰 Tools</summary>

<details>
<summary>🪛 LanguageTool</summary>

[uncategorized] ~67-~67: A punctuation mark might be missing here.
Context: ...Device`s, `ClimaComms.jl` provides the [`@threaded`](@ref ClimaComms.@threaded) m...

(AI_EN_LECTOR_MISSING_PUNCTUATION)

---

[style] ~87-~87: Using many exclamation marks might seem excessive (in this case: 7 exclamation marks for a text that’s 4127 characters long)
Context: ...guide/index.html#multiprocessor-level)  !!! note     Executing code on GPUs require...

(EN_EXCESSIVE_EXCLAMATION)

---

[uncategorized] ~93-~93: A punctuation mark might be missing here.
Context: ...zed on a GPU. See the     docstring of [`@threaded`](@ref ClimaComms.@threaded) f...

(AI_EN_LECTOR_MISSING_PUNCTUATION)

</details>

</details>

</blockquote></details>

</blockquote></details>

<details>
<summary>📜 Review details</summary>

**Configuration used: CodeRabbit UI**
**Review profile: CHILL**
**Plan: Pro**

<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between 6ea2975e4d0355f9912e00860e033dc945972413 and 6b1ca40d18851a0a67c595246757b015c4c2fd67.

</details>

<details>
<summary>📒 Files selected for processing (6)</summary>

* `.github/workflows/OS-Tests.yml` (1 hunks)
* `docs/src/index.md` (2 hunks)
* `ext/ClimaCommsCUDAExt.jl` (2 hunks)
* `src/devices.jl` (1 hunks)
* `test/hygiene.jl` (1 hunks)
* `test/runtests.jl` (1 hunks)

</details>

<details>
<summary>🚧 Files skipped from review as they are similar to previous changes (3)</summary>

* .github/workflows/OS-Tests.yml
* test/hygiene.jl
* test/runtests.jl

</details>

<details>
<summary>🧰 Additional context used</summary>

<details>
<summary>🧠 Learnings (1)</summary>

<details>
<summary>src/devices.jl (2)</summary>

```
Learnt from: dennisYatunin
PR: CliMA/ClimaComms.jl#111
File: src/devices.jl:445-453
Timestamp: 2025-05-03T03:04:41.565Z
Learning: In Julia, a for-loop evaluates to `nothing` when it completes execution, so there's no need to explicitly add `nothing` at the end of a for-loop to maintain consistent return values in control flow branches.
```

```
Learnt from: dennisYatunin
PR: CliMA/ClimaComms.jl#111
File: src/devices.jl:472-482
Timestamp: 2025-05-03T21:13:22.170Z
Learning: ClimaComms.jl is designed to work with iterators that support both `length` and `getindex` methods, as these capabilities are required for GPU compatibility. Iterators without these methods are intentionally not supported.
```

</details>

</details><details>
<summary>🪛 LanguageTool</summary>

<details>
<summary>docs/src/index.md</summary>

[uncategorized] ~67-~67: A punctuation mark might be missing here.
Context: ...Device`s, `ClimaComms.jl` provides the [`@threaded`](@ref ClimaComms.@threaded) m...

(AI_EN_LECTOR_MISSING_PUNCTUATION)

---

[style] ~87-~87: Using many exclamation marks might seem excessive (in this case: 7 exclamation marks for a text that’s 4127 characters long)
Context: ...guide/index.html#multiprocessor-level)  !!! note     Executing code on GPUs require...

(EN_EXCESSIVE_EXCLAMATION)

---

[uncategorized] ~93-~93: A punctuation mark might be missing here.
Context: ...zed on a GPU. See the     docstring of [`@threaded`](@ref ClimaComms.@threaded) f...

(AI_EN_LECTOR_MISSING_PUNCTUATION)

</details>

</details>

</details>

<details>
<summary>⏰ Context from checks skipped due to timeout of 90000ms (1)</summary>

* GitHub Check: docbuild

</details>

<details>
<summary>🔇 Additional comments (8)</summary><blockquote>

<details>
<summary>docs/src/index.md (1)</summary>

`20-22`: **Good update to supported devices.**

Documentation now correctly lists all three supported device types, removing the obsolete warning about `CPUMultiThreaded`.

</details>
<details>
<summary>ext/ClimaCommsCUDAExt.jl (4)</summary>

`53-55`: **Clear TODOs for future extensions.**

The comment appropriately documents the current limitations and future plans for supporting multi-dimensional thread blocks and multiple iterators.

---

`56-75`: **Good helper functions for CUDA kernel execution.**

These utility functions extract common CUDA-specific operations to improve code readability. The explanatory comments help understand their purpose.

---

`76-96`: **Well-implemented single-item-per-thread kernel launcher.**

The function correctly:
1. Checks for empty iterators and one-based indexing
2. Configures thread and block sizes optimally
3. Falls back to coarsening when exceeding GPU limits
4. Uses synchronous kernel execution

---

`98-130`: **Solid implementation of coarsened kernel launcher with grid-stride loop.**

The implementation properly handles:
1. Input validation
2. Thread coarsening calculations
3. Device limitations
4. Adaptive coarsening when needed

The grid-stride loop pattern reference is helpful for maintainers.

</details>
<details>
<summary>src/devices.jl (3)</summary>

`351-450`: **Comprehensive docstring for the @threaded macro.**

Excellent documentation that:
1. Explains device-specific behaviors
2. Details coarsening options for each device
3. Provides clear examples showing inference issues
4. References external resources for advanced use cases

This will be valuable for users implementing device-agnostic code.

---

`451-500`: **Well-implemented macro with flexible argument handling.**

The macro:
1. Supports optional device specification
2. Handles both symbol and integer coarsening
3. Offers block size control for GPUs
4. Prevents nested loops
5. Efficiently handles single-threaded execution

---

`502-534`: **Complete set of threaded implementations for different scheduling strategies.**

The code properly implements:
1. Dynamic, static, and greedy (Julia 1.11+) schedulers for CPUMultiThreaded
2. Manual coarsening with integer-based thread assignment
3. Version-specific features with @static guards

</details>

</blockquote></details>

</details>

<!-- This is an auto-generated comment by CodeRabbit for review status -->
state: COMMENTED
author_association: NONE
submitted_at: 2025-05-08T01:39:14Z

Review comments:
diff_hunk: @@ -376,3 +346,127 @@ macro assert(device, cond, text = nothing)
 end
 assert(::AbstractCPUDevice, cond::C, text::T) where {C, T} =
     isnothing(text) ? (Base.@assert cond()) : (Base.@assert cond() text())
+
+
+"""
+    @threaded [coarsen=...] [device] for ... end
+
+Device-flexible generalization of `Threads.@threads`, which distributes the
+iterations of a for-loop across multiple threads, with the option to control
+thread coarsening. Coarsening makes each thread evaluate more than one iteration
+of the loop, which can improve performance by reducing the runtime overhead
+of launching additional threads (though too much coarsening worsens performance
+because it reduces parallelization). The `device` is either inferred by calling
+`ClimaComms.device()`, or it can be specified manually, with the following
+device-dependent behavior:
+
+ - When `device` is a `CPUSingleThreaded()`, the loop is evaluated as-is. This
+   avoids the runtime overhead of calling `Threads.@threads` with a single
+   thread, and, when the device type is statically inferrable, it also avoids
+   compilation overhead.
+
+ - When `device` is a `CPUMultiThreaded()`, the loop is passed to
+   `Threads.@threads`. This supports three different kinds of "schedulers" for
+   determining how many iterations of the loop to evaluate in each thread:
+     1) (default) a "dynamic" scheduler that changes the number of iterations as
+        new threads are launched,
+     2) a "static" scheduler that evaluates a fixed number of iterations per
+        thread, and
+     3) a "greedy" scheduler that uses a small number of threads and continues
+        evaluating iterations in each thread until the loop is completed.
+   Setting `coarsen` to `:dynamic` or `:greedy` launches threads with those
+   schedulers, while setting it to an integer value launches threads with the
+   static scheduler. See also [`Threads.@threads`](@ref).
+
+ - When `device` is a `CUDADevice()`, the loop is compiled with `CUDA.@cuda` and
+   run with `CUDA.@sync`. Since CUDA launches all threads in parallel, only
+   static scheduling can be used. Setting `coarsen` to any symbol causes each
+   thread to evaluate a single iteration, while setting it to an integer value
+   causes each thread to evaluate at least that number of iterations. By
+   default, only one iteration is evaluated per thread. If the total number of
+   iterations in the loop is extremely large, the specified coarsening may
+   require more threads than can be simultaneously launched on the GPU, in which
+   case the amount of coarsening is automatically increased.
+
+NOTE: Whenever a value in the body of the for-loop has a type that cannot be
+inferred by the compiler, an `InvalidIRError` is thrown for a `CUDADevice()`. In
+particular, the types of global variables are not inferrable, so this macro
+must be wrapped in a function when it is used in the REPL, e.g.,
+
+```julia-repl
+julia> a = CUDA.CuArray{Int}(undef, 100); b = similar(a);
+
+julia> kernel!(a, b) = ClimaComms.@threaded for i in axes(a, 1)
+           a[i] = b[i]
+       end
+kernel! (generic function with 1 method)
+
+julia> kernel!(a, b)
+
+julia> ClimaComms.@threaded for i in axes(a, 1)
+    a[i] = b[i]
+end
+ERROR: InvalidIRError: ...
+```
+
+To fix other kinds of inference issues on GPUs, especially ones brought about by
+indexing into iterators with nonuniform element types, see
+[`UnrolledUtilities.jl`](https://github.com/CliMA/UnrolledUtilities.jl).
+"""
+macro threaded(args...)
+    usage_string = "Usage: @threaded [coarsen=...] [device] for ... end"
+    n_args = length(args)
+    1 <= n_args <= 3 || throw(ArgumentError(usage_string))
+
+    device_expr = :(ClimaComms.device())
+    coarsen_expr = :(:dynamic)
+    for arg_number in 1:(n_args - 1)
+        device_or_coarsen_expr = args[arg_number]
+        if Meta.isexpr(device_or_coarsen_expr, :(=))
+            (kwarg_name, kwarg_value) = device_or_coarsen_expr.args
+            kwarg_name == :coarsen || throw(ArgumentError(usage_string))
+            coarsen_expr = kwarg_value
+        else
+            arg_number == n_args - 1 || throw(ArgumentError(usage_string))
+            device_expr = device_or_coarsen_expr
+        end
+    end
+    if coarsen_expr isa QuoteNode
+        coarsen_expr = :(Val($(coarsen_expr)))
+    end
+
+    loop_expr = args[n_args]
+    Meta.isexpr(loop_expr, :for) || throw(ArgumentError(usage_string))
+    (var_and_itr_expr, loop_body) = loop_expr.args
+    Meta.isexpr(var_and_itr_expr, :(=)) ||
+        throw(ArgumentError("@threaded does not support nested loops"))
+    (var_expr, itr_expr) = var_and_itr_expr.args
+
+    return quote
+        $(esc(device_expr)) isa CPUSingleThreaded ? $(esc(loop_expr)) :
+        threaded(
+            $(esc(var_expr)) -> $(esc(loop_body)),
+            $(esc(device_expr)),
+            $(esc(coarsen_expr)),
+            $(esc(itr_expr)),
+        )
+    end
path: src/devices.jl
user: login: coderabbitai[bot]
body: _🛠️ Refactor suggestion_

**Ensure single-threaded branch mimics `Threads.@threads` semantics**  

The ternary currently returns the value of the loop body when `_dev isa CPUSingleThreaded`, whereas every other branch (and `Threads.@threads`) returns `nothing`.  
Returning an accidental value can silently change user code that relies on “void” semantics.

```diff
-        $(esc(device_expr)) isa CPUSingleThreaded ? $(esc(loop_expr)) :
-        threaded(
-            $(esc(var_expr)) -> $(esc(loop_body)),
-            $(esc(device_expr)),
-            $(esc(coarsen_expr)),
-            $(esc(itr_expr)),
-        )
+        let _dev = $(esc(device_expr))               # evaluate once
+            if _dev isa CPUSingleThreaded
+                $(esc(loop_expr))
+                nothing                               # keep return type consistent
+            else
+                threaded(
+                    $(esc(var_expr)) -> $(esc(loop_body)),
+                    _dev,
+                    $(esc(coarsen_expr)),
+                    $(esc(itr_expr)),
+                )
+            end
+        end
```

This also avoids evaluating `device_expr` multiple times.  



<!-- suggestion_start -->

<details>
<summary>📝 Committable suggestion</summary>

> ‼️ **IMPORTANT**
> Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

`````suggestion
    return quote
        let _dev = $(esc(device_expr))               # evaluate once
            if _dev isa CPUSingleThreaded
                $(esc(loop_expr))
                nothing                               # keep return type consistent
            else
                threaded(
                    $(esc(var_expr)) -> $(esc(loop_body)),
                    _dev,
                    $(esc(coarsen_expr)),
                    $(esc(itr_expr)),
                )
            end
        end
    end
`````

</details>

<!-- suggestion_end -->

<!-- This is an auto-generated reply by CodeRabbit -->
created_at: 2025-05-03T02:56:56Z
author_association: NONE
diff_hunk: @@ -50,4 +50,47 @@ ClimaComms.elapsed(f::F, ::CUDADevice, args...; kwargs...) where {F} =
 ClimaComms.assert(::CUDADevice, cond::C, text::T) where {C, T} =
     isnothing(text) ? (CUDA.@cuassert cond()) : (CUDA.@cuassert cond() text())
 
+function threaded(f::F, ::CUDADevice, ::Val, itr) where {F}
+    max_n_threads =
+        CUDA.attribute(CUDA.device(), CUDA.DEVICE_ATTRIBUTE_MAX_GRID_DIM_X)
+    n_threads = length(itr)
+    n_threads <= max_n_threads || return threaded(f, CUDADevice(), 2, itr)
+
+    function call_f_for_one_item()
+        item_index =
+            (CUDA.blockIdx().x - 1) * CUDA.blockDim().x + CUDA.threadIdx().x
+        item_index <= length(itr) && f(itr[item_index])
+        return nothing
+    end
+
+    kernel = CUDA.@cuda launch=false call_f_for_one_item()
+    config = CUDA.launch_configuration(kernel.fun)
+    threads_in_block = min(n_threads, config.threads)
+    n_blocks = cld(n_threads, threads_in_block)
+    CUDA.@sync kernel(; threads = threads_in_block, blocks = n_blocks)
+end
+
+function threaded(f::F, ::CUDADevice, min_items_in_thread::Int, itr) where {F}
+    max_n_threads =
+        CUDA.attribute(CUDA.device(), CUDA.DEVICE_ATTRIBUTE_MAX_GRID_DIM_X)
+    n_threads = min(cld(length(itr), min_items_in_thread), max_n_threads)
+    items_in_thread = cld(length(itr), n_threads)
+
+    function call_f_for_multiple_items()
+        thread_index =
+            (CUDA.blockIdx().x - 1) * CUDA.blockDim().x + CUDA.threadIdx().x
+        first_item_index = items_in_thread * (thread_index - 1) + 1
+        last_item_index = items_in_thread * thread_index
+        for item_index in first_item_index:min(last_item_index, length(itr))
+            f(itr[item_index])
+        end
+    end
+
+    kernel = CUDA.@cuda launch=false call_f_for_multiple_items()
+    config = CUDA.launch_configuration(kernel.fun)
+    threads_in_block = min(n_threads, config.threads)
+    n_blocks = cld(n_threads, threads_in_block)
+    CUDA.@sync kernel(; threads = threads_in_block, blocks = n_blocks)
+end
path: ext/ClimaCommsCUDAExt.jl
user: login: coderabbitai[bot]
body: _🛠️ Refactor suggestion_

**Same empty-iterator & block-limit issue for the coarsened version**

Replicate the early‐return and correct block bound logic here to avoid invalid launches and unnecessary recursion.

<!-- This is an auto-generated comment by CodeRabbit -->

✅ Addressed in commits edd91be to 05ffda2
created_at: 2025-05-03T02:56:56Z
author_association: NONE
diff_hunk: @@ -50,4 +50,47 @@
 ClimaComms.assert(::CUDADevice, cond::C, text::T) where {C, T} =
     isnothing(text) ? (CUDA.@cuassert cond()) : (CUDA.@cuassert cond() text())
 
+function threaded(f::F, ::CUDADevice, ::Val, itr) where {F}
+    max_n_threads =
+        CUDA.attribute(CUDA.device(), CUDA.DEVICE_ATTRIBUTE_MAX_GRID_DIM_X)
+    n_threads = length(itr)
+    n_threads <= max_n_threads || return threaded(f, CUDADevice(), 2, itr)
+
+    function call_f_for_one_item()
+        item_index =
+            (CUDA.blockIdx().x - 1) * CUDA.blockDim().x + CUDA.threadIdx().x
+        item_index <= length(itr) && f(itr[item_index])
+        return nothing
+    end
+
+    kernel = CUDA.@cuda launch=false call_f_for_one_item()
+    config = CUDA.launch_configuration(kernel.fun)
+    threads_in_block = min(n_threads, config.threads)
+    n_blocks = cld(n_threads, threads_in_block)
+    CUDA.@sync kernel(; threads = threads_in_block, blocks = n_blocks)
+end
path: ext/ClimaCommsCUDAExt.jl
user: login: coderabbitai[bot]
body: _⚠️ Potential issue_

**Guard against empty ranges & oversubscription check is off-by-one**  

1. Launching a kernel with `threads = 0` or `blocks = 0` is illegal.  
2. The limit you compare against (`DEVICE_ATTRIBUTE_MAX_GRID_DIM_X`) bounds *blocks*, not total threads.  The current check is stricter than required and may still exceed the block limit after coarsening.

```diff
-    max_n_threads =
+    max_blocks =
         CUDA.attribute(CUDA.device(), CUDA.DEVICE_ATTRIBUTE_MAX_GRID_DIM_X)
-    n_threads = length(itr)
-    n_threads <= max_n_threads || return threaded(f, CUDADevice(), 2, itr)
+    n_items = length(itr)
+    n_items == 0 && return                    # fast-exit
+
+    kernel = CUDA.@cuda launch=false ()->nothing
+    config = CUDA.launch_configuration(kernel.fun)
+    threads_per_block = min(n_items, config.threads)
+    n_blocks = cld(n_items, threads_per_block)
+    n_blocks <= max_blocks ||
+        return threaded(f, CUDADevice(), 2, itr)   # recurse with coarsening
```

(Full rewrite shown for clarity; adapt locally.)  



> Committable suggestion skipped: line range outside the PR's diff.

<!-- This is an auto-generated comment by CodeRabbit -->

✅ Addressed in commit 670b793
created_at: 2025-05-03T02:56:57Z
author_association: NONE
diff_hunk: @@ -376,3 +346,127 @@ macro assert(device, cond, text = nothing)
 end
 assert(::AbstractCPUDevice, cond::C, text::T) where {C, T} =
     isnothing(text) ? (Base.@assert cond()) : (Base.@assert cond() text())
+
+
+"""
+    @threaded [coarsen=...] [device] for ... end
+
+Device-flexible generalization of `Threads.@threads`, which distributes the
+iterations of a for-loop across multiple threads, with the option to control
+thread coarsening. Coarsening makes each thread evaluate more than one iteration
+of the loop, which can improve performance by reducing the runtime overhead
+of launching additional threads (though too much coarsening worsens performance
+because it reduces parallelization). The `device` is either inferred by calling
+`ClimaComms.device()`, or it can be specified manually, with the following
+device-dependent behavior:
+
+ - When `device` is a `CPUSingleThreaded()`, the loop is evaluated as-is. This
+   avoids the runtime overhead of calling `Threads.@threads` with a single
+   thread, and, when the device type is statically inferrable, it also avoids
+   compilation overhead.
+
+ - When `device` is a `CPUMultiThreaded()`, the loop is passed to
+   `Threads.@threads`. This supports three different kinds of "schedulers" for
+   determining how many iterations of the loop to evaluate in each thread:
+     1) (default) a "dynamic" scheduler that changes the number of iterations as
+        new threads are launched,
+     2) a "static" scheduler that evaluates a fixed number of iterations per
+        thread, and
+     3) a "greedy" scheduler that uses a small number of threads and continues
+        evaluating iterations in each thread until the loop is completed.
+   Setting `coarsen` to `:dynamic` or `:greedy` launches threads with those
+   schedulers, while setting it to an integer value launches threads with the
+   static scheduler. See also [`Threads.@threads`](@ref).
+
+ - When `device` is a `CUDADevice()`, the loop is compiled with `CUDA.@cuda` and
+   run with `CUDA.@sync`. Since CUDA launches all threads in parallel, only
+   static scheduling can be used. Setting `coarsen` to any symbol causes each
+   thread to evaluate a single iteration, while setting it to an integer value
+   causes each thread to evaluate at least that number of iterations. By
+   default, only one iteration is evaluated per thread. If the total number of
+   iterations in the loop is extremely large, the specified coarsening may
+   require more threads than can be simultaneously launched on the GPU, in which
+   case the amount of coarsening is automatically increased.
+
+NOTE: Whenever a value in the body of the for-loop has a type that cannot be
+inferred by the compiler, an `InvalidIRError` is thrown for a `CUDADevice()`. In
+particular, the types of global variables are not inferrable, so this macro
+must be wrapped in a function when it is used in the REPL, e.g.,
+
+```julia-repl
+julia> a = CUDA.CuArray{Int}(undef, 100); b = similar(a);
+
+julia> kernel!(a, b) = ClimaComms.@threaded for i in axes(a, 1)
+           a[i] = b[i]
+       end
+kernel! (generic function with 1 method)
+
+julia> kernel!(a, b)
+
+julia> ClimaComms.@threaded for i in axes(a, 1)
+    a[i] = b[i]
+end
+ERROR: InvalidIRError: ...
+```
+
+To fix other kinds of inference issues on GPUs, especially ones brought about by
+indexing into iterators with nonuniform element types, see
+[`UnrolledUtilities.jl`](https://github.com/CliMA/UnrolledUtilities.jl).
+"""
+macro threaded(args...)
+    usage_string = "Usage: @threaded [coarsen=...] [device] for ... end"
+    n_args = length(args)
+    1 <= n_args <= 3 || throw(ArgumentError(usage_string))
+
+    device_expr = :(ClimaComms.device())
+    coarsen_expr = :(:dynamic)
+    for arg_number in 1:(n_args - 1)
+        device_or_coarsen_expr = args[arg_number]
+        if Meta.isexpr(device_or_coarsen_expr, :(=))
+            (kwarg_name, kwarg_value) = device_or_coarsen_expr.args
+            kwarg_name == :coarsen || throw(ArgumentError(usage_string))
+            coarsen_expr = kwarg_value
+        else
+            arg_number == n_args - 1 || throw(ArgumentError(usage_string))
+            device_expr = device_or_coarsen_expr
+        end
+    end
+    if coarsen_expr isa QuoteNode
+        coarsen_expr = :(Val($(coarsen_expr)))
+    end
+
+    loop_expr = args[n_args]
+    Meta.isexpr(loop_expr, :for) || throw(ArgumentError(usage_string))
+    (var_and_itr_expr, loop_body) = loop_expr.args
+    Meta.isexpr(var_and_itr_expr, :(=)) ||
+        throw(ArgumentError("@threaded does not support nested loops"))
+    (var_expr, itr_expr) = var_and_itr_expr.args
+
+    return quote
+        $(esc(device_expr)) isa CPUSingleThreaded ? $(esc(loop_expr)) :
+        threaded(
+            $(esc(var_expr)) -> $(esc(loop_body)),
+            $(esc(device_expr)),
+            $(esc(coarsen_expr)),
+            $(esc(itr_expr)),
+        )
+    end
path: src/devices.jl
user: login: dennisYatunin
body: The return value of a for-loop in Julia is `nothing`. There is no inconsistency in return value here.
created_at: 2025-05-03T03:04:13Z
author_association: MEMBER
diff_hunk: @@ -376,3 +346,127 @@ macro assert(device, cond, text = nothing)
 end
 assert(::AbstractCPUDevice, cond::C, text::T) where {C, T} =
     isnothing(text) ? (Base.@assert cond()) : (Base.@assert cond() text())
+
+
+"""
+    @threaded [coarsen=...] [device] for ... end
+
+Device-flexible generalization of `Threads.@threads`, which distributes the
+iterations of a for-loop across multiple threads, with the option to control
+thread coarsening. Coarsening makes each thread evaluate more than one iteration
+of the loop, which can improve performance by reducing the runtime overhead
+of launching additional threads (though too much coarsening worsens performance
+because it reduces parallelization). The `device` is either inferred by calling
+`ClimaComms.device()`, or it can be specified manually, with the following
+device-dependent behavior:
+
+ - When `device` is a `CPUSingleThreaded()`, the loop is evaluated as-is. This
+   avoids the runtime overhead of calling `Threads.@threads` with a single
+   thread, and, when the device type is statically inferrable, it also avoids
+   compilation overhead.
+
+ - When `device` is a `CPUMultiThreaded()`, the loop is passed to
+   `Threads.@threads`. This supports three different kinds of "schedulers" for
+   determining how many iterations of the loop to evaluate in each thread:
+     1) (default) a "dynamic" scheduler that changes the number of iterations as
+        new threads are launched,
+     2) a "static" scheduler that evaluates a fixed number of iterations per
+        thread, and
+     3) a "greedy" scheduler that uses a small number of threads and continues
+        evaluating iterations in each thread until the loop is completed.
+   Setting `coarsen` to `:dynamic` or `:greedy` launches threads with those
+   schedulers, while setting it to an integer value launches threads with the
+   static scheduler. See also [`Threads.@threads`](@ref).
+
+ - When `device` is a `CUDADevice()`, the loop is compiled with `CUDA.@cuda` and
+   run with `CUDA.@sync`. Since CUDA launches all threads in parallel, only
+   static scheduling can be used. Setting `coarsen` to any symbol causes each
+   thread to evaluate a single iteration, while setting it to an integer value
+   causes each thread to evaluate at least that number of iterations. By
+   default, only one iteration is evaluated per thread. If the total number of
+   iterations in the loop is extremely large, the specified coarsening may
+   require more threads than can be simultaneously launched on the GPU, in which
+   case the amount of coarsening is automatically increased.
+
+NOTE: Whenever a value in the body of the for-loop has a type that cannot be
+inferred by the compiler, an `InvalidIRError` is thrown for a `CUDADevice()`. In
+particular, the types of global variables are not inferrable, so this macro
+must be wrapped in a function when it is used in the REPL, e.g.,
+
+```julia-repl
+julia> a = CUDA.CuArray{Int}(undef, 100); b = similar(a);
+
+julia> kernel!(a, b) = ClimaComms.@threaded for i in axes(a, 1)
+           a[i] = b[i]
+       end
+kernel! (generic function with 1 method)
+
+julia> kernel!(a, b)
+
+julia> ClimaComms.@threaded for i in axes(a, 1)
+    a[i] = b[i]
+end
+ERROR: InvalidIRError: ...
+```
+
+To fix other kinds of inference issues on GPUs, especially ones brought about by
+indexing into iterators with nonuniform element types, see
+[`UnrolledUtilities.jl`](https://github.com/CliMA/UnrolledUtilities.jl).
+"""
+macro threaded(args...)
+    usage_string = "Usage: @threaded [coarsen=...] [device] for ... end"
+    n_args = length(args)
+    1 <= n_args <= 3 || throw(ArgumentError(usage_string))
+
+    device_expr = :(ClimaComms.device())
+    coarsen_expr = :(:dynamic)
+    for arg_number in 1:(n_args - 1)
+        device_or_coarsen_expr = args[arg_number]
+        if Meta.isexpr(device_or_coarsen_expr, :(=))
+            (kwarg_name, kwarg_value) = device_or_coarsen_expr.args
+            kwarg_name == :coarsen || throw(ArgumentError(usage_string))
+            coarsen_expr = kwarg_value
+        else
+            arg_number == n_args - 1 || throw(ArgumentError(usage_string))
+            device_expr = device_or_coarsen_expr
+        end
+    end
+    if coarsen_expr isa QuoteNode
+        coarsen_expr = :(Val($(coarsen_expr)))
+    end
+
+    loop_expr = args[n_args]
+    Meta.isexpr(loop_expr, :for) || throw(ArgumentError(usage_string))
+    (var_and_itr_expr, loop_body) = loop_expr.args
+    Meta.isexpr(var_and_itr_expr, :(=)) ||
+        throw(ArgumentError("@threaded does not support nested loops"))
+    (var_expr, itr_expr) = var_and_itr_expr.args
+
+    return quote
+        $(esc(device_expr)) isa CPUSingleThreaded ? $(esc(loop_expr)) :
+        threaded(
+            $(esc(var_expr)) -> $(esc(loop_body)),
+            $(esc(device_expr)),
+            $(esc(coarsen_expr)),
+            $(esc(itr_expr)),
+        )
+    end
path: src/devices.jl
user: login: coderabbitai[bot]
body: @dennisYatunin You're absolutely right. In Julia, a for-loop evaluates to `nothing` when it completes execution, so there's no inconsistency in the return values between the branches of your ternary expression.

The original suggestion did point out that `device_expr` gets evaluated twice in the ternary expression, which could be inefficient if it's complex or has side effects. That part of the suggestion has merit, but the concern about inconsistent return values isn't applicable here.

---

<details>
<summary>✏️ Learnings added</summary>

```
Learnt from: dennisYatunin
PR: CliMA/ClimaComms.jl#111
File: src/devices.jl:445-453
Timestamp: 2025-05-03T03:04:41.544Z
Learning: In Julia, a for-loop evaluates to `nothing` when it completes execution, so there's no need to explicitly add `nothing` at the end of a for-loop to maintain consistent return values in control flow branches.
```

> Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure [review instructions](https://docs.coderabbit.ai/guides/review-instructions). You can manage existing learnings in the [app](https://app.coderabbit.ai/learnings).

</details>




<!-- <agentComponentType>chat-answer</agentComponentType> -->

<!-- <rawResChunk><answer>@dennisYatunin You're absolutely right. In Julia, a for-loop evaluates to `nothing` when it completes execution, so there's no inconsistency in the return values between the branches of your ternary expression.

The original suggestion did point out that `device_expr` gets evaluated twice in the ternary expression, which could be inefficient if it's complex or has side effects. That part of the suggestion has merit, but the concern about inconsistent return values isn't applicable here.

---

<details>
<summary>✏️ Learnings added</summary>

```
Learnt from: dennisYatunin
PR: CliMA/ClimaComms.jl#111
File: src/devices.jl:445-453
Timestamp: 2025-05-03T03:04:41.544Z
Learning: In Julia, a for-loop evaluates to `nothing` when it completes execution, so there's no need to explicitly add `nothing` at the end of a for-loop to maintain consistent return values in control flow branches.
```

> Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure [review instructions](https://docs.coderabbit.ai/guides/review-instructions). You can manage existing learnings in the [app](https://app.coderabbit.ai/learnings).

</details>

</answer></rawResChunk> -->

<!-- This is an auto-generated reply by CodeRabbit -->
created_at: 2025-05-03T03:04:42Z
author_association: NONE
diff_hunk: @@ -376,3 +346,132 @@ macro assert(device, cond, text = nothing)
 end
 assert(::AbstractCPUDevice, cond::C, text::T) where {C, T} =
     isnothing(text) ? (Base.@assert cond()) : (Base.@assert cond() text())
+
+
+"""
+    @threaded [coarsen=...] [device] for ... end
+
+Device-flexible generalization of `Threads.@threads`, which distributes the
+iterations of a for-loop across multiple threads, with the option to control
+thread coarsening. Coarsening makes each thread evaluate more than one iteration
+of the loop, which can improve performance by reducing the runtime overhead
+of launching additional threads (though too much coarsening worsens performance
+because it reduces parallelization). The `device` is either inferred by calling
+`ClimaComms.device()`, or it can be specified manually, with the following
+device-dependent behavior:
+
+ - When `device` is a `CPUSingleThreaded()`, the loop is evaluated as-is. This
+   avoids the runtime overhead of calling `Threads.@threads` with a single
+   thread, and, when the device type is statically inferrable, it also avoids
+   compilation overhead.
+
+ - When `device` is a `CPUMultiThreaded()`, the loop is passed to
+   `Threads.@threads`. This supports three different kinds of "schedulers" for
+   determining how many iterations of the loop to evaluate in each thread:
+     1) (default) a "dynamic" scheduler that changes the number of iterations as
+        new threads are launched,
+     2) a "static" scheduler that evaluates a fixed number of iterations per
+        thread, and
+     3) a "greedy" scheduler that uses a small number of threads and continues
+        evaluating iterations in each thread until the loop is completed.
+   Setting `coarsen` to `:dynamic` or `:greedy` launches threads with those
+   schedulers, and setting it to an integer value launches threads with the
+   static scheduler. To read more about scheduling, see the documentation for
+   [`Threads.@threads`](https://docs.julialang.org/en/v1/base/multi-threading/#Base.Threads.@threads).
+
+ - When `device` is a `CUDADevice()`, the loop is compiled with `CUDA.@cuda` and
+   run with `CUDA.@sync`. Since CUDA launches all threads in parallel, only
+   static scheduling can be used. Setting `coarsen` to any symbol causes each
+   thread to evaluate a single iteration, and setting it to an integer value
+   causes each thread to evaluate at least that number of iterations. By
+   default, only one iteration is evaluated per thread. If the total number of
+   iterations in the loop is extremely large, the specified coarsening may
+   require more threads than can be simultaneously launched on the GPU, in which
+   case the amount of coarsening is automatically increased.
+
+NOTE: When a value in the body of the loop has a type that cannot be inferred by
+the compiler, an `InvalidIRError` will be thrown during compilation for a
+`CUDADevice()`. In particular, the types of global variables are not inferrable,
+so `@threaded` must be wrapped in a function when it is used in the REPL, e.g.,
+
+```julia-repl
+julia> a = CUDA.CuArray{Int}(undef, 100); b = similar(a);
+
+julia> threaded_copyto!(a, b) = ClimaComms.@threaded for i in axes(a, 1)
+           a[i] = b[i]
+       end
+threaded_copyto! (generic function with 1 method)
+
+julia> threaded_copyto!(a, b)
+
+julia> ClimaComms.@threaded for i in axes(a, 1)
+    a[i] = b[i]
+end
+ERROR: InvalidIRError: ...
+```
+
+To fix other kinds of inference issues on GPUs, especially ones brought about by
+indexing into iterators with nonuniform element types, see
+[`UnrolledUtilities.jl`](https://github.com/CliMA/UnrolledUtilities.jl).
+"""
+macro threaded(args...)
+    usage_string = "Usage: @threaded [coarsen=...] [device] for ... end"
+    n_args = length(args)
+    1 <= n_args <= 3 || throw(ArgumentError(usage_string))
+
+    device_expr = :($ClimaComms.device())
+    coarsen_expr = :(:dynamic)
+    for arg_number in 1:(n_args - 1)
+        device_or_coarsen_expr = args[arg_number]
+        if Meta.isexpr(device_or_coarsen_expr, :(=))
+            (kwarg_name, kwarg_value) = device_or_coarsen_expr.args
+            kwarg_name == :coarsen || throw(ArgumentError(usage_string))
+            coarsen_expr = kwarg_value
+        else
+            arg_number == n_args - 1 || throw(ArgumentError(usage_string))
+            device_expr = device_or_coarsen_expr
+        end
+    end
+    if coarsen_expr isa QuoteNode
+        coarsen_expr = :(Val($(coarsen_expr)))
+    end
+
+    loop_expr = args[n_args]
+    Meta.isexpr(loop_expr, :for) || throw(ArgumentError(usage_string))
+    (var_and_itr_expr, loop_body) = loop_expr.args
+    Meta.isexpr(var_and_itr_expr, :(=)) ||
+        throw(ArgumentError("@threaded does not support nested loops"))
+    (var_expr, itr_expr) = var_and_itr_expr.args
+
+    function_expr = :($(esc(var_expr)) -> $(esc(loop_body)))
+    return quote
+        device = $(esc(device_expr))
+        device isa CPUSingleThreaded ? $(esc(loop_expr)) :
+        threaded($function_expr, device, $(esc(coarsen_expr)), $(esc(itr_expr)))
+    end
+end
+
+threaded(f::F, ::CPUMultiThreaded, ::Val{:dynamic}, itr) where {F} =
+    Threads.@threads :dynamic for item in itr
+        f(item)
+    end
+threaded(f::F, ::CPUMultiThreaded, ::Val{:static}, itr) where {F} =
+    Threads.@threads :static for item in itr
+        f(item)
+    end
+threaded(f::F, ::CPUMultiThreaded, ::Val{:greedy}, itr) where {F} =
+    Threads.@threads :greedy for item in itr
+        f(item)
+    end
+
+function threaded(f::F, ::CPUMultiThreaded, items_in_thread::Int, itr) where {F}
+    items_in_thread > 0 || throw(ArgumentError("non-positive coarsening value"))
+    n_threads = cld(length(itr), items_in_thread)
+    Threads.@threads :static for thread_index in 1:n_threads
+        first_item_index = items_in_thread * (thread_index - 1) + 1
+        last_item_index = items_in_thread * thread_index
+        for item_index in first_item_index:min(last_item_index, length(itr))
+            f(itr[item_index])
+        end
+    end
+end
path: src/devices.jl
user: login: coderabbitai[bot]
body: _💡 Verification agent_

<details>
<summary>🧩 Analysis chain</summary>

**Verify iterator length support**

On line 469, the code calls `length(itr)` which assumes the iterator has a `length` method. Consider verifying this works with all iterators you expect to support, or add a note in the documentation about this limitation.

---

🏁 Script executed:

```shell
#!/bin/bash
# Check for common iterators without length
echo "Checking common iterators that might not support length()"
rg -A 5 "\bzip\b|\brange\b|\bgenerate\b" | grep -v "length"
```

Length of output: 184

---

Let’s map out where `threaded` is actually used to see what kinds of iterators get passed in.

---

🏁 Script executed:

```shell
#!/bin/bash
# Find all calls to `threaded` to inspect iterator types in use
rg -n "threaded(" src
```

Length of output: 92

---

Let’s correctly escape the regex and locate all `threaded` invocations to see what iterators get passed in:

---

🏁 Script executed:

```shell
#!/bin/bash
# Find all calls to threaded, with escaped parenthesis
rg -n 'threaded\(' .
```

Length of output: 847

---

Here’s the static-CPU variant and its neighbors—looks like **all** CPU-threaded overloads call `length(itr)`:

---

🏁 Script executed:

```shell
#!/bin/bash
# Show CPU-threaded variants around the static case
sed -n '440,500p' src/devices.jl
```

Length of output: 1531

---



</details>

**Document indexing requirement for static coarsening**

The `threaded(f, ::CPUMultiThreaded, items_in_thread::Int, itr)` overload on lines 467–477 calls `length(itr)` and uses `itr[item_index]`, so the iterator must support both `length` and indexed access. Please update the documentation (or add a guard/fallback) to make this requirement explicit.

• File: src/devices.jl  
• Lines: 467–477  

Proposed actions:  
- Add a note in the docstring for the static‐coarsening `threaded` overload stating that `itr` must implement `length` and `getindex`.  
- (Optional) Check at runtime—e.g., `hasmethod(length, (typeof(itr),)) && hasmethod(getindex, (typeof(itr), Int))`—and throw a clear error or fall back to a simpler strategy.

<!-- This is an auto-generated reply by CodeRabbit -->
created_at: 2025-05-03T21:03:51Z
author_association: NONE
diff_hunk: @@ -376,3 +346,132 @@ macro assert(device, cond, text = nothing)
 end
 assert(::AbstractCPUDevice, cond::C, text::T) where {C, T} =
     isnothing(text) ? (Base.@assert cond()) : (Base.@assert cond() text())
+
+
+"""
+    @threaded [coarsen=...] [device] for ... end
+
+Device-flexible generalization of `Threads.@threads`, which distributes the
+iterations of a for-loop across multiple threads, with the option to control
+thread coarsening. Coarsening makes each thread evaluate more than one iteration
+of the loop, which can improve performance by reducing the runtime overhead
+of launching additional threads (though too much coarsening worsens performance
+because it reduces parallelization). The `device` is either inferred by calling
+`ClimaComms.device()`, or it can be specified manually, with the following
+device-dependent behavior:
+
+ - When `device` is a `CPUSingleThreaded()`, the loop is evaluated as-is. This
+   avoids the runtime overhead of calling `Threads.@threads` with a single
+   thread, and, when the device type is statically inferrable, it also avoids
+   compilation overhead.
+
+ - When `device` is a `CPUMultiThreaded()`, the loop is passed to
+   `Threads.@threads`. This supports three different kinds of "schedulers" for
+   determining how many iterations of the loop to evaluate in each thread:
+     1) (default) a "dynamic" scheduler that changes the number of iterations as
+        new threads are launched,
+     2) a "static" scheduler that evaluates a fixed number of iterations per
+        thread, and
+     3) a "greedy" scheduler that uses a small number of threads and continues
+        evaluating iterations in each thread until the loop is completed.
+   Setting `coarsen` to `:dynamic` or `:greedy` launches threads with those
+   schedulers, and setting it to an integer value launches threads with the
+   static scheduler. To read more about scheduling, see the documentation for
+   [`Threads.@threads`](https://docs.julialang.org/en/v1/base/multi-threading/#Base.Threads.@threads).
+
+ - When `device` is a `CUDADevice()`, the loop is compiled with `CUDA.@cuda` and
+   run with `CUDA.@sync`. Since CUDA launches all threads in parallel, only
+   static scheduling can be used. Setting `coarsen` to any symbol causes each
+   thread to evaluate a single iteration, and setting it to an integer value
+   causes each thread to evaluate at least that number of iterations. By
+   default, only one iteration is evaluated per thread. If the total number of
+   iterations in the loop is extremely large, the specified coarsening may
+   require more threads than can be simultaneously launched on the GPU, in which
+   case the amount of coarsening is automatically increased.
+
+NOTE: When a value in the body of the loop has a type that cannot be inferred by
+the compiler, an `InvalidIRError` will be thrown during compilation for a
+`CUDADevice()`. In particular, the types of global variables are not inferrable,
+so `@threaded` must be wrapped in a function when it is used in the REPL, e.g.,
+
+```julia-repl
+julia> a = CUDA.CuArray{Int}(undef, 100); b = similar(a);
+
+julia> threaded_copyto!(a, b) = ClimaComms.@threaded for i in axes(a, 1)
+           a[i] = b[i]
+       end
+threaded_copyto! (generic function with 1 method)
+
+julia> threaded_copyto!(a, b)
+
+julia> ClimaComms.@threaded for i in axes(a, 1)
+    a[i] = b[i]
+end
+ERROR: InvalidIRError: ...
+```
+
+To fix other kinds of inference issues on GPUs, especially ones brought about by
+indexing into iterators with nonuniform element types, see
+[`UnrolledUtilities.jl`](https://github.com/CliMA/UnrolledUtilities.jl).
+"""
+macro threaded(args...)
+    usage_string = "Usage: @threaded [coarsen=...] [device] for ... end"
+    n_args = length(args)
+    1 <= n_args <= 3 || throw(ArgumentError(usage_string))
+
+    device_expr = :($ClimaComms.device())
+    coarsen_expr = :(:dynamic)
+    for arg_number in 1:(n_args - 1)
+        device_or_coarsen_expr = args[arg_number]
+        if Meta.isexpr(device_or_coarsen_expr, :(=))
+            (kwarg_name, kwarg_value) = device_or_coarsen_expr.args
+            kwarg_name == :coarsen || throw(ArgumentError(usage_string))
+            coarsen_expr = kwarg_value
+        else
+            arg_number == n_args - 1 || throw(ArgumentError(usage_string))
+            device_expr = device_or_coarsen_expr
+        end
+    end
+    if coarsen_expr isa QuoteNode
+        coarsen_expr = :(Val($(coarsen_expr)))
+    end
+
+    loop_expr = args[n_args]
+    Meta.isexpr(loop_expr, :for) || throw(ArgumentError(usage_string))
+    (var_and_itr_expr, loop_body) = loop_expr.args
+    Meta.isexpr(var_and_itr_expr, :(=)) ||
+        throw(ArgumentError("@threaded does not support nested loops"))
+    (var_expr, itr_expr) = var_and_itr_expr.args
+
+    function_expr = :($(esc(var_expr)) -> $(esc(loop_body)))
+    return quote
+        device = $(esc(device_expr))
+        device isa CPUSingleThreaded ? $(esc(loop_expr)) :
+        threaded($function_expr, device, $(esc(coarsen_expr)), $(esc(itr_expr)))
+    end
+end
+
+threaded(f::F, ::CPUMultiThreaded, ::Val{:dynamic}, itr) where {F} =
+    Threads.@threads :dynamic for item in itr
+        f(item)
+    end
+threaded(f::F, ::CPUMultiThreaded, ::Val{:static}, itr) where {F} =
+    Threads.@threads :static for item in itr
+        f(item)
+    end
+threaded(f::F, ::CPUMultiThreaded, ::Val{:greedy}, itr) where {F} =
+    Threads.@threads :greedy for item in itr
+        f(item)
+    end
+
+function threaded(f::F, ::CPUMultiThreaded, items_in_thread::Int, itr) where {F}
+    items_in_thread > 0 || throw(ArgumentError("non-positive coarsening value"))
+    n_threads = cld(length(itr), items_in_thread)
+    Threads.@threads :static for thread_index in 1:n_threads
+        first_item_index = items_in_thread * (thread_index - 1) + 1
+        last_item_index = items_in_thread * thread_index
+        for item_index in first_item_index:min(last_item_index, length(itr))
+            f(itr[item_index])
+        end
+    end
+end
path: src/devices.jl
user: login: dennisYatunin
body: This package does not need to support iterators without methods for `length` or `getindex`, as such iterators are incompatible with GPU devices.
created_at: 2025-05-03T21:12:54Z
author_association: MEMBER
diff_hunk: @@ -376,3 +346,132 @@ macro assert(device, cond, text = nothing)
 end
 assert(::AbstractCPUDevice, cond::C, text::T) where {C, T} =
     isnothing(text) ? (Base.@assert cond()) : (Base.@assert cond() text())
+
+
+"""
+    @threaded [coarsen=...] [device] for ... end
+
+Device-flexible generalization of `Threads.@threads`, which distributes the
+iterations of a for-loop across multiple threads, with the option to control
+thread coarsening. Coarsening makes each thread evaluate more than one iteration
+of the loop, which can improve performance by reducing the runtime overhead
+of launching additional threads (though too much coarsening worsens performance
+because it reduces parallelization). The `device` is either inferred by calling
+`ClimaComms.device()`, or it can be specified manually, with the following
+device-dependent behavior:
+
+ - When `device` is a `CPUSingleThreaded()`, the loop is evaluated as-is. This
+   avoids the runtime overhead of calling `Threads.@threads` with a single
+   thread, and, when the device type is statically inferrable, it also avoids
+   compilation overhead.
+
+ - When `device` is a `CPUMultiThreaded()`, the loop is passed to
+   `Threads.@threads`. This supports three different kinds of "schedulers" for
+   determining how many iterations of the loop to evaluate in each thread:
+     1) (default) a "dynamic" scheduler that changes the number of iterations as
+        new threads are launched,
+     2) a "static" scheduler that evaluates a fixed number of iterations per
+        thread, and
+     3) a "greedy" scheduler that uses a small number of threads and continues
+        evaluating iterations in each thread until the loop is completed.
+   Setting `coarsen` to `:dynamic` or `:greedy` launches threads with those
+   schedulers, and setting it to an integer value launches threads with the
+   static scheduler. To read more about scheduling, see the documentation for
+   [`Threads.@threads`](https://docs.julialang.org/en/v1/base/multi-threading/#Base.Threads.@threads).
+
+ - When `device` is a `CUDADevice()`, the loop is compiled with `CUDA.@cuda` and
+   run with `CUDA.@sync`. Since CUDA launches all threads in parallel, only
+   static scheduling can be used. Setting `coarsen` to any symbol causes each
+   thread to evaluate a single iteration, and setting it to an integer value
+   causes each thread to evaluate at least that number of iterations. By
+   default, only one iteration is evaluated per thread. If the total number of
+   iterations in the loop is extremely large, the specified coarsening may
+   require more threads than can be simultaneously launched on the GPU, in which
+   case the amount of coarsening is automatically increased.
+
+NOTE: When a value in the body of the loop has a type that cannot be inferred by
+the compiler, an `InvalidIRError` will be thrown during compilation for a
+`CUDADevice()`. In particular, the types of global variables are not inferrable,
+so `@threaded` must be wrapped in a function when it is used in the REPL, e.g.,
+
+```julia-repl
+julia> a = CUDA.CuArray{Int}(undef, 100); b = similar(a);
+
+julia> threaded_copyto!(a, b) = ClimaComms.@threaded for i in axes(a, 1)
+           a[i] = b[i]
+       end
+threaded_copyto! (generic function with 1 method)
+
+julia> threaded_copyto!(a, b)
+
+julia> ClimaComms.@threaded for i in axes(a, 1)
+    a[i] = b[i]
+end
+ERROR: InvalidIRError: ...
+```
+
+To fix other kinds of inference issues on GPUs, especially ones brought about by
+indexing into iterators with nonuniform element types, see
+[`UnrolledUtilities.jl`](https://github.com/CliMA/UnrolledUtilities.jl).
+"""
+macro threaded(args...)
+    usage_string = "Usage: @threaded [coarsen=...] [device] for ... end"
+    n_args = length(args)
+    1 <= n_args <= 3 || throw(ArgumentError(usage_string))
+
+    device_expr = :($ClimaComms.device())
+    coarsen_expr = :(:dynamic)
+    for arg_number in 1:(n_args - 1)
+        device_or_coarsen_expr = args[arg_number]
+        if Meta.isexpr(device_or_coarsen_expr, :(=))
+            (kwarg_name, kwarg_value) = device_or_coarsen_expr.args
+            kwarg_name == :coarsen || throw(ArgumentError(usage_string))
+            coarsen_expr = kwarg_value
+        else
+            arg_number == n_args - 1 || throw(ArgumentError(usage_string))
+            device_expr = device_or_coarsen_expr
+        end
+    end
+    if coarsen_expr isa QuoteNode
+        coarsen_expr = :(Val($(coarsen_expr)))
+    end
+
+    loop_expr = args[n_args]
+    Meta.isexpr(loop_expr, :for) || throw(ArgumentError(usage_string))
+    (var_and_itr_expr, loop_body) = loop_expr.args
+    Meta.isexpr(var_and_itr_expr, :(=)) ||
+        throw(ArgumentError("@threaded does not support nested loops"))
+    (var_expr, itr_expr) = var_and_itr_expr.args
+
+    function_expr = :($(esc(var_expr)) -> $(esc(loop_body)))
+    return quote
+        device = $(esc(device_expr))
+        device isa CPUSingleThreaded ? $(esc(loop_expr)) :
+        threaded($function_expr, device, $(esc(coarsen_expr)), $(esc(itr_expr)))
+    end
+end
+
+threaded(f::F, ::CPUMultiThreaded, ::Val{:dynamic}, itr) where {F} =
+    Threads.@threads :dynamic for item in itr
+        f(item)
+    end
+threaded(f::F, ::CPUMultiThreaded, ::Val{:static}, itr) where {F} =
+    Threads.@threads :static for item in itr
+        f(item)
+    end
+threaded(f::F, ::CPUMultiThreaded, ::Val{:greedy}, itr) where {F} =
+    Threads.@threads :greedy for item in itr
+        f(item)
+    end
+
+function threaded(f::F, ::CPUMultiThreaded, items_in_thread::Int, itr) where {F}
+    items_in_thread > 0 || throw(ArgumentError("non-positive coarsening value"))
+    n_threads = cld(length(itr), items_in_thread)
+    Threads.@threads :static for thread_index in 1:n_threads
+        first_item_index = items_in_thread * (thread_index - 1) + 1
+        last_item_index = items_in_thread * thread_index
+        for item_index in first_item_index:min(last_item_index, length(itr))
+            f(itr[item_index])
+        end
+    end
+end
path: src/devices.jl
user: login: coderabbitai[bot]
body: @dennisYatunin Understood, thank you for clarifying. That makes perfect sense - since the package is designed to work with GPU devices, it naturally requires iterators that support both `length` and `getindex` methods, as these capabilities are fundamental for GPU compatibility.

---

<details>
<summary>✏️ Learnings added</summary>

```
Learnt from: dennisYatunin
PR: CliMA/ClimaComms.jl#111
File: src/devices.jl:472-482
Timestamp: 2025-05-03T21:13:22.142Z
Learning: ClimaComms.jl is designed to work with iterators that support both `length` and `getindex` methods, as these capabilities are required for GPU compatibility. Iterators without these methods are intentionally not supported.
```

> Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure [review instructions](https://docs.coderabbit.ai/guides/review-instructions). You can manage existing learnings in the [app](https://app.coderabbit.ai/learnings).

</details>




<!-- <agentComponentType>chat-answer</agentComponentType> -->

<!-- <rawResChunk><answer>@dennisYatunin Understood, thank you for clarifying. That makes perfect sense - since the package is designed to work with GPU devices, it naturally requires iterators that support both `length` and `getindex` methods, as these capabilities are fundamental for GPU compatibility.

---

<details>
<summary>✏️ Learnings added</summary>

```
Learnt from: dennisYatunin
PR: CliMA/ClimaComms.jl#111
File: src/devices.jl:472-482
Timestamp: 2025-05-03T21:13:22.142Z
Learning: ClimaComms.jl is designed to work with iterators that support both `length` and `getindex` methods, as these capabilities are required for GPU compatibility. Iterators without these methods are intentionally not supported.
```

> Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure [review instructions](https://docs.coderabbit.ai/guides/review-instructions). You can manage existing learnings in the [app](https://app.coderabbit.ai/learnings).

</details>

</answer></rawResChunk> -->

<!-- This is an auto-generated reply by CodeRabbit -->
created_at: 2025-05-03T21:13:23Z
author_association: NONE
diff_hunk: @@ -50,4 +50,63 @@ ClimaComms.elapsed(f::F, ::CUDADevice, args...; kwargs...) where {F} =
 ClimaComms.assert(::CUDADevice, cond::C, text::T) where {C, T} =
     isnothing(text) ? (CUDA.@cuassert cond()) : (CUDA.@cuassert cond() text())
 
+cuda_thread_index_1D() =
+    (CUDA.blockIdx().x - 1) * CUDA.blockDim().x + CUDA.threadIdx().x
+
+function cuda_device_limits_1D()
+    cuda_device = CUDA.device()
+    max_blocks =
+        CUDA.attribute(cuda_device, CUDA.DEVICE_ATTRIBUTE_MAX_GRID_DIM_X)
+    max_threads_in_block =
+        CUDA.attribute(cuda_device, CUDA.DEVICE_ATTRIBUTE_MAX_THREADS_PER_BLOCK)
+    return (; max_blocks, max_threads_in_block)
+end
+
+function threaded(f::F, ::CUDADevice, ::Val, itr) where {F}
+    length(itr) > 0 || return nothing
+
+    Base.require_one_based_indexing(itr)
+    function call_f_from_thread()
+        item_index = cuda_thread_index_1D()
+        item_index <= length(itr) && @inbounds f(itr[item_index])
+        return nothing
+    end
+    kernel = CUDA.@cuda launch=false call_f_from_thread()
+    optimal_threads_in_block = CUDA.launch_configuration(kernel.fun).threads
+
+    (; max_blocks, max_threads_in_block) = cuda_device_limits_1D()
+    blocks = min(cld(length(itr), optimal_threads_in_block), max_blocks)
+    threads_in_block = min(cld(length(itr), blocks), max_threads_in_block)
+    length(itr) <= threads_in_block * blocks ||
+        return threaded(f, CUDADevice(), 2, itr) # Coarsen threads if necessary.
+
+    CUDA.@sync kernel(; threads = threads_in_block, blocks)
+end
+
+function threaded(f::F, ::CUDADevice, min_items_in_thread::Int, itr) where {F}
+    length(itr) > 0 || return nothing
+    min_items_in_thread > 0 ||
+        throw(ArgumentError("`coarsen` value must positive"))
+
+    Base.require_one_based_indexing(itr)
+    function call_f_from_thread(items_in_thread)
+        thread_index = cuda_thread_index_1D()
+        first_item_index = items_in_thread * (thread_index - 1) + 1
+        last_item_index = items_in_thread * thread_index
path: ext/ClimaCommsCUDAExt.jl
user: login: sriharshakandala
body: This may not always be the best way to iterate with thread coarsening on a GPU. For example, one option is to use the following approach:
(pseudocode)
```
for i in thread_st:offset:thread_en
    process...f[i]
end
```
where `offset` is determined by the number of threads-per-block.
created_at: 2025-05-05T18:02:12Z
author_association: MEMBER
diff_hunk: @@ -50,4 +50,63 @@ ClimaComms.elapsed(f::F, ::CUDADevice, args...; kwargs...) where {F} =
 ClimaComms.assert(::CUDADevice, cond::C, text::T) where {C, T} =
     isnothing(text) ? (CUDA.@cuassert cond()) : (CUDA.@cuassert cond() text())
 
+cuda_thread_index_1D() =
+    (CUDA.blockIdx().x - 1) * CUDA.blockDim().x + CUDA.threadIdx().x
+
+function cuda_device_limits_1D()
+    cuda_device = CUDA.device()
+    max_blocks =
+        CUDA.attribute(cuda_device, CUDA.DEVICE_ATTRIBUTE_MAX_GRID_DIM_X)
+    max_threads_in_block =
+        CUDA.attribute(cuda_device, CUDA.DEVICE_ATTRIBUTE_MAX_THREADS_PER_BLOCK)
+    return (; max_blocks, max_threads_in_block)
+end
+
+function threaded(f::F, ::CUDADevice, ::Val, itr) where {F}
+    length(itr) > 0 || return nothing
+
+    Base.require_one_based_indexing(itr)
+    function call_f_from_thread()
+        item_index = cuda_thread_index_1D()
+        item_index <= length(itr) && @inbounds f(itr[item_index])
+        return nothing
+    end
+    kernel = CUDA.@cuda launch=false call_f_from_thread()
+    optimal_threads_in_block = CUDA.launch_configuration(kernel.fun).threads
path: ext/ClimaCommsCUDAExt.jl
user: login: sriharshakandala
body: This is just the maximum number of threads. This need not be optimal number of threads.
created_at: 2025-05-05T20:44:46Z
author_association: MEMBER

PR comments:
user: login: coderabbitai[bot]
created_at: 2025-05-03T02:56:53Z
body: <!-- This is an auto-generated comment: summarize by coderabbit.ai -->
<!-- walkthrough_start -->

## Walkthrough

This update replaces the old simple `@threaded` macro with a new device-aware version supporting CPUs and CUDA GPUs, featuring optional coarsening and device selection. It adapts execution to the device: sequential for single-threaded CPUs, multi-threaded with dynamic, static, or greedy scheduling for multi-threaded CPUs, and CUDA kernel launches for GPUs with coarsening controlling iterations per GPU thread. New `threaded` methods for `CUDADevice` enable parallel function application on GPU with one or multiple items per thread. The previous macro is removed. Tests verify the macro’s correctness across devices and coarsening modes. Additionally, the GitHub Actions cache step is upgraded to a newer version.

<!-- walkthrough_end -->

<!-- announcements_start -->

> [!TIP]
> <details>
> <summary>⚡️ Faster reviews with caching</summary>
> 
> - CodeRabbit now supports caching for code and dependencies, helping speed up reviews. This means quicker feedback, reduced wait times, and a smoother review experience overall. Cached data is encrypted and stored securely. This feature will be automatically enabled for all accounts on May 16th. To opt out, configure `Review - Disable Cache` at either the organization or repository level. If you prefer to disable all data retention across your organization, simply turn off the `Data Retention` setting under your Organization Settings.
> 
> Enjoy the performance boost—your workflow just got faster.
> 
> </details>

<!-- announcements_end -->
<!-- internal state start -->


<!-- DwQgtGAEAqAWCWBnSTIEMB26CuAXA9mAOYCmGJATmriQCaQDG+Ats2bgFyQAOFk+AIwBWJBrngA3EsgEBPRvlqU0AgfFwA6NPEgQAfACgjoCEYDEZyAAUASpETZWaCrKNwSPbABsvkCiQBHbGlcSHFcLzpIACIAUQAPGgx6XFh/NCV6ZjQGCnww/IBhAFUAEQBBUpIJeAYSaMg5SGxESkglDAwkAE1qbC6sAApbSDMARgmASh487nxW5ExIEkSyRHh8LHwAMzDYDwADQq94bMKWZkQNAAFU9MyDyGzc/IJ7bG45ilCjssrq2okA4aGAIZBkWCYOpsDCheCwvK0bB1RZbbjiTYFBQI/C+O4kDIKZytAZEAA0ywwKhOGCIew8p24kRhuGoGy2u3WzCZHgA1pRyF5kPD0O0AXUwGgiBh5uIGE9MOQ+AB3dSwfB4PyBbDwCjwukrJm1dSQEoVFDc/DfK6gjzMfDiCRszHbK17VArJLrTGoN7bHLwE6smj0sVSLz4bgs/i7AAyxUg/rEVvgAC9nVhMPR4/ZcVI+PyKILEBS0Mh/NtKGQ6vQRcd4ABZcoAenr2XKuHtVyEXjMACYAOwAZgADBojIVsBR/LCvLIKak7Tk8u9PlbcMhNnP+OQwLRTmt2WhfNxnMfIid0xiMBSBJrF2L1tL0D58MrkK6+Ns8FOPIbMN6WBvMw3jiHuB4YIBx57Pc6DJI0EYMLy2LbPARBThmNruI0eQZG0RD4NBvr5HMSTiOe8j+PaUimn8YCINwojwKh8qeoemwfnkzD8IufCnkhUrSI0lEkEyAa0vS3GqqkYaApK0qyrUCovMgvD4DUmTCaaJxnBcWFgtiiC4vAtAZugpzIG8dQIseaZ8gKJC+EwShiqhXTXqi9BqRpHgMJEzjtPgDCOOwZlvOhJkeEo4aRpQm5YIy66YKEShPlgp64DQRZXEY5S0Pu14UQu+zWHYHymTQlklfmgExqGDA5CVOTXigQElZVoSxVQ4gSYgsiIDQUlWry2wRsqiZcZABwSGMjxvNNAAsc35P4RleDRSyUHkfDVhqsJVvQ0mwKGvAAhqyA1ey44GNhIzwn52CpegtD7DOdRYg+zkkAIZa+ZsuB6neLURUoFIDbIkSQCDJClvBtBBSFsJmXMJwMLIIIAJKhMeRmUg4q2hvdXgtOydX+JETqws07lwbW+1EN1pMdSW6DIMqjm+GW6CfLMerUFFQUDXqtKeYFwUshmILYWgeDqnwkLIEwGCoRQbD0Ew3InFCHhHaG/g1CQ40MPsSEnANKCDTlBgWKaFzsMgDhOC4bgldkIr7ogwWILVIoPiMGssh+CMLDuFrefqobw+LoUtTkGunhgsgRw+5Djd+GBiEeQbo5ANgAobbQAMoCPgqapriK3aiEVXUKGMupG6GT0JtCLyG8D4AHKxAA6oXiaBh4WYzOpkXNK0FBgEmEdR4jwZM/k+xeNwY9xdTSgUANQ9b/D6KhmxyRRActxpASDzKXkFKnt8tTeM426fpAXgyxnCASWa5SQIWxZS/s8gyiGqRa6fSnDOUIbY0DnFYMgGeEsWqoAYsSQe8EHBEFIObJuEc0CQFThNKus5ZCSkQPyWgYAgghFJoMAAYuUAAitMU8pBIC602vENAmsPBvAhNrdoSAmD5hUIGdQ8gh4tAEdna6GMsCrTmJBGGddZZWgpB0LoiBei4H6PCCkhon7whrilEgrIB70BgTHdkFIiaPUwVgFYbCeQUgVi+e48gBAkDIM9TS7cSpRyFhHHYoYj74jwrQR4zw8g/xIPIeEQRdQ632LxHgzhOq7FSKgExSMWqIHVN4egLinjqSiG8LBrQr78zFrPZGglrrF1LuXCMjAn56lQkJQBoRlQIEhg+bxAMI6oBaCQb8nNFhakrG9ORJpUC6OwNsFi8B2Cs3sKyZIx5NgeD6XwNJc9NjhPkCQbkCs7J7CAbIbgtRoJrOgfkf+hlQJix8SLRMbpRGMN0QDZEHlaaHOxmKfcDUaD0CeR4VomcfRtQFtHdJpNUDkBRIgZwOc4DwNEBkrJXgcl/jLG3fIblaCXzyD5UUXIeTLFYew2G9AKxtHOR9LxgtukSQfvafwYpDGRDpp+bI14QQAHFJBuOAdOOZCCN5CLquAyBlwV7rIRrAsx9IRIBzIM3fKycDIAtauysyvsSrUJoWvNoWCyEDVJg4Y28zogAAk3yBUgBjHh/gxDbifv0U1WD36fwcr4XWYq9IAH5ohGEobiMaKrUCVjoD9JCSiyAqLURozM+UhlApauEVlMRu56h6nSUgSolJf0cogBonjDjwiUPEDQzBgllJlS6AeFI2m1GOuHJ6LDbGQyHjSXkllK4U2SsMqsGdpDbNajUEM1Sy4VyxPrWZ40UmLGVS1Npbi5je3gAISI10O75AdK9BQrB7b8D4ErOo2UmFVj8NoVotBwk8M9i0Wq1ksrxUpJCDOKrB4MBhauwRuBhHwT8s4EVfiU6G0TASdRTK7gaiIMdRkeKogbLMiI8eUMdSmQHddfQxhwBQEVXVeuhBs3KD+bulkXBeD8GEEi3lMh5BfSoKodQWgdCYZMFABFwpUQ4AIMQMgRGogKthFwKg41HbZBcFpOjKg1CaG0LoMAhgsOmAMJ6VsOkIF6XfgkTQPYOAGGiPp62lhygYx40qfm9BRNwrqsbTAaCXYMktN8CaLApreqgY8ZUXN94ePyBYlyD4DiBLPunYFGApbKkuSBwLJ8gkhIMeqeg+SKARgyA7RiDB4C2VTFEB+vwKhVBqHUDzZ6m50GvSrc2SWUspDQPyIZIWWoHG2AcUsrm/gFcBC10UBwABqx45rHLkVvBKWVqTFoBh5tUop/DRP8PQA4a6gq8gAPrrGy48fksgIsUGbhQdCLJMahBmzqAmKywA/Uva1UtvisAHHUBQR47bn7GyEq6v47qiyOSYa9dFpqgvDxhUJGxDr5ArItrsiTLBuB4EwVDCMP1fBjFKDBU+V2VgggANIesfs9xtZ42CPsgIMRbSFRZBdUm0EnvJpjOD+tyPAUQLtRExNFQEj9TjqDJyVSItIZJ+LuxNzGySSrbd5NV4ldQ6DQPFL5NA/EhHmKO6IKc6xwy0fPFVQFohNiJfzOL5hTx4SnEcHVMY4PuKMT4P9qyhEN58pF8Nar16gU6/4HrwiNW6uiga6TJrXWsF5f+IVoErXmBG5AtxDAjgXF8D8eoXZFOrcxZxXBcH9HIYC4e2S7rVPVt2Q2xE7bu39vsEO5AJ0JwKrNNduHk38eJWW5R4SIex2YnxRIOd36dNru0jL06l+r3aLmjzZ6n7yxGpN68nkQHyAQJeHEES+vyASYSSwQzEy9FukuQjJGBJmUBSK6I74/MFtFgrulFEKgtIkHN0+HOCOfusQElNfXzH2P++mpKQT1etPd3Q+I1yIGHfPOIZJFMLHSJ9LbiSFEKLuLkPKzu9DpJziCOao5I3j7hxKekyqVikPkPHAzqGAQKyL4FHswDHnVOTq1KGCPsVL5CAnMv9iWisNnm8GQuJggR4EgRuA8nwNkPEMbtxFTlzvcInghEtiCAAELbofbFjoBMof77AWayAvx5AygtDbi6z/ZDxU4KD9DcFKyoToRzZYioStJqjuwy6GQAzaCwj6S+QvpoKUhjYJJUA+BfZy5Gi/Kkx+JYIYFbAn5LDx7p4eCYhvbmgcFMJTYkBqhtBg7148BtD/Zuhz4L6QxL4JFJ6wRKCMTJA3ZMKO4e72AHLwGWFKxCw2EbjXTmBGbz5H6YFFpij/qMyYF+KGjrjM58TYBrpKTsDqCzKIBGBQAYyObEbRb3B0CPDbCTRHBqbiqICPAigHAqZuaXCaaJAaA9jAgGBQB5SaQE4JZcBjGnx0CDDbAcAcCUIUjnHvwdZ1BXEcB9ZeCK4UAADcYhSEee2WkwHmY+BwlCCx9Wzq145it2yxsxGmfwWmGxXgWxOx+UUQ+xighxQWJxZxFx9xNxMuFIYeGAy2S+eJuJQW5xkiuAzxbxuea2JA3x32Z6fxAJ3uQJsqixYJpw6mUCax2mMJ44+m/q2xSmiAFADAzYHB3YXgumPJhmkAxmpmfGFmjgYmoOuwNm1+AxN0JUp0NQ50RR7CU0x84xlaoSrwkIoQSsBUWc24HhJwh8cAIhNw5OjwW48gi6oKsk70nmyARwVgxQDYoE8ANpxxlaQ826lAqorQ56LpO+y8ZYYAqADiLibiVE+S9AvR5M6MRgWMKA5YokT8NYWkWCqc2JVodOp0+wkEvKupKJBpy4+QDga418EkkYhUvgm2RechJethU0TAiCGAj28EC2iEK2lJj2EYEkzCaITZxKnhJoBwHBj2e2s816hp6qq8nSlhzE2CDo9g6WzEsyzcyCHwXw3BHBk8kQ/Ba6HgJSbhvgkZxKyuC6U2/2XZdupIHyXKXpMhX2Oh+4MI96/0eQXg102EOCS5L6tAkQVU/gUUlhP6jEbMU2HsPUYgjQJAkImpG8um2xkAgafAnpxQhc+okQ/psWtBj8+Au+FA/QDseC5EPgTpaoGooQQWx+lA+wGQ44UA2FrmXpPp8+fpyeExJFS5HsGUL2XaU0RFtAVwepp88xkRMktZh5PBKOgKL2SINIdI5RNARA/RXAtAyhbCSkgwSg/ooEkwYMc88opx8A8QUQQRmEGRk+ZlHyDMrielROlYM6yecFMkSsPU2AWpdlwJkAAAUt4JlpAIAKZEkAYwGgUw16BwT5JIc57ZJpf5uImu9gqlYVEk9oLkboKc0esRuwgV7IohQW7FWFbogetxIeoYN5qA8cRiHyChMBU2geNwwUpkvZShKhmwdkh0bV78NwfUGcwIkAAA8hgNuJvHKJlYodlXSPAgee0SnrrIlWQBHL5f+epaGKQeQXHqNu8o3m+QmOVbaFNBSfnm2bPNiADOlaGCde8chJSVGiZbxRJIUlsB+h8FCAQo2RztlvQBXsEIudWVxiwGyA1LRc9EIC0PoVARtRJOuQ+I+XtKECsJLpJZAI9VwVbNhEuWQJ+CiGPIJFYVQPCNwSasdFzB7OeG+BHOQANFEJGaLE2kJEoCyvBtKqYqESXJqPCCMtWH+NOFaE+idaLGqvanbEsh5IBa7GDf+MkMKLCPkFglDXiKrVDLxt1DloyZiI3FuUxFlrrRnC1NkYqnkaucHh8utS+V/gYnFBhVAFytreZk8PFooMgKBRnoUNxb6RJRMXJcdL1v1mAMqFQJ8PxgjS+X1GQfdYMAcBwHpVSGHgwF1onTNbUOnRwC5XQLIAcNME0PCBIPgLyA/hJVJfaUHfVFaNIjrhHJ7PNZEBQBVR/IifQN7UJLhTxeIAHZWrrBTSQKQIetHRHE0NkFHhRC4eEOyD0odaVa1G8JnfKMbP0J2g5eVUYONfwj4CRVtbiImGNJmRueNMeWgJ5tgX+qPbSJKBfXImQNSBHKNCsKum2vBP0jMnMpeReKRbvisHeaTNWd7KaO+esLSJEGAJWRSL7QmCkfAJA/xSnkPG6o9SKXLSEaitqTyLwWDUPJzlWjzVgKhR4PGVgAMtuImVIHueStmTkIzm3AZDgoyMyIQ9dDbOULUc0ZBNSlBU0fZa0fEIeR0Z4N0axLCH0dIIMbnLskmefPkAEog0ZVib/dwNSS0tiGaZsFPZabueJV5XaV5Q6VNfILljAz3XxfqbCVKfCVkGDQo/qZAAANocEAC6TjttAAvBoN424445ddll4z40pd4yCIqo8ApeuL4uiFnFNLORSAldHf7n2f40CNdYHHWvBTLnuKJIqgwcnlYvQI9SPshahRsC3ZhbsSbaFlNJWacecZca6XIrbWSVDNDstryBfUQFcN49MIukygAN6UIAC+c0xp7QswyAJ19E25LEn8nTQcfAMDjTOUcJmkhGOt9AfhNTijaJDT1xftvFfd9xjx/TSd+lqdQzzxvTY+gzIzYcLD4jEkydBl8ojddAC1SlSzcDCD9w+ordNjWtZmxGWzRxQSdT6JkA+z3p/tiDxzx4pzy9lzFsFA1zZ6tzCx7CLIDdFlc17zO1D8XzvpPzp8fzFTAL6zbtILtTuzGJBzvdsLkLDx8LOdkFelSL92qLAzwzGLPIWLEkudblbzalT9bohLvFxLGQEcgwoVWskV0VsVYwkw/zmkVLOz9TtL0LhzDL+J8Iy2RJHAJJVzNJXLdzzDuyvRvUOLQrHzA99MbQttIriz753zTFveemBmfJYARgHUzYFF+0A0op4pBm7DJmFLAB8pVmfiypdmBgH8OCHU9gBi2CbCUQ0QlZDQ7p7iBS8jPrfrzM0JY1GZxl8IzSEWchVA/UU0aA3VF1DpuwPORAMkYwI4I4Y8b6jT0zTEszzgFbYQg2U05Q0AY1lC5ZxTfhiwTKuQYGUQK+EBJUMxrJcxBjljcj1dToLgm1mwhhGEHkXAD4FWoQxTTAJyQODz3BUxLmC2y0VbBwbxD4Lu8EfSHpnjHAy917cDTIOyZ7NGkAfYd77UCAO2HyroU4MkfrocVtbOusCT3ZjwwNQkQ4HyC0LbWoDEVGau2Jvpn7Ecjk5rHZTQiHQ8C0/7HgqE2wMkT7F1A5nxJAHjAArCOO+1h9uLh4HFpHRyCLEBPsUx6DZcFMRkPK6K+OzDksImECEKzOPJmp8je8sEEDjHvII0ilEJ7MeFEPB5uLsFez/KgBgsmtXP4tJbFnIwAOQyAoVoBoV6rrLMSC1UwOuWs63aXSBRrW3/38eyraEDlFHZaJuZT6hWzVFSmcP2UNFKB8PvICNCOJadGiOUgz2SNkuaT5kgYJutA/CVkLG3a5t6HVwFu3U2FP2jvY7ju6aQC6BTQj5jAgCDBoC3jTAePaSLt6TLsBlKU6AihoA2WIC1cUiKvoCOPwBuONcCCDduNhMGDldQAHAj59g1d1eNANdNe6RQKtdBJEjPkeOvs4sPwdeZjde9fRU05jeQCNd9iQAABUjQJ3E3U3lXHqQ4839Xp3y3bJlwa3mkERnjiHu3VBXX0gh3/XaAJ3jXiHV3o3Q3lIwSk3FXM3HqC0T3i3L3KxldiDjTG3JIHjyHrbv3nXB3C3QPIPkAC0l313kPt3sPI+dHiPAgS3KPH3jO1HlJ9HOPboe36A+PfXx3kPjXdHpPEP43yQBw7rvJEAXrBgGg2lqQXRzYouo0b4iAzY41hcYA0AeXsgzAYpIvkp0pYbKnEb4mUbDhiX2EPKuA5qXRUpoWbMw08vacboyv4nAbTCXM5UbtbwfSooqcbQl0+twusuL26AoWIIhcMzpytFJFBwzUpVzYDUL2j21TmbHwDM63F73Evvt2M017i0WXoYTNy8ddZZ55Slcfr8dIMr4ViSzEzU+kDVxv6A0ySKGVpfEcbAypSAghmo8MQkVyx4WUNdOIvgdvB6GjfRWj15+A2lDAdaCApqVENh0uORHQnULpBwfrerIQkxgY/fDF/+VROvIX7yYXogDS/DuwbR3wwj0OcXvR4giXm65A3JHrYv3rIQzYsAsgznj/Om2vIbMpGz7wJ2IqUYDG9VS2EBNlSxCDLZDSy2D/ggF4weYuYuVHcvQy1DUR22ppMflSF8A6FdEo8ALIUEKAM9jE1QdrlQTGAcAeyKjGTi9iQi2VdgH7DpF5SwKDwnQQBYvkPF4CD0UgJUYKAKipj/YMYpQTGJBBoAZBWsOCHtmgHkBR9pw0guto/DICNtjozbF3uCEEaox1A24Kdm7RcSfgOE3OMitwBYKGDd8JcNyjKHGhu9KooYA9lNCoGsd90QGAyFIOcTCI78ScD6iVDAakc9aQEfIM/mOiODYQOnLMmJGJoPgNSGwFoJPhtTI5hKT8eQBGCn7V0sE+4e1KlXpybItg1ib9nhhmAd4jcNFfquW2kFB0tUBghxpGX37sND+C9Y/hFwXpRcVqI/a/qjHi4SNVSD/EgNr09ZGBvEzYAUkKSYJloK0Qbf1H/z15ykgB1mUAfZgIYQoXQVVGqgNlgpqDmg3AKvLgXZzmxu6MLFdlzCWDR81cq4aLuj2WQixR4uFfCuAxIB90a21VGXF1kobtsohWpC+i+RUAMUuKmreliuxcQRx+gfSGLnqhiiMQr0UpbBCBiTRMwtBqbdNH0X5ba1c0HqAtOsJwIgk7qSIDLF4MOBGcz4S5NPq5nBKrdNioQ1duO21JAFksmKEKmFTQBmcFkWYZwPNgrofdZKhpUsMJwjjf1HIaYMyPzk/CPAWaQfPIMA0oKhFE4U0FYSCHKDWISUi+TFoQ3yGVlOykYWQAQBACTE/Bx9NmhCOwgwjMQ2iBfhkQ1QDowgcaOkP9UwJU15kaHDLNsE8FztDgKwydO83eiNUn4yaA8BSH3o+AVUsEBzhAUrjYiQi+YVikg2QQGJpOj1HQpSSxCwZ8kzQcQJeEliQj/4f4PZGWDsh6Jsa75dzjDkxCt4CYy9P/EAXvIyRyGkSZWMoGL7rtMsxfGCi52eg1AwGdIKUAv1CCzs4cggBTm6DYhFhoITY8vABjGzK1J8zNIwXYTDLRshITRZiPIAiKWdtAT8NQNnA+Rs0UMJkbhA/HDqIi6QZ9BSEanlC8jmyqImoTUXnr1EjSvDU/pF3P6CMWhboNoT0XEZ39VSUATdCANszNIAhj4y/iCJfFiNwgi4k/meHeTswmU2QFyFqlSS2c3ijpBYdkNupzJM2yAppFel6GYYDArGKHnhjwAEZXaxGATJwHPQiYDebghQOvCkyMZZMLGbDCRnUB4lJKy2KdAXFoCrZWQTmbCbhLo5oAAAbAIBHAABOPibQD4l9gFoYwYSX2CHCiSRw2wCYGMDQB0cBAfYAABwMBtg6kkcAOAWiSS+wAgPnvRIgCQARwdHaZKZFknaShwQ4HIMJJUl6SBA2wPsLpOEn8Shw/E2gGMAEDCT1JYwYcEOD7D8T+JcmQwLhP4kDgRwAgAcMJKHD6SFoDAfiXUG2D8TKwaAdScJIHAMAhwmUtAMJJIB9gGAYwZDnpMHD8SSeJkqAFlMSloAFoIUpQBlLqkxT6ptAISYVIHADhTIDkkcEVP4l9g6OYwOScODCk4SGJlYCyepMHBoBhw7k8yUZKmkjhEpnkujq4gHAkAhwAgCqQtFUkLRtgDATKaNNwl1TWpaAMYHR3UkkBVpC0UcGgBHC0B1Jg0qKXR1oDmStJkkhSWgEUmpTNJR0hiWMFcQzS9J3khaLFMulDhaAUU9SepJun8SXp+UuKQIB0n7SLJnUmaX9NMnxSRwdDYSUjIWgZAhJ6kgQM21kn8T3Jak/yYNIYDmTaA2wBaOFxIDCSMZUAZyYZJhkjh1JFU2gMJIWg8zop+0hgBdIcl6SkpQkxKf5LGCKSDpxk+TGNNMmgyrJA4faXdJbYOTiZYwWmdzJbbQzYpIU4Se5LQB9htgBUnmWbiqmQACpm07YKOGHB0c0pxs16epKHADhXJfE52WTJIAjh+JV07YCOHikkBeZ53c2XdIEC2T/Qvk4SfdNcleyhwdHUcC9IEANQCphkxvgtDTl9gHJaALBObO9mGysp1016bHIsn6yxgfYT2RzP4kjhPZtk2gAwB5l0cspkkoaczMgCCSxgDUZDt5OhmDS7pAk7KQ3Lo6STIpdHAcEJPOkMBEpRs2gJFNGlyyoAAcJiSZEQCsT84InZbLhn0BAA=== -->

<!-- internal state end -->
<!-- tips_start -->

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

<details>
<summary>❤️ Share</summary>

- [X](https://twitter.com/intent/tweet?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A&url=https%3A//coderabbit.ai)
- [Mastodon](https://mastodon.social/share?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A%20https%3A%2F%2Fcoderabbit.ai)
- [Reddit](https://www.reddit.com/submit?title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&text=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code.%20Check%20it%20out%3A%20https%3A//coderabbit.ai)
- [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcoderabbit.ai&mini=true&title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&summary=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code)

</details>

<details>
<summary>🪧 Tips</summary>

### Chat

There are 3 ways to chat with [CodeRabbit](https://coderabbit.ai?utm_source=oss&utm_medium=github&utm_campaign=CliMA/ClimaComms.jl&utm_content=111):

- Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  - `I pushed a fix in commit <commit_id>, please review it.`
  - `Generate unit testing code for this file.`
  - `Open a follow-up GitHub issue for this discussion.`
- Files and specific lines of code (under the "Files changed" tab): Tag `@coderabbitai` in a new review comment at the desired location with your query. Examples:
  - `@coderabbitai generate unit testing code for this file.`
  -	`@coderabbitai modularize this function.`
- PR comments: Tag `@coderabbitai` in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  - `@coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.`
  - `@coderabbitai read src/utils.ts and generate unit testing code.`
  - `@coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.`
  - `@coderabbitai help me debug CodeRabbit configuration file.`

### Support

Need help? Create a ticket on our [support page](https://www.coderabbit.ai/contact-us/support) for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

### CodeRabbit Commands (Invoked using PR comments)

- `@coderabbitai pause` to pause the reviews on a PR.
- `@coderabbitai resume` to resume the paused reviews.
- `@coderabbitai review` to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
- `@coderabbitai full review` to do a full review from scratch and review all the files again.
- `@coderabbitai summary` to regenerate the summary of the PR.
- `@coderabbitai generate docstrings` to [generate docstrings](https://docs.coderabbit.ai/finishing-touches/docstrings) for this PR.
- `@coderabbitai generate sequence diagram` to generate a sequence diagram of the changes in this PR.
- `@coderabbitai resolve` resolve all the CodeRabbit review comments.
- `@coderabbitai configuration` to show the current CodeRabbit configuration for the repository.
- `@coderabbitai help` to get help.

### Other keywords and placeholders

- Add `@coderabbitai ignore` anywhere in the PR description to prevent this PR from being reviewed.
- Add `@coderabbitai summary` to generate the high-level summary at a specific location in the PR description.
- Add `@coderabbitai` anywhere in the PR title to generate the title automatically.

### CodeRabbit Configuration File (`.coderabbit.yaml`)

- You can programmatically configure CodeRabbit by adding a `.coderabbit.yaml` file to the root of your repository.
- Please see the [configuration documentation](https://docs.coderabbit.ai/guides/configure-coderabbit) for more information.
- If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: `# yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json`

### Documentation and Community

- Visit our [Documentation](https://docs.coderabbit.ai) for detailed information on how to use CodeRabbit.
- Join our [Discord Community](http://discord.gg/coderabbit) to get help, request features, and share feedback.
- Follow us on [X/Twitter](https://twitter.com/coderabbitai) for updates and announcements.

</details>

<!-- tips_end -->
author_association: NONE
user: login: Sbozzolo
created_at: 2025-05-05T15:32:54Z
body: Could you add an entry to the NEWS file and some user documentation?

I don't see a clear place where to put this in the docs, so you could maybe add a new frequently-asked-question and discuss one example there.
author_association: MEMBER
user: login: dennisYatunin
created_at: 2025-05-05T17:34:02Z
body: > I don't see a clear place where to put this in the docs, so you could maybe add a new frequently-asked-question and discuss one example there.

 I added detailed documentation (including an example) to the docstring of `@threaded`. Are you saying I should move part of it to a separate docs page?

author_association: MEMBER
user: login: Sbozzolo
created_at: 2025-05-05T17:55:50Z
body: > > I don't see a clear place where to put this in the docs, so you could maybe add a new frequently-asked-question and discuss one example there.
> 
> I added detailed documentation (including an example) to the docstring of `@threaded`. Are you saying I should move part of it to a separate docs page?

The docstring is good. However, docstrings alone are not sufficient documentation. People reading the documentation will not know that they have to look at the docstring of `@threaded` to be able to launch a kernel with ClimaComms. Docstrings are for reference, and we still need a usage section that users can easily find in the docs. This section can be short, just show one example, and point for the docstring for more information. Given that the ClimaComms user documentation is very sparse, you could add that in the FAQ and could be "How do I directly launch a CUDA kernel with ClimaComms?" or something like that. 

author_association: MEMBER
user: login: dennisYatunin
created_at: 2025-05-07T23:36:14Z
body: @Sbozzolo, I've added a section to `index.md` for "Writing generic kernels," which provides an example and links to useful references. Please take a look if you have a chance.
author_association: MEMBER
user: login: Sbozzolo
created_at: 2025-05-08T00:15:55Z
body: > @Sbozzolo, I've added as section to `index.md` for "Writing generic kernels," which provides an example and links to useful references. Please take a look if you have a chance.

It's great, thank you!
author_association: MEMBER